Build Agents and Pools
Configuring API Access
The course is part of this learning path
Azure DevOps has many great tools for implementing and managing your build infrastructure, and this course walks you through how to use them. With a mix of theory and real-life demonstrations from the Azure portal, you will learn how to create Azure pipelines, use them to integrate 3rd party build systems, utilize agent pools, and learn how to put it all together to set up an automated workflow that can potentially save you hundreds of man-hours. So, whether you’re here to learn more about DevOps, improve your development process, or learn more about Azure DevOps in particular, this course will help get you further along your path.
For any feedback, queries, or suggestions relating to this course, please contact us at email@example.com.
- Learn how to build an Azure DevOps pipeline and add automation tasks
- Use the visual designer for adding build tasks to a pipeline
- Create a pipeline that closely matches an existing build process
- Determine a pipeline strategy based on the information given to you during a discovery process
This course is intended for:
- IT professionals looking to become certified DevOps professionals
- IT professionals experienced with other cloud providers who want to gain a better understanding of Azure
- Anyone who wants a better understanding of Azure build infrastructure processes
To get the most from this course, you should have some understanding of the software development process, at least at a high level, as well as an understanding of what DevOps is and the terminology related to it.
In this lecture, we are going to talk about build agents. A build agent is a piece of software that runs a series of build tasks called a job, on a machine. In Azure pipelines, there are two types of build agents, Microsoft-hosted agent, and self-hosted agents. We are gonna go through a deeper explanation of build agents in this module.
A Microsoft-hosted agent are agents that Microsoft provides for running your build tasks, but there is more to a Microsoft Hosted agent than just a simple piece of software. You are also getting a new virtual machine that is spun up with the latest operating system, whether it's Windows or Linux, or macOS, it will have the latest security updates, patches and maintenance updates without you needing to do anything at all. Microsoft-hosted agents can run a job directly on a virtual machine, or in a container, but for now the easiest way to understand a container is it's just a toolbox with a very specific set of tools that you hand pick to build your application. In many cases for simple to moderately complex applications a simple Microsoft-hosted agent is more than enough to satisfy the needs of the project. Best of all, this is a free service up to 1,800 minutes a month, with a maximum build job length of six hours, for private projects and completely free for up to 10 parallel jobs for public projects.
Next, we have Self-Hosted Agents. A self-hosted agent is an agent that you install on your own machine whether that is a virtual machine in the cloud managed by you or an on-premise server that you also manage yourself, you are responsible for making sure that the operating system is kept up to date with the latest security patches and maintenance updates. Not only can you use self-hosted agents in Azure pipelines, but also in Team Foundation Server if you happen to have an on-premise TFS server. This agent can be installed on any machine you have access to, however that machine will also need to be able to access Azure, either through the internet or over a private express route connection.
As a first step, Microsoft recommends trying the Microsoft-hosted agent first to see if it does all that you need it to do for you, if not you can check the logs that are created in the build process to see where you might need more than what the Microsoft-hosted agent can offer. There are however a few guidelines that can tell you quickly if you need to host your own agents on your own machine.
The simplest factor is project size, the maximum project size is 10 gigs including build output. Microsoft Hosted agents use a Standard_DS2_v2 virtual machine image. This image has two virtual CPU's seven Gigs of Ram, 14 Gigs of temporary SSD storage. This is just an average machine in most respects. If you need more information about this image, you can find it here at this URL. If you need more powerful hardware, then self-hosted is likely the way to go.
Some of the other limitations of the Microsoft-hosted agents are that you cannot sign into the agent machine, you cannot drop artifacts to a UNC file share, and you cannot run XAML builds.
Microsoft-hosted agents also use a standardized list of operating systems and software configurations. There are eight options, four versions of Windows, two versions of Ubuntu, and two versions of macOS. You can find the list at this URL.
So, if you need a version of an operating system or Visual Studio that's not part of the standard list you will need to use a self-hosted build agent instead.
If only the version of the software installed on the agent machine is a concern and not the operating system itself, Microsoft-hosted agents also have the option of using either a Windows or Linux container and using your pipeline to install the version-specific software and tools you need to build your project.
There are however some unique requirements and limitations for each operating system. You can find the specific requirements and limitations of the container Jobs here at this URL.
In this topic on build agents, I would be remiss if I did not discuss pricing and costs, since one of the main determining factors of the cost of Azure pipelines, is the length of time and concurrency of running our build jobs. As I mentioned before, pricing is affected by the visibility of your project. A free tier is limited to 1,800 minutes or 30 hours of build time per month, with a maximum job length of six hours and can only run a single job at a time. For public projects, the build minutes restriction is removed, the job length, however, is still six hours. In the free tier with a public project gives you 10 parallel jobs. Once you go above the free tier though you will need to pay for each parallel job, and buying the first parallel job for a private project does not actually allow you to run more than one job at a time, it simply removes the build minutes restriction and you will need to buy a second parallel job if you wanna run two parallel jobs at the same time.
This brings us to Agent pools. Instead of dealing with each agent individually, sometimes it makes more sense to manage a group of agents. Agent pools are just that, a way to manage groups of agents rather than each individual agent on its own. In Azure Pipelines the agent pools are configured in the organization and project settings depending on the user's access permissions and these pools are accessible based on the level at which they are created. Therefore build agent machines are registered in the organization agent pool, and the organization level pool can be referenced by any project-level pool in a one to many mapping. Project pools, however, can only be referenced by one project at a time.
For example, let's say we have an organization called Org1, and we create an agent pool called Org1 Agent Pool, and we add 2 build agent machines to this pool called Build Agent A and Build Agent B. Let's assume that we have project X and Y. In project X, we create a project agent pool called Pool X, and we reference the Org Agent Pool. Project X would then be able to use both Build Agent A and Build Agent B in its build pipeline to build project X. Now let's create an agent pool in project Y. called Pool Y at the project level. If we tried to reference another agent pool we would not be able to see the Pool X. We would only be able to see Org1 Agent Pool. If we choose not to reference an existing organization level agent pool when creating this new pool, a new pool would be created at the organization and project level named Pool Y and the new organization-level pool would not contain any build agents. With Azure DevOps Server, the new name for what was formally Team Foundation Server, agent pools are shared across the entire server so you can utilize agent pools across projects and collections.
The default agent pool type is for registering self-hosted agents. The Azure pipelines hosted pool can be used with any of the standard machine configurations from the Microsoft-hosted agent virtual machine images list. As with most things in Azure pipelines, agent pools can be managed in several ways, from the web UI, YAML, and from the Azure DevOps Command Line Interface.
In our next video, I will show you how to create a self-hosted agent, and how to create and configure an agent pool for our project.
About the Author
As well being the owner and CTO of Sharp Solutions Group, a software development and IT staffing company based in the Philippines, Kelso is a Microsoft Certified professional and an avid knowledge seeker. His belief is that you need to learn something new each day to stay on top of the constantly changing IT world. He is an avid gamer (both video games and board games) and lives in the Philippines with his wife and soon-to-be-delivered son.