Build Agents and Pools
Configuring API Access
The course is part of this learning path
Azure DevOps has many great tools for implementing and managing your build infrastructure, and this course walks you through how to use them. With a mix of theory and real-life demonstrations from the Azure portal, you will learn how to create Azure pipelines, use them to integrate 3rd party build systems, utilize agent pools, and learn how to put it all together to set up an automated workflow that can potentially save you hundreds of man-hours. So, whether you’re here to learn more about DevOps, improve your development process, or learn more about Azure DevOps in particular, this course will help get you further along your path.
For any feedback, queries, or suggestions relating to this course, please contact us at email@example.com.
- Learn how to build an Azure DevOps pipeline and add automation tasks
- Use the visual designer for adding build tasks to a pipeline
- Create a pipeline that closely matches an existing build process
- Determine a pipeline strategy based on the information given to you during a discovery process
This course is intended for:
- IT professionals looking to become certified DevOps professionals
- IT professionals experienced with other cloud providers who want to gain a better understanding of Azure
- Anyone who wants a better understanding of Azure build infrastructure processes
To get the most from this course, you should have some understanding of the software development process, at least at a high level, as well as an understanding of what DevOps is and the terminology related to it.
In this lecture, we are going to talk about concurrent or parallel jobs and some of the strategies for determining how many we need. Recommending a strategy for running build jobs in parallel or concurrently requires several considerations. Concurrent jobs or as they are now called in Azure DevOps, parallel jobs. Azure DevOps provides the ability to run one parallel job for free. Now the term job has multiple meanings in Azure DevOps and the context is important.
A pipeline can have several jobs, which in this case is a task or series of tasks to be run in a pipeline. A parallel job is what runs a pipeline job and one parallel job is required to execute one pipeline job.
Azure DevOps gives one parallel job to every organization for free. Private projects have a build time limit of 1800 minutes or 30 hours, it also can only run one pipeline job at a time. Public projects, which is projects that are part of the Azure Pipelines public project, or pipelines that build the public repository from GitHub, the build time limit time is removed and you get 10 free parallel jobs.
There's also a slight difference in the free tier offering when it comes to types of agents being used. With Microsoft-hosted agents, there is a maximum time limit for the job of 360 minutes or six hours. If you choose to use a self-hosted agent then there's no time limit on the self-hosted jobs. You can also get one additional free self-hosted parallel job for every user in your organization that has an active Visual Studio Enterprise subscriber account.
So how do we determine if we need additional parallel jobs? According to Microsoft, the simple estimate would be to run one parallel job for every four to five users in your organization.
Some scenarios where you might consider additional parallel jobs are: you have multiple teams of developers working on a single application, if your continuous integration triggers multiple builds from multiple branches, it's recommended that you have one parallel job for each branch, or if you develop multiple applications in a single DevOps Organization, the simple recommendation is one parallel job per application.
This can be extended as needed by your organization. Let's consider this scenario.
You have a single organization, your organization has four development teams of five people on each team, working on three different applications. The first application is a website for your company, second application is an internal desktop application and the third application is a XAML application for your warehouse to provide updated inventory levels on demand. It requires two XAML build controllers. Your web application has a master, develop, test, stage and release branch, and your CI pipeline will build and deploy to a deployment pipeline in Azure if after deploying to the test server, all of the automated tests pass. Your internal application is a very small utility that takes a couple of minutes to build and deploy, it just has a master branch and when commits are made to the master branch it triggers a build and drops the output into an artifact's directory if successful. Your XAML application has a master, develop, test and release branch. All code committed to each of the branches triggers a build job and an artifact drop, but deployments are done manually.
Using the guidelines provided previously, we would need four parallel jobs for the development teams, each team having four to five members each. Three for our applications, one per application. In our website pipeline, we have five branches but only two of them actively use CI triggers. So that's only an additional two rather than five. Also, server jobs, either Azure Pipelines or TFS, and deploying to a deployment group using a release pipeline do not take up a parallel job.
Though there could be an argument made here for five depending on the size and build time of the application in question.
We've already counted a parallel job for our internal application, and since this is a small application, it doesn't take very long to build and deploy an additional parallel job for the master branch is likely overkill. Our last application already has a parallel job, but this is a XAML application and it requires two build controllers. XAML build controllers require one parallel job per controller and we can count the one for the application and one additional for our second XAML build controller.
So, another simple way to determine how many parallel jobs you need without initially spending any money is if the number of builds queued starts to exceed the number of parallel jobs you have and the queue delays starts to take more time than is acceptable, simply add additional parallel jobs until the queue delay time is back down to an acceptable level. The exception to this rule is XMAL, XAML always requires one parallel job per build controller and must use a self-hosted agent and self-hosted parallel job.
An important note here, when you first decide to buy your first parallel job, that will only remove the time restriction from the free tier, if you need to actually run two jobs in parallel, you will have to purchase an additional parallel job.
How do I know if I have more builds queued than parallel jobs? Well, I am glad you asked. If you wanna check out your parallel job usage information, it's good to know where to find it. Parallel job information can be found in the organizational settings by selecting Parallel Jobs from the Menu.
This page will show you a lot of useful information, like how many minutes of your free tier you have used, how many parallel jobs you have, and it gives a breakdown of where your parallel jobs are coming from; free tier, visual studio enterprise subscriber or paid. This is also a nice way to visualize the free tier limitations.
So now that we have a better understanding of parallel jobs and we have some strategies on how to determine how many parallel jobs we might need. I look forward to seeing you in our next module.
As well being the owner and CTO of Sharp Solutions Group, a software development and IT staffing company based in the Philippines, Kelso is a Microsoft Certified professional and an avid knowledge seeker. His belief is that you need to learn something new each day to stay on top of the constantly changing IT world. He is an avid gamer (both video games and board games) and lives in the Philippines with his wife and soon-to-be-delivered son.