Azure DevOps is a tool for planning, auditing, code versioning, code integrating, testing, artifact storing, and deploying. It is the primary tool for continuous integration and deployment. This course will focus on the most important parts of Azure Pipelines in implementing an end-to-end continuous integration strategy.
Build triggers are essential for automated code processing and will reduce the workload of any team governing the process. You will learn which build triggers exist, why you may want to use them, and a strategy for implementing them.
Hybrid builds allow for flexibility in workflow, security, and processes by integrating builds from more than one pipeline or tool.
Parallel builds will speed up processing of the workflow where it makes sense to do so. You will learn how to use parallel or multi-agent builds, when to use them, and what their benefits are.
Azure DevOps is but one tool in the CI ecosystem. This course will also touch upon other build tools as well as recommendations for their use and integration into your workflow.
To round out the course, we will set up a completely automated continuous integration (CI) workflow that will provide a foundation for a secure, repeatable, auditable, and complete CI solution for your projects.
Learning Objectives
- Maximize automation strategy with build triggers
- Understand hybrid build concepts
- Speed up pipelines with parallel builds
- Learn about build tools and Azure integration
Intended Audience
- Anyone wanting to learn the continuous integration material for Microsoft's AZ-400 exam
Prerequisites
- A basic understanding of workflow and the CI build pipeline process
- A good understanding of the development lifecycle
- It would be advantageous to have a basic understanding of YAML, although it's not required
Resources
The GitHub repository for this course is at https://github.com/cloudacademy/azure-continuous-integration-build.
Build triggers initiate builds when some designated action or actions are performed and any required criteria is met.
We will cover triggers in detail over the next several minutes. For context, here is a trigger example. The snippet shows all of the types of triggers that can be defined in the azure-pipelines.yml file, branch triggers, tag triggers, path triggers, and pull request triggers.
Triggers are evaluated before the build runtime since they determine if the build is run at all. Thus, variables and other event data are not available when the build is triggered.
There are two categories of triggers, automated and manual.
Automated triggers are event or schedule based and are essential when automation is desired. Automated triggers are usually accompanied by supporting criteria, such as a specific branch or directory being updated.
Manual triggers are a way to start a CI build process without using an automated trigger. Manual triggers can be performed at will and without meeting automated trigger criteria. Builds can be manually started from the Azure DevOps Pipelines portal, the Azure CLI, or the Azure DevOps API.
Automated build triggers can be further refined into types:
- Repository triggers start a build based on a repository commit, or a pull request being initiated or updated.
- Schedule triggers can be planned to start builds at a predetermined time or interval.
- Pipeline triggers can kick off another pipeline upon completion with optional criteria such as successful or failed.
Triggers can also have criteria defined that will further refine when a trigger kicks off a build:
- Branch or tag is used to include or exclude branches and/or tags that can trigger the build.
- Path is used to include or exclude a directory or file path that can trigger a build.
- Pull request is used to specify a list of branches as the pull request target to include or exclude to determine whether to kick off a build when a pull request is opened or updated.
There are also some additional options that can be associated with triggers:
- Batch is a boolean value indicating whether to start another build while another is running for the same branch.
- None will disable any automatic triggers, making the only way to run a build, manually.
There are some important distinctions between what triggers can be used in conjunction with the resources being used as the trigger. Most notably are the differences between the pull request triggers when used with the various version control systems and Git hosts.
For instance, Azure Git Repositories use branch policies to control pull request triggers. Pull requests do not apply to Team Foundation Version Control.
Next, we will look at strategy around using build triggers. Careful design of trigger strategy can greatly impact overall goals. What does that mean exactly?
Begin by asking a simple question, when should this build run? Most commonly, the answer is when the code changes.
Another often-used trigger is when a pull request is initiated or updated. This helps ensure that code passes checks prior to merging to a main branch, provided the pipeline is set up properly.
A somewhat less common scenario is to start a build at a certain time or interval. This could be used to run daily tasks or to bulk build all repository changes at the end of day.
It is also possible to trigger a loosely coupled build based off of the completion of another pipeline. A good example would be to have a build that serves as a base for other builds, and then all upstream builds could use the dependency in a new build.
Whatever the need, there is a way to automate the CI build to ensure code quality and availability is always on par.
Branch triggers are the most common type of repository trigger. Triggers are specified in the azure-pipelines.yml file with the keyword trigger. Branch triggers specify which branches should start a build when updated.
The default configuration for a branch trigger is all branches. This means that a push to any branch will start a build for the branch. This will also trigger on branches when they are first created. For example, if a new branch is created from master on the git host or locally and pushed, a build will be started for the new branch.
This default behavior can also explicitly be defined using an asterisk. Since an asterisk is a reserved character in YAML, any text beginning with an asterisk will need to be quoted.
Trigger definitions are defined as an array of branches. This allows multiple branches to be included with this syntax.
This simplified syntax works only when we only want to include branches. More verbose syntax can be used to configure more options.
Adding the keyword branches under triggers and then include under branches is equivalent to what we just had.
The exclude keyword can also be used in the more verbose syntax. This works well when all branches except the ones listed should trigger a build.
Include and exclude can be used in conjunction, but specifying either include or exclude, implies that any branches not listed are excluded if include is used, and included if exclude is used in the filter.
For instance, if master, dev, and feature branches exist and only the master branch is listed under include, then dev and feature branches are implicitly excluded. The same goes for master being listed under exclude, dev and feature branches are implicitly included.
Using include and exclude appropriately allows us to use the shortest path, but it also allows for flexibility when used in conjunction with wildcards. Wildcards allow specifying branches where more than one possibility may exist.
There are two wildcards, an asterisk is used for zero or more characters, and a question mark, a single character.
It's a common practice to name all feature branches with the prefix feature/. All branches that begin with the prefix feature/ can be specified using an asterisk wildcard.
The question mark can be used to stand in for a single character. For instance, if there are multiple branches that begin with wip- followed by a single digit then the filter could match on wip-?.
The wildcards can also be used in the middle or the beginning of a branch name. Just remember that the text will need to be quoted if beginning with a special character.
Let's look at more advanced use case where both include and exclude are used with wildcards. All feature branches are included in the branch triggers except the excluded branches that match features/wip-?-signin-form-*.
Tags can also serve as triggers. They can be specified under the branches keyword and included or excluded just like branch names using Git syntax for tags.
As an alternative, tags can be included and excluded under their own keyword at the same level as branches without the Git syntax.
In either case, the same rules apply to using wildcards with tags as they do with branches.
By default, tags do not trigger pipelines. So, they must have a definition, or the default behavior will stand. Being able to include or exclude branches as triggers is a powerful feature, but it doesn't allow for filtering across branches. It is very useful to be able to trigger or not trigger a build based on what files were changed.
For instance, triggering a build when only the README file is updated may be a waste of resources. The README is at the root of the repository, so it can be added without a directory path. Paths are also case sensitive.
Now changes to the README on any branch, will not trigger a build, however, if any files outside the exclusion are changed in the same push, then the build will still trigger, and that's more often what we want.
The default behavior for paths is to include the root of the repository. This is equivalent to using the asterisk wildcard under include. This means that any files that change will trigger a build, provided that the branch requirement is met.
In fact, one requirement of using paths is that branch triggers are defined.
Entire directories can also be included or excluded. For example, a directory called development can be added to the include path.
Wildcard rules are different for paths than that of tags and branches. An optional asterisk can be added to the end of the path, however, it is equivalent to just specifying the directory. The asterisk is only allowed to be the last character and question marks are not allowed at all.
Include and exclude can be used together. A path can be excluded and child paths included since the path reaches deeper into the file structure. For instance, a directory named development can be excluded but its subdirectory named main can be included.
Branches, tags, and paths can be used together but special consideration must be applied.
Tags and paths are evaluated as an or when used with branch filters, thus, if a branch is a trigger and either a tag or path is included, either implicitly or explicitly, then the build is run, provided the criteria is met.
For instance, if the branch master is updated with tag version 3.1, then the build is run regardless of the path criteria. The same goes for the development/main path being met and the tag criteria not.
In another scenario, if the master branch is updated and neither the path or the tag criteria is met then the build will not run.
Lastly, in the case that the branch criteria is not met, like the test branch is updated, then the build will not run because the test branch is excluded regardless of any tag or path criteria.
Paths defined for Azure DevOps Repos must be done in the portal. Although GitHub is being used for this course, it is worth showing the Azure path configuration.
I'll traverse over to the Branches page for the project git repository that was created with the project. Clicking on the Options menu and then selecting the branch policies item will take us to the branch policies for the master branch on this repository. A build policy can be added to the build validation. Path filters can be added to the Path filter textbox. All paths are relative to the root, just like in the YAML. Separate paths with a semicolon and exclude paths with the exclamation point before the path. Remember that paths are case sensitive.
I can't save the changes because a build pipeline is required in which to apply the policy. Since I have not made a pipeline that utilizes the Azure Git repository, then I won't be able to save my changes. This is something to be aware of before trying to define paths.
Pull request triggers are used to start a build when the pull request target branch matches the PR trigger criteria. Pull request triggers and normal triggers can be defined in the same Azure pipelines file, but it isn't required.
Pull request triggers help ensure that the branch intended to be merged with the target branch doesn't break the build. This is done by merging the branch into the target branch outside of the git host, and then running that through the build process. By default, if the build does not fail then it will be deemed as valid, and can be permanently merged to the target branch, provided that review criteria is met.
As a safeguard against allowing non-validated code to be merged, any updates will invalidate the pull request build. In regard to the build, it will need to be run again in order to become valid.
The PR trigger structure is very similar to the normal trigger structure.
By default, all pull requests are triggers. This is equivalent to placing an asterisk under the pr keyword.
A PR can be set to include or exclude branches and paths but not tags, and the same wildcard rules also apply.
One difference between normal triggers and pull request triggers is the autoCancel boolean.
The default autoCancel value is true, which will invalidate any builds where the code has changed. The autoCancel keyword can be specified with a false value so that changes to files after the pull request is opened won't invalidate the build. In other words, if the pull request build passes and then changes are made to the pull request code, then the source branch will still be able to merge with the target. Leaving this as the default will help ensure the integrity of the target branch.
For reviewers of pull requests, it may be convenient to perform some actions within the pull request's detail page, such as manually running a build by issuing a comment on the pull request.
For example, a command to run a pull request's associated Azure Pipeline can be accomplished with /AzurePipelines run in the pull request's comments. /azp can be used in place of /AzurePipelines for brevity.
There are a few other commands and use cases that I urge you to explore.
Unfortunately, these are specific to the Git host, so if available and you would like to use them, then further research is warranted.
It may be that a build should never be run automatically.
Specifying trigger none will disable automatic triggers, effectively making this pipeline a manual only start regardless of the branch, tag, or path.
Pull request triggers can also be set to none, so regardless of the pull request target, the build will not start automatically.
You may have foreseen an issue where starting a build every time code is pushed can tax resources. This is especially wasteful when not all code pushes need to run through the build process.
There are a couple of ways to mitigate this scenario, skip comments and batch builds.
Builds can be skipped by adding special text to the commit comment. Here are the possible variations.
For instance, within the message for the git commit, adding the bracketed skip ci text will skip the build.
This is very useful for work in progress, also known as WIP, pushes.
Another way to take better advantage of resources is to batch them. Normally, each push that matches the trigger criteria for a branch starts a new build. This is great for individualized feedback but if pushes are made in rapid succession, it could fill the build queue, depending on the number of agents and the build time required.
Builds on the same branch can be batched to conserve resources by using the keyword batch. By default, batch is false. Specifying a value of true will enable batch builds.
Once enabled, if a build for a branch is not in progress, then one will be started, provided the branch trigger criteria is met. If code is pushed again before the prior build completes, then the build will not start until the prior one finishes. So far, this may be a disadvantage if more than one agent is available. The potential resource conservation comes into play when more pushes are made before the initial build completes.
Once the blocking build has completed, then all pushes that have accumulated for the branch will be run together. Effectively, this will run the last push on the branch.
This may not be the desired behavior if build feedback is needed for each push. Using skip ci may be a better option in this case.
Lectures
Cory W. Cordell is an accomplished DevOps Architect, Software Engineer, and author. He started his DevOps career as a DevOps Engineer for a large bank where he helped implement DevOps practices and tooling and establish a DevOps culture.
Cory then accepted a position with a global firm to build a DevOps department. He led a team of DevOps Engineers to establish best practices and train development teams on tooling and those practices. He worked to help development teams migrate their applications to Azure Kubernetes Service and establish pipelines to build, test, and deploy code. Realizing that a substantial gap existed in the toolchain, he developed an application to aid in infrastructure tracking and to provide UI abilities for teams to view application status for their software.
Cory is now enjoying working as a contractor and author.