Through sections one and two of this class, we have gradually ascended a ladder of automation, if you will. We started by talking about compliance and creating policies to ensure our system is secure. Then we figured out how to transparently automate much of that configuration using DSC. And then we automated even further by using Azure Automation State Configuration and VM agents with custom scripts to ensure things stay compliant. Now it's time to assume our ultimate form of automation: Using secure pipelines to guarantee that every code or config change deployed to our system is automatically checked, tested, and released with full compliance and security totally assured.

This is where the fun begins. So let's start by talking about pipelines. A software pipeline is a process for getting a code change from a developer's laptop out into the real world with end users. Obviously we don't want to do this in a cavalier fashion. We don't want to just SCP a script up to a server and restart it and call it a day. We want to test things out. We want to check code quality. We want to make sure everything is reviewed and compliant and safe before finally deploying to our production environment.

So the modern paradigm for this is known as CI/CD, or, continuous integration and continuous delivery. In the DevOps world we use tools like Github, Jenkins, Gitlab, CircleCI to automate a lot of this work. Microsoft Azure offers its own very robust solution in the form of Azure Pipelines.

Azure Pipelines are much like other CI/CD services. They let you automate building, testing, deployment of software. And the set up is very simple. All you need to do is define an organization in the Azure DevOps portal and make sure that your code is hosted in some version control system like Github. You can create and configure your pipeline from the browser. However, to keep a proper infrastructure as code methodology, you will want to put all of your pipeline configuration in a yaml file, something named azure-pipelines.yml. This is the file Azure will care about, and this is where you will define your pipeline steps and tasks.

Now, there's many different stages, you know, test stage, deployment stage, within a pipeline. One of the most important features though specific to Azure Pipelines is the gate. Gates are very important. Our release pipeline has these different steps and between these steps we can define gates. Gates are basically conditions or checks that ensure our software meets some standard before moving on to the next step. If our release fails the check performed by the gate, it will stop the CI/CD process and give us an error message. This is an essential component of automating security compliance in our software delivery process.

Now there are many kinds of gates that we can select. In the web UI in the DevOps dashboard we can see the different options. There are delay gates that simply pause the pipeline for some amount of time, as well as manual intervention gates that stop everything until a human performs some action, perhaps a check or something. There are also gates that run Azure Functions. There are gates that check Azure monitor alerts. There are gates that invoke REST APIs. But perhaps most importantly, there are gates that check compliance with security policies. Now that last one we'll come back to in a bit.

So now, a secure software pipeline does not start with security policies and checks and gates or test environments running a build of the app. Really, the pipeline starts with a solid code review process. And that process should incorporate automated code quality checks, Linters, manual reviews by other engineers. This is really where you start. And we can kick off this process for a specific branch, if we're using git, and a code hosting service like Github, and then have merge requests that launch deeper tests. Code that passes initial checks and manual review should undergo static analysis.

Now, ideally you also are checking OSS licenses and third party dependencies as well. But this can be a big pain, but there are some tools that can help, tools like WhiteSource software. They can help you with it. Code that has made it this far, where you're at least checking the third party dependencies and open sourced licenses. Once it's gotten past all of that analysis, you wanna deploy to a test environment. What that deploy looks like exactly will depend on your application. It will depend on your specific architecture. It might entail spinning up a virtual machine, or deploying a container, or updating a storage bucket, or something else. Since our focus is on security we won't delve too deeply into that, into infrastructure minutiae. We will instead walk through a simple example of that in our demo anyway. And what we wanna focus on is that once our software is in a test environment, we need to do some form of security scanning. Scanning is really critical and there are a lot of tools.

Microsoft, in their documentation, they recommend using OWASP ZAP for handling automated penetration testing. Now, what is OWASP ZAP? Okay, OWASP, O-W-A-S-P, stands for Open Web Application Security Project, and ZAP is one of their tools, it stands for Zed Attack Proxy tool. The OWASP group does a lot of great work in security. It's a big, long-lived international group. They work with an open source tool set, so very easy to audit and look at and configure. OWASP ZAP is one of their most popular tools for web application security scanning. And it's readily integrated into an Azure pipeline process as part of your gates, if you will.

The challenge that we face with CI/CD pipeline, in a CI/CD pipeline context, with this tool, is that really thorough scans that crawl through an entire website and attempt a wide variety of attacks, these take a long time to run. It's really common for these scans to run anywhere from tens of minutes to several hours for a more complex app. So therefore, if you make these tests into a gate for any and all code changes, it will slow your development cycle down to a crawl. And so, for this reason, it's recommended that you have a more simple automated scan for CI/CD. And again, OWASP ZAP, they have tools for this, simple test scans. And then, separately, you have a nightly full active scan that does a comprehensive test in a QA or a dev environment. So this is kind of a split approach.

Penetration testing is really important for application security, however we have to also ensure that our infrastructure meets compliance standards. And fortunately this is readily handled by using gates in our pipelines. As we mentioned, in the Azure DevOps dashboard we can create pre-deployment conditions as gates. And as mentioned, there are many types, but the one that's really interesting to us here is the Security and Compliance Assessment type of gate. This type of gate ensures that a release in our pipeline will fail if it violates a policy defined in Azure Security Center.

Now, this feature lets us incorporate compliance and security right in our release process. It relies on us first selecting the right security policies defined in Azure Security Center. Now remember, good policies alone don't make us secure; compliance with good policies are what make us secure. And therefore, if we can make the check for the compliance automatic, then we can make security automatic. And fortunately, Azure Security Center includes several industry standards right out of the box, so you can just select the one that best matches your product without actually having to define an entire set of security standards and policies from scratch.

So, there you have it, whew! This was a doozy of a lesson, a lot of material, but hopefully chock-full of useful stuff. If we can put all or most of this into practice, we can sleep comfortably knowing that our Azure infrastructure is as secure as can be. Check out the demo for this section for a walkthrough of how to set up a pipeline and use gates. And, once you're done with that, you're finished with all of our lesson content, so, see you there. Thank you.

Lectures

Course Introduction - Compliance & Security Scanning - Security Center Demo - Preventing Drift - ARM, Activity Log & Track Changes Demo - Desired State Configuration (DSC) - Azure Desired State Configuration Demo - Azure Automation State Configuration - VM Agents & Extensions - VM Agents & Extensions Demo - Azure Pipelines & Gates Demo - Course Summary