Security and Compliance Pipelines
The course is part of this learning path
Microsoft Azure is a robust, feature-rich cloud platform used by a growing number of technology companies. With its vast array of services, a key challenge to administering an Azure environment is security. How can we ensure that our Azure infrastructure meets strict security standards? This course offers the answer.
In three concise units, the student will learn all about compliance and cloud security. The course delves into several key Azure components, including Azure DevOps, Azure Security Center, Desired State Configuration, and Azure Pipelines. After completing the lessons and watching the video demonstrations, the student will be equipped with the knowledge to automate critical security tasks to ensure a thoroughly hardened cloud architecture.
This skill set will serve infrastructure developers working with live environments or seeking to pass certification exams. Most importantly, it will help students understand cloud security in a comprehensive and thorough way.
For feedback, queries, or suggestions relating to this course, please contact us at firstname.lastname@example.org.
- Scan infrastructure using Azure tools to prevent drift leading to compliance violations
- Automate configuration using Azure Automation and Desired State Configuration
- Create secure and compliant software pipelines in Azure
This course is intended for:
- Those looking to learn more about the security and compliance features in Azure
- People studying for Microsoft's AZ-400 exam
To get the most from this course, you should already have a basic understanding of Microsoft Azure as well as some knowledge of programming and cloud infrastructure.
Through sections one and two of this class, we have gradually ascended a ladder of automation, if you will. We started by talking about compliance and creating policies to ensure our system is secure. Then we figured out how to transparently automate much of that configuration using DSC. And then we automated even further by using Azure Automation State Configuration and VM agents with custom scripts to ensure things stay compliant. Now it's time to assume our ultimate form of automation: Using secure pipelines to guarantee that every code or config change deployed to our system is automatically checked, tested, and released with full compliance and security totally assured.
This is where the fun begins. So let's start by talking about pipelines. A software pipeline is a process for getting a code change from a developer's laptop out into the real world with end users. Obviously we don't want to do this in a cavalier fashion. We don't want to just SCP a script up to a server and restart it and call it a day. We want to test things out. We want to check code quality. We want to make sure everything is reviewed and compliant and safe before finally deploying to our production environment.
So the modern paradigm for this is known as CI/CD, or, continuous integration and continuous delivery. In the DevOps world we use tools like Github, Jenkins, Gitlab, CircleCI to automate a lot of this work. Microsoft Azure offers its own very robust solution in the form of Azure Pipelines.
Azure Pipelines are much like other CI/CD services. They let you automate building, testing, deployment of software. And the set up is very simple. All you need to do is define an organization in the Azure DevOps portal and make sure that your code is hosted in some version control system like Github. You can create and configure your pipeline from the browser. However, to keep a proper infrastructure as code methodology, you will want to put all of your pipeline configuration in a yaml file, something named azure-pipelines.yml. This is the file Azure will care about, and this is where you will define your pipeline steps and tasks.
Now, there's many different stages, you know, test stage, deployment stage, within a pipeline. One of the most important features though specific to Azure Pipelines is the gate. Gates are very important. Our release pipeline has these different steps and between these steps we can define gates. Gates are basically conditions or checks that ensure our software meets some standard before moving on to the next step. If our release fails the check performed by the gate, it will stop the CI/CD process and give us an error message. This is an essential component of automating security compliance in our software delivery process.
Now there are many kinds of gates that we can select. In the web UI in the DevOps dashboard we can see the different options. There are delay gates that simply pause the pipeline for some amount of time, as well as manual intervention gates that stop everything until a human performs some action, perhaps a check or something. There are also gates that run Azure Functions. There are gates that check Azure monitor alerts. There are gates that invoke REST APIs. But perhaps most importantly, there are gates that check compliance with security policies. Now that last one we'll come back to in a bit.
So now, a secure software pipeline does not start with security policies and checks and gates or test environments running a build of the app. Really, the pipeline starts with a solid code review process. And that process should incorporate automated code quality checks, Linters, manual reviews by other engineers. This is really where you start. And we can kick off this process for a specific branch, if we're using git, and a code hosting service like Github, and then have merge requests that launch deeper tests. Code that passes initial checks and manual review should undergo static analysis.
Now, ideally you also are checking OSS licenses and third party dependencies as well. But this can be a big pain, but there are some tools that can help, tools like WhiteSource software. They can help you with it. Code that has made it this far, where you're at least checking the third party dependencies and open sourced licenses. Once it's gotten past all of that analysis, you wanna deploy to a test environment. What that deploy looks like exactly will depend on your application. It will depend on your specific architecture. It might entail spinning up a virtual machine, or deploying a container, or updating a storage bucket, or something else. Since our focus is on security we won't delve too deeply into that, into infrastructure minutiae. We will instead walk through a simple example of that in our demo anyway. And what we wanna focus on is that once our software is in a test environment, we need to do some form of security scanning. Scanning is really critical and there are a lot of tools.
Microsoft, in their documentation, they recommend using OWASP ZAP for handling automated penetration testing. Now, what is OWASP ZAP? Okay, OWASP, O-W-A-S-P, stands for Open Web Application Security Project, and ZAP is one of their tools, it stands for Zed Attack Proxy tool. The OWASP group does a lot of great work in security. It's a big, long-lived international group. They work with an open source tool set, so very easy to audit and look at and configure. OWASP ZAP is one of their most popular tools for web application security scanning. And it's readily integrated into an Azure pipeline process as part of your gates, if you will.
The challenge that we face with CI/CD pipeline, in a CI/CD pipeline context, with this tool, is that really thorough scans that crawl through an entire website and attempt a wide variety of attacks, these take a long time to run. It's really common for these scans to run anywhere from tens of minutes to several hours for a more complex app. So therefore, if you make these tests into a gate for any and all code changes, it will slow your development cycle down to a crawl. And so, for this reason, it's recommended that you have a more simple automated scan for CI/CD. And again, OWASP ZAP, they have tools for this, simple test scans. And then, separately, you have a nightly full active scan that does a comprehensive test in a QA or a dev environment. So this is kind of a split approach.
Penetration testing is really important for application security, however we have to also ensure that our infrastructure meets compliance standards. And fortunately this is readily handled by using gates in our pipelines. As we mentioned, in the Azure DevOps dashboard we can create pre-deployment conditions as gates. And as mentioned, there are many types, but the one that's really interesting to us here is the Security and Compliance Assessment type of gate. This type of gate ensures that a release in our pipeline will fail if it violates a policy defined in Azure Security Center.
Now, this feature lets us incorporate compliance and security right in our release process. It relies on us first selecting the right security policies defined in Azure Security Center. Now remember, good policies alone don't make us secure; compliance with good policies are what make us secure. And therefore, if we can make the check for the compliance automatic, then we can make security automatic. And fortunately, Azure Security Center includes several industry standards right out of the box, so you can just select the one that best matches your product without actually having to define an entire set of security standards and policies from scratch.
So, there you have it, whew! This was a doozy of a lesson, a lot of material, but hopefully chock-full of useful stuff. If we can put all or most of this into practice, we can sleep comfortably knowing that our Azure infrastructure is as secure as can be. Check out the demo for this section for a walkthrough of how to set up a pipeline and use gates. And, once you're done with that, you're finished with all of our lesson content, so, see you there. Thank you.
Course Introduction - Compliance & Security Scanning - Security Center Demo - Preventing Drift - ARM, Activity Log & Track Changes Demo - Desired State Configuration (DSC) - Azure Desired State Configuration Demo - Azure Automation State Configuration - VM Agents & Extensions - VM Agents & Extensions Demo - Azure Pipelines & Gates Demo - Course Summary
Jonathan Bethune is a senior technical consultant working with several companies including TopTal, BCG, and Instaclustr. He is an experienced devops specialist, data engineer, and software developer. Jonathan has spent years mastering the art of system automation with a variety of different cloud providers and tools. Before he became an engineer, Jonathan was a musician and teacher in New York City. Jonathan is based in Tokyo where he continues to work in technology and write for various publications in his free time.