Observability in AWS
The course is part of this learning path
This course covers the core learning objective to meet the requirements of the 'Architecting for Management & Governance in AWS - Level 2' skill
- Understand the different AWS management services available to monitor the performance of a solution
- Apply Amazon CloudWatch monitoring contols to respond to system-wide performance changes
- Apply AWS Config controls to manage compliance based upon business guidelines
Hello and welcome to this lecture where we'll begin to look at how to best manage your compliance that you need to adhere to within your AWS environment.
Your compliance requirements can come from many different sources, for example, you may have a requirement for your environment to be HIPAA compliant if you're managing healthcare records. Or perhaps your security theme requires certain criteria to be adhered to such as ensuring SSH is not applied to a specific security groups. Or your operations team may dictate to you that certain EC2 types should only be used for certain environments to conform to internal standards. Maintaining compliance and internal standards for numerous parties, both internal and external, can be difficult to manage and can lead to mistakes, which in turn will eventually lead to non-compliance services making their way into production. This can be a huge risk from an audit perspective and also from a security stance.
AWS Config allows you to utilize config rules to help you manage and organizes compliance which acts as an automatic resource compliance checker. When a change is made to a resource, AWS Config will check to see if the resource matches the rule with the help of a lambda function and if so, it will check the compliance of that resource against the rule following the changes made.
There are two different types of Config Rules within AWS Config, custom config rules and AWS Managed Config Rules. AWS Managed Rules are a set of predefined rules that cover a lot of best practices, so it's always worth browsing these rules first before setting up your own as there is a chance that the rule may already exist. At the time of writing this course these Managed Rules cover five different topic areas, compute, database, management tools, security, identity and compliance, and storage. Do bear in mind that you can edit specific parameters of Managed Rules within in these topics, and so if there is a rule that very closely matches your requirements, then you may be able to edit it by selecting different triggers and parameters to reflect the changes that you need. This can save you a lot of time trying to recreate your own from scratch. Later in this lecture I will provide a demonstration of how to modify these managed rules, along with how to create your own custom rules.
Before creating, modifying and configuring your Config Rules you should first identify what compliance and standards that you need to adhere to. Define the requirements from all parties but remember, there is a limit of 50 config rules per region before you need to contact AWS to request an increase.
Let's consider a simple scenario on how these Config Rules could be used. Let's say we are security architects responsible for ensuring specific security requirements are met in the following deployment. A new web application is being released into the production environment that consists of multi-tiered infrastructure, with front-end web servers in a public subnet, application servers with EBS and a back-end RDS database. The application and the environment itself is expected to scale with demand, both up and down.
We have been told that from a security perspective SSH must not be used for the web server security group under any circumstance. Where used, all EBS volumes must be encrypted, the RDS database must be configured for high availability at all times and data stored by RDS must be encrypted. Now obviously, during deployment we can oversee the implementation and ensure that each of these elements are met. We can make sure the security groups are correctly configured, storage encryption is activated and that multi AZ is configured for high availability for the RDS instance. So, why the need for these config rules?
Well, now let's consider the following. Once the initial deployment was carried out and security checks were carried out manually, the environment was then handed over to support and operations to maintain and look after. Over time the application scaled up, additional resources were added and there may have been some ad hoc incidence and general maintenance of the environment which was carried out. The support and operations team, although they were made aware of the security requirements, they may not have adhered to them at all times, perhaps due to human error, lack of knowledge or laziness. Your environment now has a situation where some volumes are not encrypted and SSH has been activated on the web server security group perhaps to troubleshoot an existing incident.
As you can see, over time your environment changes and maintaining the same level of security implemented at the start of the project may not be continuous throughout its lifetime, which leads to mistakes and security holes within your infrastructure. AWS Config Rules can notify you of these security misconfigurations. If for example, in this situation during deployment we activated the followed AWS Managed Rules, restricted-SSH, which would check whether security groups would disallow incoming SSH traffic, encrypted-volumes, this would check whether EBS volumes are encrypted, RDS-multi-az-support checks to ensure high availability is configured for your RDS instance, and rds-storage-encrypted, and this checks if storage encryption is enabled for your RDS instances. With these config rules in place, a notification would have been sent to the configuration stream stating that a resource had changed its state from being compliant to non-compliant.
These notifications could have been configured programmatically to notify your security team, who could have then investigated as to why this had happened, and understand who made the change through the use of the configuration item by utilizing the CloudTrail information recorded when the change happened to the resource. By using Config Rules it allows the appropriate action to take place when a notification is received of a non-compliant resource, which may include resolving the security issue identified and ensuring the resource becomes compliant again. Remember, just because a resource is flagged as non-compliant, does not mean it is removed from your environment. It will continue to function.
Identify who or what made the change. If an employee made the change advise and educate them on the importance of security and why that resource requires set configurations. If it was a service, check your automation configurations to ensure it doesn't happen again. So, as you can see using AWS Config Rules allows a continuous monitoring solution for a wide range of uses to ensure compliance is maintained within your environment.
The example we have used was a simple security scenario, but the scope is huge, especially when implementing custom config rules. Before we finish this lecture, I just want to provide a quick demo to show you how to select and modify an AWS Managed Rule and also how to configure a custom rule where I will show you the different elements involved in the process.
Okay, so I'm at the AWS Config dashboard and at the Ruled page. So firstly, what we're going to take a look at is how to modify the parameters of an AWS Managed Rule.
So, if we go to Add Rule, and for this demonstrations I'm going to use the Desired Instance Type Managed Rule, so this checks whether your EC2 instances of the specified instance type.
So, we have a name at the top which we can edit if we need to. The description of the rule, it checks whether your EC2 instances are of a particular type. Here's the managed rule name which is the lambda function and here we have the triggers. Now, there's two different trigger types here. When there's changes to configurations or you can have periodic triggers, so at a set period of time. So, we got it as configuration changes for this managed rule.
Now look at the scope of changes. This identifies what resources are in scope for this rule. At the minute it's selected Resources and under Resources we have EC2 instances. We can also select tags and then add our own tag key pairs in there, so whatever we enter there the managed rule will only apply to any resources that have those tags within them or all changes. But I'm going to leave it as Resources, so we want this rule to run against any EC2 instances, and then triggered when there's any configuration changes.
Now, down here we can edit the parameters. So, we want to specify which instant types that we want to have running within our environment. Now, for our environment we want to make sure we're just running T2.small instances or M4.large. So, just take a look at this AWS Managed Rule again. We have the name, the description, the actual lambda function itself, which is the evaluation of the rule, we have the trigger type and we've left it as configuration changes, and we have the scope of change, we've selected a resource, so we want this to be based on all EC2 instances, rather than specific tags. And then for the instance type we've edited and added values of T2.small and M4.large. Now, depending what kind of AWS Managed Rule you have, there'll be different configuration parameters that you can change. I just wanted to highlight here that you can add your own parameters for specific AWS Managed Rules, and in this instance we've added our EC2 instance types here. Then once you've done that, you just click on save and then AWS Config will go off and evaluate your current environment. And then after a period of time, that evaluation will come back and give you your compliance results.So, what I'll do, I'll pause the video here, wait for that to finish and then show you the results.
Okay, so that evaluation has now been completed and we can see that under our new AWS Managed Rule, the desired instance type we have two non-compliant resources. So, let's take a look. And here we have our two EC2 instances, both coming back with non-compliance checks. Now, we can take a look at these instances. If we follow around inside here you can see Managed Resource and this will give us a link to each instance. So, let's take a look. So, we can see here that this instance type is actually a T2.nano, which is non-compliant with our T2 small and M4 large that we specified in the rule.
So, if we go back to Config and take a look at the second instance, and we can see here that this is a T2.micro, so again, non-compliant. So, that's the end of this demo on showing you how to edit the parameters of a AWS Managed Config Rule.
We'll now take a quick look at how to create your own config rule, which is very, very similar.
Okay, so to create your own config rule is very similar steps. Again, click on Add Rule and instead of selecting one of the predefined AWS Managed Rules as before, all we do is click on Add Custom Rule, and here you add a number of details. So, you can add your own name, so Cloud Academy Rule, description, this is a test for a demo. Here is where you'll enter the ARN of the lambda function that you've created. That will then evaluate your resources to see if they comply with this rule. Now, I'm not going to go and create a new lambda function because it's out of scope of this course, but for those who are experienced with lambda, then this where you would enter your ARN function.
Now, further down this is where we add our trigger types like we just discussed with the AWS Managed Rules. You can have the trigger to happen every time there's a configuration change, or over a specific period. So, with regards to the period, you can select 1 hour, 3 hours, 6 hours, 12 or 24. So, this rule will run at that set period regardless of if there's any configuration changes, but you can have both ticked regardless.
Now going down to the scope of changes, and again as we mentioned within the AWS Managed Rules, it can be against your resources, particular tags or for both. So, if we select Resources we can specify our resource type, for example, VPCs, subnets, security groups, so you can have multiple resource types applying to the same rule.
And then again, down here you can have your rule parameters. Here you would define any values that your lambda function would need to evaluate for this rule. And once you've entered all your information, then all you'd need to do is then just to click on save and then that will go ahead and create your custom rule with your own lambda function and again, it'll follow the same process of evaluation and then return any non-compliant results.
And that brings us to the end of this demonstration. For a full list of current AWS Managed Rules that we spoke about within this lecture, then please visit this link. That brings us to the end of this lecture. Coming up next we're going to look at some different use cases for AWS Config.
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.
To date, Stuart has created 150+ courses relating to Cloud reaching over 180,000 students, mostly within the AWS category and with a heavy focus on security and compliance.
Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.