Hello, and welcome to this lecture, where we will look at how AWS CloudTrail interacts with AWS CloudWatch and SNS to create a monitoring solution. In addition to S3, the logs from CloudTrail can be sent to CloudWatch Logs, which allow metrics and thresholds to be configured, which in turn can utilize SNS notifications for specific events relating to API activity.

CloudWatch allows for any event created by CloudTrail to be monitored. This enables a whole host of security monitoring checks to be utilized. A great example of this is to be notified when certain API calls request any significant changes to your security groups or network access controllers within your VPC.

Other examples of these checks that are common within organizations are API calls relating to starting, stopping, rebooting and terminating EC2 instances. If instances are being created that shouldn't be, your AWS costs could rise dramatically and quickly. Also, if instances are being rebooted or stopped, this could have a severe impact on your services if they're not configured in a highly available and resilient solution.

Also, changes to security policies within IAM and S3. If changes are being made to your policies that shouldn't be, access can be inadvertently removed for authorized users and access granted to unauthorized users, having a massive impact on operational services. Even a minor change to a policy can pave the way for an untrusted user to exploit the error.

Looking at failed login attempts to the Management Console. Monitoring failed attempts here can help to prevent unauthorized access at your environment's front door.

API calls that result in failed authorization. Not only does CloudTrail track successful API calls, whereby the correct authorization was met by the authenticated identity, but it also tracks unsuccessful API requests, too, which would likely be due to permissions applied. Special attention should be applied to these unsuccessful attempts, as this could be a malicious user trying to gain access. However, it could also be a legitimate user trying to access a resource they should have access to for their role, but the incorrect permissions have been applied with their associated IAM policy.

To configure CloudTrail to use CloudWatch, you must first create a trail. Once your trail has been created, you can then configure it to use an existing CloudWatch Log Group, or have CloudTrail create a new one. Having CloudTrail create a new one for you is recommended if it's the first time doing this, as CloudTrail will take care of all the necessary roles, permissions, and policies required.

You may be wondering why roles and policies are required. So let me give you a high-level overview of the simple process that takes place when sending CloudTrail logs to CloudWatch. When a log file is created by CloudTrail, it is sent to your selected S3 bucket and your chosen CloudWatch Log Group, assuming your trail has been configured for this feature. To allow CloudTrail to deliver these logs to CloudWatch, CloudTrail must have the correct permissions. And these are gained by assuming a role with the relevant permissions needed to run two CloudWatch APIs. The first one is CreateLogStream and this enables CloudTrail to create a CloudWatch Logs log stream in the log group. And the second one is PutLogEvents, and this allows CloudTrail to deliver CloudTrail events to the CloudWatch Logs log stream. Then finally, CloudTrail will then deliver logs to the CloudWatch Logs.

When using the AWS Management Console, you can have CloudTrail create this role for you along with the correct policy. By default, the role is called CloudTrail_CloudWatchLogs_Role. For those that are curious, the policy for this role looks as follows.

It's important to point out that CloudWatch Log Events have a size limitation of 256 kilobytes on the events that they can process. Therefore, any events that are larger than 256 kilobytes will not be sent to CloudWatch by CloudTrail. Now that you have your logs with the associated events being sent to CloudWatch, you must now configure CloudWatch to perform analysis of your CloudTrail events within the log files. This is done by configuring and adding metric filters to the log within CloudWatch. These metric filters allow to search and count a specific value or term within your events in your log file, which then allows for customizable thresholds to be applied against them. When creating these metric filters, you must create a filter pattern, which determines what exactly you want CloudWatch to monitor and extract from your files. These filter patterns are fully customizable strings, but as a result, a very specific pattern syntax is required. So if you are creating these for the first time, you must understand the correct syntax. AWS has a great page within their documentation that gives full examples of how to create your filter patterns, which can be found here.

Just to reiterate what we've spoken about so far, I want to provide a demonstration on how to edit an existing trail to configure it to send logs to CloudWatch Logs. I will then configure a metric filter with the associated metric pattern. Finally, I will set up an SNS alert to notify me when a particular threshold is met, so let's take a look.

Okay, so what we need to start with is going into CloudTrail to edit an existing trail to enable CloudWatch Logs. So if I go down to Management Tools and click on CloudTrail, and then across to Trails, and this will show our existing trail that we created earlier in the previous demo, which is ca-trail. As you can currently see, under CloudWatch Logs Log group, there's no log group selected. So if we go into the trail, and then scroll down to CloudWatch Logs, click on Configure, and then we can get CloudTrail to automatically set up this group, and it will create the necessary roles and permissions, etc. So let's call this CloudTrail/Demo.

And then click on Continue, so we've given it a name, and here it just gives a message to say that for CloudTrail to deliver event logs to CloudWatch Logs, it needs to assume a role with permissions to run two API calls, which are these two here. If we go down into the details, we can see that the IAM role that it's going to use is the CloudTrail_CloudWatchLogs_Role, and we'll ask it to create a new policy. And here's the policy document, which I showed earlier on the presentation. So go down to Allow. And then if we scroll down to our CloudWatch Logs section, we can now see that we have a log group created in CloudWatch called CloudTrail/Demo.

So if we now go across to CloudWatch, if we click on Logs on the left-hand side here, we can see that we have our log group that was just created by CloudTrail, and it's CloudTrail/Demo. Now if we go into our log group, select it, you'll see this log stream, which is the incoming stream of events being sent from CloudTrail. Now as we've only just started, there's only a few events coming in here, so you might want to wait a few minutes before setting up your metric filters to give you more of a test pattern to search on. So what I might do is just leave it a couple of minutes for some more events to start streaming in before we set up our metric filters here, just so we have something to search on. Okay, so I've left it a few minutes. So let's go back into the log group, and you can see we've now got a couple of streams, and if we go into these, we can see there's a lot more events.

So if we go back a couple of pages, back to our log group, now we need to create our metric filters to allow us to define what we want to search on within our logs. So if we select the tick next to our log group, and then go up to Create Metric Filter, and here within the metric filter, we need to define a filter pattern. Now as I explained earlir, filter pattern will define what we're actually searching for within our logs. So for this example, I'll keep it fairly simple. I'm goning to search for any API call that's being made from my machine, so from my IP address. So for that, I need to enter the following command: sourceIPAddress = 2.218.11.188, which is my IP address. And now we can test to make sure that that filter pattern is okay using this Test Pattern box here, and what that does, that will run this test filter on some log data we see from this log here, and the output of that log is in this box here. So all we need to do is click on Test Pattern, and we can see at the bottom here that it found 47 matches out of 50 events in the sample log. So we know that the syntax is okay for this filter pattern, so I'm going to go ahead and assign this metric. And we can see up here that we've got our filter name and our filter pattern, and I'm going to create a new namespace for this metric, and I'll call it Demo, and metric name will be IPAddress. And then all we need to do is click on Create Filter. Now as you can see, our filter has been created, and we have the details in this screen here.

Now what we can do at this point is create an SNS alarm so we can be notified if a certain threshold was met. So let's go ahead and do that. So the first thing that we need to do is add a name. So I'm going to call this SourceIPAddress, and the description will be Too many calls from my IP. Now I'm going to set this to be 30. So whenever my IP address is used as a source IP address that is greater or equal to 30 times for one consecutive period over five minutes, then I'll want it to set to a state of an alarm, and I'm going to want to be notified, so I'm going to enter a new list, give this a new topic, SourceIPAddressAlarm, and I want that to be sent to myself. So as we can already see, with the current data it's got, that it has already breached the alarm, but it has dropped back down below, so we'll see how this goes. And we'll create the alarm. And this is a message just to say that I need to subscribe to that AWS notification, and I can do that in just a few moments.

So if we go across to Alarms, we can see that we have our SourceIPAddress alarm in the state of OK. So at the minute, it's currently below the 30 threshold. As soon as it goes above that, it will alarm, and I will get a notification. Now over the past few minutes, I've just been having some activity within the Management Console, and as we can now see, we do have an alarm on our alert. We can see that it just crossed the threshold, and so I've received an email notification to say that it is now in a state of alarm, and if we take a quick look at that email, we can see here that it was crossed with a data point of 33, and the threshold was 30.

So that is how you set up CloudTrail to use CloudWatch with the inclusion of SNS to create alarms against API activity.