Hello and welcome to this lecture where I shall explain what CloudWatch Logs are, how they work and how they are configured. As we know CloudWatch is a monitoring service that is used to collate and collect metrics on resources running on your AWS account allowing you to monitor their performance and respond to alerts that meet to find thresholds. In addition to this, Amazon CloudWatch is a powerful tool that allows you to collect logs of your applications and a number of different AWS services. 

When data is fed into Cloudwatch Logs you are able to monitor the logstream in real time and set up metric filters to search for specific events that you need to be alerted on or respond to. This allows CloudWatch Logs to act as a central repository for real-time monitoring of log data. Let me explain the different components of this feature to give you a better understanding of how it all fits together. Starting with the Unified CloudWatch Agent. 

With the installation of the Unified CloudWatch Agent you are able to collect logs and additional metric data from EC2 instances as well from on-premise services running either a Linux or Windows operating system. Interestingly this metric data is in addition to the D4EZ2 metrics that CloudWatch automatically configures for you. The list of these additional metrics collected by the agent can be found at this link here. The agent can be installed on a number of different operating systems and at the time of writing this course the operating systems versions supported are as follows. 

To install the agent and configure it requires a number of different steps. To install the agent on your EC2 instances you need to perform the following, firstly you need to create a role and attach it to the instance with permissions allowing CloudWatch to collect data from the instances in addition to interacting with AWS systems manager SSM. You then need to download and install the agent onto the EC2 instance. And lastly configure and start the CloudWatch Agent. The most efficient way of completing this installation and configuration is with the use of the EC2 systems manager service known as SSM. You will need to create two roles. One role will be used to install the agent and also to send the additional metrics gathered to CloudWatch. The other role is used to communicate with the parameter store within SSM, to store a configuration file of the agent which then can be shared with other EC2 instances. 

From a security perspective it's best that only one of your EC2 instances has this permission to write to the parameter store. Once the agent configuration file is stored on SSM there is no need for other EC2 instances to do the same. In fact once the write has been completed, this role should be detached from the EC2 instance and the other role applied which simply allows the agent to send data to CloudWatch. 

The role with the additional permissions for SSM needs to be configured as follows when creating a role. The options, 'select the type of trusted identity' needs to be 'AWS service'. The option to 'choose the service that will use this role' needs to be 'EC2 Allows EC2 instances to call AWS services on your behalf'. The 'Attach Permissions Policies' needs to be 'CloudWatch Agent Admin Policy' and the 'Amazon EC2 role for SSM'. 

The role that is simply used to install the agent and send data back to CloudWatch needs the following configuration, the 'select type of trusted identity' needs to be 'AWS service'. The option 'choose the service that will use this role' needs to be 'EC2 Allows EC2 instances to call AWS services on your behalf'. And finally under the 'Attach Permissions Policies' it needs to be 'CloudWatch Agent Server Polic'y and 'Amazon EC2 Role for SSM'. 

Once your roles are created you can then associate the role shown in orange here, which I shall call 'CloudWatch Agent Admin Role' to your EC2 instance that will store the configuration file in the parameter store. 

From the EC2 instance with additional permissions that will be saved in the configuration file with in the parameter store of SSM, you must then install the agent which can be done using systems manager or it can be downloaded from an S3 public link either for Linux or Windows. However as mentioned earlier I will explain how to do this via SSM in particular the run command function. As a prerequisite to the CloudWatch Agent installation you'll need to verify that your EC2 instance has access to the internet to communicate with SSM and CloudWatch endpoints. In addition to this you must also have the SSM agent installed. For some AMI's as stated on the screen the agent may already be installed. For more information on how to install or update your SSM agent on your EC2 instance please see the following link. Let me now perform a very quick demonstration to show you how to install the actual CloudWatch agent on an Amazon Linux EC2 instance using the SSM role command. 

Start of demonstration 1

Okay so I'm within my AWS management console in the EC2 dashboard and as you can see I have an EC2 instance here named Logging Server and I have it in a VPC that has internet access. I have the SSM agent installed because it's based on one of the latest Linux AMI's comes by default with this, and I've also attached the CloudWatch Agent Admin Role which I discussed in the previous section so I've met all the prerequisites to install the CloudWatch Agent. 

Now what I want to do is use the EC2 systems manager to install the CloudWatch Agent itself. So if I load up SSM which now has it's own console and on the left hand side if I go down to the run command and click on run command to create a new command. What I want to select for the command document is the AWS configure AWS package. So if you just select the entry and then scroll down. And then next I need to select which EC2 instance that I want as the target. So which instance I'm going to install this package on and here we can see the Logging Server so I would just highlight that. And as you can see it's added our EC2 instance already up there. 

Now we come down to the command parameters and the action I want to perform is an install. The name of the package is Amazon CloudWatch Agent and I want to install the latest version. You can add some other comments here and some timeouts, for this demonstration I'm just going to leave those default. If you wanted to perform this same installation via the AWS CLI then you can click on the AWS command line interface command and then based on the above parameters you can simply cut and paste this command and run that command. But for this demonstration I'm going to use the run command within SSM. 

So if I click on run we can see here that the command ID was successfully sent. It's currently in progress and that will just take a moment to process and install the agent, then we should get a return of successful. So if I go back to the main dashboard of the run command we can see here that it was a success. This is the package that we just run using using this command ID, so now the agent is successfully installed. 

As you can see here I have a previous command that failed so I just want to show you that worst on here. So if we select that command and go to view details we can drill down to understand why this failed. And if we scroll down to the bottom you can see here that it failed. Select the instance ID and view the output. Now step one, we can dive deeper to see why this failed. Now if we scroll down to the error itself and here it said it failed to retrieve the manifest and it could not find the latest version of this particular package. Now what happened here was I didn't enter the correct package name. What I entered was CloudWatch Agent instead as we performed in the demonstration just now. The correct name is Amazon CloudWatch Agent. So it couldn't find the package that was available and that was the reason why it failed. 

However, going back to one that was successful, we can see here, we can drill down into this just to make sure everything was okay. Again here, success and we can view the output of this as well like we done with the failed one. And we can see here that it was all installed. It found all the files and it went through and processed everything okay. And that's it, that's how you install the CloudWatch Agent using the SSM run command. 

End of demonstration 1

On your first instance you'll need to create the CloudWatch Agent configuration file. Without doing so, you will not be able to start the agent. This file stores configuration parameters that specify which metrics and logs to capture on the instance which are then sent to CloudWatch. It can be created manually or by using a wizard. If you create it manually then you have a much wider scope for capturing elements that are not included within the wizard. Let me show you via another demonstration how to configure the agent using the wizard. 

Start of demonstration 2

So I'm now on my Logging Server instance where we installed the agent and now need to configure it. So to configure it we'll run this command here and this will launch the Amazon CloudWatch Agent configuration wizard. And it just goes through a number of questions before it completes so let's take a look. So the first question it asks which operating system your using Linux or Windows, take the default choice of Linux. And if we're going to try and collect logs on EC2 instances or On-Premises So it's EC2 for us. 

We then have a question if we want to monitor any host metrics such as CPU, memory, etc, I'm going to set a default of yes. And if we want to monitor CPU metrics per core and here additional CloudWatch charges may apply. For this demonstration I'm going to say no. We then have a question if we want to add any EC2 dimensions such as the instance ID or the instance type into our metrics if the information is available. Say yes. Then we have a question about how often CloudWatch will collect these metrics. Whether that's one second, 10 seconds 30 seconds or 60. I'll accept the default of every minute, 60 seconds and then asks which default metric config we want whether that's basic, standard, advanced or none. I'll accept the default of basic for this demonstration. It then just shows you an example of that configuration that you just selected. If you're happy with that you can say one for yes or two for no. And then select a different default metrics config so I will just select yes for this demonstration, I'm happy with that. 

Now it asks the question if we have an existing CloudWatch log agent configuration file that we want to import. At the moment we don't but what we're trying to do is get in the position of creating one to then import into the parameter store of SSM. So at this moment our default choice is no which is correct. It then asks if we want to monitor any log files on this EC2 instance. I'm going to say no just for this demonstration because all we are trying to do is configure the CloudWatch agent at the moment. Our next question asks if we want to store the config in the SSM parameter store, and here yes we do cause we want to upload this file to the parameter store to allow all other EC2 instances to connect to the SSM parameter store to download the configuration file to prevent us from doing this on each and every instance. So I'm going to say yes which is number one that we do want to store the config in the SSM parameter store. 

It then asks what default name you want to give this configuration file. And it just gives you a little message there saying you should use the prefix of Amazon CloudWatch hyphen if you're using any of the AWS managed policies. So I would go with their suggested name of Amazon CloudWatch Linux and you shouldn't run into any issues there. It then asks you which region you want to store the config in the parameter store in and it gives a default choice. I'm just going accept that default. It then asks a question about which credentials should be used to send the config to the parameter store. I'm going accept the default credentials there which should have access and we now have a message that it successfully put the config to the parameter store Amazon CloudWatch-Linux. And that's it, that's the end of the wizard so it's a very simple and quick and easy wizard. And now your CloudWatch agent configuration file is configured and it's been uploaded to the parameter store so now any other EC2 instances can click to that parameter store and simply download it and have the agent running very quickly and easily. 

End of demonstration 2

Now the configuration file is configured and successfully copied to the SSM parameter store I simply now need to start the agent and again I will use the systems manager service to complete this file with the run command. Let's take a look. 

Start of demonstration 3

Okay so the final stage is to start the agent and again I'm going do this from the AWS systems manager so I'm at the systems manager dashboard. Again I'm going use the run command so so it's over on the left hand side here. Click on run command. Then click on the orange run command button and this will allow us to start a new command for the command document. What we need to look for is the following, which is Amazon CloudWatch Manage Agent. And here we have the command document here. So I'm going to select that. 

Scroll down, we then need to select our target instance and we have our Logging Server here. Now the command parameters, for the action we want to select configure rather than stop because we're going to select the configuration file that we uploaded to the parameter store first using the EC2 mode rather than On-Premises. The optional configuration source which is SSM which is where we stored the configuration file so we'll leave that as SSM. Now in the optional configuration location we need to enter the name of the file that we stored it as. And if you can remember that was Amazon CloudWatch-Linux. So we'll paste that in. 

Now under optional restart we want that as yes because it will then start the agent once it's pulled the information from SSM. If we scroll down to the bottom simply click on run. And now we have it, the command ID was successfully sent. If we go back to our dashboard we can see that it was a success. And here the document name was the CloudWatch Manage Agent. So the CloudWatch Agent will now be running on that EC2 instance, the Logging Server. And it's as simple as that. Now we have our logs configured and log data of our EC2 instance is being sent to CloudWatch along with the additional metric information. It's now possible to search for specific entries within the logs for points of interest. 

End of demonstration 3

That now brings me to the end of this lecture on CloudWatch logs where I explained how CloudWatch can be used to centralize login for more EC2 instances or applications running on your instances by determining logs past configured by the CloudWatch Agent and log groups within CloudWatch. Coming up next I should be talking about CloudTrail logs.