Sending Log Files to Cloudwatch


Course Introduction
Course Conclusion
Start course
1h 34m

Please note: this course has been outdated and replaced with two new courses: An Overview of Amazon CloudWatch and Building CloudWatch Dashboards

CloudWatch is a monitoring service for cloud resources in the applications you run on Amazon Web Services. CloudWatch can collect metrics, set and manage alarms, and automatically react to changes in your AWS resources. Amazon Web Services Cloudwatch can monitor AWS resources such as Amazon EC2 instances, DynamoDB tables, and Amazon RDS DB instances. You can also create custom metrics generated by your applications and services and any log files your applications generate. You’ll see how we can use Amazon CloudWatch to gain system-wide visibility into resource utilization, application performance and operationally you’ll use these insights to keep applications running smoothly. This course includes a high-level overview of how to monitor EC2, monitor other Amazon resources, monitor custom metrics, monitor and store logs, set alarms, graph and view statistics, and how to monitor and react to resource changes.

Intended Audience

  • Systems Admins
  • Operational Support
  • Solution Architects working on AWS Certification
  • Anyone concerned about monitoring data or AWS recurring billing


  • AWS Console Login
  • General knowledge of how to launch an Elastic Compute Cloud (EC2) instance on either Linux or Windows
  • View CloudWatch Documentation at
  • An operational EC2 (Windows/Linux)

Learning Objectives

  • Monitor EC2 and other AWS resources
  • Build custom metrics
  • Monitor and store log information from Linux instances
  • Set alarms for metrics to take action on an instance or auto-scaling group
  • Create a dashboard to monitor EC2 instances
  • React to load to trigger auto scaling horizontally within AWS.

This Course Includes

  • Over 90 minutes of high-definition video
  • Console demos

What You'll Learn

  • Course Intro: What to expect from this course
  • Getting Started: How to launch an EC2 instance
  • Building a Dashboard: How to take the metrics from the instance and create a dashboard
  • Monitoring EC2 Instances: How and why you should be monitoring the environment in Amazon Web Services
  • Sending Log Files to Cloudwatch: A lesson on the importance of sending log files to Cloudwatch
  • Alarms: How to specify alarms
  • Course Conclusion: Course summary

This is CloudWatch lesson five, sending log files to CloudWatch. My name's Michael Bryant and I'll be your instructor for this section.

Sending log files to CloudWatch simplifies administration, could eliminate local logins, and promotes easier parsing of collected data. It's possible in your particular situation you have a network operation center that monitors multiple servers. In your production environment, you may have many servers collecting all sorts of login data from Apache Access logs to system logs, and a variety of custom application logs.

If you don't have a central repository already, you can use CloudWatch to collect those logs and actually look at them via the Amazon console. You can also export logs that you've imported into CloudWatch to third party tools. This can be extremely helpful in the event of nefarious activity, such as hacking.

You could export all of your access logs, for example, to a tool such as Athena or another third party tool like Splunk, and then you could perform detailed analysis of the data to find out who's been accessing your network.

It is fair to say that CloudWatch logging is fairly comprehensive as a logging service. However, it's also fair to say that the analytics tools do not rival those of third party capabilities. While the third party tools can be important in any production environment for the analysis of the data collected, many of those tools are very expensive and focus on the analytics versus CloudWatch's collection of the data. It should be noted that the third party tools available for data analytics are well outside the scope of this course.

In this demonstration, we will send a log file to CloudWatch and then view it in the Amazon console. We'll be using the EC2 instance we created earlier.

By now you know I never miss an opportunity to remind you to check the current Amazon Whitepapers. In this case, for logging, I recommend that you search for an Amazon whitepaper entitled What is Amazon CloudWatch Logs. This will bring you to the top level page of the user guide for CloudWatch login.

You'll notice on the left side that we have a hierarchical menu and in those menus you'll find how to set up CloudWatch logging on various flavors of Linux. This would be very helpful if you're running something other than the Amazon Linux Ami that we're running and have been using throughout this course. It is necessary to set up different flavors of Linux, such as Scent or Ubuntu in a slightly different way.

If you're familiar with those versions of Linux, the whitepapers in here can walk you through the steps, which are a little bit different. I would recommend if you're using a different flavor of Linux to read the whitepaper first and then pay special attention to the prerequisite section. In that section of the whitepaper, you'll find all the required supporting files and install packages that you'll need to install Amazon CloudWatch logging.

In the demonstration I'm about to present, we're going to add CloudWatch logging to an existing EC2 instance. It is possible that you'd want to enable CloudWatch logging on the launch of a new instance. This is particularly important if you're using auto scaling and the number of instances that you have is dynamically changing in your environment. If you look at the whitepapers on the left, there's a full explanation of how to do what we are about to do manually in an automated fashion so that your servers will launch with CloudWatch enabled by default.

Join me in the Amazon console now, and we're going to verify that the role we created earlier has the correct permissions for CloudWatch logging. We'll click on IAM, and then take a look at the roles that we created. As we can see, we have CloudWatch full access. Let's go ahead and attach a policy for CloudWatch logs full access to the role that we created earlier. We'll validate that the policy is there by clicking show policy, and we'll see that with the action of logs, we are allowing to all resources.

What we've just accomplished here is we've attached the policy named CloudWatch logs full access to the existing role. In my case, the role name is Cloud Academy. If you have a different role name from the earlier lesson, of course you would attach the CloudWatch logs full access policy to that role. The key point here is the instance that we're using needs to have an IAM role that includes CloudWatch logs full access, so as you can see, our role here is named Cloud Academy, and we'll go over to EC2 and find our little instance and validate that this instance, which is named Cloud Academy, has the IAM role attached.

As you can see, in my case it does not have the IAM role attached, so what we will do is we will go to actions, instance settings, detach or replace an IAM role, and then I will select the Cloud Academy role and click apply. If you get a failure in this action, you'll get a red box and it will ask you to retry. In my experience, sometimes you get that once. If you do, click retry and you'll usually get the green box the very next time that says IAM role operation succeeded.

Once you see the Cloud Academy IAM role, or whichever your role is named, you can click on that and we'll validate once again that we have the CloudWatch logs full access policy attached to our EC2 instance. What's important about this at this point is that the IAM role, acting as a role, does not have user credentials. We'll see that later in our demonstration.

If for any reason you want to create a user account and apply the policy, you'll have to generate the key pairs. You may be wondering if you can use user accounts to push data to the Amazon API for CloudWatch and CloudWatch logging. In fact, you can. I didn't show that earlier or mention it because I don't think it's the most secure method and it possibly adds administration time.

If you create a user instead of an IAM role, you'll be required to manage credentials, such as the secret key and the access key. Once those are created, you might have to recreate them in the future, which means you'd go server by server to replace the credentials. Also, once the credentials are created, they have to be controlled. Otherwise anybody who has access to the keys, in theory could access your environment.

A lot of this depends on the security posture you select and how you implement it. In my opinion, I would rather have IAM roles applied to instances in my environment, and if I need to make a change, or more importantly, if I need to revoke a permission, I can simply revoke it from the IAM role or delete the IAM role or recreate the IAM role, or make any changes that I need, but if I need to recreate or if I lose the access keys to a user, then I have to recreate those and install them on the servers. This is something you'll want to think about as you go forward in your implementation strategy.

For the purpose of this course, I'm going to show my preferred method, which is to use the IAM role for the balance of our demonstration. I'm now attached to the small instance here, our T2 micro that I created named Cloud Academy. The first thing I'm going to do is update my server. After the update completes, we'll be installing the AWS logs package. We'll install the AWS logs package now. This is the point where checking the whitepapers is very important.

If you're running the Amazon flavor of Linux, you can type sudo yum install minus Y AWS logs. The next thing we're going to do is make sure that AWS logs is always running. We'll do that by check config-ing it, so we'll type sudo check config AWS logs on. This command will complete very quickly. What this step does is assure that AWS logs are running at start up. The next thing we're going to do is assure that we are actually pushing our CloudWatch logs to the correct region.

By default, I've noticed that AWS logs sets itself up to push all of your logging data to the US East One region. That's pretty convenient if you're already in the US East One region and you want to look at your logging there, but in my case, I purposely put us in the Northern California region so that we would have this problem.

Our problem now is that our AWS logging is in fact installed, but if we select any logs to push to CloudWatch, they'll be in a different region than our instances. I don't want that. I actually want to set this up so that we're getting logs in the very same region that our servers are actually operating. I've cleared my screen to make this a little bit easier to follow.

What we need to do to change what region AWS logging is going to push to, we need to go in and configure a file known as AWS CLI config. To do that, we'll change the directory to a set dash AWS logs. Let's take a look at what's in this file. Let's take a look at what's in this directory. There's a couple of important files. The first one is the AWS CLI dot comf. Let's go ahead and edit that. You'll see that the region for default is US East One. In my case, I'm in the Northern California region, so I'll modify this file to say US West One. If you need a list of the regions, that's available online and you can search that. The regions do change periodically because Amazon is frequently adding them. Once you save this file, we're going to go ahead and start the AWS logging feature.

Okay, so we've started AWS logging. We'll now go into our Amazon console, which happens to be just behind our terminal window, and we'll take a look in CloudWatch and see what we got. You should be able to click on logs and you'll see that the log group, var log messages, is available. This should match our server instance ID. And you'll see that logging has begun.

Now this is limited to the one log file, slash var messages, that's configured to upload. However, for this one instance, we could have additional log files. Let's take a look at how to do that.

One very popular thing to monitor is Apache. Let's go ahead and install Apache server. This way we'll have an access log to monitor. One thing that's important to remember is while we did install Apache, we didn't actually start the service, so there won't be much for us to log, if anything. Let's go ahead and start that service.

Okay, we've started the service, and we should have also started logging along with it. The Apache logs are actually protected from the default EC2 user. Let's enable root mode by typing sudo SU. Let's then find something that we want to log. In this case, we're going to look in the var log HTTP directory.

Go ahead and change your directory accordingly. Once you're there, let's take a look by either using LL or LS, whichever is your preferred method, and see what we can log. It looks like the error underscore log has some data. I purposely tried to start Apache earlier without upgrading my credentials, which caused an error.

We can take a look at this log by simply tailing it. As you can see, there are multiple log entries in the error log. What I'd like to be able to do is instead of logging in to this server, finding the certificates, and all of that sort of thing, I'd like to be able to look at the error log in CloudWatch. Let's go ahead and set that up.

Alright, what we need to do now is make a change to our AWS log configuration. I'll go ahead and clear the screen and then we'll change the directory to at set AWS logs. Looking in that directory, we see a couple of files. The key file here will be the AWS logs dot comf. Remember, if you do not have the root privilege, you'll need to type pseudo for this command.

So let's nano AWS logs dot comf. In here, you'll find the CloudWatch logs agent configuration file. This is how you'll set up each of the log files that you want to push to CloudWatch. At the very bottom, you're always provided default messages log. This provided the messaging that we saw before, the var log messages log file.

In this case, right underneath we'll create another entry, which is this little paragraph of information. I want to send the HTTPD error log file, so I'll go ahead and basically copy what's there and make the changes necessary so that the log file, error underscore log, gets pushed to CloudWatch.

Through the magic of copy and paste, what I've done is I've created another paragraph right under the var log messages that says var log HTTPD error, then the date and time format as specified just as above, and the file that I'm targeting is the var log HTTP error underscore log file that we just tailed a moment ago. I want this to go in the log group name var log HTTPD error. You'll want to save this file by exiting and then we'll do service, AWS logs, restart. You'll see that the AWS logs stopped and started successfully. Once you've updated that file, let's take a look back in the console at CloudWatch.

Let's click on logs, and you'll see that we have the var log HTTP error log group. This is the instance that we want to see, and you'll notice that we have the exact same logging that we saw in the tailing operation we did earlier is now available in CloudWatch. We no longer have to attach to the server to get this information. You may find this incredibly valuable, particularly if you have custom applications that produce log files.

You could use the same model just by updating the AWS logs comf file with the relevant information, you could push all of your application data right to CloudWatch logging. This could save you a ton of time and this information is very easy to export.

One other thing that most system administrators are very interested in is the Apache access log to see who's accessing your web server. It could be that you have an IP address that you've been tracking and you might want to block, or you may want to validate traffic. In either case, it's fairly easy to do. It follows the exact same process.

What you would do is go into the AWS CloudWatch logs agent configuration file, just as we did, and you would add another paragraph. You would add var logs HTTPD access for example, and then you would change the file name to var log HTTPD access underscore log. Also, be sure to change the log root name as well to var logs HTTPD access, or access log at your choice, and that way you'll separate the log files so that you can look at them easily.

As we've demonstrated, you can continue this process of adding these little configuration paragraphs to the AWS logs dot comf file for pretty much any file you want to push to CloudWatch and monitor. You just repeat the process over and over until you're pushing all the data to CloudWatch that you desire. As you can see, I've created no actual or simulated load on this server, so the number of log entries is quite small. However, in a real production environment, that might not be the case.

Let's go over here and click on logs, and take a look at the options for expiring events. This is pretty important. These events are actually stored in S3, so if you select never expire as your retention method, you're going to continue to push logs to S3 forever. I recommend that you not do that. In this case, for this server, we want to keep two weeks of logs on the var log HTTP error log. We will accept the warning that reducing the retention period will permanently delete older log data and say okay.

You'll now notice that our var log HTTPD error log now will expire after 14 days. You'll want to talk to your IT security people and system administrators to find out what's appropriate for you, and then once you've made a selection, be sure to go back and update your expired events after metric. That way you don't keep log files forever, as again, you will pay for the storage that the log files take up. In this lesson, we learned the fundamentals of how to send a log file that's being generated by our Linux box to CloudWatch to make it easier to review from the Amazon console. This also prevented us from the need to log in to look at what's happening to our instance and see what's going on in our environment.

It's important to note that we did everything manually. I need to point that out because if your environment is much more dynamic, for example if you're using auto scaling, where servers are scaling in and scaling out dynamically based on load, you're going to want to update your launch configuration to push the files to CloudWatch as part of your build.

In this lesson we provided our EC2 instance the ability to push log files to CloudWatch, we discussed the advantages of it, and as a reminder, you should always consult your AWS pricing structure and consider the volume of data and retention requirements prior to implementation. I hope you enjoyed this lesson and it gave you some thoughts about how you might use this in your own environments.

About the Author
Michael Bryant
Network Engineer

Network engineer and program analyst.