AWS Logging Mechanisms
This course is part 2 of a 2-part course series that focuses on a number of key AWS services and how they perform logging and monitoring across your environment. Being able to monitor data provides a number of key benefits to your organization, such as compliance, incident detection and resolution, trend analysis and much more. Collating data and statistics about your solutions running within AWS also provides the ability to optimize its performance. This series looks at how to implement, configure, and deploy logging and monitoring mechanisms using the following AWS services and features.
- Amazon CloudFront Access Logs
- VPC Flow Logs
- AWS Config Configuration History
- Filtering and searching data using Amazon Athena
- Amazon CloudWatch - CloudWatch Monitoring Agent
- AWS CloudTrail Logs
- Monitoring CloudTrail Logs with CloudWatch Metric Filters
- Amazon S3 Access Logs
The course for Part 1 can be found here
By the end of this course series you will be able to:
- Understand why and when you should enable logging of key services
- Configure logging to enhance incident resolution and security analysis
- Understand how to extract specific data from logging data sets
The content of this course is centered around security and compliance. As a result, this course is beneficial to those who are in the roles or their equivalent of:
- Cloud Security Engineers
- Cloud Security Architects
- Cloud Administrators
- Cloud Support & Operations
- Compliance Managers
This is an advanced level course series and so you should be familiar with the following services and understand their individual use case and feature sets.
- Amazon CloudWatch
- AWS CloudTrail
- Amazon EC2
- AWS Config
- Amazon S3
- EC2 Systems Manager (SSM)
This course includes
If you have thoughts or suggestions for this course, please contact Cloud Academy at firstname.lastname@example.org.
Hello and welcome to this final lecture within this course. In this lecture, I want to summarize and highlight the key points from the previous lectures.
I started by talking about Amazon CloudFront Access Logs. During this lecture, we learned the following points. Amazon CloudFront is AWS's content delivery network that speeds up distribution of your static and dynamic content through its worldwide network of edge locations. When a user requests content via CloudFront, the request is routed to the closest edge location providing the lowest latency. CloudFront Access Logs can record the request from each user requesting access to your website and distribution. And the logs are stored on S3 for durable and persistent storage. The logging process for CloudFront takes place at the edge location and on a per distribution basis. And the log files capture data over a period of time and are dependent on the amount of requests received. CloudFront retains the logs until they are ready to be delivered on S3. And to enable the logging for your distribution, you must have FULL_CONTROL on the ACL for the S3 Bucket, along with the s3:GetBucketAcl permission and the s3:PutBucketAcl permission. And log output varies on the distribution type, whether this be Web or RTMP. And by enabling cookie logging, it will include all cookie information with your CloudFront Access Log data.
Following CloudFront, I then looked at network logging at the VPC level through the use of VPC Flow Logs. In this lecture, I explained that VPC Flow Logs allow you to capture IP traffic information that flows between your network interfaces and your resources within your VPC. The log data generated by VPC Flow Logs is then sent to CloudWatch Logs. And if you are running VPC peered connections, then you'll only be able to see Flow Logs of peered VPCs in the same account. Flow Logs are not available for EC2-Classic environments. And once a VPC Flow Log has been created, it can't be changed. The following traffic is not monitored and captured by the logs: DHCP traffic within the VPC, traffic from instances destined for the Amazon DNS Server, any traffic destined to the IP address of the VPC default router, traffic to and from 169.254.169.254 and 169.254.169.123, traffic relating to an Amazon Windows activation license, and traffic between a Network Load Balancer Network Interface and an Endpoint Network Interface. Flow Logs can be set up for a network interface on an instance, your subnet, or your entire VPC. And each interface that sends data to CloudWatch will do so in its own stream. Specific permissions are needed to allow VPC Flow Logs to push data to CloudWatch, as well as permissions to assume the role with the required permissions, these being CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups, and DescribeLogStreams. And the log files have the following syntax.
Next, I reviewed how AWS Config uses configuration history to display all changes within your environment. During this lecture, we learned that AWS Config is a great security and compliance tool that integrates well with many other AWS services. The Configuration History uses Configuration Items, CIs, to collate and produce a history of changes to a particular resource. AWS Config sends a Configuration History file for each resource type to an S3 Bucket that is selected during the setup of AWS Config. And this configuration file is typically delivered every six hours. A Configuration Item is comprised of a JSON file that holds the configuration information, relationship information, and other metadata as a point in time snapshot view of a supported resource. A CI is created every time a supported resource has a change made to its configuration. And a CI consists of the following sections: metadata, attributes, relationships, current configuration, and related events. And it's important to note that you can aggregate Configuration History files from multiple accounts into one S3 Bucket.
The final lecture focused on how to retrieve data from your logs that are being stored on Amazon S3, and this was shown via a demonstration.
That now brings me to the end of this lecture and to the end of part two of this two-part course series. If you have now viewed both parts, you should now have a deeper understanding of some of the logging capabilities that AWS provides across a number of key services. You will also have an insight into how these logs are constructed and how to search for specific information that you might need to use in your day-to-day operations.
If you have any feedback on this course, positive or negative, please do contact us at email@example.com. Your feedback is greatly appreciated.
Thank you for your time, and good luck with your continued learning of cloud computing. Thank you.
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.
To date, Stuart has created 150+ courses relating to Cloud reaching over 180,000 students, mostly within the AWS category and with a heavy focus on security and compliance.
Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.