Monitor Like a DevOps Pro: Build A Log Aggregation System in AWS

Lab Steps

Logging in to the Amazon Web Services Console
Navigating to Your Cloud's Lambda Function
Creating Some Logs
Viewing Logs Manually
Launching the Elasticsearch Domain
Sending CloudWatch Logs to Elasticsearch
Discovering and Searching Events
Visualizing Aggregated Events
Creating a Kibana Dashboard
Validate AWS Lab

The hands-on lab is part of these learning paths

DevOps Engineer – Professional Certification Preparation for AWS
course-steps 35 certification 5 lab-steps 18 quiz-steps 2 description 3
SysOps Administrator – Associate Certification Preparation for AWS
course-steps 34 certification 5 lab-steps 30 quiz-steps 4 description 5
DevOps Fundamentals
course-steps 4 certification 1 lab-steps 3

Ready for the real environment experience?

Time Limit2h
star star star star star-half


Lab Overview

Modern cloud environments are increasingly complex distributed systems with numerous software components. The challenge of maintaining moving parts and tracking changes in your AWS systems continues to grow but there are solutions. Some natural questions are:

  • How can you understand, at a high level, what is happening in your cloud?
  • Can you track usage trends over time?
  • Can you debug any issues that might arise?
  • Can you search through logs without combing through files on many disks?

The answer to each is yes you can! A sophisticated tool called a log aggregation system gathers operational information and logs from across your entire cloud. The log aggregation system is an advanced DevOps technique that enables you to quickly search your logs and graph any trends arising from structured logs.

In this Lab, you will create a distributed, scalable log aggregation system within AWS running on AWS Elasticsearch Service. This Log Aggregation System will ingest as much of your CloudWatch log stream events as you want, events generated from AWS EC2 Instances, Lambda functions, Databases, and anything else you want to submit log events from.

Lab Objectives

Upon completion of this Lab, you will be able to:

  • Compare and contrast the log analysis capabilities in CloudWatch Logs and Elastic Stack, particularly Elasticsearch and Kibana
  • Subscribe AWS Elasticsearch Service to CloudWatch to automatically stream log events to Elasticsearch
  • Search and discover log events using Kibana
  • Create Kibana visualizations and dashboards to monitor the state of your cloud

Lab Prerequisites

You should be familiar with:

  • Basic CloudWatch concepts
  • Elastic Stack experience is useful but not required

Lab Environment

Before completing the Lab instructions, the environment will look as follows:

After completing the Lab instructions, the environment should look similar to:


May 27th, 2020 - Updated a lab step's instructions to account for long Elasticsearch deployment times

October 4th, 2019 - Because of CloudFormation changes, we have updated instructions and screenshots.

May 9th, 2019 - Removed unnecessary permissions from the Lambda DynamoDB role and improved the explanation of how Lambda functions map to CloudWatch log streams

January 11th, 2019 - Fixed an issue that caused the streaming of CloudWatch Logs to ElasticSearch to fail

January 10th, 2019 - Added a validation Lab Step to check the work you perform in the Lab

June 5, 2018 - Complete update (easier to follow instructions and screenshots, update to Elastic Stack version 6)

About the Author


Nothing gets me more excited than the AWS Cloud platform! Teaching cloud skills has become a passion of mine. I have been a software and AWS cloud consultant for several years. I hold all 5 possible AWS Certifications: Developer Associate, SysOps Administrator Associate, Solutions Architect Associate, Solutions Architect Professional, and DevOps Engineer Professional. I live in Austin, Texas, USA, and work as development lead at my consulting firm, Tuple Labs.