Monitor Like a DevOps Pro: Build A Log Aggregation System in AWS


1191 students completed the lab in ~1h:12m

Total available time: 2h:0m

706 students rated this lab!

Lab Overview

Modern cloud environments are increasingly complex distributed systems with numerous software components. The challenge of maintaining moving parts and tracking changes in your AWS systems continues to grow but there are solutions. Some natural questions are:

  • How can you understand, at a high level, what is happening in your cloud?
  • Can you track usage trends over time?
  • Can you debug any issues that might arise?
  • Can you search through logs without combing through files on many disks?

The answer to each is yes you can! A sophisticated tool called a log aggregation system gathers operational information and logs from across your entire cloud. The log aggregation system is an advanced DevOps technique that enables you to quickly search your logs and graph any trends arising from structured logs.

In this Lab, you will create a distributed, scalable log aggregation system within AWS running on AWS Elasticsearch Service. This Log Aggregation System will ingest as much of your CloudWatch log stream events as you want, events generated from AWS EC2 Instances, Lambda functions, Databases, and anything else you want to submit log events from.

Lab Objectives

Upon completion of this Lab, you will be able to:

  • Compare and contrast the log analysis capabilities in CloudWatch Logs and Elastic Stack, particularly Elasticsearch and Kibana
  • Subscribe AWS Elasticsearch Service to CloudWatch to automatically stream log events to Elasticsearch
  • Search and discover log events using Kibana
  • Create Kibana visualizations and dashboards to monitor the state of your cloud

Lab Prerequisites

You should be familiar with:

  • Basic CloudWatch concepts
  • Elastic Stack experience is useful but not required

Lab Environment

Before completing the Lab instructions, the environment will look as follows:

After completing the Lab instructions, the environment should look similar to:


June 5, 2018 - Complete update (easier to follow instructions and screenshots, update to Elastic Stack version 6)

Follow these steps to learn by building helpful cloud resources

Logging in to the Amazon Web Services Console

Your first step to start the Lab experience

Navigating to Your Cloud's Lambda Function

Navigate to a simple AWS Lambda function that is used to generate logs to aggregate

Creating Some Logs

Perform a test invocation of the Lambda function to generate logs in CloudWatch

Viewing Logs Manually

Review the data and interfaces available in CloudWatch Logs

Launching the Elasticsearch Domain

Launch the AWS Elasticsearch Service Domain which is the main component of the Elastic Stack

Sending CloudWatch Logs to Elasticsearch

Publish logs from the CloudWatch logs group to Elasticsearch

Discovering and Searching Events

Produce more log events and use the discovery and search functionality in the Elastic stack system

Visualizing Aggregated Events

Build a stacked area chart of request types to the Lambda API over time using Kibana's visualization functionality

Creating a Kibana Dashboard

Create a Kibana dashboard to group together all the useful logging visualizations you created