In this quick lesson, I want to go over a powerful feature of Amazon CloudWatch - Cloudwatch Subscriptions, which can help you get access to a real-time feed of log events from CloudWatch logs - and can then deliver those logs to other services such as Kinesis streams, Firehouse, or Lambda for custom processing and analysis.

At its base level, Amazon Cloudwatch Logs provide a way to monitor, store, and access the log files from your various Amazon EC2 instances, CloudTrails, and other sources. It allows you to centralize these logs from all of your disparate systems and applications within a single place. This by itself is already very handy because from there we can easily view our logs, search through them, or filter them based on specific criteria.

But with amazon CloudWatch subscriptions we can start to do all of that in real time.

Getting Started With Amazon Cloudwatch Subscription

The first step to utilizing Amazon Cloudwatch Subscriptions is to create the receiving resource that will take in the logs created by the other systems. This resource could be a Kinesis stream for example, which is able to handle high volumes of data coming in quickly.

In order to direct the right information into the receiving resource, the Kinesis stream in this example, we need to set up a subscription filter. The subscription filter helps to define a pattern that we will look through the logs, to find matching event data, and deliver that information to our receiving resource.

Each subscription filter contains a few key elements that you need to set up and be aware of.

The first element is the Log Group Name - Which is the log group you are associating the subscription filter with. Any log events that are created within this log group will be subject to filtering, and when any matching elements are found - they will be sent on to the destination service.

The second element is the Filter Pattern itself - This is a description of how CloudWatch logs should interact with any data in the log events, and how it should filter that data - including restricting what data makes it to the destination resource.

Here are a few examples of these filter patterns:

1. Matching
1. You can match with everything - meaning all log events will get passed into the destination resource
2. You can do single term Matching - only passing in log events that contain a term like “ERROR”
3. You can Include a term AND exclude term by adding a ‘-’ that means with the previous example you could exclude any errors that exited by using “ERROR” - “Exiting”
4. You can require matching multiple terms - this could look like “ERROR Exception” which would match any strings that contained both of those words like [ERROR] Terrible Exception has Occurred!
2. Or Match
1. Allows you to match multiple terms with the ‘?’ operator - for example you could look for “?ERROR ?WARN to find any messages that were errors or warnings and send them onwards to the destination resource
2. You also have the ability to or match on specific word, positioned within a string. The format of this is a little different from the above though: When specifying a specific word required at a specific spot in a string it looks like [w1=ERROR, w2] You can also set this up with an OR operator as well which takes the form of [w1=ERROR || w1=WARN, w2] - the double pipe denoting or here instead of the ? operator - you can thank AWS for that strange change choice

Alright back to the subscription filter elements

The third element of the subscription filter is the Destination ARN This is the ARN of the destination resource we are going to send all matched data to. Again that is probably a kinesis stream, or maybe even a lambda function.

The fourth element is a simple role ARN  this is the role that can grant CloudWatch logs the permission that are required to post the filter data into the destination resource. Everything in AWS needs to have permissions to access each other, and this is a part of that system.

And the final element is the distribution method - this element is for when you use kinesis as the destination resource. By default, the data will be grouped by log stream. You can have it distribute log data randomly as well for a more even distribution.

Logs that are sent to a receiving service through a subscription filter are Base64 encoded and compressed with the gzip format.

Cross Account log data sharing

Amazon Cloudwatch Subscription log data can also be shared across accounts. Being able to collaborate with owners of different AWS accounts is vital for multi account structures and organizations.

Unlike the previous example destination we talked about, where you could send your log data to Lambda, Kinesis Data Firehouse Streams, or Kinesis streams. Cross account sharing is solely limited to Kinesis streams, so bear that in mind when designing your architectures around this problem.

To share log data across accounts you will need to establish a log data sender and recipient:

The log data sender gets the data from the data source, and lets Cloudwatch logs know when that data is ready to be sent to a specified destination. 

The log data recipient sets up a destination and lets Cloudwatch logs know that the recipient wants to receive log data.

Both of these elements require you to create a CloudWatch Logs Destination. A destination a made up of some familiar elements:

1. A Destination Name 
2. Target ARN
3. Role ARN
4. Access Policy

They are very similar to the elements of the subscription filter we spoke of earlier.

There is an important caveat to be aware of, which is that the log group and the destination must be in the same AWS region. However, the kinesis stream that the destination points to, can be in a different region. 

Analyzing and visualizing your data

Once you have your data coming in the way you expect it to, be that from another account or straight from the source, you can manipulate or visualize that information any way you see fit.

Many people use Kibana Dashboards to get a handle on data visualization as it's pretty simple to use and there are a ton of examples out there. Kibana runs on top of ElasticSearc h in order to provide its services - yet you may have noticed there was no mention of elastic search until now.  

Cloudwatch Log Subscriptions does not natively support pushing data in ElasticSearch, but we can move it into kinesis as we have already shown earlier. From kinesis, we can begin to export the data over to elastic search using a CloudWatch Logs Subscription Consumer.

AWS says the following about this - “The CloudWatch Logs Subscription Consumer is a specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter.”

Here is a link to the GitHub where you can download a sample project that contains this Specialized stream reader. I wish this was a native option with the services itself, but for the moment you will have to get it from the link here.

But with this new addition, we have the power to really see into our data and have the ability to make key decisions based on it.