Real-Time Application Monitoring with Amazon Kinesis

Amazon Kinesis is a real-time data streaming service that makes it easy to collect, process, and analyze data so you can get quick insights and react as fast as possible to new information. 

With Amazon Kinesis you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, social media feeds, etc., into your databases, data lakes, and data warehouses. You can also build your own real-time applications using this data. 

Looking to immediately get some deep info on Kinesis? Check out Cloud Academy’s Getting Started with Amazon Kinesis Learning Path, which includes courses, labs, and exams to fill your knowledge bucket.

Getting Started with Amazon Kinesis Learning Path

Two key characteristics of Kinesis are that it enables you to process and analyze data as it arrives, and then it responds in real-time instead of having to wait until all your data is collected before the processing can begin. You can understand what Kinesis is designed to do my remembering that it can:

  1. Collect data quickly
  2. Analyze data quickly

How you collect, store, process, and analyze your data is up to you — it can range from complete customization of the system to a turnkey approach using the PaaS offering.

Why use Amazon Kinesis? Why use real-time analytics?

People choose Kinesis because it’s an effective, real-time analytics service — but why are we so concerned about real-time data?

The chart below shows the function and value of data over time. 

Chart of real-time data's value over time

The value of data to be preventative or predictive diminishes (rather quickly) over time. The data still has value, it just becomes useful for a different role — for carefully looking at past trends, as opposed to continually acting on what’s happening in the present moment.

Speed — Kinesis is good for companies that already have terabytes of data in the cloud

Data shows that 20% of companies already have migrated to the cloud. If you already have your data living in robust, backed-up resources, it makes sense to use a service designed to play nicely with those resources. You can reduce your time to market and get valuable insights more quickly.

Scalability — elasticity is helpful, especially for network data

If you’re analyzing network data or anything that’s dynamic in nature, it’s important to be aware of cost concerns. You don’t want to commit to services that you don’t need, especially when your data can change at a moment’s notice. And you want to be prepared for spikes that will occur. Kinesis’s scalability helps you stay lean with costs and flexible with demand.

Ease — get benefits of a managed service

Use the AWS Console to quickly start up a Kinesis Firehose stream. In just a few clicks, you’ll be able to see your data coming in,  and you won’t have to worry about resource allocation or administration.

What services does Kinesis offer?

Kinesis Data Streams

Note: Originally, this product was called Kinesis Streams — now we have Kinesis Data Streams and Kinesis Video Streams.

Characteristics

  • Custom real-time processing
  • Fastest Kinesis offering with < 1-second processing latency
  • Customization requires coding overhead from you

How is it architected?

Since Kinesis Data Streams is a fully customizable offering, it’s good to know some basics about how it’s built. It’s components are:

  • Record
    • A record is the unit of data in an Amazon Kinesis stream. Each record in the stream is composed of a sequence number, a partition key, and a data blob. A data blob is the data of interest your data producer adds to a stream.
  • Shard
    • The data records in the stream are distributed into shards. A shard is the base throughput unit of a Kinesis Data Stream, meaning it’s how you gauge the amount of bandwidth you have. The overall capacity of your stream is a function of the number of shards that you specify for that stream. More shards = more capacity. You can always increase your number of shards.
  • Producers
    • The producers are the mechanisms that continuously push data to streams. A web service sending log data to a stream is an example of a producer. 
  • Consumers
    • Consumers receive records from Kinesis Data Streams and process them. These consumers are known as Amazon Kinesis Streams applications. Consumers can store the output using an AWS service such as Amazon DynamoDB, Amazon Redshift, or Amazon S3. An Amazon Kinesis application is a data consumer that reads and processes data from an Amazon Kinesis Stream and typically runs on a fleet of EC2 instances. You need to build your applications using either the Amazon Kinesis API or the Amazon Kinesis Client Library (KCL).

Kinesis Video Streams

This is a purpose-built video streaming analysis service that integrates with AWS Machine Learning (ML)/Artificial Intelligence (AI) offerings.

Characteristics

    • Processes video/binary data
    • Makes it easy to capture live video, play it back, and store it for real-time and batch-oriented ML-driven analytics such as Rekognition or TensorFlow
    • Offers security to input (secure connection to camera) and for the processed data (encrypted in storage)

Kinesis Firehose

This is a managed service to help you quickly get near-real-time data streams collected and into storage for analysis.

Characteristics

    • Slower than Kinesis Data Streams; has data latency of >60 seconds
    • You just need to dump the data somewhere (S3, Redshift, Elasticsearch Service, + third party services)
    • No coding necessary; just use the AWS Console to get started quickly

Kinesis Data Analytics

Allows you to quickly create SQL code that continuously reads, processes, and stores data in near real-time. 

Characteristics

    • Real-time processing
    • Fully managed service
    • A quick way to get insight into data streams by just writing some SQL
    • Scalable, which helps optimize performance and cost

What are some Amazon Kinesis use cases?

One way to think about use cases for Kinesis is by considering what some sources of real-time streaming data are, for example:

  • Mobile apps, such as any that continually collects your current GPS position
  • Open-source tools such as Open Web Analytics (OWA) or Matomo, used to track visitors’ website behavior
  • Application logs that continually collect the behavior and operating performance of server applications. 
  • IoT data, sent from sensors in all our household objects that have internet connectivity
  • Social apps such as Twitter and Facebook, exposing user posts and commentary

And a next step in learning about use cases is to check out our course on Working with Amazon Kinesis Analytics. You’ll get valuable, actionable insight on the nuts and bolts of a key Kinesis service.

Kinesis Analytics Course

Wrapping it up

Amazon Kinesis acts as the front end of your streaming data and — depending on your business needs — can be as complicated and as real-time as you want. Coupled with its integration with other AWS services, Kinesis can help you process, react, and take action so that your business move swiftly on its needs.

Joe Nemer

Written by

Joe Nemer

Joe is a Technical Researcher at Cloud Academy and works to help readers connect concepts in ways they haven't thought of before. Side interests include all sorts of waves — ocean waves, sine waves, just not goodbye waves.


Related Posts

Chris Gambino
Chris Gambino
— July 16, 2019

Introduction to Streaming Data

Designing a streaming data pipeline presents many challenges, particularly around specific technology requirements. When designing a cloud-based solution, an architect is no longer faced with the question, “How do I get this job done with the technology we have?” but rather, “What is th...

Read more
  • amazon kinesis
  • Big Data
  • Data scoping
  • IoT
  • Streaming data