In this course, we take a look at streaming data, why it's important, and how Amazon Kinesis is used to stream data into the AWS cloud.
You'll learn what data streaming is, the problems it solves, and, how Amazon Kinesis addresses them.
We'll also cover, at a very high level, what services exist inside Amazon Kinesis. These are Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams.
Learning Objectives
- Understand the fundamentals of streams processing
- Learn about the features of Amazon Kinesis
- Learn about the services that make up Amazon Kinesis
Intended Audience
This course is intended for people that want to learn about streaming data, why it's important, and how Amazon Kinesis is used to send data into the AWS cloud.
Prerequisites
- This course assumes no prior knowledge of Amazon Kinesis, streaming data, or its internals.
- A general understanding of the AWS cloud.
This has been a high-level overview of Amazon Kinesis. There's a lot more to the service. However, this should get you started using it and help answer some basic questions about what the service does and how it can be used.
As a quick review, Amazon Kinesis is a managed service from AWS designed to stream data into the AWS Cloud.
Kinesis Video Streams is used for binary-encoded data. For text-based data, the streaming frameworks are Kinesis Data Streams and Kinesis Data Firehose.
Kinesis Data Streams is used to collect and process large streams of data in real time. Once data has been put into a stream, it is available for retrieval in less than a second.
Kinesis Data Streams are composed of Shards. Shards are the base unit of throughput. If more throughput is needed, add a shard. If there is too much throughput, remove a shard.
The data stored in a Kinesis Data Stream is immutable. It cannot be changed or deleted. Updates to data come in the form of new records.
Data stays in the stream until it expires and data can expire even if it is unprocessed.
Kinesis Data Firehose is a streaming service that is designed to move data into an AWS data store or to an HTTP endpoint. Data is collected in batches and put into delivery streams based on either size of a batch or a regular time interval.
Data Firehouse is a fully managed service from AWS. Streams are provisioned and scaled automatically by AWS as are the consumers. It is not possible to create custom consumers of Firehose streams. However, It is possible to transform the data using either an AWS Lambda function or using conversions built into Firehose.
Kinesis Data Analytics can be used to transform and analyze data in real time. It can read data from both Kinesis Data Firehose as well as Kinesis Data Streams.
When using Data Analytics with Firehouse, streams can only be queried with SQL. With Kinesis Data Streams, Data Analytics can use Apache Flink to create Java or Scala applications to filter, aggregate, and transform data for advanced analytics in real time.
This brings me to the end of this lecture. Thank you for watching and letting me be part of your cloud journey.
If you have any feedback, positive or negative, please contact us at support@cloudacademy.com, your input on our content is greatly appreciated.
I'm Stephen Cole for Cloud Academy, thank you for watching!
Stephen is the AWS Certification Specialist at Cloud Academy. His content focuses heavily on topics related to certification on Amazon Web Services technologies. He loves teaching and believes that there are no shortcuts to certification but it is possible to find the right path and course of study.
Stephen has worked in IT for over 25 years in roles ranging from tech support to systems engineering. At one point, he taught computer network technology at a community college in Washington state.
Before coming to Cloud Academy, Stephen worked as a trainer and curriculum developer at AWS and brings a wealth of knowledge and experience in cloud technologies.
In his spare time, Stephen enjoys reading, sudoku, gaming, and modern square dancing.