This course will provide you with a good foundation to better understand Amazon Kinesis, along with helping you to get started with building streamed solutions. In this course, we'll put a heavier emphasis on hands-on demos along with breaking down the concepts and introducing you to the components that make up Amazon Kinesis.
- People working with Big Data
- Business intelligence
- Demonstrate knowledge of Amazon Kinesis, what are the core aspects of the service (streams, firehose, analytics), their common use cases, big data use cases, and how do you access / integrate with the Kinesis service Demonstrate how to work with and create Kinesis streams
- Demonstrate how to work with Kinesis Firehose and how to use Firehose with Redshift
- Set up monitoring and analysis of the stream services
- Understand how to extract analysis from Kinesis
This Course Includes
- 45 minutes of high-definition video
- Live demonstrations on key course concepts
What You'll Learn
- What is Streaming Data: An overview of streaming data and it’s common uses.
- Setting Up Kinesis Agent: In this demo, we're working with installing the Amazon Kinesis Stream agent.
- Kinesis Streams: An overview of Kinesis Streams, what they do, and common use cases.
- Performing Basic Stream Operations: In this demo, we'll be pulling a basic Amazon Kinesis stream from the command line.
- Firehose: In this lesson, we'll be discussing the fully managed solution, Amazon Kinesis Firehose.
- Firehose Delivery Stream: In this demo, we're going to set up an Amazon Kinesis Firehose stream.
- Testing Delivery Stream: In this lesson, we're going to do a quick follow up to the Firehose stream, and test the data delivery.
- Kinesis Analytics: In this lesson, we'll go over the analytics components of Kinesis.
- Kinesis Analytics Demo: In this demo, we're going to begin working with Amazon Kinesis Analytics.
- Kinesis Features Comparison: In this lesson, we'll compare some products within the Amazon Kinesis suite, as well as some other Amazon services.
- Course Conclusion: A wrap-up and review of the course.
28/05/2019 - Re-record of lectures to improve audio
Amazon Kinesis Streams is a solution that collects, processes, and delivers stream data into applications, data warehouses, or database tables. It can collect data from thousands of sources, and deliver data through multiple destinations and parallel processes. Amazon Kinesis Streams enables you to build customized applications to ingest streaming data to meet a wide range of use cases. It enables you to build customized applications, to consume data for dashboards, and to provide real-time monitoring and alerts.
Amazon Kinesis Streams works with other Amazon services including Amazon S3, Amazon RedShift, Amazon Elastic MapReduce, and DynamoDB. Here's a diagram of a high-level architecture of an Amazon Kinesis Stream implementation. Producers push streaming data to the Kinesis Stream's pipeline. That data is further processed and pushed out to other AWS services such as Amazon S3, DynamoDB, or RedShift. Or in some cases, the stream data can be pushed along to another Kinesis stream, or a Kinesis Firehose, or a Kinesis analytic service.
A Kinesis stream is an ordered sequence of data records, with a data record being a unit of stored data in the Kinesis stream. A data record is made up of a sequence number, a partition key, and the data itself. Producers put records into an Amazon Kinesis stream while a consumer gets records from an Amazon Kinesis Stream. Amazon Kinesis applications are a consumer of Kinesis streams, and they generally will run on an EC2 instance you provision and manage. Shards are uniquely identified groups of data records in the Kinesis stream. A stream is normally made up of one or more shards. A partition key is used to group data by a shard in a Kinesis stream. While each data record has a sequence number that is unique within each shard, the Kinesis client library is compiled within your application to enable full or parts of consumption of data from the stream.
Data records are accessible by default for 24 hours, from the time they are added to the stream. That retention period is configurable in an hourly increment from 24 to 168 hours. Kinesis streams utilize Amazon CloudWatch and Amazon CloudTrail for monitoring and logging purposes. There's additional monitoring for other Kinesis components which includes tools like the stream's Agent which has its own logging mechanism.
About the Author
Richard Augenti is a DevOps Engineer with 23 years of IT professional experience and 7 years of cloud experience with AWS and Azure. He has been engaged with varying sized projects with clients all across the globe including most sectors. He enjoys finding the best and most efficient way to make things work so, working with automation, cloud technologies, and DevOps has been the perfect fit. When Richard is not engaged with work, he can also be found presenting workshops and talks at user conferences on cloud technologies and other techie talks.