1. Home
  2. Training Library
  3. Designing for Disaster Recovery & High availability in AWS - Level 2

Kinesis Data Firehose

The second type of consumer to a Kinesis Data Stream can be an Amazon Kinesis Data Firehose delivery stream.  As the name suggest a Firehose delivery stream can pick up large data sets, transform and load them to destinations like Amazon S3, DynamoDB, Amazon EMR, OpenSearch, Splunk, DataDog, NewRelic, Dynatrace, SumoLogic, LogicMonitor, MongoDB, HTTP endpoints and Amazon Redshift  as destinations.   

Kinesis firehose manages all the infrastructure, storage, networking and configuration required to ingest and store your data to a destination. It’s fully managed which means you do not have to provision, deploy, maintain hardware, software or write any application to manage the process. It scales automatically and like many other AWS storage services it replicates data across three facilities in a region. 

Kinesis Firehose buffers the input stream to a predefined size and for a predefined time before loading it to destinations. The Buffer Size is in MBs and go from 1MB to 128MB for S3, from 1MB to 100MB for OpenSearch and 0.2MB up to 3MB for lambda functions. 

The Buffer Interval is in seconds and goes from 60 to 900- seconds.

Kinesis Firehose will store data for up to 24 hours if the delivery destination is unavailable unless the source is a  Kinesis DataStream in which case it will be retained according to the data stream configuration not firehose. 

In the case of putting data to Amazon RedShift, Kinesis Firehose uses Amazon S3 as the first step before loading data to your RedShift  Cluster

Kinesis data Firehose does not use shards and is fully automated in terms of scalability. Kinesis firehose can compress and encrypt data before delivering it to storage destinations. 

For Amazon S3, OpenSearch and Splunk destinations, if data is transformed you can optionally back up the source data to another S3 bucket.   

Firehose operates fast BUT NOT in real time. You should expect latency of 60 seconds or more when using Kinesis Firehose to store to destinations.  Also, for Kinesis Firehose you pay for the amount of data going through it. 

Kinesis Data Firehose is usually the delivery service used to get Kinesis Data stream records to AWS Storage services. Message producers to Kinesis data firehose are not limited to kinesis data streams and any application can produce messages for kinesis firehose to deliver to AWS Storage services. The Kinesis Agent is a pre-fabricated Java application which once installed and configured collects and send data to your delivery stream. You can install the Kinesis Agent on linux systems for web servers, log servers and database servers.  The agent is also available on GitHub.  The Amazon Linux, Red Hat and Microsoft windows operating systems are supported.

Both Kinesis Data Streams and Kinesis Firehose are part of the Kinesis streaming data platform which includes Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams.

Difficulty
Intermediate
Duration
1h
Students
5
Description

This course covers the core learning objective to meet the requirements of the 'Designing for disaster recovery & high availability in AWS - Level 2' skill

Learning Objectives:

  • Analyze the amount of resources required to implement a fault-tolerant architecture across multiple AWS availability Zones
  • Evaluate an effective AWS disaster recovery strategy to meet specific business requirements
  • Understand SLA for AWS services to ensure the high availability of a given AWS solution
  • Analyze which AWS services can be leveraged to implement a decoupled solution
About the Author
Students
225572
Labs
1
Courses
215
Learning Paths
175

Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.

To date, Stuart has created 150+ courses relating to Cloud reaching over 180,000 students, mostly within the AWS category and with a heavy focus on security and compliance.

Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.

He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.

In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.

Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.