What is Amazon MSK?
What is Amazon MSK?
4h 57m

This section of the AWS Certified Solutions Architect - Professional learning path introduces common AWS solution architectures relevant to the AWS Certified Solutions Architect - Professional exam and the services that support them. These services form a core component of running resilient and performant architectures. 

Want more? Try a Lab Playground or do a Lab Challenge!

Learning Objectives

  • Learn how to utilize managed services and serverless architectures to minimize cost
  • Understand how to use AWS services to process streaming data
  • Discover AWS services that support mobile app development
  • Understand when to utilize serverless services within your AWS solutions
  • Learn which AWS services to use when building a decoupled architecture

Amazon Managed Streaming for Apache Kafka or Amazon MSK allows you to run applications that utilize Apache Kafka within AWS. Kafka provides a platform for stream processing and operates as a publisher/subscriber based durable messaging system. Its key features are the ability to intake data with extreme fault tolerance, allowing for continuous streams of these records that preserve the integrity of the data, including the order in which it was received.

Apache Kafka then acts as a buffer between these data producing entities and the customers that are subscribed to it. Subscribers receive information from Kafka topics on a first in, first out basis or FIFO, allowing the subscriber to have a correct timeline of the data that was produced.

Kafka is an open-source technology that allows for a large number of community-driven tools and add-ons. This makes it very customizable and gives developers the freedom to build and create what they need. Using Amazon MSK, you can use the native Apache Kafka APIs to create datalakes, stream information to multiple sources, and power data analytic applications and pipelines, again all within AWS.

The main reason you'd wanna use Amazon MSK over rolling your own implementation of Kafka is that Amazon MSK is a fully managed service. This means you don't need to take care of any servers, you don't need to worry about any upgrades and you also don't need to bother with handling Apache Zookeeper, a required coordinating software that deals with orchestrating the cluster task and maintaining the state of the cluster in general.

About the Author
Learning Paths

Danny has over 20 years of IT experience as a software developer, cloud engineer, and technical trainer. After attending a conference on cloud computing in 2009, he knew he wanted to build his career around what was still a very new, emerging technology at the time — and share this transformational knowledge with others. He has spoken to IT professional audiences at local, regional, and national user groups and conferences. He has delivered in-person classroom and virtual training, interactive webinars, and authored video training courses covering many different technologies, including Amazon Web Services. He currently has six active AWS certifications, including certifications at the Professional and Specialty level.