The course is part of this learning path
This brief course covers the fundamentals of Amazon MSK, including what the service is, how it works, and how to provision an Amazon MSK cluster. You will also be guided through how Amazon MSK fits into a functional architecture.
If you have any feedback relating to this course, feel free to reach out to us at support@cloudacademy.com.
Learning Objectives
- Learn about the Amazon MSK service and how it works
- Learn how to provision an MSK cluster
- Understand how Amazon MSK fits into a functional architecture
Intended Audience
This lecture is perfect for anyone with no previous knowledge of Amazon MSK, who wants to learn more about the service, as well as those who are interested in taking the AWS Certified Data Analytics - Specialty (DAS-C01) Certification.
Prerequisites
To get the most out of this course, you should have a basic general understanding of cloud computing, preferably with Amazon Web Services experience. It would also be beneficial to have some basic knowledge of streaming data services such as Amazon Kinesis and Apache Kafka.
There are two elements you need to provision when creating your Amazon MSK clusters: Broker instances and Broker storage. A Broker instance is a worker node that helps to manage the Kafka cluster. Your clusters can have multiple brokers, but can also operate as a single node, if necessary. Broker instances can be run within the same availability zone or across many availability zones to create a highly available cluster, something that many architectures will require.
One of the large benefits of using Amazon MSK over a roll-your-own version of Kafka is that Amazon will keep an eye on these broker nodes and replace them if they become unhealthy. Otherwise, you would have to manage all of that yourself. Broker storage is where all of your data will be kept that comes streaming into Amazon MSK.
Within Amazon, this storage is housed within EBS volumes, and gains all the protections that EBS provides, like durability and fault tolerance. You can also encrypt these data volumes using Amazon EBS Server-side encryption and AWS KMS, the Key Management Service. An interesting note about storage however, is that once you have assigned your Broker storage, you can only increase the amount of storage, it can not be decreased. The broker's storage can be autoscaled upwards if you do reach your maximum capacity.
If you already have a Kafka cluster that you are managing yourself, either on-premises or within the cloud, you can migrate over to Amazon MSK. There are many third-party and open-source tools like MirrorMaker which can help replicate the data from your Kafka cluster into Amazon MSK.
William Meadows is a passionately curious human currently living in the Bay Area in California. His career has included working with lasers, teaching teenagers how to code, and creating classes about cloud technology that are taught all over the world. His dedication to completing goals and helping others is what brings meaning to his life. In his free time, he enjoys reading Reddit, playing video games, and writing books.