The course is part of these learning paths
Amazon Web Services Elastic MapReduce
Amazon Web Services Lambda
In this course for the Big Data Specialty Certification, we learn how to identify the appropriate data processing technologies needed for big data scenarios. We explore how to design and architect a data processing solution, and explore and define the operational characteristics of big data processing. Intended audience: This course is intended for students wanting to extend their knowledge of the data processing options available in AWS.
While there are no formal pre-requisites students will benefit from having a basic understanding of cloud computing services Recommended course - Compute Fundamentals
- Recognize and explain how to identify the appropriate data processing technologies needed for big data scenarios.
- Recognize and explain how to design and architect a data processing solution.
This Course Includes:
75 minutes of high-definition video.
What You'll Learn:
- Course Intro: What to expect from this course
- Amazon Elastic MapReduce Overview: In this lesson we discuss how EMR allows you to store and process data
- Amazon Elastic MapReduce Architecture: In this lesson you’ll learn about EMR’s clustered architecture.
- Amazon Elastic MapReduce in Detail: In this lesson we’ll dig deeper into EMR storage options, resource management, and processing options.
- Amazon Elastic MapReduce Reference Architecture: Best practices for using EMR.
- Amazon Lambda Introduction: This lesson will kick off our discussion of Lambda and how it’s used in Big Data scenarios.
- Amazon Lambda Overview: This lesson discusses how Lambda allows you to run code for virtually any type of application or backend service with no administration.
- AWS Lambda Architecture: In this lesson we’ll discuss generic Lambda architecture and and Amazon’s serverless service.
- AWS Lambda in Detail: In this lesson we’ll dig into Events and Service Limits.
- AWS Lambda Reference Architecture: In this lesson we'll look at a real life scenario of how lambda can be used.
Okay, as we come to the end of this module on AWS Lambda, let's have a quick look at an example of a reference architecture from AWS where AWS Lambda can be used. When we look at this scenario, we're looking at a typical internet of things solution where we're streaming data from connected devices to enable it to be used for analysis and ingestion and synchronization. In this scenario, the devices are connected through Amazon Kinesis and are streaming the data through to make it available to multiple Lambda functions.
The first Lambda function is grabbing the data from Kinesis and passing it through into a Amazon DynamoDB database. At the same time, it's passing the same data through to AWS CloudWatch to enable it to be used for simple monitoring of aggregated metrics. The Lambda function two is processing the same event data. But in this case, it's storing those event data into Amazon S3 to allow it to be stored for cost-effective long-term usage. And the last function, the Lambda function three, is enabling the connected devices to call the data that's stored in DynamoDB and bring it back when it needs to be used for synchronization purposes.
Lastly, we can then connect Amazon EMR to the DynamoDB or S3 data to enable us to analyze the information that's been stored and at the same time, we can also load that data into Amazon Redshift to enable it to be queried using our standard SQL query tools. So that brings us to the end of the AWS Lambda module. I look forward to speaking with you again.
About the Author
Shane has been emerged in the world of data, analytics and business intelligence for over 20 years, and for the last few years he has been focusing on how Agile processes and cloud computing technologies can be used to accelerate the delivery of data and content to users.
He is an avid user of the AWS cloud platform to help deliver this capability with increased speed and decreased costs. In fact its often hard to shut him up when he is talking about the innovative solutions that AWS can help you to create, or how cool the latest AWS feature is.
Shane hails from the far end of the earth, Wellington New Zealand, a place famous for Hobbits and Kiwifruit. However your more likely to see him partake of a good long black or an even better craft beer.