Performing Data Collection using AWS Data Pipeline
Introduction to Collecting Data

This course is designed to show you how to use the AWS Data Pipeline service for data collection requirements. 

Learning Objectives

  • Recognize and explain the operational characteristics of a data collection system.
  • Recognize and explain how a collection system can be designed to handle the frequency of data change and the type of data being ingested.
  • Recognize and identify properties that may need to be enforced by a collection system.
  • Recognize and explain AWS Data Pipeline core concepts.

Intended Audience

This course is intended for students looking to increase their knowledge of data collection methods and techniques with big data solutions.

What You'll Learn

  • Introduction to Data Pipeline: In this lesson, we'll discuss the basics of Data Pipeline.
  • AWS Data Pipeline Architecture: In this lesson, we'll go into more detail about the architecture that underpins the AWS Data Pipeline Big Data Service.
  • AWS Data Pipeline Core Concepts: In this lesson, we'll discuss how we define data nodes, access, activities, schedules, and resources.
  • AWS Data Pipeline Reference Architecture: In this lesson, we'll look at a real-life scenario of how data pipeline can be used.

Hello, and welcome to another Big Data on AWS course from Cloud Academy. In this course, we focus on the Amazon Big Data services which are designed to collect data. This course is part of a larger learning path that covers the broad range of Big Data services available from AWS.

This course assumes you have a good understanding of cloud computing in AWS, and that you are proficient with provisioning and using services within AWS. Ideally, you will also have some background understanding of Big Data. There are a large number of AWS Big Data services available and this course is designed to provide the initial core concepts required for each of these services, and to assist people in passing the AWS Big Data Specialist exam.

A little bit about me. My name is Shane Gibson, and I have worked in the area of data and business intelligence for over 20 years. And for the last three years, I've been focusing on how we can use agile processes and cloud computing technologies to accelerate the delivery of data and content to our users. I was born and still live in New Zealand. I love craft beer and good coffee. You can learn more about me by following either on my Twitter or my LinkedIn. At the end of this course, you'll be able to describe in detail how Amazon Big Data services can be used to collect data within a Big Data solution. In this Big Data on AWS learning path, we cover many AWS Big Data services that can be used to collect, store, process, analyze, visualize and secure Big Data.

In this course, we provide three modules which cover the Big Data collection services of AWS Data Pipeline, Amazon Kinesis and AWS Snowball. Each of these three Big Data collection services can be used on their own, or in combination with each other to provide processing capabilities for your Big Data solution. Each of these collection services have specific strengths that make them more suitable for the collection of different types and volumes of data, and we discuss these as we progress through the course. In each of the modules we cover, which processing and storage patterns that collections service fits within, the architecture of the service, as well as the core concepts that help you understand that service in detail.

We also cover the service limits for each service where applicable. At the end of the three modules, we will have a wrap up with a quick overview of the reference architecture which uses these services. So let's begin and find out how we can collect Big Data using Amazon Web Services capabilities.

About the Author

Shane has been emerged in the world of data, analytics and business intelligence for over 20 years, and for the last few years he has been focusing on how Agile processes and cloud computing technologies can be used to accelerate the delivery of data and content to users.

He is an avid user of the AWS cloud platform to help deliver this capability with increased speed and decreased costs. In fact its often hard to shut him up when he is talking about the innovative solutions that AWS can help you to create, or how cool the latest AWS feature is.

Shane hails from the far end of the earth, Wellington New Zealand, a place famous for Hobbits and Kiwifruit. However your more likely to see him partake of a good long black or an even better craft beer.