Distributed Machine Learning Concepts
Distributed Machine Learning Demonstration
Course Review
The course is part of these learning paths
This training course begins with an introduction to the concepts of Distributed Machine Learning. We'll discuss the reasons as to why and when you should consider training your machine learning model within a distributed environment.
Apache Spark
We’ll introduce you to Apache Spark and how it can be used to perform machine learning both at scale and speed. Apache Spark is an open-source cluster-computing framework.
Amazon Elastic Map Reduce
We’ll introduce you to Amazon’s Elastic MapReduce service, or EMR for short. EMR provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data. EMR can be easily configured to host Apache Spark.
Spark MLlib
We’ll introduce you to MLlib which is Spark’s machine learning module. We’ll discuss how MLlib can be used to perform various machine learning tasks. For this course, we'll focus our attention on decision trees as a machine learning method which the MLlib module supports. A decision tree is a type of supervised machine learning algorithm used often for classification problems.
AWS Glue
We’ll introduce you to AWS Glue. AWS Glue is a fully managed extract, transform, and load service, ETL for short. We’ll show you how AWS Glue can be used to prepare our datasets before they are used to train our machine learning models.
Demonstration
Finally, we’ll show you how to use each of the aforementioned services together to launch an EMR cluster configured and pre-installed with Apache Spark for the purpose of training a machine learning model using a decision tree. This demonstration will provide an end-to-end solution that provides machine learning predictive capabilities.
Intended Audience
The intended audience for this course includes:
- Data scientists and/or data analysts
- Anyone interested in learning and performing distributed machine learning, or machine learning at scale
- Anyone with an interest in Apache Spark and/or Amazon Elastic MapReduce
Learning Objectives
By completing this course, you will:
- Understand what Distributed machine learning is and what it offers
- Understand the benefits of Apache Spark and Elastic MapReduce
- Understand Spark MLlib as machine learning framework
- Create your own distributed machine learning environment consisting of Apache Spark, MLlib, and Elastic MapReduce.
- Understand how to use AWS Glue to perform ETL on your datasets in preparation for training a your machine learning model
- Know how to operate and execute a Zeppelin notebook, resulting in job submission to your Spark cluster
- Understand what a machine learning Decision Tree is and how to code one using MLlib
Prerequisites
The following prerequisites will be both useful and helpful for this course:
- A background in statistics or probability
- Basic understanding of data analytics
- General development and coding experience
- AWS VPC networking and IAM security experience (for the demonstrations)
Course Agenda
The agenda for the remainder of this course is as follows:
- We’ll discuss what Distributed Machine Learning is and when and why you might consider using it
- We’ll review the Apache Spark application, and its MLlib machine learning module
- We’ll review the Elastic MapReduce service
- We’ll provide an understanding what a Decision Tree is - and what types of analytical problems it is suited towards
- We’ll review the basics of using Apache Zeppelin notebooks - which can be used for interactive machine learning sessions
- We’ll review AWS Glue. We’ll show you how you can use AWS Glue to perform ETL to prepare our datasets for ingestion into a machine learning pipeline.
- Finally - We’ll present a demonstration of a fully functional distributed machine learning environment implemented using Spark running on top of an EMR cluster
Feedback
If you have thoughts or suggestions for this course, please contact Cloud Academy at support@cloudacademy.com.
- [Instructor] Welcome back. In this lecture we'll introduce you to Decision Trees. In Machine Learning, a decision tree is a type of predictive modeling approach. In the demonstration we give later on, we'll implement a Decision Tree to provide us with an ability to make predictions. Let's continue on and dive a bit deeper into the details of what a Decision Tree is and how they are created.
To start with, Decision Trees are Machine Learning models that are in the form of tree structures. Decision trees are one of the most popular and widely used Machine Learning algorithms. A Decision Tree is a type of supervised learning algorithm and is nothing more than a tree in which each non-leaf node represents a decision between a set of choices in where the leaf nodes are the final decision or classification.
There are two types of Decision Trees used in Machine Learning. The Classification Tree. An analysis where the predicted outcome is the class to which the data belongs. For example, the color is red or the person is male. Regression Tree. An analysis where the predicted outcome can be considered a real number. For example, someone's income or the time it will take to solve a customer inquiry. Let's provide a quick example of how a Decision Tree works.
A typical example involves trying to predict tomorrow's weather based on several features we know about today's weather. Let's assume we have been collecting the temperature, wind, and pressure attributes for each day of the past year, and whether or not it rained the following day. Using this type of data set together with a Decision Tree algorithm we can train our Machine Learning model to help us forecast for possibility of rain or not anytime in the future based on the weather attributes of the day before.
The decision tree presented on this slide is an example of what this model might end up looking like. Decision Tree's are a popular tool for Machine Learning practitioners for various reasons. Let's cover off some of the important advantages associated with Decision Trees. Decision Tree's are simple to understand and interpret. People are able to understand Decision Tree models after a brief explanation as they mirror human decision making.
Decision Trees are able to handle both numerical and categorical data. Decision Trees requite little or minimal data preparation. And finally, Decision Trees perform well with very large data sets.
That concludes our lecture on Decision Trees. In the next lecture, we'll introduce you to Amazon Elastic MapReduce. Here, we'll learn about the EMR service and how it enables us to quickly spin up a cluster compute environment. Go ahead and close this lecture and we'll see you shortly in the next one.
Jeremy is a Content Lead Architect and DevOps SME here at Cloud Academy where he specializes in developing DevOps technical training documentation.
He has a strong background in software engineering, and has been coding with various languages, frameworks, and systems for the past 25+ years. In recent times, Jeremy has been focused on DevOps, Cloud (AWS, Azure, GCP), Security, Kubernetes, and Machine Learning.
Jeremy holds professional certifications for AWS, Azure, GCP, Terraform, Kubernetes (CKA, CKAD, CKS).