Apache Spark is an open-source framework for doing big data processing. It was developed as a replacement for Apache Hadoop’s MapReduce framework. Both Spark and MapReduce process data on compute clusters, but one of Spark’s big advantages is that it does in-memory processing, which can be orders of magnitude faster than the disk-based processing that MapReduce uses. Not only does Spark handle data analytics tasks, but it also handles machine learning.
In 2013, the creators of Spark started a company called Databricks. The name of their product is also Databricks. It’s a cloud-based implementation of Spark with a user-friendly interface for running code on clusters interactively.
Microsoft has partnered with Databricks to bring their product to the Azure platform. The result is a service called Azure Databricks. One of the biggest advantages of using the Azure version of Databricks is that it’s integrated with other Azure services. For example, you can train a machine learning model on a Databricks cluster and then deploy it using Azure Machine Learning Services.
In this course, we will start by showing you how to set up a Databricks workspace and a cluster. Next, we’ll go through the basics of how to use a notebook to run interactive queries on a dataset. Then you’ll see how to run a Spark job on a schedule. After that, we’ll show you how to train a machine learning model. Finally, we’ll go through several ways to deploy a trained model as a prediction service.
Learning Objectives
- Create a Databricks workspace, cluster, and notebook
- Run code in a Databricks notebook either interactively or as a job
- Train a machine learning model using Databricks
- Deploy a Databricks-trained machine learning model as a prediction service
Intended Audience
- People who want to use Azure Databricks to run Apache Spark for either analytics or machine learning workloads
Prerequisites
- Prior experience with Azure and at least one programming language
Additional Resources
The GitHub repository for this course is at https://github.com/cloudacademy/azure-databricks.
Welcome to “Running Spark on Azure Databricks”. My name’s Guy Hummel. I’m a Microsoft certified Azure Solutions Architect, and I’m the Azure Content Lead at Cloud Academy. If you have any questions, feel free to connect with me on LinkedIn and send me a message, or send an email to support@cloudacademy.com.
This course is intended for people who want to use Azure Databricks to run Apache Spark for either analytics or machine learning workloads or both.
To get the most from this course, you should have some prior experience with Azure and at least one programming language. It would also be helpful to have some basic knowledge of both SQL and machine learning, although that’s not strictly necessary. This course is full of hands-on examples, so I recommend that you try performing these tasks yourself on your own Azure account. If you don’t already have one, then you can create a free trial account.
We’ll start with an overview of Spark and Databricks. Then I’ll show you how to set up a Databricks workspace and a cluster. Next, we’ll go through the basics of how to use a notebook to run interactive queries on a dataset. Then you’ll see how to run a Spark job on a schedule. After that, I’ll show you how to train a machine learning model. Finally, I’ll go through several ways to deploy a trained model as a prediction service.
By the end of this course, you should be able to create a Databricks workspace, cluster, and notebook; run code in a Databricks notebook either interactively or as a job; train a machine learning model using Databricks; and deploy a Databricks-trained machine learning model as a prediction service.
We’d love to get your feedback on this course, so please give it a rating when you’re finished.
Now, if you’re ready to learn how to run Spark on an Azure Databricks cluster, then let’s get started.
Guy launched his first training website in 1995 and he's been helping people learn IT technologies ever since. He has been a sysadmin, instructor, sales engineer, IT manager, and entrepreneur. In his most recent venture, he founded and led a cloud-based training infrastructure company that provided virtual labs for some of the largest software vendors in the world. Guy’s passion is making complex technology easy to understand. His activities outside of work have included riding an elephant and skydiving (although not at the same time).