image
Running an Experiment Script
Start course
Difficulty
Intermediate
Duration
1h 23m
Students
1351
Description

Learn how to operate machine learning solutions at cloud scale using the Azure Machine Learning SDK. This course teaches you to leverage your existing knowledge of Python and machine learning to manage data ingestion, data preparation, model training, and model deployment in Microsoft Azure.

If you have any feedback related to this course, please contact us at support@cloudacademy.com.

Learning Objectives

  • Create an Azure Machine Learning workspace using the SDK
  • Run experiments and train models using the SDK
  • Optimize and manage models using the SDK
  • Deploy and consume models using the SDK

Intended Audience

This course is designed for data scientists with existing knowledge of Python and machine learning frameworks, such as Scikit-Learn, PyTorch, and Tensorflow, who want to build and operate machine learning solutions in the cloud.

Prerequisites

  • Fundamental knowledge of Microsoft Azure
  • Experience writing Python code to work with data using libraries such as Numpy, Pandas, and Matplotlib
  • Understanding of data science, including how to prepare data and train machine learning models using common machine learning libraries, such as Scikit-Learn, PyTorch, or Tensorflow

Resources

The GitHub repo for this course, containing the code and datasets used, can be found here: https://github.com/cloudacademy/using-the-azure-machine-learning-sdk 

Transcript

We've now had a look at how to run an experiment in line. There's a much more flexible way of going, going about this, and that's to create a separate script for the experiment and store it in the folder along with any associated files. And then use Azure ML to run the experiment based on the script in the folder.

First, let's create a folder for experiment files and copy the data into it. I'm gonna use the os modules and the shutil modules to help us do this here. So to create a folder for the experiment files I specify the folder name, the experiment folder we'll be using. We use make directory from the os module. And then we use shutil to copy a diabetes data file into the folder we've just specified, the details of the folder we just specified.

So with that done we will then create a Python script containing the code for experiment and save it into the experiment folder. So we use magic command write file to write the file to the folder that we specified. We need our run object, we need pandas, and we need our os libraries. And note here we're getting the experiment run context.

So, I mean this code is a simplified version of the in line version we saw earlier. But as you can see we're using the get context method to retrieve the experiment run context when the script is run. So what it will do, the script will do, it will load the diabetes data from the folder where the script is located. It creates a folder name outputs, and it writes the sample files to it. And this folder is automatically uploaded to the experiment run.

So we count and log label counts as we did earlier on to get a sense of diabetics versus non-diabetics. And that's before with what we did with our in line experiment. We save a sample of the data in the outputs folder okay? And then we complete the run.

We are almost ready to run the experiment. But there's just a few continuation issues we need to deal with first. So we need to create a run configuration that defines a Python code execution environment for the script.

In this case, it will automatically create a conda environment with some default Python packages installed. And then secondly we need to create a script configuration that identifies the Python script file to be run in the experiment, an environment in which to run it in. So the following cells we've got here, allows us to specify a run configuration detail. As well as our script run configuration.

As mentioned earlier our run configuration automatically creates a conda environment with some default Python packages installed. With our script run config, we specify the file that needs to be run in the environment. Then we submit the experiment, this os space. And then we can use run details, the widgets we imported earlier. We can use those to show details and then we can run and invoke weight for completion. Do note that this time around it will take a little bit longer because we obviously need to set up a few things as the conda environment needs to get created.

So now we can get our logged metrics. We can use run.get_metrics to get that information. And I clicked through that and print that to the screen. So observations in terms of the row count, as before 10,000. And we've got information logged into our output folders as well.

About the Author
Students
1350
Courses
1

Kofi is a digital technology specialist in a variety of business applications. He stays up to date on business trends and technology and is an early adopter of powerful and creative ideas.
His experience covers a wide range of topics including data science, machine learning, deep learning, reinforcement learning, DevOps, software engineering, cloud computing, business & technology strategy, design & delivery of flipped/social learning experiences, blended learning curriculum design and delivery, and training consultancy.