hands-on lab

Using Azure Data Factory Pipelines to Copy Data

Intermediate
Up to 1h 15m
2,239
4.6/5
Get guided in a real environmentPractice with a step-by-step scenario in a real, provisioned environment.
Learn and validateUse validations to check your solutions every step of the way.
See resultsTrack your knowledge and monitor your progress.

Description

Azure Data Factory (ADF) is a managed cloud service for ingesting, preparing and transforming data from multiple sources. ADF provides code-free, visual data pipeline interface to describe workflows allowing data engineers and non-expert data integrators alike to accomplish complex data manipulation tasks. Over 90 data sources are supported including Azure services, Amazon Web Services, Google Cloud services, Teradata, and Salesforce.

This lab introduces you to Azure Data Factory. You will get acquainted with ADF by performing an Azure Blob Storage data movement operation using an ADF pipeline.

Warning: Currently, Data Factory UI is only officially supported by Microsoft Edge and Google Chrome web browsers. It is also recommended to use incognito mode to avoid conflicts with other Microsoft accounts impacting Data Factory UI.

Learning Objectives

Upon completion of this intermediate-level lab, you will be able to:

  • Create an Azure Data Factory
  • Understand and initialize ADF linked services
  • Initialize ADF datasets
  • Develop basic data pipelines in Data Factory UI
  • Learn to Debug and Trigger ADF pipelines
  • Triggers ADF pipeline runs on a schedule

Intended Audience

  • Candidates for Microsoft Azure Data Engineering Certifications
  • Data Engineers

Prerequisites

Familiarity with the following will be beneficial but is not required:

  • Basic understanding of Azure Data Factory
  • Basic understanding of Azure Blob Storage

The following courses can be used to fulfill the prerequisite:

Portions of this lab's content have been adapted from attributed sources.

Updates

October 3rd, 2023 - Resolved deployment issue

Environment before

Environment after

About the author

Avatar
Logan Rakai, opens in a new tab
Lead Content Developer - Labs
Students
220,634
Labs
223
Courses
9
Learning paths
57

Logan has been involved in software development and research since 2007 and has been in the cloud since 2012. He is an AWS Certified DevOps Engineer - Professional, AWS Certified Solutions Architect - Professional, Microsoft Certified Azure Solutions Architect Expert, MCSE: Cloud Platform and Infrastructure, Google Cloud Certified Associate Cloud Engineer, Certified Kubernetes Security Specialist (CKS), Certified Kubernetes Administrator (CKA), Certified Kubernetes Application Developer (CKAD), and Certified OpenStack Administrator (COA). He earned his Ph.D. studying design automation and enjoys all things tech.

LinkedIn, Twitter, GitHub

Covered topics

Lab steps

Logging in to the Microsoft Azure Portal
Creating an Azure Data Factory
Creating a linked service
Creating the Data Factory Datasets
Creating a Data Factory Pipeline
Debugging the Data Factory Pipeline
Triggering and Monitoring the Pipeline
Triggering the Pipeline on a Schedule