image
Running and Monitoring the Data Flow Demo
Start course
Difficulty
Intermediate
Duration
1h 5m
Students
2200
Ratings
4.6/5
Description

In this course, we're going to review the features, concepts, and requirements that are necessary for designing data flows and how to implement them in Microsoft Azure. We’re also going to cover the basics of data flows, common data flow scenarios, and what all is involved in designing a typical data flow.

Learning Objectives

  • Understand key components that are available in Azure that can be used to design and deploy data flows
  • Know how the components fit together

Intended Audience

This course is intended for IT professionals who are interested in earning Azure certification and for those who need to work with data flows in Azure.

Prerequisites 

To get the most from this course, you should have at least a basic understanding of data flows and what they are used for.

Transcript

Hello and welcome back. So, before we go ahead and run this data flow, let's bounce back into our sample data. Now, if you remember, we created the sink that's going to dump this data into an output folder or an output container. If we go into sample data, we don't currently have a output here yet. And that's because we didn't actually run the data flow. When you run these data previews, nothing gets written. So, we have to do is actually run the data flow for that output to happen, and that's what we're going to do here. 

To make this pipeline run, we're going to go up into the pipeline canvas here. This is where the entire pipeline is defined. It's where everything starts. And what we're going to do here is click 'Debug', and this is going to trigger a run of this pipeline and all the data flow that happens within it. We can see the status is queued down here. We can see it's now in progress. Now, once this completes, we get the glasses icon here. If we click on the glasses icon, we see the monitoring pane. And this monitoring pane allows us to view the number of rows that were processed and the time spent on others rows.

Now, what we can do here is we can click on each transformation to see what happened at that particular stage. So, for example with the SourceMovieDB, we can see the rows that were calculated, status, columns, all that information. We can then look at FilterYears, this was the year filtering transformation. And then we can go into the AggregateComedyRatings. And then lastly, we can check the sink. And this tells us that we've calculated 83 rows that were written and two columns, which is how the tutorial was supposed to go. And that is how you design a data flow that includes multiple transformations. So that, let's call it a wrap.

 

About the Author
Students
91021
Courses
89
Learning Paths
56

Tom is a 25+ year veteran of the IT industry, having worked in environments as large as 40k seats and as small as 50 seats. Throughout the course of a long an interesting career, he has built an in-depth skillset that spans numerous IT disciplines. Tom has designed and architected small, large, and global IT solutions.

In addition to the Cloud Platform and Infrastructure MCSE certification, Tom also carries several other Microsoft certifications. His ability to see things from a strategic perspective allows Tom to architect solutions that closely align with business needs.

In his spare time, Tom enjoys camping, fishing, and playing poker.