In this course, we're going to review the features, concepts, and requirements that are necessary for designing data flows and how to implement them in Microsoft Azure. We’re also going to cover the basics of data flows, common data flow scenarios, and what all is involved in designing a typical data flow.
Learning Objectives
- Understand key components that are available in Azure that can be used to design and deploy data flows
- Know how the components fit together
Intended Audience
This course is intended for IT professionals who are interested in earning Azure certification and for those who need to work with data flows in Azure.
Prerequisites
To get the most from this course, you should have at least a basic understanding of data flows and what they are used for.
Hello and welcome back. In this demonstration here, I'm going to show you how to create a pipeline that includes data flow activity. So, we are right where we left off in the last demonstration, we're at the overview page for our data factory. What we need to do here to create our pipeline is open the data factory studio. Now, once we're in our studio, what we want to do is select the orchestrate option. So, we'll go ahead and do that. And now this is where things get interesting.
Now, as I mentioned earlier, I'm following one of Microsoft's tutorials to walk you through this. So, if I'm going to follow that tutorial, I'm going to follow the naming conventions for the most part as well. So, what we're going to do here in our orchestration page here for our pipeline is we're going to give our pipeline a name over here in the general tab and we'll call this TransformMovies. Now after naming our pipeline, what we're going to do here is go under activities and since we're going to move and transform data, that's what we're going to select here, move and transform. Now, since we're building a data flow, we're going to take the data flow activity here and drag that onto our canvas.
Now, what we're going to do here is go into settings and we can select a data flow here, which we don't have any yet or we can create a new one. And we'll call our new data flow TransformMovies. Now because we're going to do some interactive testing, we need to enable the data flow debug option here by turning the toggle switch on and it gives me a warning here about debugging, not being in west central, we're fine. So, we're okay. And this data flow debug takes a few minutes to enable, so we'll give this a few seconds here or maybe even a few minutes for it to come on. Now as we wait for the debug to be enabled, I just want to point out these two tabs here.
We have TransformMovies with the picture of a pipe here and we have the TransformMovies with the little icon here from a cube to what looks like a triangle I guess. The TransformMovies here is the pipeline, hence the pipe icon here. And then this little icon here tells us we're working on our data flow. So, let this debug do its thing and we'll come back when it's ready. Now, if we move over to TransformMovies in the pipeline, we can see that the data flow debug is showing a green checkmark. I'm guessing if we bounce back over here, we also have the green checkmark.
So, at this point we have the pipeline and the data flow identified or defined, I should say. What we'll do here is we'll take a break. I'll pick up in the next demonstration where we'll actually start to build the transformation logic in this data flow canvas. So, let's call it a wrap for now and I'll see you over in the next demonstration where we'll pick back up.
Tom is a 25+ year veteran of the IT industry, having worked in environments as large as 40k seats and as small as 50 seats. Throughout the course of a long an interesting career, he has built an in-depth skillset that spans numerous IT disciplines. Tom has designed and architected small, large, and global IT solutions.
In addition to the Cloud Platform and Infrastructure MCSE certification, Tom also carries several other Microsoft certifications. His ability to see things from a strategic perspective allows Tom to architect solutions that closely align with business needs.
In his spare time, Tom enjoys camping, fishing, and playing poker.