Data Flow Basics
Data Flow Components
Building a Dataflow with Azure Data Factory
The course is part of these learning pathsSee 3 more
In this course, we're going to review the features, concepts, and requirements that are necessary for designing data flows and how to implement them in Microsoft Azure. We’re also going to cover the basics of data flows, common data flow scenarios, and what all is involved in designing a typical data flow.
- Understand key components that are available in Azure that can be used to design and deploy data flows
- Know how the components fit together
This course is intended for IT professionals who are interested in earning Azure certification and for those who need to work with data flows in Azure.
To get the most from this course, you should have at least a basic understanding of data flows and what they are used for.
So, why should the cloud be used for data flow? Well, for starters, a key benefit of using the cloud for data flow is the fact that you don’t have to worry about setting up or maintaining DMZs. Instead of needing to figure out what devices need to be opened to the internet. The cloud kind of serves as a natural barrier, I guess you could say, so you don’t have worry about what ports do and don’t need to be opened.
When deciding how to leverage the cloud for data flow, it’s important to consider the usual suspects like cost, speed, latency, etc. While pulling data into Azure, there are going to be no costs, because it’s considered ingress. However, pulling that data out later is going to incur some kind of cost. It’s also important to consider latencies and bandwidth for both moving data TO azure and FROM azure. With that said, leveraging ExpressRoute and peering would ensure data travels through a private connection, which would maintain a steady, and lower, latency than if it was traveling over the public internet.
Other benefits to the cloud for data flow processing include the number of services that are available to facilitate the data plow process to begin with. Leveraging PaaS services and features allows you to essentially hit the ground running. Simply use the services you need to design and manage your data flows without any worry about maintenance of the platform or services. The amount of work necessary to get up and running is minimal. To add to that, with such a wide array of services available, you’re likely to have access to technologies that you simply wouldn’t have access to on-pre, - stuff like machine learning and artificial intelligence would fall into this bucket.
Leveraging the cloud provides elasticity as well. It allows you to scale almost infinitely if necessary. This is not something that’s easily accomplished on-prem. Cost and time would quickly become barriers to doing so. Instead, the cloud offers the ability to scale up and down as required, while allowing you to pay only for the resources you use. This is especially helpful when dealing with uneven or inconsistent workflows that need to be processed.
These benefits will, obviously, vary on a service-by-service basis.
Tom is a 25+ year veteran of the IT industry, having worked in environments as large as 40k seats and as small as 50 seats. Throughout the course of a long an interesting career, he has built an in-depth skillset that spans numerous IT disciplines. Tom has designed and architected small, large, and global IT solutions.
In addition to the Cloud Platform and Infrastructure MCSE certification, Tom also carries several other Microsoft certifications. His ability to see things from a strategic perspective allows Tom to architect solutions that closely align with business needs.
In his spare time, Tom enjoys camping, fishing, and playing poker.