Managed Workflows for Apache Airflow
This course delves into Amazon-managed workflows for Apache Airflow (MWAA). This is a great service for anyone already using Apache Airflow, and wanting to find a better way to deal with setting up the service, scheduling, and managing their workflow.
- Understand how Amazon-managed workflows for Apache Airflow is implemented within AWS
- Learn about DAGs (Directed Acyclic Graphs), which Apache Airflow uses to run your workflows
- Understand the key components required to set up your own Managed Airflow environment
This is a great service for anyone already using Apache Airflow, and wanting to find a better way to deal with setting up the service, scheduling, and managing their workflow.
To get the most out of this course, you should have a decent understanding of cloud computing and cloud architectures, specifically with Amazon Web Services. You should also have some background knowledge about Apache Airflow, however, that is not a hard requirement. Basic knowledge of ELT pipelines and state machines would also be beneficial.
Your Managed airflow environment will need to live within a VPC and there are a number of important requirements here that we should take into consideration.
The first thing to keep in mind, much like the s3 buckets, is that you can not change your VPC or subnet selections once the environment is created.
Another important requirement is that you must have private subnets within this VPC that are able to connect to the internet. If you do not have such a VPC already set up, the service does have a quick create button that can manage the creation of a multi-AZ, VPC with all the required bells and whistles for you. Neat.
When building the network portion of your MWAA environment, you will need to decide how you wish to access the webserver for Airflow.
You can choose between a private network - where you would need to access the server through a NAT or some kind of secure login behind your VPC. This would be the right option if you want the AIrflow UI to only be visible within a corporate network.
We also have an option for a public network - where the UI would be accessible by secure login over the generic internet (through a URL). This is less secure, but requires no extra setup and makes it accessible from anywhere.
William Meadows is a passionately curious human currently living in the Bay Area in California. His career has included working with lasers, teaching teenagers how to code, and creating classes about cloud technology that are taught all over the world. His dedication to completing goals and helping others is what brings meaning to his life. In his free time, he enjoys reading Reddit, playing video games, and writing books.