Setting Up Your S3 Buckets to Work With MWAA

Start course
Overview
Difficulty
Intermediate
Duration
15m
Students
66
Ratings
5/5
starstarstarstarstar
Description

This course delves into Amazon-managed workflows for Apache Airflow (MWAA). This is a great service for anyone already using Apache Airflow, and wanting to find a better way to deal with setting up the service, scheduling, and managing their workflow.

Learning Objectives

  • Understand how Amazon-managed workflows for Apache Airflow is implemented within AWS
  • Learn about DAGs (Directed Acyclic Graphs), which Apache Airflow uses to run your workflows
  • Understand the key components required to set up your own Managed Airflow environment

Intended Audience

This is a great service for anyone already using Apache Airflow, and wanting to find a better way to deal with setting up the service, scheduling, and managing their workflow.

Prerequisites

To get the most out of this course, you should have a decent understanding of cloud computing and cloud architectures, specifically with Amazon Web Services. You should also have some background knowledge about Apache Airflow, however, that is not a hard requirement. Basic knowledge of ELT pipelines and state machines would also be beneficial.

Transcript

In order to use MWAA you will need to provide it with a DAG, your custom plugins, and any python dependencies your solution requires. These materials will need to live in a specific S3 folder that will hold all of your supporting files.

I recommend you create the bucket before provisioning your MWAA environment, but you have the option to create it at runtime as well.

First off, it's important to note that you can not change the bucket name after it has been created - so make sure it's something you are happy with. Additionally, Make sure your name is also DNS compliant.

Second, Any bucket that is used for an Amazon MWAA environment will need to be configured with versioning enabled and to block all public access.

Third, Your bucket MUST be located within the same AWS Region as the MWAA environment that you are trying to create.

Within this bucket, you will need a folder for your DAGs to live at the bare minimum. You will probably also need another folder for your plugins, and another for your requirements.

 

Other than that, you are all set on the bucket front.

About the Author

William Meadows is a passionately curious human currently living in the Bay Area in California. His career has included working with lasers, teaching teenagers how to code, and creating classes about cloud technology that are taught all over the world. His dedication to completing goals and helping others is what brings meaning to his life. In his free time, he enjoys reading Reddit, playing video games, and writing books.