MWAA vs. Step Functions

Start course
Overview
Difficulty
Intermediate
Duration
15m
Students
66
Ratings
5/5
starstarstarstarstar
Description

This course delves into Amazon-managed workflows for Apache Airflow (MWAA). This is a great service for anyone already using Apache Airflow, and wanting to find a better way to deal with setting up the service, scheduling, and managing their workflow.

Learning Objectives

  • Understand how Amazon-managed workflows for Apache Airflow is implemented within AWS
  • Learn about DAGs (Directed Acyclic Graphs), which Apache Airflow uses to run your workflows
  • Understand the key components required to set up your own Managed Airflow environment

Intended Audience

This is a great service for anyone already using Apache Airflow, and wanting to find a better way to deal with setting up the service, scheduling, and managing their workflow.

Prerequisites

To get the most out of this course, you should have a decent understanding of cloud computing and cloud architectures, specifically with Amazon Web Services. You should also have some background knowledge about Apache Airflow, however, that is not a hard requirement. Basic knowledge of ELT pipelines and state machines would also be beneficial.

Transcript

This is a big question. Should you use managed workflows for apache airflow or should you use AWS Step Functions. Well, my advice here will come down to a few things. First off if you are already using Apache airflow and you would just like to have it managed for you, then the choice is pretty obvious. 

However, if you are just starting a new project and are trying to make a decision between the two options, as they both do very similar things, I think it can come down to the following.

If you will need to support many thousands of concurrent tasks or operations and have your workflow system scale up to meet this need -  I would recommend building out with AWS Step Functions. Step functions is able to handle scales that would cripple airflow's scheduler. However, if you will be supporting a lower volume of tasks, Airflow has a lot of great features and quality of life assurances that AWS Step Functions lacks.

Another thing to consider is the cost of using both systems. In general AWS Step functions is dirt cheap, while running MWAA can cost quite a lot more overall. One of the main reasons for the cost difference is that Step Functions is a serverless service, meaning you don't have an active EC2 instance you are constantly running and paying for. 

Managed workflows for apache airflow will always need at least the environment instance running. Additionally, if you need to scale, you will have to add in more worker and scheduler instances to keep up with demand. Each of those also cost extra.

If we look at just the cost for a medium environment instance running for 31 days, 24 hours a day, that gives us 744 hours of run time. This is currently charged at $0.74 per hour, costing about $550.

Although it's not an apples-to-apples comparison, if we look at AWS Step functions cost, You are charged based on the number of state transitions required to execute your application.

This means if your workflow has 10 different state transitions you would have 10 charges for that workflow. The billing is calculated at $0.025 per 1000 state transitions or $0.000025 per single transition. You also get 4000 state transitions for free per month.

So if we executed our workflow with 10 state transitions 100,000 times per month. That would be 1 million billable transitions, giving us a grand total cost of… $25. Minus some small amount from the free tier taking off 4000 transitions.

 

So that cost difference might be something to think about when determining if you want to build out using MWAA or Step Functions.

About the Author

William Meadows is a passionately curious human currently living in the Bay Area in California. His career has included working with lasers, teaching teenagers how to code, and creating classes about cloud technology that are taught all over the world. His dedication to completing goals and helping others is what brings meaning to his life. In his free time, he enjoys reading Reddit, playing video games, and writing books.