The DynamoDB-Caused AWS Outage: What We Have Learned

Over the course of a few hours this past September 20, some of the Internet’s most popular sites like Netflix, Airbnb, and IMDb – along with other AWS customers – suffered major latency and even some outages. The proximate cause? Amazon’s Status dashboard told the story of this AWS outage:
Amazon's Status dashboard
The official note announcing the AWS outage read:

Between 2:13 AM and 8:15 AM PDT we experienced high error rates for API requests in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.”

A six-hour AWS outage will almost certainly translate to catastrophic failure for someone. This is especially true considering that this outage had an impact on as many as 22 AWS services, including DynamoDB, CloudWatch, Auto-Scaling, Simple Email Service (SES), Simple Notification Service (SNS), Simple Queue Service (SQS), CloudFormation, Lambda, SWF, and WorkSpaces (all in the N. Virginia region).

What caused the AWS outage?

The outage and its cause were identified by early that morning when error rates in Amazon DynamoDB started increasing. In a short time, most of the other major services in US-standard region were dragged in.
The root cause was identified as a problem with Amazon’s DynamoDB metadata service for partitioning.

An unexpected network disruption briefly affected DynamoDB’s storage servers ability to communicate with its metadata services. When the network issue was resolved, many storage servers simultaneously tried to load the metadata. While this usually goes off seamlessly, in this instance, the extra traffic caused the metadata service responses to exceed the retrieval and transmission time allowed by storage servers, causing storage servers to reject any further requests. After many unsuccessful attempts to bring down the load and increase the capacity of the metadata service, the servers needed to shut down.

The impact cascaded through the AWS system, dragging down other services that use DynamoDB to store their internal tables. After six hours of firefighting, AWS engineers brought up the capacity of metadata service significantly, the metadata service is successfully reactivated, and storage servers are brought back up to full operation.

What was done in the aftermath of the AWS outage?

Amazon has taken many preventive actions to avoid any recurrence of similar events. The capacity of the metadata service has already been increased significantly, stricter monitoring is put in place to identify the membership size and arrive at the correct capacity. For the longer term, Amazon plans to segment the DynamoDB service so that instances of the metadata service each serve only portions of the storage server fleet.

Lessons learned from the AWS outage

Outages are bound to happen, whether your infrastructure is in an on-premise data center or the cloud. But to minimize your risk, your architecture should be built with a philosophy of “failure is bound to happen”.  Netflix, the media giant that relies heavily on AWS for its operation, quickly recovered from this crisis. They attribute their resilience to what they call chaos engineering.

With its experience from past AWS outages, Netflix regularly deploys its Simian Army: software that deliberately attempts to disrupt its systems. Chaos Monkey shutdowns their production system randomly. Chaos Gorilla simulates an availability-zone failure and Latency Monkey introduces latency on the network. By constantly testing itself with failures, Netflix barely blinked this time around, as it quickly redirected traffic from the impacted AWS region to datacenters in an unaffected area.

Netflix also maintains active-active replication for critical data. Though it cost them 25% more on their AWS bill, it serves them (and their customers) very well fends in just the kind of emergency we’re talking about.

Capacity planning and stricter monitoring of newer services is a must. In our case, an important element of the problem was the increased metadata generated by a new feature called the Global Secondary Index (GSI). GSI allows users to access the table using an alternate key. With GSI, the partition per table increased significantly for some very large tables. With a larger volume of data, the processing time inside the metadata service for some membership requests began to exceed the retrieval allowed time by storage servers. Due to the limited capacity of the metadata service, this quickly became an outage. According to Amazon

We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests.

Amazon quickly apologized to customers, while noting that DynamoDB has effectively enjoyed 100 percent uptime in the past three years.

“We apologize for the impact to affected customers. While we are proud of the last three years of availability on DynamoDB (it’s effectively been 100%), we know how critical this service is to customers, both because many use it for mission-critical operations and because AWS services also rely on it. For us, availability is the most important feature of DynamoDB, and we will do everything we can to learn from the event and to avoid a recurrence in the future.”

Amazon did its best given the problem it faced. They have also been careful to provide helpful and reassuring communication in the AWS outage. As a user of a massive service like AWS, their customers should also shoulder their share of the responsibility, to design and operate their infrastructures more like Netflix.

If you want to get a jump start on DynamoDB, check out Cloud Academy’s Working with Amazon DynamoDB Course.

Learn how to create Amazon DynamoDB tables, add indexes, and query your data in the Introduction to DynamoDB Hands-on Lab.

Avatar

Written by

Chandan Patra

Cloud Computing and Big Data professional with 10 years of experience in pre-sales, architecture, design, build and troubleshooting with best engineering practices. Specialities: Cloud Computing - AWS, DevOps(Chef), Hadoop Ecosystem, Storm & Kafka, ELK Stack, NoSQL, Java, Spring, Hibernate, Web Service


Related Posts

Avatar
Stuart Scott
— October 16, 2019

AWS Security: Bastion Host, NAT instances and VPC Peering

Effective security requires close control over your data and resources. Bastion hosts, NAT instances, and VPC peering can help you secure your AWS infrastructure. Welcome to part four of my AWS Security overview. In part three, we looked at network security at the subnet level. This ti...

Read more
  • AWS
Avatar
Sudhi Seshachala
— October 9, 2019

Top 13 Amazon Virtual Private Cloud (VPC) Best Practices

Amazon Virtual Private Cloud (VPC) brings a host of advantages to the table, including static private IP addresses, Elastic Network Interfaces, secure bastion host setup, DHCP options, Advanced Network Access Control, predictable internal IP ranges, VPN connectivity, movement of interna...

Read more
  • AWS
  • best practices
  • VPC
Avatar
Stuart Scott
— October 2, 2019

Big Changes to the AWS Certification Exams

With AWS re:Invent 2019 just around the corner, we can expect some early announcements to trickle through with upcoming features and services. However, AWS has just announced some big changes to their certification exams. So what’s changing and what’s new? There is a brand NEW ...

Read more
  • AWS
  • Certifications
Alisha Reyes
Alisha Reyes
— October 1, 2019

New on Cloud Academy: ITIL® 4, Microsoft 365 Tenant, Jenkins, TOGAF® 9.1, and more

At Cloud Academy, we're always striving to make improvements to our training platform. Based on your feedback, we released some new features to help make it easier for you to continue studying. These new features allow you to: Remove content from “Continue Studying” section Disc...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
  • ITIL® 4
  • Jenkins
  • Microsoft 365 Tenant
  • New content
  • Product Feature
  • Python programming
  • TOGAF® 9.1
Avatar
Stuart Scott
— September 27, 2019

AWS Security Groups: Instance Level Security

Instance security requires that you fully understand AWS security groups, along with patching responsibility, key pairs, and various tenancy options. As a precursor to this post, you should have a thorough understanding of the AWS Shared Responsibility Model before moving onto discussi...

Read more
  • AWS
  • instance security
  • Security
  • security groups
Avatar
Jeremy Cook
— September 17, 2019

Cloud Migration Risks & Benefits

If you’re like most businesses, you already have at least one workload running in the cloud. However, that doesn’t mean that cloud migration is right for everyone. While cloud environments are generally scalable, reliable, and highly available, those won’t be the only considerations dri...

Read more
  • AWS
  • Azure
  • Cloud Migration
Joe Nemer
Joe Nemer
— September 12, 2019

Real-Time Application Monitoring with Amazon Kinesis

Amazon Kinesis is a real-time data streaming service that makes it easy to collect, process, and analyze data so you can get quick insights and react as fast as possible to new information.  With Amazon Kinesis you can ingest real-time data such as application logs, website clickstre...

Read more
  • amazon kinesis
  • AWS
  • Stream Analytics
  • Streaming data
Joe Nemer
Joe Nemer
— September 6, 2019

Google Cloud Functions vs. AWS Lambda: The Fight for Serverless Cloud Domination

Serverless computing: What is it and why is it important? A quick background The general concept of serverless computing was introduced to the market by Amazon Web Services (AWS) around 2014 with the release of AWS Lambda. As we know, cloud computing has made it possible for users to ...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
Joe Nemer
Joe Nemer
— September 3, 2019

Google Vision vs. Amazon Rekognition: A Vendor-Neutral Comparison

Google Cloud Vision and Amazon Rekognition offer a broad spectrum of solutions, some of which are comparable in terms of functional details, quality, performance, and costs. This post is a fact-based comparative analysis on Google Vision vs. Amazon Rekognition and will focus on the tech...

Read more
  • Amazon Rekognition
  • AWS
  • Google Cloud Platform
  • Google Vision
Alisha Reyes
Alisha Reyes
— August 30, 2019

New on Cloud Academy: CISSP, AWS, Azure, & DevOps Labs, Python for Beginners, and more…

As Hurricane Dorian intensifies, it looks like Floridians across the entire state might have to hunker down for another big one. If you've gone through a hurricane, you know that preparing for one is no joke. You'll need a survival kit with plenty of water, flashlights, batteries, and n...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
  • New content
  • Product Feature
  • Python programming
Joe Nemer
Joe Nemer
— August 27, 2019

Amazon Route 53: Why You Should Consider DNS Migration

What Amazon Route 53 brings to the DNS table Amazon Route 53 is a highly available and scalable Domain Name System (DNS) service offered by AWS. It is named by the TCP or UDP port 53, which is where DNS server requests are addressed. Like any DNS service, Route 53 handles domain regist...

Read more
  • Amazon
  • AWS
  • Cloud Migration
  • DNS
  • Route 53
Alisha Reyes
Alisha Reyes
— August 22, 2019

How to Unlock Complimentary Access to Cloud Academy

Are you looking to get trained or certified on AWS, Azure, Google Cloud Platform, DevOps, Cloud Security, Python, Java, or another technical skill? Then you'll want to mark your calendars for August 23, 2019. Starting Friday at 12:00 a.m. PDT (3:00 a.m. EDT), Cloud Academy is offering c...

Read more
  • AWS
  • Azure
  • cloud academy content
  • complimentary access
  • GCP
  • on the house