1. Home
  2. Training Library
  3. Designing for Disaster Recovery & High availability in AWS - Level 2

Advanced SNS Messaging with FIFO SNS Topics

Let’s summarize the message for this segment in a single statement and then flesh it out. Here it goes. 

“Using FIFO SNS Topics in combination with FIFO queues we can accomplish idempotency at the messaging layer instead of having to modify our applications.”

Now that is a major statement for developers and it’s actually very good news because it makes our work easier on a fairly complicated use-case.  Let’s break it down.  

In the model of “one to many” message distribution that SNS implements the notion of message order is not guaranteed.  SNS will attempt to deliver messages from the publisher in the order they were published into the topic. 

However, network issues could potentially result in out-of-order messages at the subscriber end.  Also most of the time each message will be delivered to your application exactly once, the distributed nature of Amazon SNS and transient network conditions could result in occasional, duplicate messages at the subscriber end.  As developers we should design applications with components such that processing a message more than once does not create any errors or inconsistencies.   This is usually called idempotency.  An application component is idempotent if an identical request can be made one or more times in a row and leave the server in the same state.  All read operations are idempotent by nature.  However, write, or modify operations need to be designed carefully such that if the same modification is requested more than once for reasons outside our control, the change ends up happening once and only once.  

Consider for a moment a request to transfer money from a bank account to make a payment.  If for any reason our request does not complete successfully including an acknowledgment of the transfer, repeating the request which would be natural to do in this case should not and never result in the transfer happening twice.  That would be considered a major malfunction and needs to be anticipated and remediated accordingly by developers of the application when leveraging messages to trigger requests.  

The process of accomplishing this was a bit complicated in that as developers we needed to add a message ID manually in order to help preserve the order of messages published to the topic.  We also needed to implement code for removing duplicate messages when needed.  Especially, if there was an issue with the original request and we end up getting the same request twice. 

These days, by using an SNS FIFO Topic, you can configure a message group by including message group IDs when publishing to an SNS FIFO Topic and for each message group ID, all messages are sent and delivered in order of their arrival.  You can also avoid duplicate message deliveries for the SNS FIFO Topic by enabling content-based message deduplication. This is just a check box in the AWS console for configuring a FIFO Topic. This feature makes SNS use a SHA-256 hash to generate a message deduplication ID using the body of the message. Any duplicated message sent within a time period is accepted but not delivered. 

If an SQS FIFO queue is subscribed to an SNS FIFO Topic, the deduplication ID is passed to the queue and SQS uses it to avoid duplicate messages being sent as well.  

In short, using SNS and SQS together allows us to fan out messages as needed.  Using SNS FIFO Topics, Together with SQS FIFO Queues help us implement application components that are idempotent by design and makes things a lot simpler in the complex process of message ordering preservation and message deduplication logic.  

FIFO topics can process 300 messages per second or 10MB per second whichever limit is hit first. This limit cannot be increased. 

Using SNS FIFO topics, message ordering and deduplication can be achieved at the messaging layer.  Strictly preserved message ordering and exactly once message delivery are an advanced feature of SNS FIFO Topics. 

We can now reduce the effort required to process high throughput, consistently ordered transactions and simplify our messaging architecture. Example use cases include bank transaction logs, stock tickers, flight trackers, price updates, news broadcasting, and inventory management.

Let’s close this segment with the same statement we started because now we have a better notion of what is means and how important it is.  

“Using FIFO SNS Topics in combination with FIFO queues we can accomplish idempotency at the messaging layer instead of having to modify our applications.” 

This is a time and effort saving feature to say the least.

Difficulty
Intermediate
Duration
1h
Students
5
Description

This course covers the core learning objective to meet the requirements of the 'Designing for disaster recovery & high availability in AWS - Level 2' skill

Learning Objectives:

  • Analyze the amount of resources required to implement a fault-tolerant architecture across multiple AWS availability Zones
  • Evaluate an effective AWS disaster recovery strategy to meet specific business requirements
  • Understand SLA for AWS services to ensure the high availability of a given AWS solution
  • Analyze which AWS services can be leveraged to implement a decoupled solution
About the Author
Students
225572
Labs
1
Courses
215
Learning Paths
175

Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.

To date, Stuart has created 150+ courses relating to Cloud reaching over 180,000 students, mostly within the AWS category and with a heavy focus on security and compliance.

Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.

He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.

In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.

Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.