1. Home
  2. Training Library
  3. Serverless, Component Decoupling, and Solution Architectures (SAP-C02)

Advanced SNS Messaging with FIFO SNS Topics


Course Introduction
Utilizing Managed Services and Serverless Architectures to Minimize Cost
Decoupled Architecture
Amazon API Gateway
Advanced API Gateway
PREVIEW11m 29s
Amazon Elastic Map Reduce
Introduction to EMR
Amazon EventBridge
Design considerations

The course is part of this learning path

Let’s summarize the message for this segment in a single statement and then flesh it out. Here it goes. 

“Using FIFO SNS Topics in combination with FIFO queues we can accomplish idempotency at the messaging layer instead of having to modify our applications.”

Now that is a major statement for developers and it’s actually very good news because it makes our work easier on a fairly complicated use-case.  Let’s break it down.  

In the model of “one to many” message distribution that SNS implements the notion of message order is not guaranteed.  SNS will attempt to deliver messages from the publisher in the order they were published into the topic. 

However, network issues could potentially result in out-of-order messages at the subscriber end.  Also most of the time each message will be delivered to your application exactly once, the distributed nature of Amazon SNS and transient network conditions could result in occasional, duplicate messages at the subscriber end.  As developers we should design applications with components such that processing a message more than once does not create any errors or inconsistencies.   This is usually called idempotency.  An application component is idempotent if an identical request can be made one or more times in a row and leave the server in the same state.  All read operations are idempotent by nature.  However, write, or modify operations need to be designed carefully such that if the same modification is requested more than once for reasons outside our control, the change ends up happening once and only once.  

Consider for a moment a request to transfer money from a bank account to make a payment.  If for any reason our request does not complete successfully including an acknowledgment of the transfer, repeating the request which would be natural to do in this case should not and never result in the transfer happening twice.  That would be considered a major malfunction and needs to be anticipated and remediated accordingly by developers of the application when leveraging messages to trigger requests.  

The process of accomplishing this was a bit complicated in that as developers we needed to add a message ID manually in order to help preserve the order of messages published to the topic.  We also needed to implement code for removing duplicate messages when needed.  Especially, if there was an issue with the original request and we end up getting the same request twice. 

These days, by using an SNS FIFO Topic, you can configure a message group by including message group IDs when publishing to an SNS FIFO Topic and for each message group ID, all messages are sent and delivered in order of their arrival.  You can also avoid duplicate message deliveries for the SNS FIFO Topic by enabling content-based message deduplication. This is just a check box in the AWS console for configuring a FIFO Topic. This feature makes SNS use a SHA-256 hash to generate a message deduplication ID using the body of the message. Any duplicated message sent within a time period is accepted but not delivered. 

If an SQS FIFO queue is subscribed to an SNS FIFO Topic, the deduplication ID is passed to the queue and SQS uses it to avoid duplicate messages being sent as well.  

In short, using SNS and SQS together allows us to fan out messages as needed.  Using SNS FIFO Topics, Together with SQS FIFO Queues help us implement application components that are idempotent by design and makes things a lot simpler in the complex process of message ordering preservation and message deduplication logic.  

FIFO topics can process 300 messages per second or 10MB per second whichever limit is hit first. This limit cannot be increased. 

Using SNS FIFO topics, message ordering and deduplication can be achieved at the messaging layer.  Strictly preserved message ordering and exactly once message delivery are an advanced feature of SNS FIFO Topics. 

We can now reduce the effort required to process high throughput, consistently ordered transactions and simplify our messaging architecture. Example use cases include bank transaction logs, stock tickers, flight trackers, price updates, news broadcasting, and inventory management.

Let’s close this segment with the same statement we started because now we have a better notion of what is means and how important it is.  

“Using FIFO SNS Topics in combination with FIFO queues we can accomplish idempotency at the messaging layer instead of having to modify our applications.” 

This is a time and effort saving feature to say the least.

4h 43m

This section of the AWS Certified Solutions Architect - Professional learning path introduces common AWS solution architectures relevant to the AWS Certified Solutions Architect - Professional exam and the services that support them. These services form a core component of running resilient and performant architectures. 

Want more? Try a Lab Playground or do a Lab Challenge!

Learning Objectives

  • Learn how to utilize managed services and serverless architectures to minimize cost
  • Understand how to use AWS services to process streaming data
  • Discover AWS services that support mobile app development
  • Understand when to utilize serverless services within your AWS solutions
  • Learn which AWS services to use when building a decoupled architecture
About the Author
Learning Paths

Danny has over 20 years of IT experience as a software developer, cloud engineer, and technical trainer. After attending a conference on cloud computing in 2009, he knew he wanted to build his career around what was still a very new, emerging technology at the time — and share this transformational knowledge with others. He has spoken to IT professional audiences at local, regional, and national user groups and conferences. He has delivered in-person classroom and virtual training, interactive webinars, and authored video training courses covering many different technologies, including Amazon Web Services. He currently has six active AWS certifications, including certifications at the Professional and Specialty level.