Advanced SNS Messaging with FIFO SNS Topics


SAA-C03 Introduction
Decoupled Architecture
AWS Step Functions
AWS Step Functions
AWS Machine Learning Services
Design considerations
SAA-C03 Review
Start course
3h 46m

Domain One of The AWS Solution Architect Associate exam guide SAA-C03 requires us to be able to Design a multi-tier architecture solution so that is our topic for this section.
We cover the need to know aspects of how to design Multi-Tier solutions using AWS services. 

Want more? Try a lab playground or do a Lab Challenge!

Learning Objectives

  • Learn some of the essential services for creating multi-tier architect on AWS, including the Simple Queue Service (SQS) and the Simple Notification Service (SNS)
  • Understand data streaming and how Amazon Kinesis can be used to stream data
  • Learn how to design a multi-tier solution on AWS, and the important aspects to take into consideration when doing so
  • Learn how to design cost-optimized AWS architectures
  • Understand how to leverage AWS services to migrate applications and databases to the AWS Cloud

Let's summarize the message for this segment in a single statement and then flesh it out. Here it goes: Using FIFO SNS Topics in combination with FIFO queues, we can accomplish idempotency at the messaging layer instead of having to modify our applications. Now, that is a major statement for developers, and it's actually very good news, because it makes our work easier on a fairly complicated use case. Let's break it down. In the model of one-to-many message distribution that SNS implements, the notion of message order is not guaranteed. SNS will attempt to deliver messages from the publisher in the order that they were published into the topic. However, network issues could potentially result in out-of-order messages at the subscriber end. Also, most of the time each message will be delivered to your application exactly once. The distributed nature of Amazon SNS and transient network conditions could result in occasional duplicate message at the subscriber end.

As developers, we should design applications with components such that processing a message more than once does not create any errors or inconsistencies. This is usually called idempotency. An application component is itempotent if an identical request can be made one or more times in a row and leave the server exactly in the same state. All read operations are idempotent by nature. However, write or modify operations need to be designed carefully, such that if the same modification is requested more than once, for reasons outside our control, the change ends up happening once and only once. Consider for a moment a request to transfer money from a bank account to make a payment. If for any reason our request does not complete successfully, including an acknowledgement of the transfer, repeating the request, which would be natural to do in this case, should not and never result in the transfer happening twice. That would be considered a major malfunction and needs to be anticipated and remediated accordingly by developers of the application when leveraging messages to trigger requests.

The process of accomplishing this was a bit complicated, in that as developers we needed to add a message ID manually in order to help preserve the order of messages published to the topic. We also needed to implement code for removing duplicate messages when needed, especially if there was an issue with the original request and we end up getting the same request twice. These days, by using an SNS FIFO Topic, you can configure a message group by including a message group ID when publishing to an SNS FIFO Topic. And for each message group ID, all messages are sent and delivered in the order of their arrival. You can also avoid duplicate message deliveries for the SNS FIFO Topic by enabling content-based message deduplication. This is just a checkbox in the AWS console for configuring a FIFO Topic. This feature makes SNS use a SHA256# to generate a message deduplication ID using the actual body of the message. Any duplicated message sent within the time period is accepted but not delivered. If an SQS FIFO queue is subscribed to an SNS FIFO Topic, the deduplication ID is passed to the queue, and SQS uses it to avoid duplicate messages being sent as well.

So, in short, using SNS and SQS together allows us to find out messages as needed. Using SNS FIFO Topics together with SQS FIFO queues help us implement application components that are idempotent by design and make things a lot simpler in the complex process of message ordering preservation and message deduplication logic. FIFO Topics can process 300 messages per second, or 10 megabytes per second, whichever limit is hit first. This limit cannot be increased. Using SNS FIFO Topics, message ordering and deduplication can be achieved at the messaging layer. Strictly preserved message ordering and exactly once message delivery are an advanced feature of SNS FIFO Topics. We can now reduce the effort required to process high throughput, consistently ordered transactions, and simplify our messaging architecture. Example use cases include: bank transaction logs, stock tickers, flight trackers, price updates, news broadcasting, and inventory management. Let's close this segment with the same statement that we started it, because now we have a better notion of what it means and how important it is. Using FIFO SNS Topics in combination with FIFO queues, we can accomplish idempotency at the messaging layer instead of having to modify our applications. This is a time saving and effort saving feature, to say the least.


About the Author
Learning Paths

Andrew is fanatical about helping business teams gain the maximum ROI possible from adopting, using, and optimizing Public Cloud Services. Having built  70+ Cloud Academy courses, Andrew has helped over 50,000 students master cloud computing by sharing the skills and experiences he gained during 20+  years leading digital teams in code and consulting. Before joining Cloud Academy, Andrew worked for AWS and for AWS technology partners Ooyala and Adobe.