The course is part of this learning path
This course explores and introduces you to the concepts of both decoupled and event-driven architectures within AWS. It also provides an introduction to the Amazon Simple Queue Service, Amazon Simple Notification Service, Amazon Kinesis and AWS Lambda.
For any feedback, comments, or questions related to this course, feel free to reach out to us at firstname.lastname@example.org.
The objectives of this course are:
- To establish an understanding of what decoupled architecture is
- Establish an understanding of what event-driven architecture is
- To learn the foundations of Amazon SQS and how it is used in a decoupled environment
- To gain an awareness of the Amazon Simple Notification Service, Amazon Kinesis and AWS Lambda to understand when and why you might implement them in an event-driven solution
This course has been designed for architects who are looking to design and implement best practice solutions by utilizing services in a decoupled and/or event-driven environment.
To get the most from this course, it would be beneficial to have a basic awareness of what AWS is, in addition to understanding general infrastructure and application architectures, although this is not essential.
Hello and welcome to this lecture, which will cover the SQS Service, Simple Queue Service. With the continuing growth of microservices and the cloud best practice of designing decoupled systems, it's imperative that developers have the ability to utilize a service, or system, that handles the delivery of messages between components. And this is where SQS comes in. SQS is a fully managed service offered by AWS that works seamlessly with serverless systems, mock services, and any distributed architecture. Although it's simply a queueing service for messages between components, it does much more than that. It has the capability of sending, storing, and receiving these messages at scale without dropping message data, as well as utilizing different queue types depending on requirements. And it includes additional features, such as dead-letter queues. It is also possible to configure the service using the AWS Management Console, the AWS Sarli, or using the AWS SDKs. Let me focus on some of the components to allow to understand how the service is put together. The service itself uses three different elements. Two of which are a part of your distributed system, these being the producers and the consumers, and the third part is the actual queue, which is managed by SQS and is managed across a number of SQS servers for resiliency. Let me explain how these components work together.
The producer component of your architecture is responsible for sending messages to your queue. At this point, the SQS service stores the message across a number of SQS servers for resiliency within the specified queue. This ensures that the message remains in the queue should a failure occur with one of the SQS servers. Consumers are responsible for processing the messages within your queue. As a result, when the consumer element of your architecture is ready to process the massage from the queue, the message is retrieved and is then marked as being processed by activating the visibility timeout on the message. This timeout ensures that the same message will not be read and processed by another consumer. When the message has been processed, the consumer then deletes the message from the queue. Before moving on, I just want to point out a little more relating to the visibility timeout. As I said, when a message is retrieved by a consumer, the visibility timeout is started. The default time is 30 seconds, but it can be set up to as long as 12 hours. During this period, the consumer processes the message. If it fails to process a message, perhaps due to a communication error, the consumer will not send a delete message request back to SQS. As a result, if the visibility timeout expires and it doesn't receive the request to delete the message, the message will become available again in the queue for other consumers to process. This message will then appear as a new message to the queue. The value of your visibility timeout should be longer than it takes for your consumers to process your messages. I mentioned earlier that there were different types of queues.
These being standard queues, first-in, first-out queues, and dead-letter queues. Standard queues, which are the default queue type upon configuration, support at-least-once delivery of messages. This means that the message might actually be delivered to the queue more than once, which is largely down to the highly distributive volume of SQS servers, which would make the message appear out of its original order or delivery. As a result, the standard queue will only offer a best-effort on trying to preserve the message ordering from when the message are sent by the producers. Standard queues also offer an almost unlimited number of transactions per second, TPS, making this queue highly scalable. If message ordering is critical to your solution, then standard queues might not be the right choice for you. Instead, you would need to use first-in, first-out queues. This queue is able to ensure the order of messages is maintained, and that there are no duplication of messages within the queue. Unlike standard queues, FIFO queues do have a limited number of transactions per second. These are defaulted to 300 per second for all send and receive and delete operations. If you use batching with SQS, then this changes to 3,000. Batching essentially allows you to perform actions against 10 messages at once within a single action. So, the key takeaways between the two queues are for standard queues, you have unlimited throughput, at-least-once delivery, and best-effort ordering. And for first-in, first-out queues, you have high throughput, first-in, first-out delivery, and exactly-once processing. For both queues, it is also possible to enable encryption using server-side encryption via KMS. A dead-letter queue differs to the standard and FIFA queues as this dead-letter queue is not used as a source queue to hold messages submitted by producers. Instead, the dead-letter queue is used by the source queue to send messages that fail processing for one reason or another. This could be the result of cloud enabling your application, corruption within the message, or simply missing information within a database that no message data relates to.
By the way, if the message can't be processed by a consumer after a maximum number of tries specified, the queue will send the message to a dead-letter queue. This allows engineers to assess why the message failed, to identify where the issue is, to help prevent further messages from falling into the dead-letter queue. By viewing and analyzing the content of these messages, it might be possible to identify the problem and ascertain if the issue exists from the producer or consumer perspective. A couple of points to make with a dead-letter queue is that is must be configured as the same queue type as the source is used against. For example, if the source queue is a standard queue, the dead-letter queue must also be a standard queue type. And similarly, for FIFA queues, the dead-letter queue must also be configured as a FIFA queue. Before I end this lecture, I just want to show a quick demonstration on how to set up a queue and some of the configuration options available during this process. That now brings me to the end of this lecture which covered an introduction to the Simple Queue Service.
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.
To date, Stuart has created 90+ courses relating to Cloud reaching over 140,000 students, mostly within the AWS category and with a heavy focus on security and compliance.
Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.