CloudAcademy
  1. Home
  2. Training Library
  3. Amazon Web Services
  4. Courses
  5. Using SQS, SNS and SES in a decoupled and distributed environment

Introduction to the Simple Queue Service

The course is part of these learning paths

DevOps Engineer – Professional Certification Preparation for AWS
course-steps 35 certification 4 lab-steps 17 quiz-steps 2 description 3
Solutions Architect – Professional Certification Preparation for AWS
course-steps 45 certification 4 lab-steps 19 quiz-steps 5 description 2
SysOps Administrator – Associate Certification Preparation for AWS
course-steps 33 certification 4 lab-steps 30 quiz-steps 4 description 5
Certified Developer – Associate Certification Preparation for AWS
course-steps 27 certification 4 lab-steps 22 description 2
more_horiz See 1 more
play-arrow
Start course
Overview
DifficultyBeginner
Duration37m
Students673

Description

With this course, you will be helping and enabling your development teams to architect applications that are highly scalable and resilient using best practices when working within a decoupled environment. This will lead to increased availability of operational service to their applications helping to minimize incidents that would affect the end user.

In this course, you will learn to design greater applications using managed services in a decoupled environment along with being able to differentiate and explain the differences between SQS, SES, and SNS.

Though designed as a beginners course, a basic awareness of AWS Lambda and IAM policies would give a headstart in better understanding the core principles of this course. In this course, you will find three comprehensive lectures along with further resources that can be found the description box underneath the respective lectures.

Learning Objectives

  • Explain the difference between SQS, SNS, and SES
  • Understand when you would use each SQS, SNS, and SES within a solution
  • Identity and explain the key components of within SQS, SNS, and SES

Intended Audience

  • Developers working in a decoupled environment.
  • Cloud Solution architects.
  • Those who are looking to take the AWS Certified Developer - Associate certification.

Prerequisites

  • An awareness of IAM policies and how they are constructed.
  • For more information on IAM policies, please see the following course.
  • Basic knowledge of AWS Lambda.

Related Training Content




Transcript

Hello and welcome to this lecture, which will cover the SQS Service, Simple Queue Service. With the continuing growth of mock services and the cloud-based practice of designing decoupled systems, it's imperative that developers have the ability to utilize a service, or system, that handles the delivery of messages between components. And this is where SQS comes in. SQS is a fully managed service offered by AWS that works seamlessly with serverless systems, mock services, and any distributed architecture. Although it's simply a queueing service for messages between components, it does much more than that. It has the capability of sending, storing, and receiving these messages at scale without dropping message data, as well as utilizing different queue types depending on requirements. And it includes additional features, such as dead-letter queues. It is also possible to configure the service using the AWS Management Console, the AWS Sarli, or using the AWS SDKs. Let me focus on some of the components to allow to understand how the service is put together. The service itself uses three different elements. Two of which are a part of your distributed system, these being the producers and the consumers, and the third part is the actual queue, which is managed by SQS and is managed across a number of SQS servers for resiliency. Let me explain how these components work together.

 The producer component of your architecture is responsible for sending messages to your queue. At this point, the SQS service stores the message across a number of SQS servers for resiliency within the specified queue. This ensures that the message remains in the queue should a failure occur with one of the SQS servers. Consumers are responsible for processing the messages within your queue. As a result, when the consumer element of your architecture is ready to process the massage from the queue, the message is retrieved and is then marked as being processed by activating the visibility timeout on the message. This timeout ensures that the same message will not be read and processed by another consumer. When the message has been processed, the consumer then deletes the message from the queue. Before moving on, I just want to point out a little more relating to the visibility timeout. As I said, when a message is retrieved by a consumer, the visibility timeout is started. The default time is 30 seconds, but it can be set up to as long as 12 hours. During this period, the consumer processes the message. If it fails to process a message, perhaps due to a communication error, the consumer will not send a delete message request back to SQS. As a result, if the visibility timeout expires and it doesn't receive the request to delete the message, the message will become available again in the queue for other consumers to process. This message will then appear as a new message to the queue. The value of your visibility timeout should be longer than it takes for your consumers to process your messages. I mentioned earlier that there were different types of queues. 

These being standard queues, first-in, first-out queues, and dead-letter queues. Standard queues, which are the default queue type upon configuration, support at-least-once delivery of messages. This means that the message might actually be delivered to the queue more than once, which is largely down to the highly distributive volume of SQS servers, which would make the message appear out of its original order or delivery. As a result, the standard queue will only offer a best-effort on trying to preserve the message ordering from when the message are sent by the producers. Standard queues also offer an almost unlimited number of transactions per second, TPS, making this queue highly scalable. If message ordering is critical to your solution, then standard queues might not be the right choice for you. Instead, you would need to use first-in, first-out queues. This queue is able to ensure the order of messages is maintained, and that there are no duplication of messages within the queue. Unlike standard queues, FIFO queues do have a limited number of transactions per second. These are defaulted to 300 per second for all send and receive and delete operations. If you use batching with SQS, then this changes to 3,000. Batching essentially allows you to perform actions against 10 messages at once within a single action. So, the key takeaways between the two queues are for standard queues, you have unlimited throughput, at-least-once delivery, and best-effort ordering. And for first-in, first-out queues, you have high throughput, first-in, first-out delivery, and exactly-once processing. For both queues, it is also possible to enable encryption using server-side encryption via KMS. A dead-letter queue differs to the standard and FIFA queues as this dead-letter queue is not used as a source queue to hold messages submitted by producers. Instead, the dead-letter queue is used by the source queue to send messages that fail processing for one reason or another. This could be the result of cloud enabling your application, corruption within the message, or simply missing information within a database that no message data relates to.

 By the way, if the message can't be processed by a consumer after a maximum number of tries specified, the queue will send the message to a dead-letter queue. This allows engineers to assess why the message failed, to identify where the issue is, to help prevent further messages from falling into the dead-letter queue. By viewing and analyzing the content of these messages, it might be possible to identify the problem and ascertain if the issue exists from the producer or consumer perspective. A couple of points to make with a dead-letter queue is that is must be configured as the same queue type as the source is used against. For example, if the source queue is a standard queue, the dead-letter queue must also be a standard queue type. And similarly, for FIFA queues, the dead-letter queue must also be configured as a FIFA queue. Before I end this lecture, I just want to show a quick demonstration on how to set up a queue and some of the configuration options available during this process. That now brings me to the end of this lecture which covered an introduction to the Simple Queue Service.

About the Author

Students44084
Labs1
Courses50
Learning paths31

Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data centre and network infrastructure design, to cloud architecture and implementation.

To date Stuart has created over 40 courses relating to Cloud, most within the AWS category with a heavy focus on security and compliance

He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.

In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.

Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.