AWS SQS
When we start deploying multiple applications, they will inevitably need to communicate with each other.
We have two types of communication.
1: Synchronous communication which is basically application to application through APIs.
2: Asynchronous / Event based which is an application to queue to application.
Synchronous communication between applications can be problematic if there are sudden increases in traffic. It’s better to decouple your applications in the real world.
- SQS: Queue model
- SNS: Pub(publish)/sub(subscribe) model
- Kinesis: Real-time streaming model
Let’s get started with AWS SQS (Simple Queue Service).
AWS SQS- Simple Queue Service
What is SQS
Basically SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a consumer to process them.
SQS is a distributed queue system and a queue is a temporary repo for messages that are waiting to be processed.
AWS SQS Architecture
What Are the Main Benefits of Amazon SQS?
Security: You control who can send messages to and receive messages from an Amazon SQS queue.
Server-side encryption (SSE): It lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
Durability: To ensure the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support at-least-once message delivery, and FIFO queues support exactly-once message processing.
Availability: Amazon SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.
Scalability: Amazon SQS can process each buffered request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
Reliability: Amazon SQS locks your messages during processing so that multiple producers can send and multiple consumers can receive messages at the same time.
Customization: Your queues don’t have to be exactly alike — for example, you can set a default delay on a queue. You can store the contents of messages larger than 256 KB using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.
Types of Queue:
AWS SQS — Standard Queue
It’s fully managed, scales from 1 message per second to 10,000s per second.
Default retention of messages: 4 days, maximum of 14 days and no limit to how many messages can be in the queue.
It has low latency (<10 ms on publish and receive) and horizontal scaling in terms of number of consumers. It also can have duplicate messages (at least once delivery, occasionally). Limitation of 256KB per message sent.
AWS SQS — FIFO (First In First Out)
The most important features of FIFO queues are First-In-First-Out delivery and exactly-once processing. The order in which messages are sent and received is strictly preserved and a message is delivered once and remains until consumer processes delete it. Duplication is not allowed in FIFO and it is limited to 300 transactions/sec (TPS).
Name of the queue must end in .fifo Eg: MyTestFIFOQueue.fifo
SQS is pull based, not pushed based.
What Type of Queues you need: Standard vs FIFO
Comparison Standard VS FIFO(source AWS Official)
Amazon SQS and Amazon SNS are queue and topic services that are highly scalable, simple to use, and don’t require you to set up message brokers. Amazon MQ is a managed message broker service that provides compatibility with many popular message brokers. We recommend Amazon MQ for migrating applications from existing message brokers that rely on compatibility with APIs such as JMS or protocols such as AMQP, MQTT, Open-Wire, and STOMP.
How to Produce Messaging: SQS
We define the body, add message attributes (metadata — optional), and provide delay delivery (optional). We Get back the message identifier and an MD5 hash of the body.
SQS — Consuming Messages
Poll SQS for messages (receive up to 10 messages at a time). Process the message within the visibility timeout, and delete the message using the message ID and receipt handle.
SQS — Visibility timeout
When a consumer polls a message from a queue, the message is “invisible” to other consumers for a defined period which is the Visibility Timeout.
We can set between 0 seconds and 12 hours (default 30 is seconds). If it is too high (15 minutes) and the consumer fails to process the message, you must wait a long time before processing the message again, and if it is too low (30 seconds) and the consumer needs time to process the message (2 minutes), another consumer will receive the message and the message will be processed more than once.
AWS SQS — Delay Queue
Delay queues let you postpone the delivery of new messages to a queue for a number of seconds. If you create a delay queue, any messages that you send to the queue remain invisible to consumers for the duration of the delay period. Delay a message (consumers don’t see it immediately) up to 15 minutes and the default is 0 seconds (message is available right away). We can set a default queue level and also can override the default using DelaySeconds parameter.