AWS SQS Retention Period: Default Limits and Configuration

AWS Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. One of the critical configurations in SQS is the Retention Period.

In this guide, we will explore what the SQS Retention Period is, its default values, and what happens to messages when they expire.

<div class="toc"> <ul> <li><a href="#what-is-aws-sqs-retention-period">What is AWS SQS Retention Period?</a></li> <li><a href="#default-and-maximum-retention-periods">Default and Maximum Retention Periods</a></li> <li><a href="#message-lifecycle-and-expiration">Message Lifecycle and Expiration</a></li> <li><a href="#using-dead-letter-queues-dlq">Using Dead Letter Queues (DLQ)</a></li> <li><a href="#how-to-configure-retention-period">How to Configure Retention Period?</a></li> <li><a href="#frequently-asked-questions-faq">Frequently Asked Questions (FAQ)</a></li> </ul> </div> <h2 id="what-is-aws-sqs-retention-period">What is AWS SQS Retention Period?</h2>

The Retention Period is the amount of time a message resides in the SQS queue before it is automatically deleted. For more technical details, refer to the official <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-retention-period.html" target="_blank" rel="noopener noreferrer">AWS SQS Message Retention Period Documentation</a>.

If consumers do not process and delete the message within this timeframe, AWS SQS removes it automatically. The retention timer starts from the moment the message is added to the queue. It is important to note that the timer does not reset if the message is being processed or fails processing.

<h2 id="default-and-maximum-retention-periods">Default and Maximum Retention Periods</h2>

The retention periods for SQS queues are as follows:

Default Period: 4 Days
Minimum Period: 60 Seconds (1 Minute)
Maximum Period: 14 Days (1,209,600 Seconds)

This flexibility allows you to configure the queue according to your application needs. For instance, you might increase the retention period for debugging purposes or decrease it for time-sensitive data.

<h2 id="message-lifecycle-and-expiration">Message Lifecycle and Expiration</h2>

A typical message lifecycle involves:

Producer: Sends a message to the queue. State: "Message Available". Retention timer starts.
Consumer: Polls and receives the message. State: "In-Flight".
Success: Consumer processes and deletes the message.
Failure: If processing fails, the message returns to the queue after the Visibility Timeout.

However, when the Retention Period expires, the message is permanently deleted regardless of its state. Therefore, it is crucial to account for consumer processing times and potential delays when setting this value.

<h2 id="using-dead-letter-queues-dlq">Using Dead Letter Queues (DLQ)</h2>

To prevent data loss for expired or unprocessable messages, it is best practice to use a Dead Letter Queue (DLQ). When a message exceeds the Maximum Receive Count or expires (depending on Redrive Policy configuration), it can be moved to a DLQ.

This allows you to analyze failed messages and replay them later if necessary.

<h2 id="how-to-configure-retention-period">How to Configure Retention Period?</h2>

You can configure the Retention Period using the AWS Console, AWS CLI, or Infrastructure as Code (IaC) tools like Terraform.

<h3 id="configuring-with-aws-cli">Configuring with AWS CLI</h3>

The following command sets the retention period of an existing queue to 14 days (1,209,600 seconds). See the <a href="https://docs.aws.amazon.com/cli/latest/reference/sqs/set-queue-attributes.html" target="_blank" rel="noopener noreferrer">AWS CLI Command Reference: set-queue-attributes</a> for more options.

aws sqs set-queue-attributes     --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue     --attributes MessageRetentionPeriod=1209600

<h3 id="configuring-with-terraform">Configuring with Terraform</h3>

If you are using Terraform, you can set the message_retention_seconds argument in the aws_sqs_queue resource:

resource "aws_sqs_queue" "terraform_queue" {
  name                      = "terraform-example-queue"
  delay_seconds             = 90
  max_message_size          = 2048
  message_retention_seconds = 1209600 # 14 days
  receive_wait_time_seconds = 10
}

<h2 id="frequently-asked-questions-faq">Frequently Asked Questions (FAQ)</h2> <h3 id="faq-1">1. How long are SQS messages stored by default?</h3> The default retention period is 4 days. You can adjust this between 1 minute and 14 days. <h3 id="faq-2">2. What happens to a message when the retention period expires?</h3> The message is automatically deleted from the queue. If a Dead Letter Queue (DLQ) is configured, the message might be moved there depending on your Redrive Policy. <h3 id="faq-3">3. Can I increase the retention period beyond 14 days?</h3> No, the maximum retention period for AWS SQS is 14 days. For longer retention, consider backing up messages to durable storage like Amazon S3 or DynamoDB.

For more information, check out our AWS Consultancy and Kubernetes Consultancy services.

<br> <p>Source: <a href="https://awsfundamentals.com/blog/aws-sqs-retention-period">https://awsfundamentals.com/blog/aws-sqs-retention-period</a></p>