# SQS Messaging Patterns

Advanced messaging patterns and configurations.

## Queue Policies

### Allow SNS to Send Messages

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:my-topic"
        }
      }
    }
  ]
}
```

### Allow S3 to Send Messages

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::my-bucket"
        }
      }
    }
  ]
}
```

### Cross-Account Access

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}
```

## Encryption

### Server-Side Encryption (SSE-SQS)

```bash
aws sqs create-queue \
  --queue-name my-queue \
  --attributes '{
    "SqsManagedSseEnabled": "true"
  }'
```

### SSE with KMS

```bash
aws sqs create-queue \
  --queue-name my-queue \
  --attributes '{
    "KmsMasterKeyId": "alias/my-key",
    "KmsDataKeyReusePeriodSeconds": "300"
  }'
```

## FIFO Queue Patterns

### Message Groups for Parallel Processing

```python
import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/orders.fifo'

# Different customers can be processed in parallel
# Same customer maintains order
orders = [
    {'customer_id': 'A', 'order_id': '1'},
    {'customer_id': 'B', 'order_id': '2'},
    {'customer_id': 'A', 'order_id': '3'},  # After order 1
]

for order in orders:
    sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps(order),
        MessageGroupId=order['customer_id'],  # Group by customer
        MessageDeduplicationId=f"{order['customer_id']}-{order['order_id']}"
    )
```

### High Throughput FIFO

```bash
aws sqs create-queue \
  --queue-name my-queue.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true",
    "DeduplicationScope": "messageGroup",
    "FifoThroughputLimit": "perMessageGroupId"
  }'
```

## Batch Processing with Lambda

### Partial Batch Failure

```python
def handler(event, context):
    batch_item_failures = []

    for record in event['Records']:
        try:
            body = json.loads(record['body'])
            process_message(body)
        except Exception as e:
            # Report this specific message as failed
            batch_item_failures.append({
                'itemIdentifier': record['messageId']
            })

    return {'batchItemFailures': batch_item_failures}
```

### Configure Lambda for Partial Failures

```bash
aws lambda update-event-source-mapping \
  --uuid <mapping-uuid> \
  --function-response-types ReportBatchItemFailures
```

## Visibility Timeout Management

### Extend Visibility During Processing

```python
import boto3
import time
import threading

sqs = boto3.client('sqs')

def extend_visibility(queue_url, receipt_handle, stop_event):
    """Background thread to extend visibility."""
    while not stop_event.wait(timeout=30):
        try:
            sqs.change_message_visibility(
                QueueUrl=queue_url,
                ReceiptHandle=receipt_handle,
                VisibilityTimeout=60
            )
        except Exception:
            break

def process_long_running_message(queue_url, message):
    stop_event = threading.Event()

    # Start background visibility extender
    extender = threading.Thread(
        target=extend_visibility,
        args=(queue_url, message['ReceiptHandle'], stop_event)
    )
    extender.start()

    try:
        # Long processing...
        do_long_processing(message['Body'])

        # Delete on success
        sqs.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message['ReceiptHandle']
        )
    finally:
        stop_event.set()
        extender.join()
```

## Message Delay Patterns

### Per-Message Delay

```python
# Delay individual message up to 15 minutes
sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=json.dumps({'action': 'reminder'}),
    DelaySeconds=900  # 15 minutes
)
```

### Scheduled Processing

```python
import time

def schedule_message(queue_url, body, execute_at):
    """Schedule message for future processing."""
    delay = int(execute_at - time.time())

    if delay <= 0:
        delay = 0
    elif delay > 900:
        # For delays > 15 min, use message attribute
        # and filter on receive
        sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=json.dumps({
                **body,
                '_execute_at': execute_at
            }),
            DelaySeconds=900  # Maximum delay
        )
        return

    sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps(body),
        DelaySeconds=delay
    )
```

## Monitoring

### CloudWatch Alarms

```bash
# DLQ depth alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "SQS-DLQ-NotEmpty" \
  --metric-name ApproximateNumberOfMessagesVisible \
  --namespace AWS/SQS \
  --dimensions Name=QueueName,Value=my-queue-dlq \
  --statistic Sum \
  --period 60 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:alerts

# Queue backlog alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "SQS-Queue-Backlog" \
  --metric-name ApproximateNumberOfMessagesVisible \
  --namespace AWS/SQS \
  --dimensions Name=QueueName,Value=my-queue \
  --statistic Sum \
  --period 300 \
  --threshold 1000 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:alerts

# Age of oldest message
aws cloudwatch put-metric-alarm \
  --alarm-name "SQS-Old-Messages" \
  --metric-name ApproximateAgeOfOldestMessage \
  --namespace AWS/SQS \
  --dimensions Name=QueueName,Value=my-queue \
  --statistic Maximum \
  --period 300 \
  --threshold 3600 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:alerts
```

### Custom Metrics

```python
import boto3

cloudwatch = boto3.client('cloudwatch')

def publish_processing_metrics(queue_name, processing_time, success):
    cloudwatch.put_metric_data(
        Namespace='MyApp/SQS',
        MetricData=[
            {
                'MetricName': 'ProcessingTime',
                'Value': processing_time,
                'Unit': 'Milliseconds',
                'Dimensions': [
                    {'Name': 'QueueName', 'Value': queue_name}
                ]
            },
            {
                'MetricName': 'ProcessingSuccess' if success else 'ProcessingFailure',
                'Value': 1,
                'Unit': 'Count',
                'Dimensions': [
                    {'Name': 'QueueName', 'Value': queue_name}
                ]
            }
        ]
    )
```

## Request-Response Pattern

### Temporary Response Queue

```python
import boto3
import json
import uuid

sqs = boto3.client('sqs')

def send_request_with_response(request_queue_url, payload):
    # Create temporary response queue
    correlation_id = str(uuid.uuid4())
    response_queue = sqs.create_queue(
        QueueName=f'response-{correlation_id}',
        Attributes={'MessageRetentionPeriod': '300'}
    )
    response_queue_url = response_queue['QueueUrl']

    try:
        # Send request with reply-to
        sqs.send_message(
            QueueUrl=request_queue_url,
            MessageBody=json.dumps(payload),
            MessageAttributes={
                'ReplyTo': {
                    'DataType': 'String',
                    'StringValue': response_queue_url
                },
                'CorrelationId': {
                    'DataType': 'String',
                    'StringValue': correlation_id
                }
            }
        )

        # Wait for response
        response = sqs.receive_message(
            QueueUrl=response_queue_url,
            WaitTimeSeconds=20,
            MaxNumberOfMessages=1
        )

        if response.get('Messages'):
            return json.loads(response['Messages'][0]['Body'])

        return None
    finally:
        sqs.delete_queue(QueueUrl=response_queue_url)
```

## Large Message Pattern

### Using S3 for Large Payloads

```python
import boto3
import json
import uuid

s3 = boto3.client('s3')
sqs = boto3.client('sqs')

def send_large_message(queue_url, bucket, payload):
    """Send message > 256KB using S3."""
    payload_str = json.dumps(payload)

    if len(payload_str) > 200000:  # Use S3 for large payloads
        key = f'sqs-payloads/{uuid.uuid4()}.json'
        s3.put_object(Bucket=bucket, Key=key, Body=payload_str)

        sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=json.dumps({
                's3_bucket': bucket,
                's3_key': key
            }),
            MessageAttributes={
                'PayloadLocation': {
                    'DataType': 'String',
                    'StringValue': 's3'
                }
            }
        )
    else:
        sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=payload_str
        )

def receive_large_message(queue_url, bucket):
    """Receive message, fetch from S3 if needed."""
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MessageAttributeNames=['PayloadLocation'],
        MaxNumberOfMessages=1
    )

    if not response.get('Messages'):
        return None

    message = response['Messages'][0]
    attrs = message.get('MessageAttributes', {})

    if attrs.get('PayloadLocation', {}).get('StringValue') == 's3':
        pointer = json.loads(message['Body'])
        response = s3.get_object(
            Bucket=pointer['s3_bucket'],
            Key=pointer['s3_key']
        )
        body = json.loads(response['Body'].read())

        # Clean up S3 object
        s3.delete_object(Bucket=pointer['s3_bucket'], Key=pointer['s3_key'])
    else:
        body = json.loads(message['Body'])

    return body, message['ReceiptHandle']
```

---
name: sqs
description: AWS SQS message queue service for decoupled architectures. Use when creating queues, configuring dead-letter queues, managing visibility timeouts, implementing FIFO ordering, or integrating with Lambda.
last_updated: "2026-01-07"
doc_source: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/
---

# AWS SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service for decoupling and scaling microservices, distributed systems, and serverless applications.

## Table of Contents

- [Core Concepts](#core-concepts)
- [Common Patterns](#common-patterns)
- [CLI Reference](#cli-reference)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)
- [References](#references)

## Core Concepts

### Queue Types

| Type | Description | Use Case |
|------|-------------|----------|
| **Standard** | At-least-once, best-effort ordering | High throughput |
| **FIFO** | Exactly-once, strict ordering | Order-sensitive processing |

### Key Settings

| Setting | Description | Default |
|---------|-------------|---------|
| **Visibility Timeout** | Time message is hidden after receive | 30 seconds |
| **Message Retention** | How long messages are kept | 4 days (max 14) |
| **Delay Seconds** | Delay before message is available | 0 |
| **Max Message Size** | Maximum message size | 256 KB |

### Dead-Letter Queue (DLQ)

Queue for messages that failed processing after maxReceiveCount attempts.

## Common Patterns

### Create a Standard Queue

**AWS CLI:**

```bash
aws sqs create-queue \
  --queue-name my-queue \
  --attributes '{
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "604800",
    "ReceiveMessageWaitTimeSeconds": "20"
  }'
```

**boto3:**

```python
import boto3

sqs = boto3.client('sqs')

response = sqs.create_queue(
    QueueName='my-queue',
    Attributes={
        'VisibilityTimeout': '60',
        'MessageRetentionPeriod': '604800',
        'ReceiveMessageWaitTimeSeconds': '20'  # Long polling
    }
)
queue_url = response['QueueUrl']
```

### Create FIFO Queue

```bash
aws sqs create-queue \
  --queue-name my-queue.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true"
  }'
```

### Configure Dead-Letter Queue

```bash
# Create DLQ
aws sqs create-queue --queue-name my-queue-dlq

# Get DLQ ARN
DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-dlq \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' --output text)

# Set redrive policy on main queue
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes "{
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"
  }"
```

### Send Messages

```python
import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

# Send single message
sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=json.dumps({'order_id': '12345', 'action': 'process'}),
    MessageAttributes={
        'MessageType': {
            'DataType': 'String',
            'StringValue': 'Order'
        }
    }
)

# Send to FIFO queue
sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo',
    MessageBody=json.dumps({'order_id': '12345'}),
    MessageGroupId='order-12345',
    MessageDeduplicationId='unique-id-12345'
)

# Batch send (up to 10 messages)
sqs.send_message_batch(
    QueueUrl=queue_url,
    Entries=[
        {'Id': '1', 'MessageBody': json.dumps({'id': 1})},
        {'Id': '2', 'MessageBody': json.dumps({'id': 2})},
        {'Id': '3', 'MessageBody': json.dumps({'id': 3})}
    ]
)
```

### Receive and Process Messages

```python
import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

while True:
    # Long polling (wait up to 20 seconds)
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=20,
        MessageAttributeNames=['All'],
        AttributeNames=['All']
    )

    messages = response.get('Messages', [])

    for message in messages:
        try:
            body = json.loads(message['Body'])
            print(f"Processing: {body}")

            # Process message...

            # Delete on success
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=message['ReceiptHandle']
            )
        except Exception as e:
            print(f"Error processing message: {e}")
            # Message will become visible again after visibility timeout
```

### Lambda Integration

```bash
# Create event source mapping
aws lambda create-event-source-mapping \
  --function-name my-function \
  --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5
```

Lambda handler:

```python
def handler(event, context):
    for record in event['Records']:
        body = json.loads(record['body'])
        message_id = record['messageId']

        try:
            process_message(body)
        except Exception as e:
            # Raise to put message back in queue
            raise

    return {'batchItemFailures': []}
```

## CLI Reference

### Queue Management

| Command | Description |
|---------|-------------|
| `aws sqs create-queue` | Create queue |
| `aws sqs delete-queue` | Delete queue |
| `aws sqs list-queues` | List queues |
| `aws sqs get-queue-url` | Get queue URL by name |
| `aws sqs get-queue-attributes` | Get queue settings |
| `aws sqs set-queue-attributes` | Update queue settings |

### Messaging

| Command | Description |
|---------|-------------|
| `aws sqs send-message` | Send single message |
| `aws sqs send-message-batch` | Send up to 10 messages |
| `aws sqs receive-message` | Receive messages |
| `aws sqs delete-message` | Delete message |
| `aws sqs delete-message-batch` | Delete up to 10 messages |
| `aws sqs purge-queue` | Delete all messages |

### Visibility

| Command | Description |
|---------|-------------|
| `aws sqs change-message-visibility` | Change timeout |
| `aws sqs change-message-visibility-batch` | Batch change |

## Best Practices

### Message Processing

- **Use long polling** (WaitTimeSeconds=20) to reduce API calls
- **Delete messages promptly** after successful processing
- **Configure appropriate visibility timeout** (> processing time)
- **Implement idempotent consumers** for at-least-once delivery

### Dead-Letter Queues

- **Always configure DLQ** for production queues
- **Set appropriate maxReceiveCount** (usually 3-5)
- **Monitor DLQ depth** with CloudWatch alarms
- **Process DLQ messages** manually or with automation

### FIFO Queues

- **Use message group IDs** to partition ordering
- **Enable content-based deduplication** or provide dedup IDs
- **Throughput**: 300 msgs/sec without batching, 3000 with

### Security

- **Use queue policies** to control access
- **Enable encryption** with SSE-SQS or SSE-KMS
- **Use VPC endpoints** for private access

## Troubleshooting

### Messages Not Being Received

**Causes:**
- Short polling returning empty
- All messages in flight (visibility timeout)
- Messages delayed (DelaySeconds)

**Debug:**

```bash
# Check queue attributes
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names All

# Check approximate message counts
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names \
    ApproximateNumberOfMessages,\
    ApproximateNumberOfMessagesNotVisible,\
    ApproximateNumberOfMessagesDelayed
```

### Messages Going to DLQ

**Causes:**
- Processing errors
- Visibility timeout too short
- Consumer not deleting messages

**Redrive from DLQ:**

```bash
# Enable redrive allow policy on source queue
aws sqs set-queue-attributes \
  --queue-url $MAIN_QUEUE_URL \
  --attributes '{"RedriveAllowPolicy": "{\"redrivePermission\":\"allowAll\"}"}'

# Start redrive
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123456789012:my-queue-dlq \
  --destination-arn arn:aws:sqs:us-east-1:123456789012:my-queue
```

### Duplicate Processing

**Solutions:**
- Use FIFO queues for exactly-once
- Implement idempotency in consumer
- Track processed message IDs in database

### Lambda Not Processing

```bash
# Check event source mapping
aws lambda list-event-source-mappings \
  --function-name my-function

# Check for errors
aws lambda get-event-source-mapping \
  --uuid <mapping-uuid>
```

## References

- [SQS Developer Guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/)
- [SQS API Reference](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/)
- [SQS CLI Reference](https://docs.aws.amazon.com/cli/latest/reference/sqs/)
- [boto3 SQS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html)