Lambda + SQS — Message-Driven Serverless

DodaTech Updated 2026-06-28 6 min read

In this tutorial, you will learn about Lambda + SQS. We cover key concepts, practical examples, and best practices to help you master this topic.

AWS SQS integrates with Lambda for queue-based message processing where functions poll queues, Process messages in batches, and delete successfully handled messages automatically.

What You'll Learn

By the end of this lesson you will understand how to configure SQS as a Lambda event source, handle batch messages, implement partial batch failures, configure DLQs, and use SQS for service decoupling.

Why It Matters

SQS decouples producers from consumers. When your Lambda processes an SQS message, the producer never waits for the consumer. If processing fails, the message stays in the queue for retry. This pattern is fundamental to resilient Serverless architectures.

Real-World Use

DodaZIP's file conversion service uses SQS to queue conversion requests. When a user submits a file, the API Lambda puts a message in SQS. A separate Lambda polls the queue, performs the conversion, and stores the result -- the user never waits for the conversion to complete.

flowchart LR
    P[Producer Lambda] --> Q[SQS Queue]
    Q --> C[Consumer Lambda]
    C -->|Success| D[Delete Message]
    C -->|Failure| R[Retry / DLQ]
    Q --> DLQ[Dead-Letter Queue]
    style Q fill:#f90,color:#fff

SQS Event Source Mapping

Lambda automatically polls SQS at configurable rates and invokes your function with batches of messages. The function must return success or failure for the batch.

# sqs_handler.py
# Basic SQS Lambda handler

import json

def lambda_handler(event, context):
    for record in event["Records"]:
        message_id = record["messageId"]
        body = json.loads(record["body"])
        receipt_handle = record["receiptHandle"]
        
        print(f"Processing message {message_id}: {body}")
        
        if body.get("type") == "send_email":
            send_email(body["to"], body["subject"], body["body"])
        elif body.get("type") == "process_order":
            process_order(body["order_id"])
        else:
            print(f"Unknown message type: {body.get('type')}")
    
    return {"batchItemFailures": []}

def send_email(to, subject, body):
    print(f"  -> Sending email to {to}: {subject}")

def process_order(order_id):
    print(f"  -> Processing order {order_id}")

sqs_event = {"Records": [
    {"messageId": "m1", "receiptHandle": "h1", "body": '{"type": "send_email", "to": "alice@example.com", "subject": "Welcome", "body": "Hello!"}'},
    {"messageId": "m2", "receiptHandle": "h2", "body": '{"type": "process_order", "order_id": "ORD-001"}'}
]}
lambda_handler(sqs_event, None)

Expected output:

Processing message m1: {'type': 'send_email', 'to': 'alice@example.com', 'subject': 'Welcome', 'body': 'Hello!'}
  -> Sending email to alice@example.com: Welcome
Processing message m2: {'type': 'process_order', 'order_id': 'ORD-001'}
  -> Processing order ORD-001

Partial Batch Failures

When some messages in a batch fail, use reportBatchItemFailures to return only the failed message IDs. AWS retries only those messages.

# partial_failures.py
# Handling partial batch failures

import json

def lambda_handler(event, context):
    failed_ids = []
    
    for record in event["Records"]:
        try:
            message_id = record["messageId"]
            body = json.loads(record["body"])
            
            print(f"Processing {message_id}: {body}")
            
            if body.get("simulate_failure"):
                raise ValueError(f"Simulated failure for {message_id}")
            
            print(f"  -> Success: {message_id}")
        
        except Exception as e:
            print(f"  -> Failed: {message_id} - {e}")
            failed_ids.append({"itemIdentifier": message_id})
    
    print(f"Failed items: {len(failed_ids)}")
    return {"batchItemFailures": failed_ids}

sqs_event = {"Records": [
    {"messageId": "success-1", "receiptHandle": "h1", "body": '{"simulate_failure": false}'},
    {"messageId": "fail-1", "receiptHandle": "h2", "body": '{"simulate_failure": true}'},
    {"messageId": "success-2", "receiptHandle": "h3", "body": '{"simulate_failure": false}'}
]}
result = lambda_handler(sqs_event, None)
print(f"Returned batchItemFailures: {result}")

Expected output:

Processing success-1: {'simulate_failure': false}
  -> Success: success-1
Processing fail-1: {'simulate_failure': true}
  -> Failed: fail-1 - Simulated failure for fail-1
Processing success-2: {'simulate_failure': false}
  -> Success: success-2
Failed items: 1
Returned batchItemFailures: {'batchItemFailures': [{'itemIdentifier': 'fail-1'}]}

Dead-Letter Queues

Messages that exceed the maximum retry count are moved to a dead-letter queue for manual inspection and reprocessing.

# dlq_pattern.py
# Dead-letter queue pattern

MAX_RETRIES = 3

def lambda_handler(event, context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        attributes = record.get("attributes", {})
        receive_count = int(attributes.get("ApproximateReceiveCount", 1))
        
        print(f"Message received {receive_count} times")
        
        if receive_count > MAX_RETRIES:
            print(f"  -> Moving to DLQ: {body}")
            move_to_dlq(body)
            continue
        
        try:
            process_message(body)
        except Exception as e:
            print(f"  -> Processing failed, retry {receive_count}/{MAX_RETRIES}")
            raise

def process_message(body):
    if body.get("fragile"):
        raise Exception("Processing error")
    print(f"  -> Processed: {body}")

def move_to_dlq(body):
    print(f"  -> (DLQ) {body} stored for manual review")

messages = [
    {"messageId": "m1", "body": '{"fragile": true}', "attributes": {"ApproximateReceiveCount": "1"}},
    {"messageId": "m2", "body": '{"fragile": false}', "attributes": {"ApproximateReceiveCount": "1"}},
    {"messageId": "m3", "body": '{"fragile": true}', "attributes": {"ApproximateReceiveCount": "4"}},
]

for msg in messages:
    event = {"Records": [msg]}
    try:
        lambda_handler(event, None)
    except:
        pass

Expected output:

Message received 1 times
  -> Processing failed, retry 1/3
Message received 1 times
  -> Processed: {'fragile': false}
Message received 4 times
  -> Moving to DLQ: {'fragile': true}
  -> (DLQ) {'fragile': true} stored for manual review

SQS FIFO Queues

FIFO queues preserve message order and guarantee exactly-once processing. Lambda processes messages in order within each message group.

# fifo_pattern.py
# FIFO queue processing

import json

def lambda_handler(event, context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        group_id = record["attributes"]["MessageGroupId"]
        dedup_id = record["attributes"]["MessageDeduplicationId"]
        
        print(f"[Group: {group_id}] Message: {dedup_id}")
        print(f"  Processing: {body}")
        
        if body.get("type") == "create_user":
            print(f"  Creating user: {body['email']}")
        elif body.get("type") == "update_user":
            print(f"  Updating user: {body['email']}")

fifo_event = {"Records": [
    {"messageId": "f1", "receiptHandle": "h1", "body": '{"type": "create_user", "email": "alice@example.com"}', "attributes": {"MessageGroupId": "user-1", "MessageDeduplicationId": "u1-1"}},
    {"messageId": "f2", "receiptHandle": "h2", "body": '{"type": "update_user", "email": "alice@example.com"}', "attributes": {"MessageGroupId": "user-1", "MessageDeduplicationId": "u1-2"}}
]}
lambda_handler(fifo_event, None)

Common Mistakes

Not handling partial batch failures: Without returning failed item IDs, the entire batch is treated as failed and all messages are retried.
Setting too long visibility timeout: If the visibility timeout is shorter than function execution, the same message is processed multiple times.
Ignoring message size limits: SQS messages are limited to 256KB. Use S3 for larger payloads and send the S3 reference in SQS.
Using standard queues when ordering matters: Standard queues deliver messages at least once but may reorder. Use FIFO for strict ordering.
Not monitoring queue depth: A growing queue depth indicates the consumer cannot keep up. Set CloudWatch alarms on ApproximateNumberOfMessagesVisible.

Practice Questions

How does Lambda poll SQS? Lambda internally polls SQS at regular intervals and invokes the function with batches of messages.
What happens when a Lambda function fails to process an SQS message? The message remains in the queue and becomes visible again after the visibility timeout expires.
How do you report partial batch failures? Return a list of failed message IDs in batchItemFailures from the Lambda function.
What is the difference between standard and FIFO queues? Standard queues offer high throughput but at-least-once delivery. FIFO queues guarantee exactly-once and ordering.
Challenge: Design a multi-step order processing pipeline using multiple SQS queues where each Lambda function handles one step and passes the result to the next queue.

FAQ

What is the maximum SQS batch size for Lambda?

Up to 10,000 messages with a total payload of 6MB. Configure batchSize in the event source mapping.

Can I use SQS with FIFO and Lambda?

Yes. Lambda supports FIFO queues with ordered processing within each message group.

How does Lambda handle SQS throttling?

If Lambda throttles, messages remain in the queue and retry later. Use reserved concurrency to guarantee capacity.

What visibility timeout should I set?

Set visibility timeout to at least 6 times the function timeout to handle retries without reprocessing.

Can I filter SQS messages before Lambda processes them?

No. Lambda receives all messages from the queue. Filtering must happen in your function code.

Mini Project

Create an order processing pipeline with three stages: order intake to SQS, validation Lambda, and fulfillment Lambda. Each stage passes the order to the next queue after processing.

import json

def intake_handler(event, context):
    for record in event["Records"]:
        order = json.loads(record["body"])
        print(f"[Intake] Received order {order['order_id']}")
        print(f"[Intake] Enqueuing for validation")
    return {"batchItemFailures": []}

def validation_handler(event, context):
    for record in event["Records"]:
        order = json.loads(record["body"])
        valid = order.get("amount", 0) > 0 and order.get("email", "").count("@") == 1
        if valid:
            print(f"[Validation] Order {order['order_id']} valid -> enqueue for fulfillment")
        else:
            print(f"[Validation] Order {order['order_id']} INVALID -> moving to DLQ")

def fulfillment_handler(event, context):
    for record in event["Records"]:
        order = json.loads(record["body"])
        print(f"[Fulfillment] Processing order {order['order_id']}")
        print(f"[Fulfillment] Charging ${order['amount']}")

test_order = '{"order_id": "ORD-001", "amount": 49.99, "email": "alice@example.com"}'
intake_handler({"Records": [{"messageId": "1", "receiptHandle": "h", "body": test_order}]}, None)
validation_handler({"Records": [{"messageId": "1", "receiptHandle": "h", "body": test_order}]}, None)
fulfillment_handler({"Records": [{"messageId": "1", "receiptHandle": "h", "body": test_order}]}, None)

What's Next

Next: Lambda + Step Functions for workflow Orchestration.

← Previous Lambda + S3 — Serverless File Processing Next → Lambda + Step Functions — Serverless Workflow Orchestration

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Serverless