Cloud Message Queues — SQS, Azure Queue Storage & Pub/Sub Guide

In this tutorial, you'll learn about Cloud Message Queues. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Cloud Message Queues decouple application components by passing messages between producers and consumers asynchronously, enabling resilient, scalable Distributed Systems that tolerate failures gracefully.

What You'll Learn

You'll learn how Message Queues work, how to configure SQS, Queue Storage, and Pub/Sub, and how to implement patterns like competing consumers, dead-letter queues, and fan-out messaging in real applications.

Why It Matters

Direct coupling between services creates fragile systems. When one service slows down or fails, the failure cascades. Queues buffer the load, smooth traffic spikes, and let each component operate at its own pace. DodaZIP uses queues to distribute compression jobs across worker pools.

Real-World Use

An e-commerce platform processes 10,000 orders per minute during Black Friday. Orders go into a queue, inventory service picks them up as capacity allows, and the order service never blocks — even when downstream systems lag.

Queue Architecture

flowchart LR
  A[Order Service] --> B[Message Queue]
  B --> C[Worker 1]
  B --> D[Worker 2]
  B --> E[Worker 3]
  C --> F[Inventory DB]
  D --> F
  E --> F
  B --> G[Dead Letter Queue]
  F --> H[Notification Service]
  style B fill:#48f,color:#fff
  style G fill:#f44,color:#fff

AWS SQS

SQS offers standard queues (high throughput, at-least-once) and FIFO queues (exactly-once, ordered).

# Create a standard queue with dead-letter configuration
aws sqs create-queue \
  --queue-name order-processing \
  --attributes \
    DelaySeconds=0,\
    MaximumMessageSize=262144,\
    MessageRetentionPeriod=345600,\
    VisibilityTimeout=60

# Create a dead-letter queue
DLQ_URL=$(aws sqs create-queue \
  --queue-name order-processing-dlq \
  --query "QueueUrl" \
  --output text)

# Configure redrive policy
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/order-processing \
  --attributes \
    '{"RedrivePolicy": "{\"maxReceiveCount\":\"3\", \"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:order-processing-dlq\"}"}'

Azure Queue Storage

Azure Queue Storage integrates natively with Azure Storage accounts.

# Create a storage account and queue
az storage account create \
  --name dodatechqueues \
  --resource-group my-rg \
  --location eastus \
  --sku Standard_LRS

az storage queue create \
  --name order-processing \
  --account-name dodatechqueues

# Send a message
az storage message put \
  --queue-name order-processing \
  --content '{"orderId": "12345", "amount": 99.99}' \
  --account-name dodatechqueues

GCP Pub/Sub

Pub/Sub provides global message ingestion with push and pull subscriptions.

# Create a topic and subscription
gcloud pubsub topics create order-processing

gcloud pubsub subscriptions create order-processing-sub \
  --topic order-processing \
  --ack-deadline 60 \
  --message-retention-duration 7d

# Publish a message
gcloud pubsub topics publish order-processing \
  --message '{"orderId": "12345", "amount": 99.99}' \
  --attribute "source=web,priority=high"

Processing Messages with Code

import json
import boto3

sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/order-processing"

def process_orders():
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20
        )

        if "Messages" not in response:
            continue

        for msg in response["Messages"]:
            try:
                order = json.loads(msg["Body"])
                process_order(order)
                sqs.delete_message(
                    QueueUrl=QUEUE_URL,
                    ReceiptHandle=msg["ReceiptHandle"]
                )
            except Exception as e:
                print(f"Failed to process order: {e}")

def process_order(order):
    print(f"Processing order {order['orderId']} for ${order['amount']}")

Common Errors

1. Messages exceed size limits — SQS max is 256KB. Use S3 with a message reference for larger payloads. Pub/Sub allows up to 10MB.
2. Not handling poison pills — A malformed message causes infinite retries. Use dead-letter queues to isolate failing messages after 3-5 retries.
3. Visibility timeout too short — If a worker crashes during processing, the message reappears after timeout. For 30s jobs, set visibility timeout to at least 60s.
4. Using FIFO when standard suffices — FIFO limits throughput to 3000 msg/s. Standard queues handle virtually unlimited throughput.
5. No monitoring on queue depth — A growing queue means workers are falling behind. Set CloudWatch alarms on ApproximateNumberOfMessagesVisible.

Practice Questions

1. What is the difference between at-least-once and exactly-once delivery? At-least-once may deliver duplicates (standard SQS). Exactly-once prevents duplicates but reduces throughput (FIFO SQS, Pub/Sub).
2. What is a dead-letter queue and when should you use one? A DLQ captures messages that fail processing after a set number of retries. Use it to isolate and debug persistent failures.
3. How does Azure Queue Storage differ from Service Bus? Queue Storage is simpler and cheaper for basic queuing. Service Bus supports topics, sessions, and transactions for enterprise messaging.
4. What is backpressure and how do queues help? Backpressure is when a slow consumer blocks the producer. Queues decouple them so the producer never waits for the consumer.
5. Challenge: Design a queuing system for a video processing pipeline that accepts uploads, transcodes to 3 formats, and notifies users. Handle failures gracefully and ensure no video is lost.

Mini Project

Build a distributed order processing system:

• Orders submitted via API go to a queue
• Three worker services Process: inventory check, payment, shipping
• Failed orders go to a dead-letter queue for manual review
• Metrics: queue depth, processing time, failure rate

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.