Webhook Dead Letter Queues — Complete Guide
In this tutorial, you will learn about Webhook Dead Letter Queues. We cover key concepts, practical examples, and best practices to help you master this topic.
A dead letter queue (DLQ) stores webhook events that cannot be delivered after exhausting all retry attempts. Instead of losing these events, a DLQ preserves them for manual inspection, delayed replay, or alternative processing. This lesson covers DLQ design, implementation, and operational practices for webhook systems.
What You'll Learn
- Design a dead letter queue for undeliverable webhook events
- Implement DLQ storage with metadata for debugging
- Build a manual replay and automatic reprocessing system
- Set up alerting and monitoring for DLQ events
Why It Matters
Without a DLQ, failed webhook events are silently lost. This leads to data inconsistencies, missed business operations, and difficult debugging. A DLQ provides a safety net that preserves every event, gives operators visibility into delivery failures, and prevents data loss in production systems.
Real-World Use
- AWS SQS provides built-in dead letter queues with configurable max receive counts
- Stripe stores failed webhook deliveries for up to 30 days with manual replay in the dashboard
- GitHub shows failed delivery attempts and allows manual re-delivery from the settings page
- Enterprise message brokers (RabbitMQ, ActiveMQ) support DLX (dead letter exchange) patterns
Mermaid Flow
graph TD
A[Event Received] --> B[Delivery Attempt 1]
B -->|Fail| C[Retry with Backoff]
C --> D[Delivery Attempt N]
D -->|Fail| E{Max Retries?}
E -->|Yes| F[Move to DLQ]
E -->|No| C
F --> G[DLQ Storage]
G --> H[Operator Alert]
H --> I[Investigate Failure]
I --> J{Can Replay?}
J -->|Yes| K[Re-queue for Delivery]
J -->|No| L[Archive Manually]
K --> M[Successful Delivery]
Teacher's Corner
Explain that a DLQ is not just a storage bucket; it is a managed queue with its own monitoring and lifecycle. Compare to postal mail dead letter offices. Emphasize that DLQ events should include full metadata: event payload, consumer URL, error details, attempt history, and timestamps.
Code Examples
Example 1: Basic Dead Letter Queue in Python
import json
import sqlite3
from datetime import datetime, timedelta
class DeadLetterQueue:
def __init__(self, db_path="dlq.db"):
self.conn = sqlite3.connect(db_path)
self.conn.execute("""
CREATE TABLE IF NOT EXISTS dlq_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
event_id TEXT NOT NULL,
subscriber_url TEXT NOT NULL,
payload TEXT NOT NULL,
error_message TEXT,
attempt_count INTEGER,
last_attempt_at TEXT,
created_at TEXT NOT NULL,
status TEXT DEFAULT 'pending'
)
""")
self.conn.commit()
def add(self, event_id, subscriber_url, payload, error, attempts):
self.conn.execute("""
INSERT INTO dlq_events
(event_id, subscriber_url, payload, error_message,
attempt_count, last_attempt_at, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (event_id, subscriber_url, json.dumps(payload),
str(error), attempts, datetime.utcnow().isoformat(),
datetime.utcnow().isoformat()))
self.conn.commit()
def list_pending(self):
cur = self.conn.execute(
"SELECT * FROM dlq_events WHERE status = 'pending'"
)
return cur.fetchall()
def replay(self, event_id):
self.conn.execute(
"UPDATE dlq_events SET status = 'replaying' WHERE event_id = ?",
(event_id,)
)
self.conn.commit()
dlq = DeadLetterQueue()
dlq.add("evt-001", "https://example.com/hooks",
{"order_id": 123}, "Connection timeout", 5)
print(dlq.list_pending())
Expected Output: A list of one DLQ event with all metadata fields populated.
Example 2: Automatic DLQ with Delivery Manager
import time
import requests
from datetime import datetime
class DeliveryManager:
def __init__(self, max_retries=5, base_delay=10):
self.max_retries = max_retries
self.base_delay = base_delay
self.queue = []
self.dlq = DeadLetterQueue()
def enqueue(self, event_id, url, payload):
task = {
"event_id": event_id,
"url": url,
"payload": payload,
"attempts": 0,
"errors": []
}
self.queue.append(task)
def process(self):
for task in self.queue[:]:
try:
resp = requests.post(
task["url"],
json=task["payload"],
timeout=10
)
if resp.status_code // 100 == 2:
self.queue.remove(task)
print(f"Delivered {task['event_id']}")
else:
task["errors"].append(f"HTTP {resp.status_code}")
self._handle_failure(task)
except Exception as e:
task["errors"].append(str(e))
self._handle_failure(task)
def _handle_failure(self, task):
task["attempts"] += 1
if task["attempts"] >= self.max_retries:
self.dlq.add(
task["event_id"],
task["url"],
task["payload"],
"; ".join(task["errors"]),
task["attempts"]
)
self.queue.remove(task)
print(f"DLQ: {task['event_id']}")
else:
delay = self.base_delay * (2 ** (task["attempts"] - 1))
print(f"Retry {task['event_id']} in {delay}s")
manager = DeliveryManager(max_retries=3, base_delay=1)
manager.enqueue("evt-1", "https://httpbin.org/status/500",
{"test": True})
for _ in range(5):
manager.process()
time.sleep(0.5)
Expected Output: Event delivery fails three times, then moves to DLQ on the third failure.
Example 3: DLQ Monitoring and Alerting
import smtplib
import json
from collections import Counter
from datetime import datetime, timedelta
class DLQMonitor:
def __init__(self, dlq, alert_threshold=10, alert_window_hours=1):
self.dlq = dlq
self.alert_threshold = alert_threshold
self.alert_window_hours = alert_window_hours
def check_and_alert(self):
events = self.dlq.list_pending()
recent = [
e for e in events
if datetime.fromisoformat(e[6]) > datetime.utcnow() - timedelta(hours=self.alert_window_hours)
]
if len(recent) >= self.alert_threshold:
urls = Counter(e[2] for e in recent)
top_url = urls.most_common(1)[0]
self.send_alert(
f"DLQ alert: {len(recent)} events in {self.alert_window_hours}h. "
f"Top failing URL: {top_url[0]} ({top_url[1]} events)"
)
def send_alert(self, message):
print(f"ALERT: {message}")
monitor = DLQMonitor(dlq, alert_threshold=1)
monitor.check_and_alert()
Expected Output: ALERT: DLQ alert: 1 events in 1h. Top failing URL: https://example.com/hooks (1 events)
Common Mistakes
- Not implementing a DLQ at all, losing undeliverable events permanently
- Setting unlimited retries without a DLQ cutoff, causing infinite delivery attempts
- Storing DLQ events without subscriber URL context, making replay impossible
- Not alerting on DLQ events, letting failures go unnoticed for days
- Keeping DLQ events indefinitely without a retention or archival policy
- Not distinguishing between transient failures (should retry) and permanent failures (should not)
- Allowing automatic replay from DLQ without manual review for certain error types
Practice Questions
- What is the purpose of a dead letter queue in a webhook system?
- How does a DLQ differ from simply logging failed events?
- What metadata should be stored with each DLQ event?
- When should DLQ events be automatically replayed vs. manually reviewed?
- Challenge: Design a dead letter queue system that supports multiple tiers: a hot DLQ (accessible within seconds for replay), a warm DLQ (accessible within hours for replay), and a cold DLQ (archived after 7 days). Implement automatic promotion from hot to warm based on event age, and alert if any tier exceeds configurable thresholds.
Answer Key
1. A DLQ preserves events that cannot be delivered after exhausting retries, providing a safety net for data integrity and enabling manual or automated replay. 2. Logging provides visibility but no action. A DLQ is a managed queue with replay capability, retention policies, and alerting. 3. Event ID, subscriber URL, full payload, error messages, attempt count, timestamps of each attempt, and last failure reason. Include consumer response body if available. 4. Transient errors (network timeouts, 5xx) can often be auto-replayed after a cooldown. Permanent errors (invalid payload, 4xx authentication failures) need manual review. 5. Use a database table with a `tier` column and `created_at` timestamp. A scheduled job promotes events: hot (0-1h), warm (1-24h), cold (24h-7d). Archive to S3 after 7d. Alerts per tier threshold. Use database Partitioning by tier for query performance.FAQ
Mini Project
Build a complete DLQ system with a web interface. Create a Python Flask application that: (1) accepts webhook events and attempts delivery with configurable retry, (2) moves failed events to a DLQ table in SQLite, (3) provides /dlq endpoint listing all DLQ events with search and filter, (4) provides /dlq/<id>/replay to re-queue a single event, (5) provides /dlq/replay-all to re-queue all pending events, and (6) implements a scheduled check that alerts when DLQ count exceeds threshold.
What's Next
Now that you have a DLQ, learn how to protect your webhook system with webhook security best practices and prevent common attack vectors.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro