Webhook Dead Letter Queue — Complete Guide
In this tutorial, you will learn about Webhook Dead Letter Queue. We cover key concepts, practical examples, and best practices to help you master this topic.
Learn webhook dead letter queues: design DLQ for permanently failed deliveries, inspect failed webhooks, retry manually, alert on dead letter growth, and integrate with monitoring systems.
What You Learn
You will learn how to design and implement dead letter queues for webhook systems, store failed webhooks with error details, provide manual retry capabilities, set up alerts for dead letter accumulation, and integrate with monitoring dashboards.
Why It Matters
Webhooks that exhaust all retry attempts must not be lost. The dead letter queue captures permanent failures so operators can debug, fix, and retry. Without a DLQ, failed webhooks disappear silently, causing data loss and integration failures.
Real-World Use
DodaTech's webhook system dead letters approximately 50 webhooks per day out of 500K deliveries. The DLQ stores the full payload, delivery attempts, and error details. Operations reviews dead letters daily, resolves issues, and retries. This maintains 99.99% delivery rate.
DLQ Architecture
graph TD
Event[Event Trigger] --> Dispatch[Dispatch Service]
Dispatch --> Att1[Attempt 1]
Att1 -->|Success| Done[Done]
Att1 -->|Fail| Att2[Attempt 2 - 1min]
Att2 -->|Success| Done
Att2 -->|Fail| Att3[Attempt 3 - 5min]
Att3 -->|Success| Done
Att3 -->|Fail| Att4[Attempt 4 - 30min]
Att4 -->|Success| Done
Att4 -->|Fail| Att5[Attempt 5 - 2hr]
Att5 -->|Success| Done
Att5 -->|Fail| DLQ[Dead Letter Queue]
DLQ --> Inspect[Inspect & Debug]
Inspect -->|Fix Issue| Retry[Manual Retry]
Retry --> Dispatch
DLQ --> Alert[Alert Team]
After exhausting all retry attempts, the webhook moves to the dead letter queue. The DLQ stores the complete delivery history. Operators inspect, debug, and manually retry.
Dead Letter Queue Implementation
class DeadLetterQueue {
constructor(options = {}) {
this.storage = options.storage; // Database or Redis
this.maxEntries = options.maxEntries || 10000;
this.alertThreshold = options.alertThreshold || 100;
}
async addToDLQ(deliveryRecord) {
const entry = {
id: deliveryRecord.webhookId || `dlq_${Date.now()}`,
subscriberUrl: deliveryRecord.url,
eventType: deliveryRecord.eventType,
payload: deliveryRecord.payload,
headers: deliveryRecord.headers,
attempts: deliveryRecord.attempts,
attemptsDetails: deliveryRecord.attemptsDetails,
lastError: deliveryRecord.lastError,
lastStatusCode: deliveryRecord.lastStatusCode,
failedAt: new Date().toISOString(),
status: 'dead_lettered',
retryCount: 0,
};
await this.storage.store(`dlq:${entry.id}`, entry);
// Check if alert is needed
const count = await this.getDLQCount();
if (count >= this.alertThreshold) {
await this.sendAlert(count);
}
console.log(`Webhook ${entry.id} dead lettered`);
return entry;
}
async getDLQEntry(id) {
return this.storage.get(`dlq:${id}`);
}
async getDLQEntries(options = {}) {
const {
status,
eventType,
limit = 50,
offset = 0,
} = options;
// Filter and paginate
return this.storage.query('dlq:*', { status, eventType, limit, offset });
}
async getDLQCount() {
return this.storage.count('dlq:*');
}
async retryFromDLQ(id) {
const entry = await this.getDLQEntry(id);
if (!entry) throw new Error('DLQ entry not found');
entry.status = 'retrying';
entry.retriedAt = new Date().toISOString();
entry.retryCount++;
await this.storage.store(`dlq:${id}`, entry);
// Re-dispatch the webhook
return this.redeliver(entry);
}
async bulkRetry(eventType) {
const entries = await this.getDLQEntries({ eventType, limit: 1000 });
const results = [];
for (const entry of entries) {
try {
await this.retryFromDLQ(entry.id);
results.push({ id: entry.id, status: 'retried' });
} catch (err) {
results.push({ id: entry.id, status: 'failed', error: err.message });
}
}
return results;
}
async redeliver(entry) {
try {
const response = await fetch(entry.subscriberUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-ID': entry.id,
'X-Webhook-Retry': 'true',
},
body: JSON.stringify(entry.payload),
signal: AbortSignal.timeout(30000),
});
if (response.ok) {
entry.status = 'delivered';
entry.deliveredAt = new Date().toISOString();
await this.storage.store(`dlq:${entry.id}`, entry);
return { success: true };
}
entry.lastError = `HTTP ${response.status}`;
entry.lastStatusCode = response.status;
entry.status = 'dead_lettered';
await this.storage.store(`dlq:${entry.id}`, entry);
return { success: false, status: response.status };
} catch (err) {
entry.lastError = err.message;
entry.status = 'dead_lettered';
await this.storage.store(`dlq:${entry.id}`, entry);
return { success: false, error: err.message };
}
}
async sendAlert(count) {
console.log(`ALERT: DLQ has ${count} entries. Threshold: ${this.alertThreshold}`);
// Send to Slack, PagerDuty, email
}
}
Expected output: DLQ stores each failed webhook with complete delivery history. Operators can view, filter, and retry individual or bulk entries. Alerts fire when DLQ size exceeds threshold.
PostgreSQL DLQ Schema
CREATE TABLE webhook_dead_letter_queue (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
event_id UUID REFERENCES webhook_events(id),
subscriber_id UUID REFERENCES webhook_subscribers(id),
subscriber_url TEXT NOT NULL,
event_type TEXT NOT NULL,
payload JSONB NOT NULL,
headers JSONB,
attempt_details JSONB NOT NULL DEFAULT '[]',
max_attempts INTEGER NOT NULL,
last_error TEXT,
last_status_code INTEGER,
status TEXT NOT NULL DEFAULT 'dead_lettered'
CHECK (status IN ('dead_lettered', 'retrying', 'delivered')),
retry_count INTEGER DEFAULT 0,
failed_at TIMESTAMPTZ DEFAULT NOW(),
last_retry_at TIMESTAMPTZ,
delivered_at TIMESTAMPTZ,
notes TEXT
);
CREATE INDEX idx_dlq_status ON webhook_dead_letter_queue (status);
CREATE INDEX idx_dlq_event_type ON webhook_dead_letter_queue (event_type);
CREATE INDEX idx_dlq_failed_at ON webhook_dead_letter_queue (failed_at DESC);
CREATE INDEX idx_dlq_subscriber ON webhook_dead_letter_queue (subscriber_id);
Expected output: DLQ schema stores complete delivery context. JSONB attempt_details stores each attempt's status code, error, duration, and timestamp. Status tracks lifecycle from dead_lettered through retrying to delivered.
DLQ Dashboard API
const express = require('express');
const router = express.Router();
// List dead letter entries
router.get('/api/dlq', async (req, res) => {
const { status, eventType, limit, offset } = req.query;
const entries = await dlq.getDLQEntries({
status,
eventType,
limit: parseInt(limit) || 50,
offset: parseInt(offset) || 0,
});
const total = await dlq.getDLQCount();
res.json({ entries, total, limit, offset });
});
// Get single entry
router.get('/api/dlq/:id', async (req, res) => {
const entry = await dlq.getDLQEntry(req.params.id);
if (!entry) return res.status(404).json({ error: 'Not found' });
res.json(entry);
});
// Retry single entry
router.post('/api/dlq/:id/retry', async (req, res) => {
try {
const result = await dlq.retryFromDLQ(req.params.id);
res.json(result);
} catch (err) {
res.status(400).json({ error: err.message });
}
});
// Bulk retry by event type
router.post('/api/dlq/bulk-retry', async (req, res) => {
const { eventType } = req.body;
if (!eventType) return res.status(400).json({ error: 'eventType required' });
const results = await dlq.bulkRetry(eventType);
res.json({ total: results.length, results });
});
// DLQ statistics
router.get('/api/dlq/stats', async (req, res) => {
const stats = {
total: await dlq.getDLQCount(),
byEventType: await dlq.groupByEventType(),
bySubscriber: await dlq.groupBySubscriber(),
oldestEntry: await dlq.getOldestEntry(),
retryHistory: await dlq.getRetryHistory(),
};
res.json(stats);
});
Expected output: REST API provides full DLQ management: list entries with filters and pagination, view single entry with delivery history, retry individually or in bulk, and view DLQ statistics.
Automatic Retry from DLQ
class DLQRetryScheduler {
constructor(dlq, options = {}) {
this.dlq = dlq;
this.retryInterval = options.retryInterval || 3600000; // 1 hour
this.maxRetries = options.maxRetries || 3;
this.retryWindowDays = options.retryWindowDays || 7;
this.timer = null;
}
start() {
console.log(`DLQ retry scheduler started (every ${this.retryInterval/3600000}h)`);
this.timer = setInterval(() => this.runRetryCycle(), this.retryInterval);
}
stop() {
if (this.timer) {
clearInterval(this.timer);
this.timer = null;
}
}
async runRetryCycle() {
console.log('Running DLQ retry cycle...');
const cutoff = new Date(
Date.now() - this.retryWindowDays * 86400000
);
const entries = await this.dlq.getDLQEntries({
status: 'dead_lettered',
limit: 100,
});
let retried = 0;
let succeeded = 0;
for (const entry of entries) {
if (entry.retryCount >= this.maxRetries) {
console.log(`Skipping ${entry.id}: max retries reached`);
continue;
}
if (new Date(entry.failedAt) < cutoff) {
console.log(`Skipping ${entry.id}: outside retry window`);
continue;
}
const result = await this.dlq.retryFromDLQ(entry.id);
retried++;
if (result.success) succeeded++;
}
console.log(`DLQ retry cycle: ${retried} retried, ${succeeded} succeeded`);
}
}
Expected output: Scheduler runs periodic retry cycles, retrying dead lettered webhooks within the retry window. Webhooks beyond max retries or retry window are skipped. Success/failure is logged.
Common Mistakes
1. No Dead Letter Queue
Without a DLQ, webhooks that fail all retries are permanently lost. You cannot investigate, retry, or audit them. Always implement a DLQ for production webhook systems.
2. DLQ Without Context
Storing only the payload without delivery history makes debugging impossible. Store: all attempt details (status, error, duration, timestamp), subscriber info, and request headers.
3. No DLQ Monitoring
Dead letters accumulate silently. Set up monitoring: alert when DLQ count exceeds threshold, track DLQ growth rate, report DLQ age distribution. Investigate persistent failures promptly.
4. Manual Retry Without Rate Limiting
Operators retrying 1000 DLQ entries at once overwhelm the consumer. Implement bulk retry with concurrency limits (10-50 concurrent retries). Add delays between retry batches.
5. No DLQ Cleanup
DLQ grows unbounded without cleanup. Archive resolved entries. Delete entries older than retention period. Keep only entries within the retry window for automatic retry. Archive the rest.
Practice Questions
1. When does a webhook enter the dead letter queue?
After exhausting all retry attempts (typically 5-10 retries over 1-24 hours). The webhook is permanently failed and moved to DLQ for manual inspection and retry.
2. What information should be stored in the dead letter queue?
Full payload, subscriber URL, event type, delivery attempt details (status, error, duration per attempt), headers, failure timestamp, and retry count. This enables complete debugging.
3. How do you retry webhooks from the dead letter queue?
Manual retry via API or dashboard. Automatic retry via scheduled job (every 1-6 hours). Bulk retry by event type or subscriber. Rate-limit retries to avoid overwhelming the consumer.
4. Why should you alert on dead letter queue growth?
DLQ growth indicates systemic issues. A subscriber may have changed their URL. A provider may have changed payload format. A code bug may reject valid webhooks. Early alerting prevents data loss.
Challenge
Build a complete DLQ system: PostgreSQL-backed storage with delivery history, REST API for list/view/retry/bulk-retry, automatic retry scheduler (every hour, max 3 retries), alerting when DLQ exceeds 50 entries, and a dashboard UI showing DLQ status, statistics, and retry controls.
FAQ
Mini Project: DLQ Management Dashboard
Build a dashboard for dead letter queue management: list view with filters (status, event type, date range), entry detail with full delivery history and attempt timeline, retry button (single and bulk), automatic retry scheduler configuration, DLQ statistics with charts, and alert configuration for DLQ thresholds.
What's Next
Now that you can manage dead letter queues, learn about Rate Limiting for controlling webhook delivery throughput.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro