Webhook Retry Policy
DodaTech
6 min read
title: "Webhook Retry Policy" description: "Learn how to design webhook retry policies including exponential backoff, retry limits, queue management, and permanent failure handling." weight: 16 date: 2026-06-28 lastmod: 2026-06-28 tags: ["apis", "webhooks"]
Retry policies determine how webhook providers handle delivery failures. A well-designed retry policy balances reliability with resource usage, ensuring events are delivered without overwhelming consumers.
## What You'll Learn
- Retry triggers and conditions
- Exponential backoff strategies
- Retry limits and permanent failures
- Consumer-side retry handling
- Dead letter queues
## Why It Matters
Without a proper retry policy, temporary failures cause permanent data loss. With too aggressive retries, you risk overwhelming an already struggling consumer.
## Real-World Use
A cloud infrastructure provider uses a tiered retry policy: 5 attempts with exponential backoff (1m, 5m, 15m, 30m, 1h), then 24 hours of hourly retries, then permanent failure notification via email.
## Flow Chart
```mermaid
flowchart TD
A[Delivery Attempt] --> B{Success?}
B -->|Yes| C[Delivery Complete]
B -->|No| D{Retry?}
D -->|Yes| E[Wait: Backoff]
E --> F[Increment Attempt]
F --> A
D -->|No| G[Permanent Failure]
G --> H[Dead Letter Queue]
G --> I[Notify Admin]
Code Examples
Example 1: Exponential Backoff Implementation
class WebhookRetryService {
constructor(options = {}) {
this.maxRetries = options.maxRetries || 5;
this.baseDelay = options.baseDelay || 1000;
this.maxDelay = options.maxDelay || 3600000; // 1 hour
this.deliveries = new Map();
}
async deliver(webhook, event) {
const deliveryKey = `${webhook.id}:${event.id}`;
let attempt = 0;
while (attempt < this.maxRetries) {
attempt++;
const delivery = {
webhookId: webhook.id,
eventId: event.id,
attempt,
url: webhook.url,
timestamp: new Date().toISOString(),
};
try {
const response = await this.sendRequest(webhook.url, event);
delivery.status = 'success';
delivery.statusCode = response.status;
this.recordDelivery(deliveryKey, delivery);
return { success: true, delivery };
} catch (error) {
delivery.status = 'failed';
delivery.error = error.message;
this.recordDelivery(deliveryKey, delivery);
if (this.shouldRetry(error, attempt)) {
const delay = this.calculateBackoff(attempt);
console.log(
`Retry ${attempt}/${this.maxRetries} for ${event.id}: ` +
`waiting ${delay}ms`
);
await this.sleep(delay);
} else {
break;
}
}
}
return {
success: false,
finalAttempt: attempt,
};
}
calculateBackoff(attempt) {
// Exponential backoff with jitter
const exponentialDelay = Math.min(
this.baseDelay * Math.pow(2, attempt - 1),
this.maxDelay
);
// Add 0-1000ms random jitter
const jitter = Math.random() * 1000;
return Math.floor(exponentialDelay + jitter);
}
shouldRetry(error, attempt) {
// Don't retry client errors (4xx)
if (error.response && error.response.status >= 400 &&
error.response.status < 500 &&
error.response.status !== 429) {
return false;
}
// Don't exceed max retries
if (attempt >= this.maxRetries) {
return false;
}
return true;
}
async sendRequest(url, event) {
const response = await axios.post(url, event.payload, {
headers: event.headers,
timeout: 15000,
});
return response;
}
recordDelivery(key, delivery) {
const deliveries = this.deliveries.get(key) || [];
deliveries.push(delivery);
this.deliveries.set(key, deliveries);
}
sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
}
Expected output: Retry service with exponential backoff, jitter, and intelligent retry decision (skipping 4xx errors).
Example 2: Consumer Retry Handling
from flask import Flask, request, jsonify
import time
import logging
app = Flask(__name__)
logger = logging.getLogger(__name__)
# Rate limiting state
rate_limits = {}
@app.route('/webhook', methods=['POST'])
def handle_webhook():
# Check rate limit headers
retry_after = request.headers.get('Retry-After')
if retry_after:
# Consumer is overloaded, tell provider to slow down
return jsonify({
"error": "rate_limited",
"retry_after": 30
}), 429
# Process webhook
try:
result = process_event(request.json)
return jsonify({"status": "ok"}), 200
except TemporaryError as e:
# Temporary failure, provider should retry
logger.warning(f"Temporary error: {e}")
return jsonify({
"error": "temporary_failure",
"retry_after": 10
}), 500
except PermanentError as e:
# Permanent failure, provider should not retry
logger.error(f"Permanent error: {e}")
return jsonify({
"error": "permanent_failure",
"message": str(e)
}), 422
# Provider respects consumer response
def should_retry(response):
status = response.status_code
body = response.json()
# Never retry 2xx
if 200 <= status < 300:
return False
# Retry on 429 with respect to Retry-After
if status == 429:
return True, body.get('retry_after', 30)
# Retry on 5xx (temporary errors)
if status >= 500:
return True, body.get('retry_after', 10)
# Don't retry 4xx (client errors, except 429)
if 400 <= status < 500:
return False, None
return True, 60
Expected output: Consumer returns specific HTTP status codes and Retry-After headers to guide provider retry behavior.
Example 3: Dead Letter Queue
class DeadLetterQueue {
constructor() {
this.queue = [];
this.maxRetries = 10;
this.bucket = 'webhook-failures';
}
async handlePermanentFailure(webhook, event, deliveries) {
const failure = {
webhookId: webhook.id,
eventId: event.id,
webhookUrl: webhook.url,
eventType: event.type,
payload: event.payload,
lastAttempt: new Date().toISOString(),
attempts: deliveries.length,
deliveryLog: deliveries,
failureReason: deliveries[deliveries.length - 1]?.error,
};
// Store for later inspection
this.queue.push(failure);
await this.storeInDatabase(failure);
await this.uploadToStorage(failure);
// Notify administrators
await this.sendAlert(failure);
console.log(
`Webhook ${event.id} moved to dead letter queue. ` +
`Failed after ${deliveries.length} attempts.`
);
}
async retryFromDLQ(failureId) {
const failure = this.queue.find(f => f.eventId === failureId);
if (!failure) throw new Error('Failure not found');
// Attempt reprocessing
try {
const response = await axios.post(
failure.webhookUrl,
failure.payload,
{ timeout: 15000 }
);
if (response.status >= 200 && response.status < 300) {
// Success! Remove from DLQ
this.queue = this.queue.filter(
f => f.eventId !== failureId
);
await this.removeFromDatabase(failureId);
return { success: true };
}
} catch (error) {
return { success: false, error: error.message };
}
}
async retryAllFromDLQ() {
const results = [];
for (const failure of [...this.queue]) {
const result = await this.retryFromDLQ(failure.eventId);
results.push({ eventId: failure.eventId, ...result });
}
return results;
}
getFailedDeliveries() {
return this.queue.map(f => ({
eventId: f.eventId,
webhookUrl: f.webhookUrl,
eventType: f.eventType,
attempts: f.attempts,
lastAttempt: f.lastAttempt,
failureReason: f.failureReason,
}));
}
}
Expected output: Dead letter queue captures permanently failed webhooks, stores them for inspection, and supports manual or automatic reprocessing.
Common Mistakes
| Mistake | Explanation |
|---|---|
| Retrying on 4xx errors | Client errors (400, 404, 422) indicate permanent issues; retrying is wasted effort |
| Not using jitter | Without jitter, retries from multiple events sync up, creating thundering herd |
| Infinite retries | Always set a maximum retry limit to prevent runaway resource consumption |
| Ignoring Retry-After header | Consumers use Retry-After to signal overload; ignoring it makes things worse |
| Not logging delivery attempts | Without logs, you cannot diagnose delivery issues or measure reliability |
Practice Questions
- What HTTP status codes should trigger a retry?
- How does exponential backoff work?
- What is the purpose of jitter in retry timing?
- What is a dead letter queue and when is it used?
- How do consumers signal the provider to slow down?
Challenge
Build a webhook delivery system with a configurable retry policy: support for exponential backoff with jitter, configurable retry limits, intelligent retry decisions (skip 4xx, retry 5xx), dead letter queue with dashboard, and manual reprocessing.
FAQ
Mini Project
Build a webhook retry simulator that sends webhooks to a configurable endpoint, implements configurable retry policies, displays delivery status in real-time, and allows testing different failure scenarios with a visual retry timeline.
What's Next
Learn about webhook idempotency
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro