Webhook Retry Policy — Complete Guide
In this tutorial, you will learn about Webhook Retry Policy. We cover key concepts, practical examples, and best practices to help you master this topic.
Learn webhook retry policies: exponential backoff, retry schedules, idempotency, maximum retries, dead letter queues, and designing reliable webhook delivery for production systems.
What You Learn
You will learn how to design retry policies for webhook delivery, implement exponential backoff with jitter, configure maximum retry limits, handle success and failure scenarios, and design systems that recover from extended consumer downtime.
Why It Matters
Network failures, server crashes, and maintenance Windows cause webhook delivery failures. A well-designed retry policy recovers automatically from transient failures, prevents duplicate processing, and alerts operators about permanent failures.
Real-World Use
DodaTech's webhook dispatch system delivers 99.95% of Webhooks on first attempt. The remaining 0.05% are recovered through a 5-stage retry policy spanning 9 hours, with 95% of retries succeeding on the second or third attempt.
Retry Strategy
graph TD
Attempt1[Attempt 1: Immediate] -->|Fail| Wait1[Wait 1 minute]
Wait1 --> Attempt2[Attempt 2]
Attempt2 -->|Fail| Wait2[Wait 5 minutes]
Wait2 --> Attempt3[Attempt 3]
Attempt3 -->|Fail| Wait3[Wait 30 minutes]
Wait3 --> Attempt4[Attempt 4]
Attempt4 -->|Fail| Wait4[Wait 2 hours]
Wait4 --> Attempt5[Attempt 5]
Attempt5 -->|Fail| Wait5[Wait 6 hours]
Wait5 --> Attempt6[Attempt 6: Final]
Attempt6 -->|Fail| Dead[Dead Letter Queue]
Any Attempt -->|2xx Success| Done[Done]
Typical retry schedules follow increasing intervals: 1 minute, 5 minutes, 30 minutes, 2 hours, 6 hours. This balances quick recovery from transient failures with reasonable total retry window.
Implementing Retry with Backoff
class WebhookRetrier {
constructor(options = {}) {
this.maxAttempts = options.maxAttempts || 10;
this.initialDelayMs = options.initialDelayMs || 1000;
this.backoffFactor = options.backoffFactor || 2;
this.maxDelayMs = options.maxDelayMs || 3600000; // 1 hour
this.jitter = options.jitter || 0.1; // 10% jitter
}
getDelay(attempt) {
const delay = this.initialDelayMs *
Math.pow(this.backoffFactor, attempt - 1);
const capped = Math.min(delay, this.maxDelayMs);
const jitterAmount = capped * this.jitter *
(Math.random() * 2 - 1);
return Math.round(capped + jitterAmount);
}
async deliverWithRetry(url, payload, headers = {}) {
for (let attempt = 1; attempt <= this.maxAttempts; attempt++) {
const result = await this.deliver(url, payload, headers);
console.log(`Attempt ${attempt}/${this.maxAttempts}: ${result.status}`);
if (result.success) {
return { delivered: true, attempt };
}
// Don't retry client errors (4xx except 429)
if (result.status >= 400 && result.status < 500 &&
result.status !== 429) {
console.log(`Client error ${result.status}, not retrying`);
return { delivered: false, attempt, status: result.status };
}
if (attempt < this.maxAttempts) {
const delay = this.getDelay(attempt);
console.log(`Waiting ${delay}ms before retry ${attempt + 1}`);
await this.sleep(delay);
}
}
return { delivered: false, attempt: this.maxAttempts };
}
async deliver(url, payload, headers) {
try {
const response = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
...headers,
},
body: JSON.stringify(payload),
signal: AbortSignal.timeout(30000),
});
return {
success: response.ok,
status: response.status,
statusText: response.statusText,
};
} catch (err) {
return {
success: false,
status: 0,
error: err.message,
};
}
}
sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
}
// Test the retrier
const retrier = new WebhookRetrier({
maxAttempts: 5,
initialDelayMs: 1000,
backoffFactor: 3,
maxDelayMs: 300000,
});
const delays = [];
for (let i = 1; i <= 5; i++) {
delays.push(retrier.getDelay(i));
}
console.log('Retry delays (ms):', delays);
Expected output: Retry delays with 3x backoff and 10% jitter: approximately 1000ms, 3000ms, 9000ms, 27000ms, 81000ms. Jitter adds random variation to avoid thundering herd.
Handling Different HTTP Status Codes
function classifyResponse(status) {
if (status >= 200 && status < 300) {
return 'success';
}
if (status === 429) {
return 'rate_limited'; // Retry after backoff
}
if (status >= 400 && status < 500) {
return 'client_error'; // Don't retry
}
if (status >= 500) {
return 'server_error'; // Retry
}
return 'network_error'; // Retry
}
// Retry decision logic
function shouldRetry(status, attempt, maxAttempts) {
const category = classifyResponse(status);
switch (category) {
case 'success':
return false;
case 'client_error':
return false; // Bad request, not recoverable
case 'rate_limited':
return attempt < maxAttempts; // Retry with longer delay
case 'server_error':
return attempt < maxAttempts; // Server may recover
case 'network_error':
return attempt < maxAttempts; // Transient
default:
return false;
}
}
Expected output: 2xx stops retrying. 4xx (except 429) stops retrying (consumer bug). 429 and 5xx retry. Network errors retry. This prevents hammering a consumer that is rejecting the payload.
Rate Limit Respect
// Consumer rate limit handling
class RateLimitedRetrier {
constructor() {
this.retryAfter = 60; // Default 60 seconds
}
async deliverWithRateLimit(url, payload) {
for (let attempt = 1; attempt <= 10; attempt++) {
const response = await fetch(url, {
method: 'POST',
body: JSON.stringify(payload),
headers: { 'Content-Type': 'application/json' },
});
if (response.ok) {
return { success: true };
}
if (response.status === 429) {
// Use Retry-After header if present
const retryAfter = response.headers.get('Retry-After');
const delaySec = retryAfter
? parseInt(retryAfter)
: this.retryAfter * Math.pow(2, attempt - 1);
console.log(`Rate limited. Waiting ${delaySec}s`);
await this.sleep(delaySec * 1000);
continue;
}
// Non-retryable error
return { success: false, status: response.status };
}
return { success: false, error: 'Max retries exceeded' };
}
}
Expected output: When consumer returns 429, the provider respects the Retry-After header. If no header, it uses exponential backoff starting at 60 seconds.
Failed Webhook Recovery
// Periodic recovery of failed webhooks
class WebhookRecovery {
constructor(storage) {
this.storage = storage; // Database or in-memory store
this.recoveryInterval = 3600000; // 1 hour
}
startRecoveryCycle() {
setInterval(async () => {
const failed = await this.storage.getFailedWebhooks();
for (const webhook of failed) {
const result = await this.retryWebhook(webhook);
if (result.success) {
console.log(`Recovered webhook ${webhook.id}`);
await this.storage.markDelivered(webhook.id);
} else {
console.log(`Still failing: ${webhook.id}`);
webhook.attempts++;
await this.storage.updateAttempts(webhook.id, webhook.attempts);
}
}
}, this.recoveryInterval);
}
async retryWebhook(webhook) {
try {
const response = await fetch(webhook.url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(webhook.payload),
});
return { success: response.ok };
} catch {
return { success: false };
}
}
}
Expected output: Every hour, the system retries all failed webhooks. Successful retries are marked delivered. Persistent failures continue to be retried on the next cycle.
Common Mistakes
1. Retrying Too Aggressively
Retrying every second overwhelms the failing consumer and your own infrastructure. Use exponential backoff: 1s, 2s, 4s, 8s, 16s minimum. Add jitter to spread retries.
2. Not Differentiating Error Types
Network errors, 5xx, and 4xx have different recovery characteristics. 4xx (except 429) should not be retried. 5xx should be retried. 429 should be retried with longer delays.
3. Unlimited Retries
Without a maximum retry limit, you retry forever against a permanently dead consumer. Set a maximum (5-10 attempts) or a time window (24 hours max). Dead letter after exhaustion.
4. No Dead Letter Queue
Permanent failures are lost without a dead letter queue. Store failed webhooks with error details. Provide a dashboard to inspect and manually retry. Alert on dead letter queue growth.
5. Not Tracking Retry Metrics
Without metrics, you cannot tune your retry policy. Track: first-attempt success rate, average retry count, time-to-delivery for retried webhooks, dead letter rate. These inform policy adjustments.
Practice Questions
1. What is exponential backoff in webhook retries?
Each retry waits exponentially longer than the previous: 1s, 2s, 4s, 8s, 16s. This prevents overwhelming the consumer and spreads retry load over time.
2. Why add jitter to retry delays?
Without jitter, all retries happen simultaneously (thundering herd). Jitter randomizes delays so retries are spread evenly, preventing bursts on the consumer.
3. Which HTTP status codes should trigger a retry?
429 (rate limited), 5xx (server errors), and network errors (timeout, connection refused). 4xx (except 429) should not be retried as they indicate consumer-side issues.
4. What is the purpose of a dead letter queue in webhooks?
It stores webhooks that failed all retry attempts. Operators inspect them, debug the issue, and manually retry. It prevents permanent data loss and provides audit trail.
Challenge
Design a retry policy for a critical payment webhook system. Requirements: deliver within 5 minutes for 99.9% of webhooks, retry up to 24 hours, never lose a webhook, alert if any webhook remains undelivered for 1 hour, and respect consumer rate limits.
FAQ
Mini Project: Retry Policy Simulator
Build a simulator that models webhook delivery with configurable retry policies. Track: first-attempt success rate, average delivery time, retry distribution, dead letter rate. Compare different backoff strategies and retry counts to find optimal configuration.
What's Next
Now that you understand retry policies, learn about Idempotency to prevent duplicate processing during retries.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro