Retry Strategies Project — Build a Complete Retry System
In this tutorial, you will learn about Retry Strategies Project. We cover key concepts, practical examples, and best practices to help you master this topic.
This retry strategies project guides you through building a production-ready retry system that combines exponential backoff, jitter, circuit breakers, idempotency, and monitoring into a single resilient HTTP client.
What You'll Learn
By completing this project, you will integrate all retry concepts into a working system, handling real-world failures like network timeouts, server errors, and Rate Limiting.
Why It Matters
Individual retry tutorials teach isolated concepts. This project shows how they compose into a complete system, just like DodaTech's production resilience layer.
Real-World Use
The system mirrors DodaTech's internal HTTP client: it handles transient failures transparently, opens circuit breakers for persistent failures, provides fallback responses, and monitors retry metrics.
Project Learning Path
flowchart LR
A[Advanced Retry] --> B[Retry Project]
B --> C[Complete Client]
C --> D[Production Ready]
B --> E{You Are Here}
style E fill:#f90,color:#fff
Project Overview
Build a resilient HTTP client with:
- Exponential backoff with full jitter
- Circuit breaker for persistent failures
- Idempotency key support for POST requests
- Retry-After header respect
- Metrics and monitoring
- Fallback responses
- Configurable retry policies
Step 1: Setup
const express = require("express");
const crypto = require("crypto");
// Metrics
const metrics = {
totalRequests: 0,
retries: 0,
circuitBreakerOpens: 0,
fallbacks: 0,
successes: 0,
failures: 0
};
Step 2: Circuit Breaker
class CircuitBreaker {
constructor(options = {}) {
this.threshold = options.threshold || 5;
this.resetTimeout = options.resetTimeout || 30000;
this.failures = 0;
this.state = "closed";
this.nextAttempt = Date.now();
}
async call(fn) {
if (this.state === "open") {
if (Date.now() >= this.nextAttempt) {
this.state = "half-open";
} else {
throw new Error("circuit_open");
}
}
try {
const result = await fn();
this.failures = 0;
this.state = "closed";
return result;
} catch (err) {
this.failures++;
if (this.failures >= this.threshold || this.state === "half-open") {
this.state = "open";
this.nextAttempt = Date.now() + this.resetTimeout;
metrics.circuitBreakerOpens++;
}
throw err;
}
}
}
Step 3: Retry with Backoff
class RetryHandler {
constructor(options = {}) {
this.maxRetries = options.maxRetries || 3;
this.baseDelay = options.baseDelay || 200;
this.maxDelay = options.maxDelay || 30000;
}
getDelay(attempt) {
const exp = Math.min(this.baseDelay * Math.pow(2, attempt), this.maxDelay);
return Math.random() * exp;
}
async execute(fn) {
for (let attempt = 0; attempt < this.maxRetries; attempt++) {
try {
return await fn();
} catch (err) {
if (attempt === this.maxRetries - 1) throw err;
if (err.message === "circuit_open") throw err;
metrics.retries++;
const delay = this.getDelay(attempt);
await new Promise(r => setTimeout(r, delay));
}
}
}
}
Step 4: Idempotency Support
class IdempotencyManager {
constructor() {
this.cache = new Map();
}
generateKey() {
return crypto.randomUUID();
}
async execute(key, fn) {
if (this.cache.has(key)) {
return this.cache.get(key);
}
const result = await fn();
this.cache.set(key, result);
setTimeout(() => this.cache.delete(key), 86400000);
return result;
}
}
Step 5: Resilient HTTP Client
class ResilientClient {
constructor() {
this.circuitBreaker = new CircuitBreaker();
this.retry = new RetryHandler();
this.idempotency = new IdempotencyManager();
this.circuitBreakerByUrl = new Map();
}
getCircuitBreaker(url) {
if (!this.circuitBreakerByUrl.has(url)) {
this.circuitBreakerByUrl.set(url, new CircuitBreaker());
}
return this.circuitBreakerByUrl.get(url);
}
async request(url, options = {}) {
metrics.totalRequests++;
const cb = this.getCircuitBreaker(url);
const idempotencyKey = options.idempotencyKey;
try {
const result = await cb.call(async () => {
return this.retry.execute(async () => {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), options.timeout || 10000);
try {
const response = await fetch(url, {
method: options.method || "GET",
headers: {
"Content-Type": "application/json",
...(idempotencyKey ? { "Idempotency-Key": idempotencyKey } : {})
},
body: options.body ? JSON.stringify(options.body) : undefined,
signal: controller.signal
});
if (response.status === 429) {
const retryAfter = response.headers.get("Retry-After");
const delay = retryAfter ? parseInt(retryAfter) * 1000 : 5000;
await new Promise(r => setTimeout(r, delay));
throw new Error("rate_limited");
}
if (!response.ok && response.status >= 500) {
throw new Error(`server_error_${response.status}`);
}
if (!response.ok) {
throw new Error(`client_error_${response.status}`);
}
metrics.successes++;
return response.json();
} finally {
clearTimeout(timeout);
}
});
});
return result;
} catch (err) {
metrics.failures++;
if (options.fallback) {
metrics.fallbacks++;
return options.fallback();
}
throw err;
}
}
}
Step 6: Express Middleware Integration
const app = express();
const client = new ResilientClient();
app.get("/api/data", async (req, res) => {
try {
const data = await client.request("https://api.example.com/data", {
timeout: 5000,
fallback: () => ({ cached: true, data: [] })
});
res.json(data);
} catch (err) {
res.status(503).json({ error: "Service unavailable" });
}
});
app.get("/metrics", (req, res) => {
res.json(metrics);
});
app.listen(3000);
Step 7: Test the Client
// Simulate different failure scenarios
async function testClient() {
// Scenario 1: Successful request
const result1 = await client.request("https://httpstat.us/200");
console.log("Scenario 1:", result1);
// Scenario 2: Server error with retry
const result2 = await client.request("https://httpstat.us/503", {
fallback: () => ({ degraded: true })
});
console.log("Scenario 2:", result2);
// Scenario 3: Rate limited
const result3 = await client.request("https://httpstat.us/429", {
fallback: () => ({ retryLater: true })
});
console.log("Scenario 3:", result3);
console.log("Final metrics:", metrics);
}
Common Mistakes
Not distinguishing circuit breaker states -- The retry handler must not retry when the circuit is open.
Missing timeout on requests -- Without timeouts, a hanging request blocks the entire retry chain.
Retrying non-idempotent POST without idempotency key -- This creates duplicate resources.
Not handling the Retry-After header -- The server tells you exactly how long to wait. Ignoring it wastes retries.
No fallback for critical operations -- Every retryable operation should have a fallback, even if it is a cached response.
Practice Questions
How does the circuit breaker integrate with the retry handler? The circuit breaker wraps the retry handler. If the circuit is open, the retry handler is not called.
Why use per-URL circuit breakers? Different downstream services have different reliability. One failing service should not affect calls to healthy services.
When would the fallback be used instead of throwing? When the operation is non-critical and a degraded response is acceptable (e.g., showing cached data).
Challenge: Add automatic retry metrics reporting to the client.
FAQ
Project Extension Ideas
- Add a circuit breaker dashboard with real-time state visualization
- Implement distributed circuit breaker state using Redis
- Add automatic retry policy tuning based on success rates
- Implement retry queuing for offline resilience
- Add support for gRPC retry alongside HTTP
What's Next
Congratulations on completing the retry strategies project! Explore the circuit breaker pattern in depth. Then learn about circuit breaker states and transitions.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro