Retry with Circuit Breaker — Complete Integration Guide
In this tutorial, you will learn about Retry with Circuit Breaker. We cover key concepts, practical examples, and best practices to help you master this topic.
Combining retry strategies with circuit breakers creates a robust failure handling system where retries handle transient failures and circuit breakers prevent cascading failures of persistent problems.
What You'll Learn
By the end of this tutorial, you will integrate retry logic with circuit breakers, understand when to retry vs open the circuit, and prevent retry storms from overwhelming downstream services.
Why It Matters
Retries alone can make failures worse. A circuit breaker stops retries when the downstream service is clearly down. DodaTech combines both for resilient microservice communication.
Real-World Use
DodaZIP's conversion service retries failed conversions 3 times with backoff. If all retries fail, the circuit breaker opens and subsequent requests fail immediately for 30 seconds.
Retry + Circuit Breaker Learning Path
flowchart LR
A[Jitter] --> B[Retry + Circuit Breaker]
B --> C[Integration Pattern]
C --> D[State Machine]
B --> E{You Are Here}
style E fill:#f90,color:#fff
Why Retry Alone Is Not Enough
Retries without circuit breakers can make a failing situation worse by continuing to hammer a struggling service.
// Problem: retrying a service that is completely down
// All retries fail, wasting resources and potentially
// preventing the service from recovering
Integrated Retry with Circuit Breaker
The circuit breaker wraps retry logic. If retries fail, the circuit opens. If the circuit is open, retries are skipped entirely.
class CircuitBreakerWithRetry {
constructor(options) {
this.failureThreshold = options.failureThreshold || 5;
this.resetTimeout = options.resetTimeout || 30000;
this.retryOptions = options.retryOptions || {
maxRetries: 3,
baseDelay: 200
};
this.failureCount = 0;
this.state = "closed";
this.nextAttempt = Date.now();
}
getState() { return this.state; }
async call(fn) {
if (this.state === "open") {
if (Date.now() < this.nextAttempt) {
throw new Error("Circuit breaker is open");
}
this.state = "half-open";
}
try {
const result = await this.executeWithRetry(fn);
this.onSuccess();
return result;
} catch (err) {
this.onFailure();
throw err;
}
}
async executeWithRetry(fn) {
const { maxRetries, baseDelay } = this.retryOptions;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
return await fn();
} catch (err) {
if (attempt === maxRetries - 1) throw err;
const delay = baseDelay * Math.pow(2, attempt);
await new Promise(r => setTimeout(r, delay));
}
}
}
onSuccess() {
this.failureCount = 0;
this.state = "closed";
}
onFailure() {
this.failureCount++;
if (this.failureCount >= this.failureThreshold) {
this.state = "open";
this.nextAttempt = Date.now() + this.resetTimeout;
this.failureCount = 0;
}
}
}
Graceful Degradation with Fallback
When the circuit is open and retries are exhausted, provide a fallback response instead of failing completely.
class DegradingCircuitBreaker extends CircuitBreakerWithRetry {
constructor(options) {
super(options);
this.fallback = options.fallback || (() => {
return { error: "Service unavailable", cached: true };
});
}
async callWithFallback(fn) {
try {
return await this.call(fn);
} catch (err) {
if (this.state === "open" || err.message === "Circuit breaker is open") {
return this.fallback();
}
throw err;
}
}
}
const breaker = new DegradingCircuitBreaker({
failureThreshold: 3,
resetTimeout: 10000,
retryOptions: { maxRetries: 2, baseDelay: 100 },
fallback: () => ({ data: "cached response", fromCache: true })
});
app.get("/api/data", async (req, res) => {
const result = await breaker.callWithFallback(() => fetchFromService());
res.json(result);
});
Common Mistakes
Retrying when circuit is open -- If the circuit is open, retries waste resources and delay recovery. Fail Fast instead.
Opening circuit on transient failures -- A single timeout should not open the circuit. Use a threshold (5+ failures).
Not resetting retry count after circuit half-open success -- A successful half-open call should reset the failure count and close the circuit.
Using the same configuration for all services -- Different services need different thresholds based on their reliability.
Not logging circuit state transitions -- Circuit opening is a critical event. Log all state changes for debugging.
Practice Questions
Why combine retries with circuit breakers? Retries handle transient failures. Circuit breakers prevent retries from overwhelming a failing service.
What happens when the circuit is half-open? One request is allowed through as a probe. If it succeeds, the circuit closes. If it fails, the circuit reopens.
When should retries be skipped entirely? When the circuit is open. Retries would only waste resources and delay the downstream service's recovery.
Challenge: Design a circuit breaker that tracks failure rate over a Sliding Window.
class RateBasedCircuitBreaker {
constructor(options) {
this.windowMs = options.windowMs || 60000;
this.threshold = options.threshold || 0.5; // 50% failure rate
this.timestamps = [];
}
record(errored) {
this.timestamps.push({ time: Date.now(), errored });
this.cleanup();
}
get failureRate() {
const recent = this.timestamps.filter(t => Date.now() - t.time < this.windowMs);
const failures = recent.filter(t => t.errored).length;
return recent.length > 0 ? failures / recent.length : 0;
}
}
FAQ
Mini Project
Build a complete retry + circuit breaker system with exponential backoff, jitter, state tracking, and fallback responses.
class RetryCircuitBreaker {
constructor(options) {
this.maxRetries = options.maxRetries || 3;
this.baseDelay = options.baseDelay || 200;
this.failureThreshold = options.failureThreshold || 5;
this.resetTimeout = options.resetTimeout || 30000;
this.fallback = options.fallback;
this.state = "closed";
this.failureCount = 0;
this.nextAttempt = Date.now();
}
async execute(fn) {
if (this.state === "open") {
if (Date.now() >= this.nextAttempt) {
this.state = "half-open";
} else if (this.fallback) {
return this.fallback();
} else {
throw new Error("Circuit breaker open");
}
}
for (let attempt = 0; attempt <= this.maxRetries; attempt++) {
try {
const result = await fn();
this.onSuccess();
return result;
} catch (err) {
if (attempt < this.maxRetries) {
const delay = this.baseDelay * Math.pow(2, attempt);
await new Promise(r => setTimeout(r, delay));
}
}
}
this.onFailure();
if (this.state === "half-open" || this.fallback) {
return this.fallback ? this.fallback() : null;
}
throw new Error("All retries failed");
}
onSuccess() {
this.failureCount = 0;
this.state = "closed";
}
onFailure() {
this.failureCount++;
if (this.failureCount >= this.failureThreshold || this.state === "half-open") {
this.state = "open";
this.nextAttempt = Date.now() + this.resetTimeout;
}
}
}
What's Next
Now that you understand retries with circuit breakers, explore making operations safe for retries. Then learn about retry strategies for HTTP clients.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro