Advanced Circuit Breaker Patterns — Complete Implementation Guide
In this tutorial, you will learn about Advanced Circuit Breaker Patterns. We cover key concepts, practical examples, and best practices to help you master this topic.
Advanced circuit breaker patterns extend basic circuit breaking with bulkheads, fallback chains, rate-based state transitions, and adaptive threshold tuning for complex production systems.
What You'll Learn
By the end of this tutorial, you will implement bulkheads to isolate resources, fallback chains for graceful degradation, and self-tuning circuit breakers.
Real-World Use
DodaTech's payment system uses bulkheads: one circuit breaker per payment provider, with a fallback chain that tries providers in order.
Bulkhead Pattern
Bulkheads isolate resources by giving each dependency its own Connection Pool and thread pool, preventing one failing service from exhausting all resources.
class Bulkhead {
constructor(maxConcurrent) {
this.maxConcurrent = maxConcurrent;
this.active = 0;
this.queue = [];
}
async execute(fn) {
if (this.active >= this.maxConcurrent) {
throw new Error("Bulkhead full");
}
this.active++;
try {
return await fn();
} finally {
this.active--;
this.processQueue();
}
}
processQueue() {
if (this.queue.length > 0 && this.active < this.maxConcurrent) {
const next = this.queue.shift();
this.execute(next.fn).then(next.resolve).catch(next.reject);
}
}
}
// Usage with circuit breaker
class BulkheadCircuitBreaker {
constructor(name, options = {}) {
this.circuitBreaker = new CircuitBreaker(options);
this.bulkhead = new Bulkhead(options.maxConcurrent || 10);
this.name = name;
}
async call(fn) {
return this.bulkhead.execute(() =>
this.circuitBreaker.call(fn)
);
}
}
Fallback Chain
A fallback chain tries multiple strategies in sequence when the primary call fails.
class FallbackChain {
constructor() {
this.strategies = [];
}
addStrategy(name, fn, breaker) {
this.strategies.push({ name, fn, breaker });
return this;
}
async execute() {
const errors = [];
for (const strategy of this.strategies) {
try {
if (strategy.breaker) {
return await strategy.breaker.call(strategy.fn);
}
return await strategy.fn();
} catch (err) {
errors.push({ strategy: strategy.name, error: err.message });
}
}
throw new Error(`All strategies failed: ${JSON.stringify(errors)}`);
}
}
// Usage
const chain = new FallbackChain()
.addStrategy("primary", () => fetch("https://primary-api/data"), new CircuitBreaker({ threshold: 3 }))
.addStrategy("secondary", () => fetch("https://secondary-api/data"), new CircuitBreaker({ threshold: 5 }))
.addStrategy("cache", () => Promise.resolve({ cached: true, data: [] }));
Self-Tuning Circuit Breaker
A self-tuning circuit breaker adjusts its thresholds based on observed failure patterns and success rates.
class SelfTuningCircuitBreaker {
constructor(options = {}) {
this.baseThreshold = options.baseThreshold || 5;
this.minThreshold = options.minThreshold || 2;
this.maxThreshold = options.maxThreshold || 20;
this.resetTimeout = options.resetTimeout || 30000;
this.state = "closed";
this.failureCount = 0;
this.totalCalls = 0;
this.results = [];
this.threshold = this.baseThreshold;
}
async call(fn) {
this.totalCalls++;
if (this.state === "open") {
if (Date.now() < this.nextAttempt) throw new Error("Open");
this.state = "half-open";
}
try {
const result = await fn();
this.record(true);
return result;
} catch (err) {
this.record(false);
throw err;
}
}
record(success) {
this.results.push({ success, time: Date.now() });
this.cleanup();
if (success) {
this.failureCount = 0;
if (this.state === "half-open") {
this.state = "closed";
this.tuneThreshold();
}
} else {
this.failureCount++;
if (this.state === "half-open" || this.failureCount >= this.threshold) {
this.state = "open";
this.nextAttempt = Date.now() + this.resetTimeout;
this.tuneThreshold();
}
}
}
tuneThreshold() {
const recent = this.results.filter(r => Date.now() - r.time < 300000);
if (recent.length < 50) return;
const failures = recent.filter(r => !r.success).length;
const rate = failures / recent.length;
if (rate > 0.3) {
this.threshold = Math.max(this.minThreshold, this.threshold - 1);
} else if (rate < 0.05) {
this.threshold = Math.min(this.maxThreshold, this.threshold + 1);
}
}
cleanup() {
const cutoff = Date.now() - 600000;
this.results = this.results.filter(r => r.time > cutoff);
}
}
Common Mistakes
Bulkheads with too few concurrent slots -- Setting maxConcurrent too low causes false rejections. Monitor queue depth.
Fallback chains that mask real failures -- If the fallback always succeeds, you never know the primary is failing. Alert on fallback usage.
Self-tuning on too little data -- With few samples, the threshold fluctuates wildly. Require minimum sample size.
Not prioritizing bulkhead isolation -- Share nothing between bulkheads. Each dependency gets its own dedicated resources.
Fallbacks that are slower than the primary -- A slow fallback negates the benefit of fail-fast. Keep fallbacks fast.
Practice Questions
What problem does the bulkhead pattern solve? Resource exhaustion. One failing dependency cannot consume all connection pool or thread pool resources.
How does a fallback chain improve availability? When the primary fails, it tries secondary options. Users see a successful (possibly degraded) response.
How does self-tuning improve circuit breaker effectiveness? It adjusts to changing conditions. During stable periods, it becomes more tolerant. During unstable periods, it becomes more aggressive.
Challenge: Implement a bulkhead that times out queued requests.
class TimedBulkhead extends Bulkhead {
constructor(maxConcurrent, queueTimeout = 5000) {
super(maxConcurrent);
this.queueTimeout = queueTimeout;
}
async execute(fn) {
if (this.active >= this.maxConcurrent) {
return Promise.race([
new Promise((_, reject) => setTimeout(() => reject(new Error("Queue timeout")), this.queueTimeout)),
super.execute(fn)
]);
}
return super.execute(fn);
}
}
FAQ
Mini Project
Build an advanced circuit breaker system with bulkheads, fallback chains, and self-tuning thresholds.
class AdvancedProtectionSystem {
constructor(name, options = {}) {
this.name = name;
this.cb = new SelfTuningCircuitBreaker(options);
this.bulkhead = new Bulkhead(options.maxConcurrent || 10);
this.fallbacks = [];
}
addFallback(name, fn) {
this.fallbacks.push({ name, fn });
return this;
}
async execute(fn) {
try {
return await this.bulkhead.execute(() => this.cb.call(fn));
} catch (err) {
for (const fallback of this.fallbacks) {
try {
const result = await fallback.fn();
console.log(`Fallback "${fallback.name}" used for ${this.name}`);
return result;
} catch {}
}
throw err;
}
}
getStats() {
return {
name: this.name,
state: this.cb.state,
threshold: this.cb.threshold,
bulkheadActive: this.bulkhead.active,
bulkheadQueued: this.bulkhead.queue.length,
fallbacksConfigured: this.fallbacks.length
};
}
}
What's Next
Now that you understand advanced circuit breakers, explore testing circuit breaker implementation. Then learn about circuit breaker performance optimization.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro