Circuit Breaker in Microservices — Complete Architecture Guide
In this tutorial, you will learn about Circuit Breaker in Microservices. We cover key concepts, practical examples, and best practices to help you master this topic.
Circuit breaker in microservice architectures prevents cascading failures by isolating failing services, using distributed state management, and integrating with service meshes and API gateways.
What You'll Learn
By the end of this tutorial, you will implement distributed circuit breakers, integrate with service meshes, and design microservice architectures that gracefully degrade.
Real-World Use
DodaTech's microservice architecture uses circuit breakers at every service boundary. When the payment service fails, all upstream services detect it within seconds and Fail Fast.
Microservice Circuit Breaker Implementation
const Redis = require("ioredis");
class DistributedCircuitBreaker {
constructor(name, redis, options = {}) {
this.name = name;
this.redis = redis;
this.threshold = options.threshold || 5;
this.resetTimeout = options.resetTimeout || 30000;
this.localState = "closed";
this.localFailures = 0;
this.nextAttempt = Date.now();
}
async call(fn) {
const globalState = await this.getGlobalState();
if (globalState === "open" || this.localState === "open") {
if (Date.now() < this.nextAttempt) {
throw new Error("Circuit breaker open");
}
this.localState = "half-open";
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (err) {
this.onFailure(err);
throw err;
}
}
async getGlobalState() {
try {
return await this.redis.get(`cb:${this.name}:state`);
} catch {
return null;
}
}
async onSuccess() {
this.localFailures = 0;
this.localState = "closed";
await this.redis.set(`cb:${this.name}:state`, "closed", "EX", 60);
await this.redis.del(`cb:${this.name}:failures`);
}
async onFailure(err) {
this.localFailures++;
const globalFailures = await this.redis.incr(`cb:${this.name}:failures`);
await this.redis.expire(`cb:${this.name}:failures`, 60);
if (this.localState === "half-open" || globalFailures >= this.threshold) {
this.localState = "open";
this.nextAttempt = Date.now() + this.resetTimeout;
await this.redis.set(`cb:${this.name}:state`, "open", "EX", Math.ceil(this.resetTimeout / 1000));
}
}
}
API Gateway Integration
API gateways are the ideal place to implement circuit breakers for all downstream services.
const express = require("express");
const app = express();
const serviceBreakers = {
users: new DistributedCircuitBreaker("users", redis, { threshold: 3 }),
orders: new DistributedCircuitBreaker("orders", redis, { threshold: 5 }),
payments: new DistributedCircuitBreaker("payments", redis, { threshold: 2 })
};
async function gatewayMiddleware(req, res, next) {
const service = req.path.split("/")[1];
const breaker = serviceBreakers[service];
if (!breaker) return next();
try {
await breaker.call(async () => {
const response = await fetch(`http://${service}-service${req.path}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const data = await response.json();
res.json(data);
});
} catch (err) {
if (err.message === "Circuit breaker open") {
return res.status(503).json({
error: `${service} service unavailable`,
suggestRetry: true
});
}
res.status(502).json({ error: `Failed to reach ${service} service` });
}
}
app.use(gatewayMiddleware);
app.listen(3000);
Cascading Failure Prevention
Circuit breakers at each layer prevent failures from propagating through the service graph.
flowchart TD A[Client] --> B[API Gateway] B --> C[Circuit Breaker] C --> D[User Service] C --> E[Order Service] C --> F[Payment Service] F --> G[Circuit Breaker] G --> H[Payment Provider] style C fill:#f90,color:#fff style G fill:#f90,color:#fff
// Each service has its own circuit breaker for downstream calls
// Service A calls Service B -> Service B circuit breaker
// Service B calls Service C -> Service C circuit breaker
// A failure in Service C is contained by Service B's breaker
// Service A never sees the failure in Service C
Common Mistakes
Not sharing state across instances -- Each instance has its own breaker, limiting effectiveness. Use Redis.
Circuit breakers at every layer causing cascading opens -- A downstream circuit breaker opening causes the upstream one to open too. This is cascading and desired behavior.
Not configuring different thresholds per service -- Critical services need aggressive thresholds. Batch services can be more tolerant.
Ignoring the health check endpoint -- Circuit breakers should integrate with Kubernetes liveness/readiness probes.
No graceful degradation UI -- When services are down, the frontend should show meaningful messages, not errors.
Practice Questions
Why share circuit breaker state across instances? Without shared state, a failing service instance would only be avoided by clients that happened to call that instance.
How do circuit breakers prevent cascading failures? By failing fast at each service boundary, upstream services do not accumulate waiting requests that exhaust resources.
What is the role of circuit breakers in an API gateway? The gateway is the first line of defense. It detects failing services and stops routing traffic to them.
Challenge: Design a circuit breaker hierarchy for a three-tier microservice architecture.
// Tier 1: API Gateway circuit breakers
// Tier 2: Service-to-service circuit breakers
// Tier 3: Service-to-database circuit breakers
// Each tier opens independently based on its own thresholds
FAQ
Mini Project
Build a microservice circuit breaker system with distributed Redis state, API gateway integration, and health monitoring.
class MicroserviceCircuitBreaker {
constructor(name, redis, options = {}) {
this.name = name;
this.redis = redis;
this.threshold = options.threshold || 5;
this.resetTimeout = options.resetTimeout || 30000;
}
async wrapRequest(fn) {
if (await this.isOpen()) {
throw new Error(`Service ${this.name} is unavailable (circuit open)`);
}
try {
const result = await fn();
await this.recordSuccess();
return result;
} catch (err) {
await this.recordFailure();
throw err;
}
}
async isOpen() {
const state = await this.redis.get(`mscb:${this.name}:state`);
if (state !== "open") return false;
const nextAttempt = await this.redis.get(`mscb:${this.name}:nextAttempt`);
if (nextAttempt && Date.now() > parseInt(nextAttempt)) {
await this.redis.del(`mscb:${this.name}:state`);
return false;
}
return true;
}
async recordFailure() {
const failures = await this.redis.incr(`mscb:${this.name}:failures`);
await this.redis.expire(`mscb:${this.name}:failures`, 60);
if (failures >= this.threshold) {
await this.redis.set(`mscb:${this.name}:state`, "open");
await this.redis.set(`mscb:${this.name}:nextAttempt`, Date.now() + this.resetTimeout, "EX", Math.ceil(this.resetTimeout / 1000));
}
}
async recordSuccess() {
await this.redis.del(`mscb:${this.name}:state`);
await this.redis.del(`mscb:${this.name}:failures`);
}
}
What's Next
Now that you understand microservice circuit breakers, explore advanced circuit breaker patterns. Then learn about testing circuit breaker implementation.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro