Circuit Breaker Pattern — Resilient Microservices Communication
In this tutorial, you will learn about Circuit Breaker Pattern. We cover key concepts, practical examples, and best practices to help you master this topic.
The circuit breaker pattern detects when a downstream service is failing, stops sending requests to it, and periodically probes for recovery, preventing cascading failures and allowing services to recover.
What You'll Learn
By the end of this lesson you will implement circuit breakers with three states (closed, open, half-open), configure failure thresholds and timeouts, integrate circuit breakers with other resilience patterns, and understand the difference between circuit breakers and retries.
Why It Matters
When a microservice becomes slow or unresponsive, all upstream services that call it also become slow as they wait for responses. This cascading effect can bring down the entire system. Circuit breakers isolate failures by failing fast when a service is known to be unhealthy.
Real-World Use
DodaZIP's API Gateway implements circuit breakers for all downstream services. When the payment service starts returning 5xx errors, the circuit breaker opens after 5 failures within 30 seconds. Subsequent payment requests fail immediately with a friendly message instead of hanging until timeout.
flowchart LR
A[Closed: Normal Operation] -->|Failures exceed threshold| B[Open: Requests Blocked]
B -->|Timeout expires| C[Half-Open: Test Request]
C -->|Success| A
C -->|Failure| B
style A fill:#2d3748,color:#fff
style B fill:#e53e3e,color:#fff
style C fill:#d69e2e,color:#fff
Circuit Breaker States
The three states of a circuit breaker.
# circuit_states.py
# Circuit breaker states
def circuit_states():
print("Circuit Breaker States")
print("=" * 40)
print()
states = [
{
"state": "Closed",
"meaning": "Normal operation. Requests pass through to the service.",
"behavior": "Track failure count. If threshold exceeded, transition to Open.",
"color": "Green"
},
{
"state": "Open",
"meaning": "Service is considered down. Requests are rejected immediately.",
"behavior": "Fail fast with cached response or error. After timeout, transition to Half-Open.",
"color": "Red"
},
{
"state": "Half-Open",
"meaning": "Testing if service has recovered.",
"behavior": "Allow a limited number of test requests. If they succeed, transition to Closed. If they fail, back to Open.",
"color": "Yellow"
},
]
for s in states:
print(f"{s['state']:15s} ({s['color']})")
print(f" {s['meaning']}")
print(f" {s['behavior']}")
print()
circuit_states()
Circuit Breaker Implementation
Full circuit breaker with state machine.
# circuit_breaker_impl.py
# Circuit breaker implementation
def circuit_breaker_impl():
print("Circuit Breaker Implementation")
print("=" * 40)
print()
code = """
import time
import threading
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class CircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=30,
half_open_max_requests=3):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.half_open_max_requests = half_open_max_requests
self.state = CircuitState.CLOSED
self.failure_count = 0
self.success_count = 0
self.last_failure_time = 0
self.half_open_requests = 0
self.lock = threading.Lock()
def call(self, func, fallback=None):
"""Execute func with circuit breaker protection."""
if not self._can_proceed():
return self._get_fallback(fallback)
try:
result = func()
self._on_success()
return result
except Exception as e:
self._on_failure()
return self._get_fallback(fallback, e)
def _can_proceed(self):
with self.lock:
if self.state == CircuitState.CLOSED:
return True
if self.state == CircuitState.OPEN:
# Check if recovery timeout has elapsed
if time.time() - self.last_failure_time >= self.recovery_timeout:
self.state = CircuitState.HALF_OPEN
self.half_open_requests = 0
print("Circuit: OPEN -> HALF_OPEN")
return True
return False
# Half-Open: allow limited requests
if self.half_open_requests < self.half_open_max_requests:
self.half_open_requests += 1
return True
return False
def _on_success(self):
with self.lock:
if self.state == CircuitState.HALF_OPEN:
self.success_count += 1
if self.success_count >= self.half_open_max_requests:
self.state = CircuitState.CLOSED
self.failure_count = 0
self.success_count = 0
print("Circuit: HALF_OPEN -> CLOSED")
elif self.state == CircuitState.CLOSED:
self.failure_count = 0 # Reset on success
def _on_failure(self):
with self.lock:
self.last_failure_time = time.time()
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.OPEN
print("Circuit: HALF_OPEN -> OPEN")
return
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
print(f"Circuit: CLOSED -> OPEN "
f"({self.failure_count} failures)")
def _get_fallback(self, fallback, error=None):
if fallback:
return fallback()
if error:
raise error
raise Exception("Service unavailable (circuit open)")
"""
print(code)
circuit_breaker_impl()
Integration with Retry
Combining circuit breaker with retry logic.
# circuit_with_retry.py
# Circuit breaker + retry
def circuit_retry():
print("Circuit Breaker with Retry Integration")
print("=" * 45)
print()
code = """
import time
import random
class RetryWithCircuitBreaker:
"""Combines retry logic with circuit breaker protection."""
def __init__(self, circuit_breaker, max_retries=3,
backoff_base=1.0, backoff_multiplier=2.0):
self.circuit = circuit_breaker
self.max_retries = max_retries
self.backoff_base = backoff_base
self.backoff_multiplier = backoff_multiplier
def execute(self, func, fallback=None):
last_error = None
for attempt in range(self.max_retries + 1):
def call_func():
return func()
result = self.circuit.call(call_func)
if result is not None:
return result # Success
# Circuit is open or call failed
last_error = "Service unavailable"
if attempt < self.max_retries:
backoff = self.backoff_base * (
self.backoff_multiplier ** attempt
)
jitter = random.uniform(0, backoff * 0.1)
sleep_time = backoff + jitter
print(f"Retry {attempt + 1}/{self.max_retries} "
f"in {sleep_time:.1f}s")
time.sleep(sleep_time)
# All retries exhausted
if fallback:
return fallback()
raise Exception(f"All {self.max_retries} retries failed: {last_error}")
# Usage
cb = CircuitBreaker(failure_threshold=3, recovery_timeout=15)
retrier = RetryWithCircuitBreaker(cb, max_retries=2)
def call_payment_service():
resp = httpx.post("http://payment/charge", json={"amount": 50})
resp.raise_for_status()
return resp.json()
result = retrier.execute(call_payment_service, fallback=lambda: {"status": "queued"})
"""
print(code)
circuit_retry()
Monitoring Circuit Breakers
Tracking circuit state for operations.
# circuit_monitoring.py
# Circuit breaker monitoring
def circuit_monitoring():
print("Circuit Breaker Monitoring")
print("=" * 40)
print()
monitoring_code = """
import time
import json
class CircuitBreakerMonitor:
"""Tracks circuit breaker metrics for observability."""
def __init__(self):
self.metrics = {} # service_name -> CircuitMetrics
def record_call(self, service_name, state, success, duration_ms):
if service_name not in self.metrics:
self.metrics[service_name] = {
"calls": 0,
"successes": 0,
"failures": 0,
"rejected": 0,
"total_duration_ms": 0,
"current_state": state,
"last_failure_time": None,
"state_changes": []
}
m = self.metrics[service_name]
m["calls"] += 1
m["total_duration_ms"] += duration_ms
m["current_state"] = state
if state == "open":
m["rejected"] += 1
elif success:
m["successes"] += 1
else:
m["failures"] += 1
m["last_failure_time"] = time.time()
def get_metrics(self, service_name=None):
if service_name:
return self.metrics.get(service_name)
return self.metrics
def report(self):
"""Generate a summary report."""
report = []
for service, m in self.metrics.items():
report.append({
"service": service,
"state": m["current_state"],
"calls": m["calls"],
"success_rate": round(
m["successes"] / max(m["calls"], 1) * 100, 1
),
"rejected": m["rejected"],
"avg_duration_ms": round(
m["total_duration_ms"] / max(m["calls"], 1), 1
),
"last_failure": m["last_failure_time"]
})
return report
# Use in health endpoint
monitor = CircuitBreakerMonitor()
@app.get("/health/circuit-breakers")
def circuit_health():
return {"circuit_breakers": monitor.report()}
"""
print(monitoring_code)
circuit_monitoring()
Common Mistakes
No fallback mechanism: Opening the circuit without a fallback means users get errors instead of degraded but functional responses. Always provide a fallback.
Setting thresholds too low: A brief spike in failures (e.g., deployment restart) opens the circuit unnecessarily. Set thresholds based on normal failure rates.
Not resetting success count in half-open: If you allow 3 test requests and only 1 succeeds, the circuit should stay open. Reset success count properly on each half-open entry.
Using circuit breakers for idempotent reads only: Circuit breakers work for all operations, but be careful with writes. A write that succeeds on the server but fails to respond triggers the circuit incorrectly.
No monitoring integration: A circuit breaker you cannot observe is dangerous. Always expose circuit state, failure counts, and state changes via metrics endpoints.
Practice Questions
What are the three states of a circuit breaker? Closed (normal operation), Open (requests blocked), Half-Open (testing recovery).
What triggers the transition from Closed to Open? The failure count exceeds the configured threshold within a time window.
What happens in the Half-Open state? A limited number of test requests are allowed through. If they succeed, the circuit closes. If they fail, it reopens.
Why combine circuit breakers with fallback responses? So users still get a response (even if degraded) when the circuit is open, instead of an error.
Challenge: Implement a circuit breaker for a notification service that sends email, SMS, and push notifications. If all three channels are failing, open the circuit and queue notifications for later delivery. Include per-channel circuit breakers.
FAQ
Mini Project
Implement circuit breakers for a three-service e-commerce system: product catalog, inventory, and pricing. Each service has its own circuit breaker. If inventory is down, the product page still loads (showing stock as unknown). If pricing is down, show cached prices with a notice. If all three fail, show a cached product page.
def ecommerce_circuit_breakers():
print("E-Commerce Circuit Breaker Design")
print("=" * 45)
print()
print("Services and Circuit Breakers:")
print()
print(" Product Catalog Service")
print(" CB: 5 failures, 30s timeout")
print(" Fallback: Cached product listing")
print()
print(" Inventory Service")
print(" CB: 3 failures, 15s timeout (more critical)")
print(" Fallback: Show 'stock unknown'")
print()
print(" Pricing Service")
print(" CB: 5 failures, 60s timeout")
print(" Fallback: Last known price")
print()
print("Page Rendering Strategy:")
print(" All CBs closed -> Show full live page")
print(" Inventory CB open -> Show product + 'stock check unavailable'")
print(" Pricing CB open -> Show cached price with 'price may be outdated'")
print(" All CBs open -> Show fully cached page from CDN")
print()
print("Recovery:")
print(" Each service has /health/liveness endpoint")
print(" CB probes health endpoint in half-open state")
ecommerce_circuit_breakers()
What's Next
Next: Distributed Tracing for tracing requests across services.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro