Circuit Breaker vs Retry vs Bulkhead vs Timeout — Resilience Pattern Comparison

In this tutorial, you will learn about Circuit Breaker vs Retry vs Bulkhead vs Timeout. We cover key concepts, practical examples, and best practices to help you master this topic.

Circuit breaker, retry, bulkhead, timeout, Rate Limiting, and fallback are complementary resilience patterns that protect different failure modes. Combining them correctly creates robust systems that handle transient failures, resource exhaustion, and cascading outages.

flowchart LR
    Req[Request] --> RL[Rate Limiter]
    RL --> T[Timeout]
    T --> RT[Retry]
    RT --> CB[Circuit Breaker]
    CB --> BH[Bulkhead]
    BH --> FB[Fallback]
    FB --> Res[Response]
    style CB fill:#f90,color:#fff

What You'll Learn

• Each resilience pattern's purpose and failure mode
• When to use circuit breaker vs retry vs bulkhead
• Pattern composition order
• Anti-patterns in pattern combination
• Real-world pattern selection framework

Why It Matters

Using the wrong pattern — or composing them in the wrong order — makes failures worse. Retrying a request that takes 30 seconds without a timeout exhausts threads. Opening a circuit for every transient failure causes false positives. Understanding each pattern's job is essential for building resilient systems.

Real-World Use

DodaTech combines all patterns in its scanning API: rate limiter (100 req/s per API key), timeout (5 seconds per request), retry (2 attempts with 100ms backoff), circuit breaker (open after 5 failures), bulkhead (10 concurrent scans per service), and fallback (cached scan results). This stack handles 99.99% uptime during traffic spikes.

Pattern Comparison Table

patterns = [
    {"name": "Circuit Breaker", "handles": "Persistent failures", "prevents": "Cascading failures"},
    {"name": "Retry", "handles": "Transient failures", "prevents": "Temporary blips from causing errors"},
    {"name": "Bulkhead", "handles": "Resource exhaustion", "prevents": "One service starving others"},
    {"name": "Timeout", "handles": "Slow responses", "prevents": "Thread/connection exhaustion"},
    {"name": "Rate Limiter", "handles": "Traffic spikes", "prevents": "System overload from excessive requests"},
    {"name": "Fallback", "handles": "Complete failures", "prevents": "User-facing errors"},
]

print(f"{'Pattern':<20} {'Handles':<25} {'Prevents':<30}")
print("-" * 75)
for p in patterns:
    print(f"{p['name']:<20} {p['handles']:<25} {p['prevents']:<30}")

Expected output:

Pattern              Handles                  Prevents                      
----------------------------------------------------------------------------
Circuit Breaker      Persistent failures      Cascading failures            
Retry                Transient failures       Temporary blips from causing errors
Bulkhead             Resource exhaustion      One service starving others   
Timeout              Slow responses           Thread/connection exhaustion  
Rate Limiter         Traffic spikes           System overload from excessive requests
Fallback             Complete failures        User-facing errors

Pattern Composition

import time
import random
import threading

class ResiliencePipeline:
    def __init__(self):
        self.patterns = []

    def add_pattern(self, name, fn):
        self.patterns.append((name, fn))
        return self

    def execute(self, request_fn, *args, **kwargs):
        for name, pattern_fn in self.patterns:
            try:
                pattern_fn()
            except Exception as e:
                print(f"[{name}] blocked: {e}")
                return None
        return request_fn(*args, **kwargs)

def rate_limit():
    print("[Rate Limiter] Check passed")

def enforce_timeout():
    print("[Timeout] 5s limit set")

def retry_check():
    print("[Retry] 2 attempts configured")

def cb_check():
    print("[Circuit Breaker] Closed")

def bulkhead_check():
    print("[Bulkhead] 10 threads available")

pipeline = ResiliencePipeline()
pipeline.add_pattern("Rate Limiter", rate_limit)
pipeline.add_pattern("Timeout", enforce_timeout)
pipeline.add_pattern("Retry", retry_check)
pipeline.add_pattern("Circuit Breaker", cb_check)
pipeline.add_pattern("Bulkhead", bulkhead_check)

result = pipeline.execute(lambda: "Success")
print(f"Result: {result}")

Expected output:

[Rate Limiter] Check passed
[Timeout] 5s limit set
[Retry] 2 attempts configured
[Circuit Breaker] Closed
[Bulkhead] 10 threads available
Result: Success

When to Use Each Pattern

def recommend_patterns(failure_type):
    recommendations = {
        "transient_network": ["Retry", "Timeout"],
        "service_down": ["Circuit Breaker", "Fallback"],
        "slow_response": ["Timeout", "Circuit Breaker"],
        "resource_exhaustion": ["Bulkhead", "Rate Limiter"],
        "traffic_spike": ["Rate Limiter", "Bulkhead"],
        "dependency_failure": ["Fallback", "Circuit Breaker"],
    }
    return recommendations.get(failure_type, ["Circuit Breaker"])

scenarios = [
    ("Database connection timeout", "transient_network"),
    ("Payment service returns 503", "service_down"),
    ("Search API takes 30 seconds", "slow_response"),
    ("All threads blocked on DB queries", "resource_exhaustion"),
    ("1000x normal traffic", "traffic_spike"),
    ("Inventory service unreachable", "dependency_failure"),
]

for scenario, failure_type in scenarios:
    patterns = recommend_patterns(failure_type)
    print(f"{scenario:<45} -> {', '.join(patterns)}")

Expected output:

Database connection timeout                  -> Retry, Timeout
Payment service returns 503                  -> Circuit Breaker, Fallback
Search API takes 30 seconds                  -> Timeout, Circuit Breaker
All threads blocked on DB queries            -> Bulkhead, Rate Limiter
1000x normal traffic                         -> Rate Limiter, Bulkhead
Inventory service unreachable                -> Fallback, Circuit Breaker

Common Mistakes

• Retrying without a timeout -- retrying a slow request without a timeout multiplies the damage. Each retry attempt waits for the full response time, blocking threads and connections. Always place timeout before retry in the pipeline.
• Circuit breaker without retry -- a single transient failure opens the circuit, causing minutes of downtime for a 100ms blip. Place retry before circuit breaker so only persistent failures trigger circuit opening.
• Bulkhead without circuit breaker -- bulkhead isolates resources but does not stop requests to a failing service. Failed requests still waste bulkhead threads. Combine bulkhead with circuit breaker to stop traffic when the downstream fails.
• Rate limiting after circuit breaker -- rate limiting should happen before circuit breaker to prevent high traffic from keeping the circuit open. Rate limiting first, then circuit breaker, ensures the circuit sees controlled traffic volume.
• Using patterns in isolation -- each pattern addresses one failure mode. Using only circuit breaker without timeout means slow requests keep the circuit open longer. Using only retry without circuit breaker means persistent failures are retried endlessly. Compose all patterns in the correct order.

Practice Questions

1. What is the correct order for composing resilience patterns?
2. Why should retry come before circuit breaker?
3. What failure mode does bulkhead address that circuit breaker does not?
4. When would you use fallback without circuit breaker?
5. How do you decide which patterns to use for a given service?

Challenge

Design a resilience pipeline for a payment service: (1) identify all 6 patterns that apply to payment processing, (2) determine the correct composition order with justification for each positioning decision, (3) configure each pattern with appropriate parameters for payment operations (timeout: 5s, retry: 2 attempts with 500ms backoff, circuit: 3 failures in 60s window), (4) implement the pipeline in Python with realistic failure simulation, (5) measure the impact: simulate 1000 requests with 20% failure rate and show how many reach the user as errors vs graceful fallbacks.

FAQ

Mini Project

Build a complete resilience pipeline with all 6 patterns: (1) rate limiter: 100 req/s with token bucket algorithm, (2) timeout: configurable per-endpoint timeout (1-30s), (3) retry: 3 attempts with exponential backoff (100ms, 200ms, 400ms), (4) circuit breaker: Sliding Window of 10 requests, 50% failure rate threshold, 30s reset timeout, (5) bulkhead: 10 threads per downstream service with Semaphore, (6) fallback: tiered fallback chain (cache, static defaults, error response), (7) pipeline orchestrator that executes patterns in correct order with metrics collection per stage, (8) test script that simulates various failure modes and demonstrates the pipeline response.

What's Next

Continue with Zero-Downtime Deployments to learn rolling updates with circuit breakers. Then explore Observability for advanced circuit breaker monitoring.