Chaos Testing Circuit Breakers — Validating Resilience with Controlled Failures

In this tutorial, you will learn about Chaos Testing Circuit Breakers. We cover key concepts, practical examples, and best practices to help you master this topic.

Chaos testing for circuit breakers validates that circuit breakers open at the right thresholds, fallbacks activate correctly, half-open probes detect recovery, and the system handles cascading failures without unexpected behavior.

flowchart LR
    Test[Chaos Test Suite] --> Injection[Failure Injection]
    Injection -->|Latency| LatencyTests[Slow Responses]
    Injection -->|Crash| CrashTests[Service Down]
    Injection -->|Errors| ErrorTests[Hight Error Rate]
    LatencyTests --> Verify{Verify Behavior}
    CrashTests --> Verify
    ErrorTests --> Verify
    Verify --> Opens[Circuit Opens]
    Verify --> Fallback[Fallback Activated]
    Verify --> Recovers[Circuit Recovers]

What You'll Learn

• Failure injection strategies
• Latency and error injection
• Deterministic circuit breaker testing
• Chaos Mesh integration
• Recovery validation

Why It Matters

Without chaos testing, circuit breakers may never have been tested in production-like failure scenarios. Configuration errors (wrong thresholds, missing fallbacks) only surface during real outages. Chaos testing validates circuit breakers work correctly before the real outage.

Real-World Use

DodaTech runs weekly chaos experiments that inject 5-second latency into the payment service. Circuit breakers must open within 10 requests, fallbacks return cached responses, and recovery occurs within 30 seconds of latency normalization. Any failure triggers an alert.

Failure Injection

import time
import random
import threading

class FailureInjector:
    def __init__(self, target):
        self.target = target
        self.inject_latency = False
        self.latency_ms = 0
        self.inject_errors = False
        self.error_rate = 0
        self.inject_crash = False

    def wrap(self, fn):
        def wrapper(*args, **kwargs):
            if self.inject_crash:
                raise ConnectionError("Service crashed (injected)")

            if self.inject_latency:
                time.sleep(self.latency_ms / 1000)

            if self.inject_errors and random.random() < self.error_rate:
                raise ConnectionError("Injected error")

            return fn(*args, **kwargs)
        return wrapper

    def start_latency_injection(self, ms=2000):
        self.inject_latency = True
        self.latency_ms = ms
        print(f"[Chaos] Injecting {ms}ms latency")

    def start_error_injection(self, rate=0.5):
        self.inject_errors = True
        self.error_rate = rate
        print(f"[Chaos] Injecting {rate*100}% error rate")

    def stop_all(self):
        self.inject_latency = False
        self.inject_errors = False
        self.inject_crash = False
        print("[Chaos] Injection stopped")

injector = FailureInjector("payment-service")

def mock_payment_call(amount):
    time.sleep(0.1)
    return f"Paid ${amount}"

wrapped_call = injector.wrap(mock_payment_call)

for i in range(4):
    if i == 1:
        injector.start_latency_injection(500)
    try:
        result = wrapped_call(100)
        print(f"Call {i+1}: {result}")
    except Exception as e:
        print(f"Call {i+1}: {e}")
    time.sleep(0.1)

injector.stop_all()

Expected output:

Call 1: Paid $100
[Chaos] Injecting 500ms latency
Call 2: Paid $100
Call 3: Paid $100
Call 4: Paid $100
[Chaos] Injection stopped

Circuit Breaker Test

import time
import threading
import random

class TestCircuitBreaker:
    def __init__(self, name, fail_threshold=5, recovery_timeout=5):
        self.name = name
        self.fail_threshold = fail_threshold
        self.recovery_timeout = recovery_timeout
        self.failures = 0
        self.state = 'CLOSED'
        self.last_failure = 0

    def call(self, fn, *args, **kwargs):
        if self.state == 'OPEN':
            if time.time() - self.last_failure > self.recovery_timeout:
                self.state = 'HALF_OPEN'
            else:
                return None

        try:
            result = fn(*args, **kwargs)
            self.failures = 0
            if self.state == 'HALF_OPEN':
                self.state = 'CLOSED'
            return result
        except Exception:
            self.failures += 1
            self.last_failure = time.time()
            if self.failures >= self.fail_threshold:
                self.state = 'OPEN'
            return None

    def get_state(self):
        return self.state

def test_circuit_breaker_opens():
    cb = TestCircuitBreaker("test", fail_threshold=3, recovery_timeout=10)
    failed_fn = lambda: (_ for _ in ()).throw(Exception("fail"))

    for i in range(5):
        cb.call(failed_fn)

    assert cb.state == 'OPEN', f"Expected OPEN, got {cb.state}"
    print("PASS: Circuit opens after threshold failures")

def test_circuit_breaker_recovers():
    cb = TestCircuitBreaker("test", fail_threshold=3, recovery_timeout=1)
    failed_fn = lambda: (_ for _ in ()).throw(Exception("fail"))

    for _ in range(5):
        cb.call(failed_fn)
    assert cb.state == 'OPEN'

    time.sleep(1.5)
    result = cb.call(lambda: "success")
    assert result == "success"
    assert cb.state == 'CLOSED', f"Expected CLOSED, got {cb.state}"
    print("PASS: Circuit recovers after reset timeout")

def test_fallback_activated():
    cb = TestCircuitBreaker("test", fail_threshold=2, recovery_timeout=30)
    failed_fn = lambda: (_ for _ in ()).throw(Exception("fail"))

    for _ in range(3):
        cb.call(failed_fn)
    assert cb.state == 'OPEN'

    result = cb.call(failed_fn)
    assert result is None, f"Expected None fallback, got {result}"
    print("PASS: Fallback activated when circuit is open")

test_circuit_breaker_opens()
test_circuit_breaker_recovers()
test_fallback_activated()

Expected output:

PASS: Circuit opens after threshold failures
PASS: Circuit recovers after reset timeout
PASS: Fallback activated when circuit is open

Common Mistakes

• Only testing happy path -- most circuit breaker tests verify normal operation. Test the unhappy path: what happens when the circuit opens, how fallbacks behave, what state transitions occur.
• No timing tests -- circuit breaker recovery timeout and half-open probing depend on timing. Use time manipulation (monotonic clock injection) for deterministic timing tests. Skip time.sleep in unit tests.
• Testing with real downstream services -- chaos tests should inject failures at the circuit breaker boundary, not at the actual downstream service. Mock the downstream and control its behavior in the test.
• No verification of metrics -- circuit breaker tests should verify that metrics are emitted correctly. Assert that state transition counters increment, failure counts are accurate, and fallback metrics are recorded.
• Not testing concurrent access -- circuit breakers in multi-threaded environments need concurrent access tests. Spawn multiple threads that simultaneously call the circuit breaker and verify state consistency.

Practice Questions

1. How do you inject latency into circuit breaker tests?
2. What is the advantage of deterministic timing in circuit breaker tests?
3. How do you test concurrent circuit breaker access?
4. What metrics should you verify in circuit breaker chaos tests?
5. How does Chaos Mesh inject failures into circuit breakers?

Challenge

Build a chaos test suite for circuit breakers: (1) inject latency (1s, 2s, 5s) and verify circuit breaker opens correctly at each threshold, (2) inject burst errors (10 consecutive, then recovery) and verify circuit opens and recovers, (3) inject intermittent errors (50% error rate) and verify circuit breaker handles partial failures correctly, (4) concurrent access test: 10 threads calling the circuit breaker simultaneously with injected failures, (5) half-open probe validation: inject failure during probe, verify circuit reopens, (6) fallback test: inject failures and verify the correct fallback is returned for each service, (7) produce a chaos test report showing circuit breaker behavior for each failure scenario.

FAQ

Mini Project

Build a Chaos Engineering framework for circuit breakers: (1) failure injection library (latency, errors, crashes) with configurable schedules, (2) automated test suite that validates circuit breaker behavior for each failure type, (3) Chaos Mesh workflow for Kubernetes: inject failures into circuit breaker-managed services, (4) deterministic clock for fast unit tests, (5) concurrent access test harness for multi-threaded circuit breaker validation, (6) test report generator showing pass/fail for each failure scenario with metrics, (7) CI integration: run chaos tests in a staging environment before production deployment.

What's Next

Continue with Self-Healing Systems to learn autonomous recovery patterns. Then explore Predictive Circuit Breaking for Machine Learning-driven circuit breakers.