Circuit Breaker Orchestration — Coordinating Resilience Across Distributed Systems

In this tutorial, you will learn about Circuit Breaker Orchestration. We cover key concepts, practical examples, and best practices to help you master this topic.

Circuit breaker orchestration coordinates resilience decisions across multiple services by managing global circuit states, synchronizing recovery attempts, and preventing cascading failures through coordinated circuit opening and closing.

flowchart TD
    Orchestrator[Orchestrator] --> CB1[CB: Payment]
    Orchestrator --> CB2[CB: Inventory]
    Orchestrator --> CB3[CB: Shipping]
    CB1 -->|Open| Alert[Alert]
    CB2 -->|Open| Alert
    Orchestrator -->|Global Open| CB_All[All Circuits]
    CB_All --> Degraded[Degraded Mode]
    Degraded -->|Recover| Orchestrator
    style Orchestrator fill:#f90,color:#fff

What You'll Learn

• Global circuit state coordination
• Coordinated recovery strategies
• Cascading failure prevention
• Multi-service circuit synchronization
• Degraded mode orchestration

Why It Matters

Without orchestration, each service's circuit breaker operates independently. When Service A's circuit opens, Service B still sends traffic to the failing downstream, wasting resources. Orchestration ensures that related circuits open and close together, preventing the thundering herd problem during recovery.

Real-World Use

DodaTech's order processing pipeline uses orchestrated circuit breakers. When the payment service circuit opens, the orchestrator also opens the order creation circuit and the notification circuit, entering a "checkout degraded" mode. All three circuits recover together after a health check confirms the payment service is operational.

Global Circuit State Manager

import time
import threading

class GlobalCircuitState:
    def __init__(self):
        self.circuits = {}
        self.global_state = 'NORMAL'
        self._lock = threading.Lock()

    def register(self, name, depends_on=None):
        with self._lock:
            self.circuits[name] = {
                'state': 'CLOSED',
                'depends_on': depends_on or [],
                'failures': 0,
                'last_change': time.time()
            }

    def record_failure(self, name):
        with self._lock:
            if name not in self.circuits:
                return
            self.circuits[name]['failures'] += 1
            if self.circuits[name]['failures'] >= 3:
                self._set_state(name, 'OPEN')
                self._cascade(name)

    def record_success(self, name):
        with self._lock:
            if name not in self.circuits:
                return
            self.circuits[name]['failures'] = 0
            if self.circuits[name]['state'] == 'OPEN':
                self._set_state(name, 'HALF_OPEN')

    def _set_state(self, name, state):
        self.circuits[name]['state'] = state
        self.circuits[name]['last_change'] = time.time()

    def _cascade(self, name):
        for cb_name, cb in self.circuits.items():
            if name in cb['depends_on']:
                cb['state'] = 'OPEN'
                cb['last_change'] = time.time()
                print(f"Cascade: {cb_name} opened due to {name} failure")

    def get_state(self, name):
        with self._lock:
            return self.circuits.get(name, {}).get('state', 'UNKNOWN')

    def get_global_state(self):
        with self._lock:
            open_circuits = [n for n, c in self.circuits.items() if c['state'] == 'OPEN']
            if open_circuits:
                return f"DEGRADED: {', '.join(open_circuits)}"
            return 'NORMAL'

gcs = GlobalCircuitState()
gcs.register('payment')
gcs.register('order', depends_on=['payment'])
gcs.register('notification', depends_on=['payment'])

for i in range(5):
    gcs.record_failure('payment')
    print(f"After failure {i+1}: payment={gcs.get_state('payment')}, order={gcs.get_state('order')}, notif={gcs.get_state('notification')}")
    print(f"  Global: {gcs.get_global_state()}")

Expected output:

After failure 1: payment=CLOSED, order=CLOSED, notif=CLOSED
  Global: NORMAL
After failure 2: payment=CLOSED, order=CLOSED, notif=CLOSED
  Global: NORMAL
Cascade: order opened due to payment failure
Cascade: notification opened due to payment failure
After failure 3: payment=OPEN, order=OPEN, notif=OPEN
  Global: DEGRADED: payment, order, notification
After failure 4: payment=OPEN, order=OPEN, notif=OPEN
  Global: DEGRADED: payment, order, notification
After failure 5: payment=OPEN, order=OPEN, notif=OPEN
  Global: DEGRADED: payment, order, notification

Coordinated Recovery

import time
import random

class CoordinatedRecovery:
    def __init__(self, global_state):
        self.global_state = global_state
        self.recovery_group = []
        self.recovery_in_progress = False

    def mark_for_recovery(self, *circuits):
        self.recovery_group = list(circuits)

    def attempt_recovery(self):
        if self.recovery_in_progress:
            return False
        self.recovery_in_progress = True

        health_results = {}
        for circuit in self.recovery_group:
            healthy = random.random() < 0.7
            health_results[circuit] = healthy
            print(f"Health check {circuit}: {'PASS' if healthy else 'FAIL'}")

        all_healthy = all(health_results.values())
        if all_healthy:
            for circuit in self.recovery_group:
                self.global_state.record_success(circuit)
            print("All circuits healthy. Recovery complete.")
        else:
            failed = [c for c, h in health_results.items() if not h]
            print(f"Recovery deferred. Still failing: {failed}")

        self.recovery_in_progress = False
        return all_healthy

gcs = GlobalCircuitState()
gcs.register('payment')
gcs.register('order', depends_on=['payment'])
gcs.register('notification', depends_on=['payment'])

for i in range(3):
    gcs.record_failure('payment')

recovery = CoordinatedRecovery(gcs)
recovery.mark_for_recovery('payment', 'order', 'notification')

for attempt in range(3):
    print(f"\nRecovery attempt {attempt + 1}:")
    if recovery.attempt_recovery():
        break

Expected output:

Cascade: order opened due to payment failure
Cascade: notification opened due to payment failure

Recovery attempt 1:
Health check payment: PASS
Health check order: FAIL
Health check notification: PASS
Recovery deferred. Still failing: ['order']

Recovery attempt 2:
Health check payment: PASS
Health check order: PASS
Health check notification: PASS
All circuits healthy. Recovery complete.

Cascading Failure Prevention

class CascadingFailurePrevention:
    def __init__(self, max_cascading_circuits=3):
        self.max_cascading = max_cascading_circuits
        self.open_count = 0

    def should_open(self, circuit_name, reason):
        if self.open_count >= self.max_cascading:
            print(f"BLOCKED: {circuit_name} would exceed max cascading circuits")
            return False
        print(f"ALLOWED: Opening {circuit_name} due to {reason}")
        self.open_count += 1
        return True

    def on_recovery(self, circuit_name):
        self.open_count = max(0, self.open_count - 1)
        print(f"Recovered: {circuit_name}. Open count: {self.open_count}")

prevention = CascadingFailurePrevention(max_cascading_circuits=2)

for circuit in ['payment', 'order', 'shipping', 'notification']:
    prevention.should_open(circuit, "dependency failure")

Expected output:

ALLOWED: Opening payment due to dependency failure
ALLOWED: Opening order due to dependency failure
BLOCKED: shipping would exceed max cascading circuits
BLOCKED: notification would exceed max cascading circuits

Common Mistakes

• No cascading limit -- one service failure can open every downstream circuit, taking down the entire system. Set a maximum number of cascading circuit openings per failure event to contain Blast Radius.
• Synchronous state synchronization across regions -- waiting for circuit state consensus across regions adds latency and reduces availability. Use asynchronous state propagation with Conflict Resolution based on local observations.
• Coordinated recovery without exponential backoff -- all circuits probing the recovering service simultaneously creates a thundering herd. Stagger recovery probes with jitter: circuit 1 probes after 5s, circuit 2 after 7s, circuit 3 after 11s.
• Ignoring circuit state staleness -- a circuit state that is 60 seconds old may be irrelevant. Attach timestamps to state announcements and ignore states older than a configurable threshold (e.g., 10 seconds for fast-changing systems).
• Over-orchestration -- not every circuit needs coordination. Only orchestrate circuits that share a common dependency. Independent circuits should operate independently to maintain isolation and avoid unnecessary coupling.

Practice Questions

1. Why might you want coordinated circuit breaker state across services?
2. What is the thundering herd problem in circuit breaker recovery?
3. How do you prevent cascading failures when orchestrating circuit breakers?
4. When should you NOT orchestrate circuit breakers?
5. How do you handle state staleness in distributed circuit breaker coordination?

Challenge

Build a circuit orchestrator for an e-commerce system: (1) circuits: payment, inventory, shipping, notifications, recommendations, (2) dependency graph: order depends on payment + inventory, checkout depends on order + notification, recommendations is independent, (3) cascade logic: when payment opens, cascade to order and checkout but not recommendations, (4) coordinated recovery: only attempt recovery when all dependencies are healthy, (5) max cascade limit of 2 circuits per failure event, (6) asynchronous state propagation with 5-second max staleness, (7) degraded mode: when checkout is degraded, serve cached product data and show "checkout unavailable" banner.

FAQ

Mini Project

Build a circuit orchestrator: (1) dependency graph configuration defining circuit relationships, (2) global state manager that tracks all circuit states with timestamps, (3) cascade logic with configurable maximum cascade depth, (4) coordinated recovery with staggered probes and exponential backoff, (5) degraded mode definitions per circuit group (checkout, catalog, account), (6) state propagation via message bus with 5-second staleness threshold, (7) monitoring dashboard showing circuit groups and global state, (8) override endpoint for operators to manually open or close circuit groups.

What's Next

Continue with Pattern Comparison to compare circuit breakers with other resilience patterns. Then explore Zero-Downtime Deployments for rolling updates with circuit breakers.