Saga Pattern with Circuit Breakers — Resilience in Distributed Transactions

In this tutorial, you will learn about Saga Pattern with Circuit Breakers. We cover key concepts, practical examples, and best practices to help you master this topic.

Saga pattern with circuit breakers provides resilience for distributed transactions by triggering compensating transactions when a circuit opens mid-saga, preventing partial updates from leaving the system in an inconsistent state.

flowchart LR
    Saga[Saga Orchestrator] -->|Step 1| CB1[Circuit: Payment]
    CB1 -->|Open| Comp1[Compensate: None]
    Saga -->|Step 2| CB2[Circuit: Inventory]
    CB2 -->|Open| Comp2[Compensate: Refund Payment]
    Saga -->|Step 3| CB3[Circuit: Shipping]
    CB3 -->|Open| Comp3[Compensate: Restock Inventory + Refund]
    style CB2 fill:#f90,color:#fff

What You'll Learn

• Compensating transactions on circuit open
• Saga orchestration with circuit state awareness
• Choreography-based saga circuit breaking
• Circuit-aware saga rollback strategies
• Partial saga recovery on circuit recovery

Why It Matters

Without circuit-aware sagas, a circuit opening mid-Transaction leaves the system in an inconsistent state. The payment was charged but the inventory was not deducted. Saga compensation ensures that when a circuit opens, all previous steps are rolled back atomically.

Real-World Use

DodaTech's order processing saga has 4 steps: payment authorization, inventory reservation, shipping scheduling, and notification. If the inventory service circuit opens after payment succeeds, the saga orchestrator triggers a payment refund compensation. The order is marked as "failed" and the user receives a cancellation notice.

Saga Orchestrator with Circuit Awareness

import time
import random

class CircuitAwareSaga:
    def __init__(self, saga_id):
        self.saga_id = saga_id
        self.steps = []
        self.completed = []
        self.failed = False

    def add_step(self, name, execute_fn, compensate_fn, circuit_check_fn=None):
        self.steps.append({
            'name': name,
            'execute': execute_fn,
            'compensate': compensate_fn,
            'circuit_check': circuit_check_fn or (lambda: True)
        })

    def execute(self):
        print(f"Saga {self.saga_id} starting...")
        for step in self.steps:
            circuit_ok = step['circuit_check']()
            if not circuit_ok:
                print(f"  Circuit OPEN for {step['name']}. Starting compensation...")
                self._compensate()
                self.failed = True
                return False

            try:
                result = step['execute']()
                self.completed.append(step)
                print(f"  Step '{step['name']}' succeeded")
            except Exception as e:
                print(f"  Step '{step['name']}' failed: {e}")
                self._compensate()
                self.failed = True
                return False

        print(f"Saga {self.saga_id} completed successfully")
        return True

    def _compensate(self):
        print("  Compensation phase:")
        for step in reversed(self.completed):
            try:
                step['compensate']()
                print(f"    Compensated: {step['name']}")
            except Exception as e:
                print(f"    Compensation failed for {step['name']}: {e}")

circuit_states = {"payment": "CLOSED", "inventory": "OPEN", "shipping": "CLOSED"}

def check_circuit(name):
    return circuit_states.get(name) == 'CLOSED'

saga = CircuitAwareSaga("order-123")
saga.add_step("charge_payment",
    lambda: print("    Payment charged: $29.99"),
    lambda: print("    Payment refunded: $29.99"),
    lambda: check_circuit("payment"))
saga.add_step("reserve_inventory",
    lambda: print("    Inventory reserved: SKU-456"),
    lambda: print("    Inventory released: SKU-456"),
    lambda: check_circuit("inventory"))
saga.add_step("schedule_shipping",
    lambda: print("    Shipping scheduled: UPS Ground"),
    lambda: print("    Shipping cancelled: UPS Ground"),
    lambda: check_circuit("shipping"))

saga.execute()
print(f"Saga failed: {saga.failed}")

Expected output:

Saga order-123 starting...
  Step 'charge_payment' succeeded
  Circuit OPEN for reserve_inventory. Starting compensation...
  Compensation phase:
    Compensated: charge_payment
Saga failed: True

Circuit-Aware Saga Rollback Strategies

import time
import random

class SagaRollbackStrategy:
    def __init__(self):
        self.strategies = {}

    def register(self, step_name, rollback_type, max_retries=3):
        self.strategies[step_name] = {
            'type': rollback_type,
            'max_retries': max_retries
        }

    def execute_rollback(self, step_name, compensate_fn):
        strategy = self.strategies.get(step_name, {'type': 'immediate', 'max_retries': 3})
        print(f"  Rollback strategy for '{step_name}': {strategy['type']}")

        if strategy['type'] == 'immediate':
            for attempt in range(strategy['max_retries']):
                try:
                    compensate_fn()
                    print(f"    Compensated (attempt {attempt+1})")
                    return True
                except Exception as e:
                    if attempt == strategy['max_retries'] - 1:
                        print(f"    Failed after {strategy['max_retries']} attempts")
                        return False

        elif strategy['type'] == 'deferred':
            print(f"    Queued for deferred compensation")
            return True

        elif strategy['type'] == 'circuit-aware':
            for attempt in range(strategy['max_retries']):
                if random.random() < 0.3:
                    print(f"    Circuit still open, deferring (attempt {attempt+1})")
                    continue
                compensate_fn()
                print(f"    Compensated after circuit recovery (attempt {attempt+1})")
                return True
            print(f"    Circuit did not recover, manual intervention required")
            return False

        return False

strategies = SagaRollbackStrategy()
strategies.register("charge_payment", "immediate", 3)
strategies.register("reserve_inventory", "circuit-aware", 5)
strategies.register("schedule_shipping", "deferred", 1)

for step in ["charge_payment", "reserve_inventory", "schedule_shipping"]:
    strategies.execute_rollback(step, lambda s=step: print(f"      Compensating {s}"))

Expected output:

  Rollback strategy for 'charge_payment': immediate
    Compensated (attempt 1)
  Rollback strategy for 'reserve_inventory': circuit-aware
    Circuit still open, deferring (attempt 1)
    Circuit still open, deferring (attempt 2)
    Compensated after circuit recovery (attempt 3)
  Rollback strategy for 'schedule_shipping': deferred
    Queued for deferred compensation

Common Mistakes

• No compensation for circuit-open failures -- when a circuit opens mid-saga, the saga assumes the step failed without compensating previous steps. Always add compensation handlers for every saga step, triggered by circuit open events.
• Synchronous compensation waiting for circuit recovery -- waiting for a circuit to close before compensating blocks the saga. Queue compensation requests and execute them asynchronously when the circuit recovers.
• Compensating transactions calling the same failed service -- if the payment circuit is open, a refund compensation that also calls the payment service will also fail. Compensation paths must use different communication channels or cached data.
• No idempotency in compensation handlers -- circuit breaker recovery may trigger compensation multiple times. Ensure compensation handlers are idempotent: calling refund twice should not refund twice.
• Ignoring partial sagas during circuit recovery -- when a circuit closes after being open for 5 minutes, there may be stale sagas waiting for compensation. Implement a saga recovery scan that detects incomplete sagas and triggers pending compensations.

Practice Questions

1. Why do sagas need circuit breaker awareness?
2. What is a compensating transaction in saga context?
3. How does a circuit-aware saga orchestrator differ from a regular orchestrator?
4. What compensation strategies are available for circuit-open failures?
5. How do you handle saga recovery after a circuit closes?

Challenge

Build a circuit-aware saga framework: (1) saga orchestrator that checks circuit state before each step, (2) automatic compensation on circuit open with configurable Strategy per step (immediate, deferred, circuit-aware), (3) idempotent compensation handlers that can be called multiple times safely, (4) compensation queue that retries failed compensations when circuits recover, (5) saga state persistence in PostgreSQL with status tracking (PENDING, COMPLETED, COMPENSATING, COMPENSATED, FAILED), (6) saga recovery scanner that detects incomplete sagas and triggers pending compensations, (7) dashboard showing active sagas, pending compensations, and circuit state impact on saga progress.

FAQ

Mini Project

Build a saga framework with circuit breaker integration: (1) saga orchestrator that checks circuit state for each step, (2) 3 saga types: order (payment, inventory, shipping), refund (refund, restock, notify), and account (create, notify, provision), (3) per-step compensation strategies: immediate retry (3 attempts), deferred (queue for background processing), circuit-aware (retry on circuit recovery), (4) saga state machine: PENDING -> EXECUTING -> COMPLETED / COMPENSATING -> COMPENSATED / FAILED, (5) compensation queue with exponential backoff (1s, 2s, 4s, 8s, max 60s), (6) saga recovery scanner that runs every 60 seconds to detect and retry failed compensations, (7) dashboard showing saga status, circuit state at each step, and compensation queue depth.

What's Next

Continue with Production Readiness for the production readiness checklist. Then explore Best Practices for a complete summary of circuit breaker patterns.