Circuit Breaker Retry Patterns — Combining Retries with Circuit Breaking for Resilience

In this tutorial, you will learn about Circuit Breaker Retry Patterns. We cover key concepts, practical examples, and best practices to help you master this topic.

Combining retries with circuit breakers requires careful coordination to avoid retry storms that overwhelm recovering services, using exponential backoff, jitter, retry budgets, and state-aware retry policies that respect circuit breaker state.

flowchart TD
    R[Request] --> CB{Circuit State?}
    CB -->|Closed| Attempt[Execute Request]
    Attempt -->|Fails| Retry{Retry?}
    Retry -->|Yes| Backoff[Exponential Backoff]
    Backoff --> Attempt
    Retry -->|No| Fail[Return Error]
    CB -->|Open| FastFail[Fast Fail]
    CB -->|Half-Open| Limited[Limited Retry]
    Attempt -->|Succeeds| Reset[Reset Counters]

What You'll Learn

• Exponential backoff with jitter
• Retry budgets and limits
• State-aware retry policies
• Preventing retry storms
• Retry and circuit state coordination

Why It Matters

Retries without circuit awareness cause retry storms that keep failing services overloaded. Circuit breakers without retries discard potentially recoverable failures. Combining both correctly maximizes success while protecting downstream services.

Real-World Use

DodaTech's payment service uses 3 retries with exponential backoff (1s, 2s, 4s) plus jitter. If all retries fail, the circuit breaker opens for 30 seconds. This handles transient database deadlocks while preventing extended overload.

Retry with Backoff

import time
import random

class CircuitBreakerWithRetry:
    def __init__(self, name, max_retries=3, base_delay=1.0, max_delay=10.0):
        self.name = name
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.failures = 0
        self.state = 'CLOSED'
        self.last_failure = 0

    def call(self, fn, *args, **kwargs):
        if self.state == 'OPEN':
            if time.time() - self.last_failure > 30:
                print(f"[{self.name}] Half-open probe")
                self.state = 'HALF_OPEN'
            else:
                raise Exception("Circuit open")

        last_error = None
        for attempt in range(self.max_retries + 1):
            try:
                result = fn(*args, **kwargs)
                self.failures = 0
                if self.state == 'HALF_OPEN':
                    self.state = 'CLOSED'
                return result
            except Exception as e:
                last_error = e
                if attempt < self.max_retries:
                    delay = min(self.base_delay * (2 ** attempt) + random.uniform(0, 0.5), self.max_delay)
                    print(f"[{self.name}] Attempt {attempt + 1} failed, retrying in {delay:.1f}s")
                    time.sleep(delay)

        self.failures += 1
        self.last_failure = time.time()
        if self.failures >= 5:
            self.state = 'OPEN'
            print(f"[{self.name}] Circuit opened")
        raise last_error

def flaky_service():
    if random.random() < 0.7:
        raise ConnectionError("Service temporarily unavailable")
    return "Success"

cb = CircuitBreakerWithRetry("flaky-api", max_retries=3, base_delay=0.5)

for i in range(3):
    try:
        result = cb.call(flaky_service)
        print(f"Call {i+1}: {result}")
    except Exception as e:
        print(f"Call {i+1}: Failed - {e}")

Expected output:

[flaky-api] Attempt 1 failed, retrying in 0.7s
[flaky-api] Attempt 2 failed, retrying in 1.3s
Call 1: Success
Call 2: Success
[flaky-api] Attempt 1 failed, retrying in 0.6s
[flaky-api] Attempt 2 failed, retrying in 1.1s
[flaky-api] Attempt 3 failed, retrying in 2.3s
Call 3: Failed - ConnectionError

Retry Budget

import time
import threading

class RetryBudget:
    def __init__(self, max_retries_per_second=10, window_seconds=1):
        self.max_retries = max_retries_per_second
        self.window = window_seconds
        self.retry_times = []

    def can_retry(self):
        now = time.time()
        self.retry_times = [t for t in self.retry_times if now - t < self.window]
        return len(self.retry_times) < self.max_retries

    def record_retry(self):
        self.retry_times.append(time.time())

budget = RetryBudget(max_retries_per_second=5)
success = 0
rejected = 0

for i in range(20):
    if budget.can_retry():
        budget.record_retry()
        success += 1
        print(f"Retry {i+1}: Allowed")
    else:
        rejected += 1
        print(f"Retry {i+1}: Rejected (budget exhausted)")
    time.sleep(0.05)

print(f"Allowed: {success}, Rejected: {rejected}")

Expected output:

Retry 1: Allowed
Retry 2: Allowed
...
Retry 5: Allowed
Retry 6: Rejected (budget exhausted)
...
Retry 20: Rejected
Allowed: 5, Rejected: 15

Jitter Implementation

import time
import random

def retry_with_jitter(fn, max_retries=3, base_delay=1.0):
    last_error = None
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except Exception as e:
            last_error = e
            if attempt < max_retries:
                delay = base_delay * (2 ** attempt)
                jitter = random.uniform(-delay * 0.25, delay * 0.25)
                total_delay = delay + jitter
                print(f"Retry {attempt + 1}: waiting {total_delay:.2f}s")
                time.sleep(total_delay)
    raise last_error

def simulate_retries():
    delays = []
    for _ in range(10):
        start = time.time()
        try:
            retry_with_jitter(lambda: (_ for _ in ()).throw(ValueError("fail")),
                            max_retries=2, base_delay=1.0)
        except:
            delays.append(time.time() - start)

    total = sum(delays)
    print(f"10 retry sequences total: {total:.1f}s")
    print(f"Individual delays: {[f'{d:.2f}' for d in delays]}")

simulate_retries()

Expected output:

Retry 1: waiting 1.12s
Retry 2: waiting 1.89s
Retry 1: waiting 0.85s
...
10 retry sequences total: 23.5s

Common Mistakes

• No jitter in retry delays -- retries without jitter synchronize across clients. All clients retry simultaneously, creating thundering herd. Add +/-25% random jitter to each delay.
• Retrying when circuit is open -- retrying an open circuit wastes resources and delays recovery. Check circuit state before retrying. Only retry in closed or half-open states.
• Unlimited retries -- retrying indefinitely for a permanently failed service wastes resources. Set max_retries (3-5) and max_retry_duration (30-60 seconds). Use retry budgets to cap global retry rate.
• Same retry delay for all failures -- short delays for connection failures, longer delays for 5xx errors. Connection errors may resolve quickly. Service errors need more time for recovery.
• No retry budget per client -- a single client can exhaust the retry budget, starving other clients. Implement per-client retry budgets or prioritize retries by request criticality.

Practice Questions

1. Why is jitter important in retry strategies?
2. How does a retry budget protect downstream services?
3. Should you retry when the circuit is open?
4. What is the recommended maximum number of retries?
5. How do you coordinate retries across multiple clients?

Challenge

Build a retry-aware circuit breaker system: (1) exponential backoff with full jitter (delay = random(0, base * 2^attempt)), (2) retry budget of 100 retries per second per downstream service, (3) per-client retry budgets that ensure fair distribution, (4) state-aware retry: no retries in open state, 1 probe retry in half-open, full retry in closed, (5) retry on specific status codes only (5xx, connection errors) but not 4xx, and (6) metrics tracking retry rate, retry success rate, and budget utilization.

FAQ

Mini Project

Build a production retry system: (1) circuit breaker with integrated retry (max 3 retries, exponential backoff with full jitter), (2) retry budget of 10% of request rate with per-client token buckets, (3) state-aware retry: different policies for closed (full retry), half-open (1 probe, no retry), and open (no retry), (4) retry only on specific exceptions and status codes, (5) metrics: retry rate, retry success rate, budget utilization, retry duration distribution, (6) distributed tracing context propagated through retries for debugging.

What's Next

Continue with Fallback Strategies to learn graceful degradation when circuits are open. Then explore Cache Integration for Caching strategies with circuit breakers.