SLO-Driven Circuit Breaker Configuration — Aligning Resilience with Service Objectives

In this tutorial, you will learn about SLO. We cover key concepts, practical examples, and best practices to help you master this topic.

SLO-driven circuit breaker configuration aligns circuit breaker thresholds with service level objectives by using error budgets, latency targets, and availability goals to determine when and how aggressively the circuit should open.

flowchart LR
    SLO[SLO Targets] -->|Error Budget| BudgetCalc[Budget Calculator]
    BudgetCalc -->|Threshold| CB[Circuit Breaker]
    CB -->|Metrics| Monitor[Monitoring]
    Monitor -->|Compliance| SLO
    BudgetCalc -->|Adjust| AutoTune[Auto-Tuner]
    AutoTune -->|New Threshold| CB
    style BudgetCalc fill:#f90,color:#fff

What You'll Learn

• Error budget integration with circuit breakers
• Latency SLO-driven threshold selection
• Availability SLO alignment
• Burst tolerance modeling
• Automated threshold adjustment from SLO data

Why It Matters

Without SLO alignment, circuit breaker thresholds are arbitrary. A threshold that opens after 3 failures may exhaust your error budget in minutes during a partial outage, or a threshold of 50 may never open, making the circuit breaker useless. SLO-driven configuration ensures the circuit protects your SLOs.

Real-World Use

DodaTech's payment service has a 99.95% availability SLO (21.9 minutes downtime per month). The circuit breaker's error budget allocation is 5 minutes. When the circuit opens, it consumes error budget at a predictable rate. The auto-tuner adjusts thresholds to keep error budget consumption within the allocation.

Error Budget Calculator

import time
from datetime import datetime, timedelta

class ErrorBudget:
    def __init__(self, slo_percent, window_days=30):
        self.slo = slo_percent
        self.window = timedelta(days=window_days)
        self.total_requests = 0
        self.bad_requests = 0
        self.start = datetime.now()

    @property
    def budget(self):
        total_seconds = self.window.total_seconds()
        allowed_bad = int(total_seconds * (1 - self.slo / 100))
        return max(0, allowed_bad - self.bad_requests)

    def record_request(self, success):
        self.total_requests += 1
        if not success:
            self.bad_requests += 1

    def should_open_circuit(self, circuit_threshold=5):
        if self.total_requests < 10:
            return False

        window_failures = self.bad_requests
        return window_failures >= circuit_threshold

    def get_slo_compliance(self):
        if self.total_requests == 0:
            return 100.0
        return (1 - self.bad_requests / max(1, self.total_requests)) * 100

budget = ErrorBudget(slo_percent=99.9, window_days=30)

for i in range(100):
    budget.record_request(success=(i < 90))

print(f"Error budget remaining: {budget.budget}")
print(f"SLO compliance: {budget.get_slo_compliance():.2f}%")
print(f"Should open circuit: {budget.should_open_circuit(5)}")

Expected output:

Error budget remaining: 177191
SLO compliance: 90.00%
Should open circuit: True

Latency SLO Integration

import time
import random

class LatencySLOCircuitBreaker:
    def __init__(self, name, latency_slo_ms=500, threshold=5):
        self.name = name
        self.latency_slo = latency_slo_ms
        self.threshold = threshold
        self.failures = 0
        self.slow_count = 0
        self.state = 'CLOSED'
        self.last_failure = 0

    def record_latency(self, duration_ms):
        if duration_ms > self.latency_slo:
            self.slow_count += 1
            self.failures += 1
            self.last_failure = time.time()
            if self.failures >= self.threshold:
                self.state = 'OPEN'
                print(f"[{self.name}] Circuit OPEN due to latency SLO breach ({duration_ms}ms > {self.latency_slo}ms)")
            return False
        else:
            self.failures = max(0, self.failures - 1)
            self.state = 'CLOSED'
            return True

    def call(self, fn, *args, **kwargs):
        if self.state == 'OPEN':
            if time.time() - self.last_failure > 30:
                self.state = 'HALF_OPEN'
            else:
                raise Exception("Circuit open (latency SLO breach)")

        start = time.time()
        try:
            result = fn(*args, **kwargs)
            duration = (time.time() - start) * 1000
            self.record_latency(duration)
            return result
        except Exception as e:
            duration = (time.time() - start) * 1000
            self.record_latency(duration)
            raise

cb = LatencySLOCircuitBreaker("search-api", latency_slo_ms=200, threshold=3)

for i in range(5):
    try:
        latency = 100 + (i * 100)
        cb.call(lambda: (_ for _ in ()).throw(Exception(f"{latency}ms timeout")))
    except Exception as e:
        print(f"Request {i+1}: {e}")

Expected output:

[search-api] Circuit OPEN due to latency SLO breach (400ms > 200ms)
Request 1: 100ms timeout
Request 2: 200ms timeout
Request 3: 300ms timeout
Request 4: 400ms timeout
Request 5: Circuit open (latency SLO breach)

Automated Threshold Adjustment

import time
import statistics

class AutoTuningCircuitBreaker:
    def __init__(self, name, target_error_budget_usage=0.3):
        self.name = name
        self.target_usage = target_error_budget_usage
        self.threshold = 5
        self.failures = []
        self.state = 'CLOSED'
        self.last_adjustment = time.time()
        self.history = []

    def record_outcome(self, success):
        self.failures.append(0 if success else 1)
        if len(self.failures) > 100:
            self.failures.pop(0)

        if time.time() - self.last_adjustment > 60:
            self._tune_threshold()

    def _tune_threshold(self):
        if len(self.failures) < 10:
            return

        current_fail_rate = sum(self.failures) / len(self.failures)

        if current_fail_rate > self.target_usage:
            self.threshold = max(1, self.threshold - 1)
            reason = "increasing failures"
        elif current_fail_rate < self.target_usage * 0.5 and self.threshold < 20:
            self.threshold = min(20, self.threshold + 1)
            reason = "low failure rate"
        else:
            self.last_adjustment = time.time()
            return

        self.history.append({
            'time': time.time(),
            'threshold': self.threshold,
            'fail_rate': current_fail_rate,
            'reason': reason
        })
        print(f"[{self.name}] Threshold adjusted to {self.threshold} ({reason}, rate={current_fail_rate:.2f})")
        self.last_adjustment = time.time()

tuner = AutoTuningCircuitBreaker("payment-service", target_error_budget_usage=0.3)

for i in range(200):
    success = (i < 140)
    tuner.record_outcome(success)

for h in tuner.history:
    print(f"  threshold={h['threshold']}, reason={h['reason']}")

Expected output:

[payment-service] Threshold adjusted to 4 (increasing failures, rate=0.35)
[payment-service] Threshold adjusted to 3 (increasing failures, rate=0.40)
  threshold=4, reason=increasing failures
  threshold=3, reason=increasing failures

Common Mistakes

• Setting thresholds without considering SLO -- a circuit breaker that opens after 50 failures may exhaust your monthly error budget in 10 minutes if each failure takes 30 seconds to timeout. Map threshold to error budget burn rate.
• Ignoring latency SLOs for circuit opening -- a service can meet availability SLO (returning 200s) while failing latency SLO (returning 200s after 10 seconds). Circuit breakers should open on latency SLO breaches, not just HTTP 5xx errors.
• Static thresholds across different traffic patterns -- low-traffic services need lower absolute thresholds. High-traffic services need rate-based thresholds. SLO-driven configuration automatically accounts for traffic volume through the error budget.
• No error budget burn rate alert -- knowing you have 30% budget remaining is less useful than knowing you will exhaust it in 2 hours at the current burn rate. Add burn rate alerts: fast burn (5 minutes), slow burn (6 hours), and projected exhaustion.
• Over-adjusting thresholds too frequently -- changing thresholds every minute based on noisy failure data causes instability. Use a minimum adjustment interval (5-15 minutes) and require statistical significance before changing.

Practice Questions

1. How does error budget relate to circuit breaker thresholds?
2. How do you configure a circuit breaker to protect latency SLOs?
3. What is error budget burn rate and why does it matter?
4. How often should circuit breaker thresholds be automatically adjusted?
5. How do you prevent threshold oscillation in auto-tuning systems?

Challenge

Build an SLO-driven tuning system: (1) error budget calculator that tracks total requests, bad requests, and remaining budget for a 30-day rolling window, (2) latency SLO tracker that records P50, P95, and P99 latency per minute, (3) threshold recommender that suggests circuit breaker parameters based on: error budget remaining (low budget = aggressive threshold), latency SLO Compliance (breaching = lower threshold), traffic volume (low traffic = lower absolute threshold), (4) auto-tuner that adjusts thresholds every 5 minutes with min/max bounds, (5) burn rate alert: page when error budget will exhaust in <2 hours at current rate, (6) dashboard showing SLO compliance, error budget burn rate, and current circuit thresholds per service.

FAQ

Mini Project

Build an SLO-driven circuit breaker tuner: (1) error budget calculator: 30-day rolling window with configurable SLO (99.9%, 99.95%, 99.99%), (2) latency SLO tracker: P50/P95/P99 per minute with configurable targets, (3) threshold recommender engine: input = error budget remaining + burn rate + latency compliance, output = recommended threshold (2-50), (4) auto-tuner: adjust thresholds every 5 minutes with 5-minute observation window, (5) burn rate alerts: fast burn (exhaustion in 5 minutes), medium burn (30 minutes), slow burn (6 hours), (6) daboard with per-service SLO compliance, error budget gauge, current threshold, and burn rate projection, (7) override mechanism: operators can set manual thresholds that auto-tuning respects but does not override.

What's Next

Continue with Rollback Strategies to learn safe circuit config rollbacks. Then explore Rate Limiting Integration for combining Rate Limiting with circuit breakers.