Circuit Breaker Observability — Complete Monitoring and Visualization Guide

In this tutorial, you will learn about Circuit Breaker Observability. We cover key concepts, practical examples, and best practices to help you master this topic.

Circuit breaker observability provides real-time visibility into circuit states, transition rates, failure counts, and health scores through Prometheus metrics, structured logging, and Grafana dashboards that enable rapid incident response.

flowchart LR
    CB[Circuit Breaker] -->|Metrics| Prometheus[Prometheus]
    CB -->|Logs| ELK[ELK Stack]
    CB -->|Traces| Jaeger[Jaeger]
    Prometheus -->|Alerts| Alertmanager[Alertmanager]
    Prometheus -->|Data| Grafana[Grafana Dashboard]
    Alertmanager -->|Notify| Pager[PagerDuty/Slack]
    style Grafana fill:#f90,color:#fff

What You'll Learn

• Circuit breaker metrics and instrumentation
• Prometheus metric exposition
• Grafana dashboard design for circuits
• Structured logging for state transitions
• Circuit health scoring and trending

Why It Matters

Without observability, circuit breakers are invisible. You don't know the circuit is open until users complain about errors. Proper observability lets you see circuit state changes in real time, analyze failure trends, and set alerts before users are affected.

Real-World Use

DodaTech's circuit breaker dashboard tracks 200+ services across 5 data centers. Each circuit has a health score (0-100) based on open duration, failure rate, and recovery time. The operations team gets paged when any critical circuit stays open for more than 5 minutes.

Prometheus Metrics Exposition

import time
import random
from dataclasses import dataclass, field
from typing import List

@dataclass
class CircuitMetrics:
    name: str
    state: str = 'CLOSED'
    total_calls: int = 0
    failed_calls: int = 0
    open_count: int = 0
    half_open_count: int = 0
    total_open_duration: float = 0.0
    last_state_change: float = field(default_factory=time.time)

class InstrumentedCircuitBreaker:
    def __init__(self, name, threshold=3):
        self.metrics = CircuitMetrics(name=name)
        self.threshold = threshold
        self.failures = 0
        self.last_failure = 0

    def call(self, fn, *args, **kwargs):
        self.metrics.total_calls += 1

        if self.metrics.state == 'OPEN':
            if time.time() - self.last_failure > 30:
                self._transition_to('HALF_OPEN')
            else:
                self.metrics.failed_calls += 1
                raise Exception("Circuit open")

        try:
            result = fn(*args, **kwargs)
            self.failures = 0
            if self.metrics.state == 'HALF_OPEN':
                self._transition_to('CLOSED')
            return result
        except Exception as e:
            self.failures += 1
            self.metrics.failed_calls += 1
            self.last_failure = time.time()
            if self.failures >= self.threshold:
                self._transition_to('OPEN')
            raise

    def _transition_to(self, new_state):
        old_state = self.metrics.state
        self.metrics.state = new_state
        now = time.time()

        if old_state == 'OPEN':
            self.metrics.total_open_duration += now - self.metrics.last_state_change

        self.metrics.last_state_change = now

        if new_state == 'OPEN':
            self.metrics.open_count += 1
        elif new_state == 'HALF_OPEN':
            self.metrics.half_open_count += 1

        print(f"[METRIC] circuit_transition{{name=\"{self.metrics.name}\",from=\"{old_state}\",to=\"{new_state}\"}} 1")

    def get_prometheus_metrics(self):
        return f"""
# HELP circuit_state Current circuit breaker state (0=CLOSED, 1=HALF_OPEN, 2=OPEN)
# TYPE circuit_state gauge
circuit_state{{name="{self.metrics.name}"}} {['0', '1', '2'][['CLOSED', 'HALF_OPEN', 'OPEN'].index(self.metrics.state)]}
# HELP circuit_calls_total Total calls to circuit breaker
# TYPE circuit_calls_total counter
circuit_calls_total{{name="{self.metrics.name}"}} {self.metrics.total_calls}
# HELP circuit_failures_total Total failed calls
# TYPE circuit_failures_total counter
circuit_failures_total{{name="{self.metrics.name}"}} {self.metrics.failed_calls}
# HELP circuit_open_duration_seconds Total time circuit has been open
# TYPE circuit_open_duration_seconds counter
circuit_open_duration_seconds{{name="{self.metrics.name}"}} {self.metrics.total_open_duration:.2f}
"""

cb = InstrumentedCircuitBreaker("payment-service", threshold=3)

for i in range(5):
    try:
        cb.call(lambda: (_ for _ in ()).throw(Exception("error")))
    except Exception:
        pass
    time.sleep(0.1)

print(cb.get_prometheus_metrics())

Expected output:

[METRIC] circuit_transition{name="payment-service",from="CLOSED",to="OPEN"} 1

# HELP circuit_state Current circuit breaker state (0=CLOSED, 1=HALF_OPEN, 2=OPEN)
# TYPE circuit_state gauge
circuit_state{name="payment-service"} 2
# HELP circuit_calls_total Total calls to circuit breaker
# TYPE circuit_calls_total counter
circuit_calls_total{name="payment-service"} 5
# HELP circuit_failures_total Total failed calls
# TYPE circuit_failures_total counter
circuit_failures_total{name="payment-service"} 5
# HELP circuit_open_duration_seconds Total time circuit has been open
# TYPE circuit_open_duration_seconds counter
circuit_open_duration_seconds{name="payment-service"} 0.00

Structured Logging for Circuit Transitions

import json
import time

class StructuredCircuitLogger:
    def __init__(self, service):
        self.service = service

    def log_transition(self, circuit_name, from_state, to_state, reason, context=None):
        entry = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "level": "WARN" if to_state == "OPEN" else "INFO",
            "service": self.service,
            "circuit": circuit_name,
            "event": "circuit_transition",
            "from": from_state,
            "to": to_state,
            "reason": reason,
            "duration_open_ms": 0,
            "failure_count": 0,
            "trace_id": context.get("trace_id", "") if context else ""
        }
        return json.dumps(entry)

    def log_call(self, circuit_name, result, duration_ms):
        entry = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "level": "ERROR" if result == "failure" else "INFO",
            "service": self.service,
            "circuit": circuit_name,
            "event": "circuit_call",
            "result": result,
            "duration_ms": duration_ms
        }
        return json.dumps(entry)

logger = StructuredCircuitLogger("payment-service")
print(logger.log_transition("payment-db", "CLOSED", "OPEN",
                            "5 consecutive failures", {"trace_id": "abc123"}))
print(logger.log_call("payment-db", "failure", 2500))

Expected output:

{"timestamp": "2026-06-28T00:00:00Z", "level": "WARN", "service": "payment-service", "circuit": "payment-db", "event": "circuit_transition", "from": "CLOSED", "to": "OPEN", "reason": "5 consecutive failures", "duration_open_ms": 0, "failure_count": 0, "trace_id": "abc123"}
{"timestamp": "2026-06-28T00:00:00Z", "level": "ERROR", "service": "payment-service", "circuit": "payment-db", "event": "circuit_call", "result": "failure", "duration_ms": 2500}

Common Mistakes

• No circuit breaker metrics at all -- the most common mistake. Circuit breakers become invisible without metrics. Always expose circuit state, failure count, and transition events as Prometheus metrics or equivalent.
• Only tracking circuit state without failure rate -- knowing the circuit is open is useful, but knowing the failure rate before it opened helps with tuning. Track Sliding Window failure rate as a separate metric.
• No alerting on state transitions -- open circuits detected by user complaints are already too late. Set up alerts on circuit.state == OPEN for critical services with 1-minute evaluation Windows.
• Logging without structured format -- unstructured log messages like "Circuit opened" cannot be parsed by log aggregation tools. Use JSON-structured logs with circuit name, from_state, to_state, reason, and service name as separate fields.
• Not correlating circuit metrics with other signals -- a circuit opening because of a database failure should correlate with database latency metrics. Build dashboards that overlay circuit states with CPU, memory, latency, and error rates.

Practice Questions

1. What Prometheus metrics should every circuit breaker expose?
2. How do you set up alerting for circuit breaker state changes?
3. What fields should structured circuit logs include?
4. How do you build a circuit breaker health score?
5. How do circuit metrics correlate with other observability signals?

Challenge

Build a complete observability stack: (1) circuit breaker library that exposes Prometheus metrics (state gauge, call counter, failure counter, open duration, transition events), (2) structured JSON logging with trace ID correlation, (3) Grafana dashboard with: circuit state timeline per service, failure rate heatmap, open duration distribution, transition event log panel, (4) alerting rules: critical service circuit open for >5 minutes, any circuit transitioning 5+ times in 10 minutes (flapping), failure rate >50% in 5-minute window, (5) health score formula: 100 - (open_duration_ratio * 30 + failure_rate * 40 + flapping_penalty * 30), (6) Slack alerts with circuit name, current state, duration, and Grafana dashboard link.

FAQ

Mini Project

Build a circuit breaker observability framework: (1) metrics collector that tracks: state gauge, total calls, failed calls, open duration, transition events, failure rate per time window, (2) Prometheus /metrics endpoint with proper HELP/TYPE comments, (3) structured JSON logger with correlation ID from trace context, (4) Grafana dashboard with 6 panels: circuit state timeline, failure rate heatmap, call volume, open duration distribution, recent transition events table, health score gauge per service, (5) alertmanager config for circuit open, flapping, and high failure rate alerts, (6) Slack Webhook integration sending transition notifications with service name, circuit name, old/new state, and duration, (7) health score API endpoint returning score per service with breakdown.

What's Next

Continue with SLO-Driven Configuration to learn SLO-aligned circuit tuning. Then explore Rollback Strategies for safe circuit config rollbacks.