Distributed Tracing with Circuit Breakers — Complete Observability Guide

In this tutorial, you will learn about Distributed Tracing with Circuit Breakers. We cover key concepts, practical examples, and best practices to help you master this topic.

Distributed tracing with circuit breakers provides end-to-end visibility into circuit state transitions across Microservices by instrumenting circuit breaker calls with OpenTelemetry spans, tags, and events.

flowchart LR
    A[Service A] -->|Span: CB Check| B{Circuit Open?}
    B -->|No| C[Service B]
    B -->|Yes| D[Fallback]
    C -->|Span: CB Record| E{Success?}
    E -->|Yes| F[Return]
    E -->|No| G[Count Failure]
    G --> H{Threshold?}
    H -->|Yes| I[Open Circuit]
    I --> J[Emit Event]
    style J fill:#f90,color:#fff

What You'll Learn

• OpenTelemetry instrumentation for circuit breakers
• Span context propagation through circuit breaker calls
• Circuit state as span attributes
• Trace-based failure analysis
• Distributed circuit breaker visualization

Why It Matters

Without distributed tracing, circuit breaker state changes are invisible across service boundaries. When Service A opens its circuit breaker for Service B, you need to see the full request path: which request triggered the open, what the downstream response was, and how the fallback behaved. Distributed tracing connects these dots.

Real-World Use

DodaTech uses OpenTelemetry-instrumented circuit breakers across all microservices. When the payment service circuit opens, a trace shows the exact sequence: 5 failed payment requests with 503 responses, the circuit transition event, fallback execution with cached data, and recovery probe results. Mean time to resolution dropped from 45 minutes to 12 minutes.

OpenTelemetry Instrumentation

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import time, random

tracer = trace.get_tracer(__name__)

class TracedCircuitBreaker:
    def __init__(self, name, threshold=3, reset_timeout=30):
        self.name = name
        self.threshold = threshold
        self.reset_timeout = reset_timeout
        self.failures = 0
        self.state = 'CLOSED'
        self.state_version = 0
        self.last_failure = 0

    def call(self, fn, *args, **kwargs):
        with tracer.start_as_current_span(f"circuit_breaker.{self.name}") as span:
            span.set_attribute("circuit.name", self.name)
            span.set_attribute("circuit.state", self.state)
            span.set_attribute("circuit.version", self.state_version)

            if self.state == 'OPEN':
                if time.time() - self.last_failure > self.reset_timeout:
                    self.state = 'HALF_OPEN'
                    self.state_version += 1
                    span.add_event("circuit.transition", {"from": "OPEN", "to": "HALF_OPEN"})
                else:
                    span.add_event("circuit.blocked", {"reason": "open"})
                    span.set_status(Status(StatusCode.ERROR, "circuit open"))
                    raise Exception("Circuit breaker is open")

            try:
                result = fn(*args, **kwargs)
                self.failures = 0
                if self.state in ('HALF_OPEN', 'OPEN'):
                    old_state = self.state
                    self.state = 'CLOSED'
                    self.state_version += 1
                    span.add_event("circuit.transition", {"from": old_state, "to": "CLOSED"})
                span.set_status(Status(StatusCode.OK))
                return result
            except Exception as e:
                self.failures += 1
                self.last_failure = time.time()
                span.record_exception(e)
                span.set_attribute("circuit.failure_count", self.failures)
                if self.failures >= self.threshold:
                    old_state = self.state
                    self.state = 'OPEN'
                    self.state_version += 1
                    span.add_event("circuit.transition", {"from": old_state, "to": "OPEN"})
                span.set_status(Status(StatusCode.ERROR, str(e)))
                raise

cb = TracedCircuitBreaker("payment-service", threshold=3)
for i in range(5):
    try:
        cb.call(lambda: (_ for _ in ()).throw(Exception("timeout")))
    except Exception:
        print(f"Request {i+1}: failed, circuit state={cb.state}")

Expected output:

Request 1: failed, circuit state=CLOSED
Request 2: failed, circuit state=CLOSED
Request 3: failed, circuit state=OPEN
Request 4: failed, circuit state=OPEN
Request 5: failed, circuit state=OPEN

Trace-Based Dependency Graph

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
import json

trace.set_tracer_provider(TracerProvider())
span_processor = SimpleSpanProcessor(ConsoleSpanExporter())
trace.get_tracer_provider().add_span_processor(span_processor)

tracer = trace.get_tracer("circuit-tracer")

class DependencyTracker:
    def __init__(self):
        self.dependencies = {}

    def record_call(self, service, success, circuit_state):
        key = (service, circuit_state)
        if key not in self.dependencies:
            self.dependencies[key] = {'calls': 0, 'failures': 0}
        self.dependencies[key]['calls'] += 1
        if not success:
            self.dependencies[key]['failures'] += 1

    def get_health_report(self):
        report = []
        for (service, state), stats in self.dependencies.items():
            report.append({
                'service': service,
                'circuit_state': state,
                'total_calls': stats['calls'],
                'failure_rate': round(stats['failures'] / stats['calls'], 2)
            })
        return report

tracker = DependencyTracker()
tracker.record_call("payment-service", True, "CLOSED")
tracker.record_call("payment-service", False, "CLOSED")
tracker.record_call("payment-service", False, "CLOSED")
tracker.record_call("payment-service", False, "CLOSED")
tracker.record_call("payment-service", False, "OPEN")

report = tracker.get_health_report()
for entry in report:
    print(f"{entry['service']} ({entry['circuit_state']}): {entry['total_calls']} calls, {entry['failure_rate']*100}% failures")

Expected output:

payment-service (CLOSED): 4 calls, 75.0% failures
payment-service (OPEN): 1 calls, 100.0% failures

Common Mistakes

• Not propagating trace context through circuit breaker calls -- without context propagation, circuit breaker spans appear as orphaned traces. Always pass trace headers (traceparent, tracestate) through the circuit breaker to the downstream service and fallback.
• Missing circuit state in span attributes -- without circuit state tags, you cannot filter traces by circuit status. Always add circuit.name, circuit.state, and circuit.version as span attributes for trace-level filtering.
• Recording circuit breaker events but not the cause -- recording "circuit opened" without the triggering exception makes debugging impossible. Record the exception details, response status code, and response time as span events.
• No sampling Strategy for circuit breaker traces -- high-traffic services generate millions of spans. Use head-based sampling with circuit breaker state as a sampling decision factor: always sample traces where the circuit transitions state.
• Ignoring trace correlation between upstream and downstream circuit breakers -- nested circuit breakers create a chain of state transitions. Use a trace ID to correlate the upstream circuit opening because the downstream circuit opened, versus the upstream circuit opening for a different reason.

Practice Questions

1. How does OpenTelemetry instrumentation help diagnose circuit breaker issues?
2. What span attributes should every circuit breaker call include?
3. How do you propagate trace context through circuit breaker fallbacks?
4. What sampling strategy is appropriate for circuit breaker traces?
5. How do you visualize distributed circuit breaker state across microservices?

Challenge

Build a distributed tracing system for a 3-service architecture: (1) Service A calls Service B which calls Service C, (2) each service has a circuit breaker instrumented with OpenTelemetry, (3) when Service C fails, the circuit breaker state change propagates as a span event visible in all parent traces, (4) Jaeger or Zipkin visualization shows the full circuit breaker chain, (5) a dashboard displays the current circuit state for each service with trace links.

FAQ

Mini Project

Build a fully instrumented circuit breaker demo: (1) 3 microservices with circuit breakers instrumented using OpenTelemetry Python SDK, (2) each circuit breaker records spans with circuit name, state, failure count, and threshold attributes, (3) span events for every state transition with cause and timestamp, (4) trace context propagation across HTTP calls and fallback execution, (5) Jaeger backend receiving spans with a dashboard showing circuit states per service, (6) alerting rule that triggers on circuit.state=OPEN span events with trace ID in the alert payload, (7) a health report generator that uses trace data to calculate circuit availability per service over time Windows.

What's Next

Continue with Configuration Management to learn dynamic configuration for circuit breakers. Then explore Observability for advanced monitoring patterns.