Skip to content

Circuit Breaker Monitoring — Complete Implementation Guide

DodaTech Updated 2026-06-28 6 min read

In this tutorial, you will learn about Circuit Breaker Monitoring. We cover key concepts, practical examples, and best practices to help you master this topic.

Circuit breaker monitoring tracks state transitions, failure rates, probe results, and recovery patterns to provide visibility into downstream service health and circuit breaker effectiveness.

What You'll Learn

By the end of this tutorial, you will instrument circuit breakers with metrics, set up alerts for open circuits, and build dashboards to visualize circuit breaker health.

Why It Matters

An unmonitored circuit breaker is a blind spot. You need to know when circuits open, how often they transition, and whether probes are succeeding.

Real-World Use

DodaTech's monitoring dashboard shows circuit breaker states for all downstream services. An open circuit triggers an immediate alert to the on-call engineer.

Monitoring Learning Path

flowchart LR
  A[Half-Open Probes] --> B[Monitoring]
  B --> C[Metrics]
  B --> D[Alerting]
  B --> E[Dashboards]
  B --> F{You Are Here}
  style F fill:#f90,color:#fff

Circuit Breaker Metrics

Track key metrics for each circuit breaker: state, failure count, request count, and transition events.

class MetricCircuitBreaker {
  constructor(name) {
    this.name = name;
    this.state = "closed";
    this.metrics = {
      totalCalls: 0,
      successfulCalls: 0,
      failedCalls: 0,
      rejectedCalls: 0,
      stateTransitions: {
        open: 0,
        close: 0,
        halfOpen: 0
      },
      lastStateChange: null
    };
  }

  async call(fn) {
    this.metrics.totalCalls++;

    if (this.state === "open") {
      if (Date.now() < this.nextAttempt) {
        this.metrics.rejectedCalls++;
        throw new Error("Circuit open");
      }
      this.changeState("half-open");
    }

    try {
      const result = await fn();
      this.metrics.successfulCalls++;
      this.onSuccess();
      return result;
    } catch (err) {
      this.metrics.failedCalls++;
      this.onFailure();
      throw err;
    }
  }

  changeState(state) {
    this.state = state;
    this.metrics.lastStateChange = Date.now();
    if (state === "open") this.metrics.stateTransitions.open++;
    if (state === "closed") this.metrics.stateTransitions.close++;
    if (state === "half-open") this.metrics.stateTransitions.halfOpen++;
  }

  getReport() {
    return {
      name: this.name,
      state: this.state,
      metrics: this.metrics,
      health: {
        successRate: this.metrics.totalCalls > 0
          ? (this.metrics.successfulCalls / this.metrics.totalCalls * 100).toFixed(1) + "%"
          : "N/A",
        rejectionRate: this.metrics.totalCalls > 0
          ? (this.metrics.rejectedCalls / this.metrics.totalCalls * 100).toFixed(1) + "%"
          : "N/A"
      }
    };
  }
}

Prometheus Integration

Export circuit breaker metrics to Prometheus for centralized monitoring and alerting.

const prometheus = require("prom-client");

const circuitState = new prometheus.Gauge({
  name: "circuit_breaker_state",
  help: "Circuit breaker state (0=closed, 1=half-open, 2=open)",
  labelNames: ["name"]
});

const circuitTransitions = new prometheus.Counter({
  name: "circuit_breaker_transitions_total",
  help: "Total circuit breaker state transitions",
  labelNames: ["name", "to_state"]
});

const circuitRejected = new prometheus.Counter({
  name: "circuit_breaker_rejected_total",
  help: "Total rejected requests due to open circuit",
  labelNames: ["name"]
});

const circuitCallDuration = new prometheus.Histogram({
  name: "circuit_breaker_call_duration_seconds",
  help: "Duration of circuit breaker wrapped calls",
  labelNames: ["name", "result"],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5]
});

class PrometheusCircuitBreaker {
  constructor(name) {
    this.name = name;
    this.state = "closed";
    this.failureCount = 0;
    this.nextAttempt = Date.now();
    circuitState.labels(name).set(0);
  }

  async call(fn) {
    const start = Date.now();

    if (this.state === "open") {
      if (Date.now() < this.nextAttempt) {
        circuitRejected.labels(this.name).inc();
        circuitCallDuration.labels(this.name, "rejected").observe((Date.now() - start) / 1000);
        throw new Error("Open");
      }
      this.setState("half-open");
    }

    try {
      const result = await fn();
      this.failureCount = 0;
      if (this.state === "half-open") this.setState("closed");
      circuitCallDuration.labels(this.name, "success").observe((Date.now() - start) / 1000);
      return result;
    } catch (err) {
      this.failureCount++;
      if (this.state === "half-open" || this.failureCount >= 5) {
        this.setState("open");
        this.nextAttempt = Date.now() + 30000;
      }
      circuitCallDuration.labels(this.name, "failure").observe((Date.now() - start) / 1000);
      throw err;
    }
  }

  setState(state) {
    const stateValues = { closed: 0, "half-open": 1, open: 2 };
    this.state = state;
    circuitState.labels(this.name).set(stateValues[state]);
    circuitTransitions.labels(this.name, state).inc();
  }
}

Alerting Rules

Set up alerts for circuit breaker events that require human attention.

// Alert conditions:
// 1. Circuit open for more than 5 minutes
// 2. More than 10 circuit transitions in 10 minutes
// 3. High rejection rate (> 50%)

class CircuitBreakerAlert {
  constructor(name, alertFn) {
    this.name = name;
    this.alert = alertFn;
    this.transitionHistory = [];
  }

  onTransition(event) {
    this.transitionHistory.push({ time: Date.now(), ...event });
    this.cleanup();

    const recentTransitions = this.transitionHistory.filter(
      t => Date.now() - t.time < 600000
    );

    if (recentTransitions.length > 10) {
      this.alert({
        severity: "warning",
        message: `Circuit ${this.name}: ${recentTransitions.length} transitions in 10 minutes`,
        transitions: recentTransitions
      });
    }

    if (event.to === "open") {
      setTimeout(() => {
        if (this.state === "open") {
          this.alert({
            severity: "critical",
            message: `Circuit ${this.name}: open for 5+ minutes`,
            time: new Date().toISOString()
          });
        }
      }, 300000);
    }
  }

  cleanup() {
    const cutoff = Date.now() - 3600000;
    this.transitionHistory = this.transitionHistory.filter(t => t.time > cutoff);
  }
}

Common Mistakes

  1. Not monitoring circuit breaker state -- You cannot know if circuit breakers are working without monitoring. Always expose state metrics.

  2. Alerting on every state change -- State changes are normal. Alert on patterns: rapid cycling, long-open circuits, or high rejection rates.

  3. Not correlating circuit breaker events with downstream incidents -- Circuit state changes should correlate with deployment events or infrastructure issues.

  4. Using too few monitoring dimensions -- Track per-service, per-endpoint, and per-instance metrics for granular visibility.

  5. Ignoring half-open probe results -- Probe successes and failures are leading indicators of recovery or persistent problems.

Practice Questions

  1. What metrics should you track for each circuit breaker? State, call count, success count, failure count, rejection count, transition count, and call duration.

  2. How do you distinguish normal from problematic circuit behavior? Normal: occasional openings with quick recovery. Problematic: rapid cycling, extended open periods, high rejection rates.

  3. Why correlate circuit breaker events with deployments? A circuit opening immediately after a deployment indicates the deployment caused the failure.

  4. Challenge: Build a dashboard that shows circuit breaker health over time.

// Dashboard panels:
// 1. Current state (gauge per service)
// 2. Transition rate (graph over time)
// 3. Rejection rate (percentage)
// 4. Top failing services (table)
// 5. Recent state changes (log)

FAQ

What is the most important circuit breaker metric?

Current state and transition rate. A circuit that keeps opening and closing indicates an unstable service.

How do I monitor circuit breakers in production?

Export metrics to Prometheus via a /metrics endpoint. Use Grafana for dashboards and Alertmanager for alerts.

Should I monitor circuit breakers per instance or globally?

Both. Per-instance shows local issues. Global metrics show service-wide problems.

How do I track circuit breaker performance impact?

Measure request latency with and without circuit breaker rejection. Rejected requests are much faster (fail-fast).

What is a healthy circuit breaker pattern?

Mostly closed state with occasional brief openings. Rarely half-open. Probes succeed quickly when the service recovers.

Mini Project

Build a circuit breaker monitoring system with Prometheus metrics, health endpoint, and state change history.

class MonitoredCircuitBreaker {
  constructor(name) {
    this.name = name;
    this.state = "closed";
    this.failureCount = 0;
    this.successCount = 0;
    this.totalCalls = 0;
    this.rejectedCalls = 0;
    this.history = [];
    this.nextAttempt = Date.now();
  }

  async call(fn) {
    this.totalCalls++;

    if (this.state === "open") {
      if (Date.now() < this.nextAttempt) {
        this.rejectedCalls++;
        return { rejected: true, state: "open" };
      }
      this.transition("half-open");
    }

    try {
      const result = await fn();
      this.successCount++;
      this.failureCount = 0;
      if (this.state === "half-open") this.transition("closed");
      return result;
    } catch (err) {
      this.failureCount++;
      if (this.state === "half-open" || this.failureCount >= 5) {
        this.transition("open");
        this.nextAttempt = Date.now() + 30000;
      }
      throw err;
    }
  }

  transition(to) {
    this.history.push({ from: this.state, to, time: new Date().toISOString() });
    this.state = to;
  }

  getHealth() {
    return {
      name: this.name,
      state: this.state,
      uptime: this.totalCalls - this.rejectedCalls,
      rejected: this.rejectedCalls,
      total: this.totalCalls,
      recentTransitions: this.history.slice(-10)
    };
  }
}

What's Next

Now that you understand circuit breaker monitoring, explore using circuit breakers for HTTP calls. Then learn about circuit breakers for database connections.

← Previous Half-Open Probes — Complete Implementation Guide Next → HTTP Circuit Breaker — Complete Implementation Guide

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Circuit Breaker Pattern