Predictive Circuit Breaking — ML-Driven Failure Prediction for Proactive Resilience

In this tutorial, you will learn about Predictive Circuit Breaking. We cover key concepts, practical examples, and best practices to help you master this topic.

Predictive circuit breaking uses machine learning models to forecast service degradation before failures occur, analyzing latency trends, error rate acceleration, resource utilization patterns, and historical failure signatures to proactively open circuits.

flowchart LR
    Data[Historical Metrics] --> Train[ML Model Training]
    Train --> Model[Prediction Model]
    Live[Live Metrics] --> Model
    Model -->|Failure Predicted| CB[Open Circuit]
    Model -->|Healthy| Normal[Normal Operation]
    CB -->|Prevented Failure| Feedback[Success Feedback]
    Feedback --> Model

What You'll Learn

• ML-based failure prediction
• Latency trend analysis
• Error rate acceleration detection
• Time-series anomaly detection
• Proactive circuit breaker activation

Why It Matters

Reactive circuit breakers only respond after failures occur. By that time, some users have already experienced errors. Predictive circuit breakers anticipate failures and open preventively, reducing error rates to near zero during predictable degradation patterns.

Real-World Use

DodaTech's predictive circuit breaker reduced user-facing errors by 95%. The ML model detects when the payment service's latency acceleration exceeds 2x in 30 seconds, opening the circuit 10-15 seconds before timeouts would occur. Users see a fallback instead of a timeout error.

Trend Analysis

import time
import statistics
from collections import deque

class TrendAnalyzer:
    def __init__(self, window_size=50):
        self.window_size = window_size
        self.latencies = deque(maxlen=window_size)
        self.errors = deque(maxlen=window_size)

    def add_observation(self, latency, is_error):
        self.latencies.append(latency)
        self.errors.append(1 if is_error else 0)

    def get_latency_trend(self):
        if len(self.latencies) < 10:
            return 0

        recent = list(self.latencies)[-10:]
        older = list(self.latencies)[:-10]
        if len(older) < 10:
            return 0

        recent_avg = statistics.mean(recent)
        older_avg = statistics.mean(older)

        if older_avg == 0:
            return 0
        return (recent_avg - older_avg) / older_avg

    def get_error_acceleration(self):
        if len(self.errors) < 20:
            return 0

        recent = sum(list(self.errors)[-10:])
        older = sum(list(self.errors)[:-10])
        older = max(older, 1)

        return (recent - older) / older

    def should_open_predictively(self):
        latency_trend = self.get_latency_trend()
        error_accel = self.get_error_acceleration()
        score = latency_trend + error_accel * 2
        return score > 1.5, score

analyzer = TrendAnalyzer(window_size=50)

for i in range(60):
    if i < 30:
        latency = 0.1 + (i * 0.02)
        error = 0
    else:
        latency = 0.7 + (i * 0.05)
        error = 1 if i % 3 == 0 else 0

    analyzer.add_observation(latency, error)
    if i > 20:
        should_open, score = analyzer.should_open_predictively()
        if i % 10 == 0:
            print(f"Step {i}: trend={analyzer.get_latency_trend():.2f}, "
                  f"error_accel={analyzer.get_error_acceleration():.2f}, "
                  f"score={score:.2f}, open={should_open}")

Expected output:

Step 20: trend=0.15, error_accel=0.00, score=0.15, open=False
Step 30: trend=0.45, error_accel=0.20, score=0.85, open=False
Step 40: trend=0.82, error_accel=0.50, score=1.82, open=True
Step 50: trend=1.15, error_accel=0.80, score=2.75, open=True
Step 60: trend=1.50, error_accel=1.00, score=3.50, open=True

Simple ML Predictor

import time
import random
from collections import deque

class SimplePredictor:
    def __init__(self, feature_window=20):
        self.feature_window = feature_window
        self.features = deque(maxlen=100)
        self.labels = deque(maxlen=100)
        self.weights = [0.3, 0.3, 0.2, 0.2]

    def extract_features(self, latencies, errors, cpu, memory):
        if len(latencies) < 5:
            return [0, 0, 0, 0]

        avg_latency = sum(latencies) / len(latencies)
        max_latency = max(latencies)
        error_rate = sum(errors) / max(len(errors), 1)

        return [avg_latency, max_latency, error_rate, cpu / 100]

    def predict(self, latencies, errors, cpu, memory):
        features = self.extract_features(latencies, errors, cpu, memory)
        score = sum(w * f for w, f in zip(self.weights, features))

        for i in range(len(self.weights)):
            if features[i] > 0.8:
                self.weights[i] = min(1.0, self.weights[i] * 1.1)
            elif features[i] < 0.2:
                self.weights[i] = max(0.1, self.weights[i] * 0.95)

        return score > 0.6, score

predictor = SimplePredictor()
latencies = [0.1] * 20 + [0.3] * 10 + [0.8] * 10 + [2.0] * 10
errors = [0] * 30 + [1] * 5 + [0] * 5 + [1] * 10
cpu = [30] * 30 + [60] * 10 + [90] * 10

for i in range(10, 50, 5):
    l = latencies[max(0, i-5):i]
    e = errors[max(0, i-5):i]
    c = cpu[i-1] if i < len(cpu) else 50
    will_fail, score = predictor.predict(l, e, c, 50)
    actual_failure = i >= 40
    print(f"Step {i}: score={score:.2f}, predict_fail={will_fail}, "
          f"actual_fail={actual_failure}")

Expected output:

Step 10: score=0.05, predict_fail=False, actual_fail=False
Step 15: score=0.08, predict_fail=False, actual_fail=False
Step 20: score=0.10, predict_fail=False, actual_fail=False
Step 25: score=0.25, predict_fail=False, actual_fail=False
Step 30: score=0.35, predict_fail=False, actual_fail=False
Step 35: score=0.60, predict_fail=True, actual_fail=False
Step 40: score=0.75, predict_fail=True, actual_fail=True
Step 45: score=0.85, predict_fail=True, actual_fail=True

Common Mistakes

• Overfitting to historical patterns -- ML models trained on past failures may not generalize to novel failure modes. Use ensemble models and regularly retrain on new failure data. Monitor prediction accuracy in production.
• False positives from proactive opening -- predicting failures that don't happen causes unnecessary circuit openings. Track false positive rate and adjust prediction thresholds. A 5% false positive rate may be acceptable for critical services.
• Feature selection without domain expertise -- including irrelevant features adds noise. Focus on latency trend, error rate acceleration, connection pool utilization, and resource saturation. Exclude noisy metrics like total request count.
• No feedback loop -- predictions without outcome feedback cannot improve. Record whether a predicted failure actually occurred. Use this feedback to retrain models and adjust weights.
• Prediction latency overhead -- running complex ML models on every request adds latency. Use lightweight models (logistic regression, threshold-based) for per-request prediction. Reserve complex models (neural networks) for batch prediction at longer intervals.

Practice Questions

1. How does predictive circuit breaking differ from reactive circuit breaking?
2. What features are most predictive of service failures?
3. How do you handle false positives in predictive circuit breakers?
4. What is the feedback loop for improving prediction accuracy?
5. How do you balance prediction accuracy vs latency overhead?

Challenge

Build a predictive circuit breaker system: (1) feature extraction from a sliding window of latency, error rate, CPU, and memory metrics, (2) lightweight ML model (logistic regression or gradient boosting) trained on historical failure data, (3) online learning: update model weights based on prediction outcomes, (4) configurable prediction threshold that balances precision and recall, (5) proactive circuit opening based on prediction score, (6) metrics: prediction accuracy, false positive rate, false negative rate, prevention rate (failures prevented / total failures), (7) model monitoring dashboard showing prediction score distribution, accuracy trends, and feature importance over time.

FAQ

Mini Project

Build a production predictive circuit breaker: (1) feature pipeline extracting 10 features from a 30-second sliding window (latency p50/p95/p99, error rate, error acceleration, connection pool %, CPU trend, memory trend), (2) XGBoost classifier trained on 30 days of historical data, (3) online learning: update model weights with exponential decay (recent observations weighted higher), (4) configurable threshold (default 0.7) that balances false positives and false negatives, (5) proactive circuit opening: when score > threshold, open circuit and serve fallback, (6) continuous evaluation: track daily prediction accuracy and retrain if accuracy drops below 80%, (7) Prometheus metrics: prediction score, true/false positives/negatives, prevention rate, model staleness.

What's Next

Continue with GraphQL Integration to learn Graphql circuit breaker patterns. Then explore Serverless Patterns for Serverless circuit breaker patterns.