Rate Limiting Integration with Circuit Breakers — Combined Resilience Strategy

In this tutorial, you will learn about Rate Limiting Integration with Circuit Breakers. We cover key concepts, practical examples, and best practices to help you master this topic.

Rate limiting integration with circuit breakers creates a layered resilience strategy where rate limiting controls traffic volume before it reaches the circuit breaker, preventing high traffic from overwhelming the circuit and keeping the circuit closed during legitimate traffic spikes.

flowchart LR
    Client[Client] --> RL[Rate Limiter]
    RL -->|Allowed| CB[Circuit Breaker]
    RL -->|Blocked| Reject[429 Too Many Requests]
    CB -->|Closed| Service[Backend Service]
    CB -->|Open| Fallback[Fallback Response]
    Service -->|Failure Feedback| RL
    Service -->|Failure Feedback| CB
    style RL fill:#f90,color:#fff

What You'll Learn

• Token bucket with circuit breaker coordination
• Rate limiting feedback loops to circuit breaker
• Coordinated rate limiting and circuit opening
• Adaptive rate limiting based on circuit state
• Rate limit header propagation

Why It Matters

Without rate limiting before the circuit breaker, a traffic spike can open the circuit through sheer volume even if the failure rate is low. Rate limiting ensures the circuit breaker sees a controlled traffic volume, so its decisions are based on actual failure rates rather than request volume.

Real-World Use

DodaTech's API gateway combines rate limiting and circuit breakers: rate limiter (100 req/s per API key), then circuit breaker (open after 5% failure rate in 60-second window). During a DDoS attempt, the rate limiter blocked 95% of requests, keeping the circuit breaker closed for legitimate traffic.

Token Bucket with Circuit Breaker Feedback

import time
import threading

class TokenBucket:
    def __init__(self, rate, burst):
        self.rate = rate
        self.burst = burst
        self.tokens = burst
        self.last_refill = time.time()
        self.lock = threading.Lock()

    def consume(self, tokens=1):
        with self.lock:
            self._refill()
            if self.tokens >= tokens:
                self.tokens -= tokens
                return True
            return False

    def _refill(self):
        now = time.time()
        elapsed = now - self.last_refill
        self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
        self.last_refill = now

    def set_rate(self, new_rate):
        with self.lock:
            self.rate = new_rate

class RateLimitingCircuitBreaker:
    def __init__(self, name, rate=10, burst=20, cb_threshold=3):
        self.name = name
        self.rate_limiter = TokenBucket(rate, burst)
        self.cb_threshold = cb_threshold
        self.failures = 0
        self.state = 'CLOSED'
        self.last_failure = 0

    def call(self, fn, *args, **kwargs):
        if not self.rate_limiter.consume():
            raise Exception("Rate limit exceeded")

        if self.state == 'OPEN':
            if time.time() - self.last_failure > 30:
                self.state = 'HALF_OPEN'
            else:
                raise Exception("Circuit open")

        try:
            result = fn(*args, **kwargs)
            self.failures = 0
            if self.state in ('HALF_OPEN', 'OPEN'):
                self.state = 'CLOSED'
            return result
        except Exception as e:
            self.failures += 1
            self.last_failure = time.time()
            if self.failures >= self.cb_threshold:
                self.state = 'OPEN'
                self.rate_limiter.set_rate(max(1, self.rate_limiter.rate * 0.5))
                print(f"[{self.name}] Circuit OPEN. Rate limit reduced to {self.rate_limiter.rate:.1f} req/s")
            raise

cb = RateLimitingCircuitBreaker("payment-service", rate=5, burst=10, cb_threshold=3)

for i in range(10):
    try:
        success = i < 1
        if not success:
            cb.call(lambda: (_ for _ in ()).throw(Exception("failure")))
        else:
            cb.call(lambda: "ok")
    except Exception as e:
        print(f"Request {i+1}: {e}")
    time.sleep(0.05)

Expected output:

[payment-service] Circuit OPEN. Rate limit reduced to 2.5 req/s
Request 2: failure
Request 3: failure
Request 4: failure
Request 5: Circuit open
Request 6: Circuit open
Request 7: Circuit open
Request 8: Circuit open
Request 9: Circuit open
Request 10: Circuit open

Coordinated Rate Limiting

import time

class CoordinatedRateLimiter:
    def __init__(self, default_rate=100):
        self.default_rate = default_rate
        self.circuit_states = {}
        self.rates = {}

    def set_circuit_state(self, service, state):
        old_state = self.circuit_states.get(service)
        self.circuit_states[service] = state

        if state == 'OPEN' and old_state != 'OPEN':
            self.rates[service] = max(1, self.rates.get(service, self.default_rate) * 0.25)
            print(f"[{service}] Circuit OPEN: rate reduced to {self.rates[service]} req/s")
        elif state == 'HALF_OPEN':
            self.rates[service] = int(self.default_rate * 0.5)
            print(f"[{service}] Circuit HALF_OPEN: rate set to {self.rates[service]} req/s")
        elif state == 'CLOSED' and old_state in ('OPEN', 'HALF_OPEN'):
            self.rates[service] = self.default_rate
            print(f"[{service}] Circuit CLOSED: rate restored to {self.rates[service]} req/s")

    def get_rate(self, service):
        return self.rates.get(service, self.default_rate)

coordinator = CoordinatedRateLimiter(default_rate=100)

print(f"Initial: payment={coordinator.get_rate('payment')} req/s")
coordinator.set_circuit_state("payment", "OPEN")
print(f"After open: payment={coordinator.get_rate('payment')} req/s")
coordinator.set_circuit_state("payment", "HALF_OPEN")
print(f"After half-open: payment={coordinator.get_rate('payment')} req/s")
coordinator.set_circuit_state("payment", "CLOSED")
print(f"After closed: payment={coordinator.get_rate('payment')} req/s")

Expected output:

Initial: payment=100 req/s
[payment] Circuit OPEN: rate reduced to 25.0 req/s
After open: payment=25.0 req/s
[payment] Circuit HALF_OPEN: rate set to 50 req/s
After half-open: payment=50 req/s
[payment] Circuit CLOSED: rate restored to 100 req/s
After closed: payment=100 req/s

Common Mistakes

• Rate limiting after circuit breaker -- rate limiting after the circuit breaker means the circuit breaker sees unthrottled traffic, causing it to open during traffic spikes. Always place rate limiting before the circuit breaker.
• No rate limit reduction when circuit opens -- if the circuit opens but the rate limiter keeps admitting traffic at full speed, the fallback system gets overwhelmed. Reduce the rate when the circuit opens to protect the fallback.
• Rate limiter and circuit breaker with different time windows -- a rate limiter with a 1-second window and a circuit breaker with a 60-second window see different traffic patterns. Align measurement windows for coordinated behavior.
• No rate limit header propagation -- when a circuit opens and the rate limiter reduces the rate, clients need to know the new limit. Include RateLimit-Remaining and Retry-After headers in circuit breaker responses too.
• Static rate limits without circuit state feedback -- rate limits that never adjust to circuit state miss an opportunity to reduce pressure on failing systems. Use circuit state as a signal to dynamically adjust rate limits.

Practice Questions

1. Why should rate limiting be placed before the circuit breaker?
2. How does circuit state affect rate limit configuration?
3. What happens to the rate limiter when the circuit opens?
4. How do you propagate rate limit information through circuit breaker responses?
5. What are the failure modes of combining rate limiting with circuit breakers?

Challenge

Build a combined rate limiting and circuit breaker system: (1) token bucket rate limiter (100 req/s, 200 burst) that feeds into a circuit breaker, (2) circuit breaker with Sliding Window (60s, 10% failure threshold), (3) feedback loop: when circuit opens, reduce rate by 75%, when half-open, set rate to 50% of normal, when closed, restore to 100%, (4) rate limit headers in all responses (RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset), (5) coordinated dashboard showing rate limit and circuit state per service, (6) burst protection: allow short bursts during circuit recovery without overwhelming, (7) proportional rate reduction based on failure severity (timeouts = 50% reduction, 5xx = 75% reduction, connection errors = 90% reduction).

FAQ

Mini Project

Build an integrated rate limiting and circuit breaker gateway: (1) token bucket rate limiter per API key (100 req/s, 200 burst), (2) sliding window circuit breaker per upstream service (60s window, 10% failure threshold, 30s reset), (3) feedback controller that adjusts rate limits based on circuit state: open = 25%, half-open = 50%, closed = 100%, (4) adaptive rate reduction by failure type: timeout = 50%, 5xx = 75%, connection error = 90%, (5) rate limit headers in all responses including blocked circuit breaker responses, (6) Prometheus metrics: rate limit allowance vs usage, circuit state, combined blocked count, (7) Grafana dashboard with overlaid rate limit and circuit state timelines.

What's Next

Continue with Security Patterns for circuit breaker security considerations. Then explore Multi-Datacenter Patterns for cross-region circuit breaker deployment.