Webhook Circuit Breaker — Complete Guide to Preventing Cascading Failures
In this tutorial, you will learn about Webhook Circuit Breaker. We cover key concepts, practical examples, and best practices to help you master this topic.
Webhook Circuit Breaker Pattern protects providers from non-responsive consumers by monitoring delivery failures and temporarily suspending deliveries to unhealthy endpoints, preventing cascading failures.
What You'll Learn
- How the circuit breaker pattern applies to Webhooks
- Monitoring consumer health and failure thresholds
- Automatic recovery and half-open state
Why It Matters
A slow or dead webhook consumer can exhaust provider resources (connections, threads, queue capacity), impacting delivery to healthy consumers. Circuit breakers isolate failures.
Real-World Use
Durga Antivirus Pro webhook system monitors delivery success rates per consumer. If failure rate exceeds 50% in 5 minutes, the circuit opens and deliveries are paused for 15 minutes before retrying.
flowchart LR
A["Circuit State"] --> B["CLOSED"]
B -->|"Failures > Threshold"| C["OPEN"]
C -->|"Timeout Expired"| D["HALF-OPEN"]
D -->|"Test Succeeds"| B
D -->|"Test Fails"| C
style B fill:#dbeafe,stroke:#2563eb
Code Examples
import time
from collections import deque
class CircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=300, half_open_max=3):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.half_open_max = half_open_max
self.state = 'CLOSED'
self.failures = deque(maxlen=failure_threshold)
self.last_failure_time = 0
self.half_open_attempts = 0
def record_failure(self):
self.failures.append(time.time())
self.last_failure_time = time.time()
if len(self.failures) >= self.failure_threshold:
self.state = 'OPEN'
print("Circuit OPEN - deliveries paused")
def record_success(self):
if self.state == 'HALF-OPEN':
self.half_open_attempts = 0
self.state = 'CLOSED'
self.failures.clear()
print("Circuit CLOSED - deliveries resumed")
def allow_request(self):
if self.state == 'CLOSED':
return True
if self.state == 'OPEN':
if time.time() - self.last_failure_time > self.recovery_timeout:
self.state = 'HALF-OPEN'
self.half_open_attempts = 0
print("Circuit HALF-OPEN - testing")
return True
return False
if self.state == 'HALF-OPEN':
if self.half_open_attempts < self.half_open_max:
self.half_open_attempts += 1
return True
return False
return False
Expected output: Circuit transitions between CLOSED, OPEN (paused), and HALF-OPEN (testing) states.
// Consumer health monitor
class ConsumerHealthMonitor {
constructor() {
this.consumers = new Map(); // { url: { failures, lastFailure, circuitState } }
}
recordDelivery(url, success) {
if (!this.consumers.has(url)) {
this.consumers.set(url, { failures: 0, lastFailure: 0, circuitState: 'CLOSED' });
}
const consumer = this.consumers.get(url);
if (success) {
if (consumer.circuitState === 'HALF_OPEN') {
consumer.circuitState = 'CLOSED';
consumer.failures = 0;
}
} else {
consumer.failures++;
consumer.lastFailure = Date.now();
if (consumer.failures >= 5) {
consumer.circuitState = 'OPEN';
console.log(`Circuit opened for ${url}`);
setTimeout(() => {
consumer.circuitState = 'HALF_OPEN';
console.log(`Circuit half-open for ${url}`);
}, 300000); // 5 min recovery
}
}
}
canDeliver(url) {
const consumer = this.consumers.get(url);
if (!consumer) return true;
return consumer.circuitState !== 'OPEN';
}
}
Expected output: Health monitor tracks failures per consumer; circuits open after 5 consecutive failures.
# Integration with webhook delivery system
class WebhookCircuitBreaker:
def __init__(self):
self.breakers = {} # subscriber_id -> CircuitBreaker
def get_breaker(self, subscriber_id):
if subscriber_id not in self.breakers:
self.breakers[subscriber_id] = CircuitBreaker()
return self.breakers[subscriber_id]
def deliver(self, subscriber_id, url, event):
breaker = self.get_breaker(subscriber_id)
if not breaker.allow_request():
self.queue_for_retry(subscriber_id, event)
return {'status': 'queued', 'reason': 'circuit_open'}
try:
resp = requests.post(url, json=event, timeout=10)
if resp.ok:
breaker.record_success()
return {'status': 'delivered'}
else:
breaker.record_failure()
return {'status': 'failed', 'code': resp.status_code}
except Exception as e:
breaker.record_failure()
return {'status': 'error', 'error': str(e)}
Expected output: Circuit breaker integrated with delivery; failed deliveries queued when circuit is open.
Common Mistakes
1. Circuit Opening Too Quickly
Opening the circuit after 1-2 failures causes flapping. Set threshold to 5-10 failures in a window.
2. No Half-Open State
Once open, a circuit stays open forever without half-open testing. Always implement automatic recovery testing.
3. Same Threshold for All Consumers
Critical consumers need lower thresholds; non-critical ones can tolerate more failures.
4. Not Notifying Consumer When Circuit Opens
Consumers are unaware their webhook endpoint is paused. Send alert emails when circuit opens.
5. Discarding Events During Open State
Events arriving when the circuit is open are lost. Queue them for delivery when the circuit closes.
Practice Questions
- What are the three states of a circuit breaker?
- Why does the circuit breaker protect the provider?
- What is the purpose of the half-open state?
- How do you determine the failure threshold?
- What happens to events when the circuit is open?
Answers:
- Closed (normal), Open (paused), Half-Open (testing recovery).
- It prevents a failing consumer from exhausting provider resources and impacting other consumers.
- Half-open allows limited test deliveries to verify if the consumer has recovered.
- Based on normal failure rates: set threshold at 3-5x the expected failure rate in a window.
- Queue events for later delivery, or move to dead-letter queue if circuit stays open too long.
Challenge: Build a circuit breaker system for webhook delivery with: configurable thresholds per consumer, three circuit states, automatic half-open testing, event queuing during open state, and notification when circuit opens.
FAQ
Mini Project
Build a circuit breaker for webhook delivery with: per-consumer failure tracking, three state transitions, configurable thresholds and recovery timeout, event queuing during open state, automatic half-open testing, and Grafana metrics showing circuit states.
What's Next
Learn about Webhook delivery guarantees for reliable delivery, or explore Webhook dead letter queues for handling persistently failing events.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro