Monitoring Webhooks — Complete Guide
In this tutorial, you will learn about Monitoring Webhooks. We cover key concepts, practical examples, and best practices to help you master this topic.
Learn webhook monitoring: track delivery metrics, monitor success rates, set up health checks, build dashboards for webhook performance, and alert on delivery failures and anomalies.
What You Learn
You will learn how to monitor webhook delivery systems: track key metrics (delivery rate, success rate, latency), build dashboards for real-time visibility, set up alerts for failures and anomalies, and implement health checks for subscribers.
Why It Matters
Webhook delivery failures happen silently. A subscriber changes their URL, a network route breaks, or a provider changes payload format. Without monitoring, you discover delivery failures when customers complain. Proactive monitoring catches issues before they affect users.
Real-World Use
DodaTech's webhook monitoring system tracks 500K daily deliveries across 12000 subscribers. Real-time dashboards show delivery success rates by provider, event type, and subscriber. Alerts fire when any subscriber's success rate drops below 99%. Average detection time for issues is 2 minutes.
Key Metrics
class WebhookMetrics {
constructor() {
this.metrics = {
totalDelivered: 0,
totalFailed: 0,
totalRetried: 0,
totalDeadLettered: 0,
deliveriesByProvider: new Map(),
deliveriesByEventType: new Map(),
latencyBuckets: new Array(10).fill(0), // 0-100ms, 100-200ms, etc
lastMinuteDeliveries: [],
startTime: Date.now(),
};
}
recordDelivery(delivery) {
this.metrics.totalDelivered++;
// By provider
const provider = delivery.provider || 'unknown';
const providerCount = this.metrics.deliveriesByProvider.get(provider) || 0;
this.metrics.deliveriesByProvider.set(provider, providerCount + 1);
// By event type
const eventType = delivery.eventType || 'unknown';
const eventCount = this.metrics.deliveriesByEventType.get(eventType) || 0;
this.metrics.deliveriesByEventType.set(eventType, eventCount + 1);
// Latency
if (delivery.durationMs !== undefined) {
const bucketIndex = Math.min(
Math.floor(delivery.durationMs / 100),
this.metrics.latencyBuckets.length - 1
);
this.metrics.latencyBuckets[bucketIndex]++;
}
// Last minute sliding window
this.metrics.lastMinuteDeliveries.push({
timestamp: Date.now(),
success: delivery.success,
durationMs: delivery.durationMs,
});
// Keep only last 60 seconds
const cutoff = Date.now() - 60000;
this.metrics.lastMinuteDeliveries =
this.metrics.lastMinuteDeliveries.filter(d => d.timestamp > cutoff);
}
recordFailure(delivery) {
this.metrics.totalFailed++;
this.recordDelivery({ ...delivery, success: false });
}
getCurrentRate() {
return this.metrics.lastMinuteDeliveries.length;
}
getSuccessRate() {
const lastMinute = this.metrics.lastMinuteDeliveries;
if (lastMinute.length === 0) return 100;
const successful = lastMinute.filter(d => d.success).length;
return (successful / lastMinute.length * 100).toFixed(2);
}
getAverageLatency() {
const withLatency = this.metrics.lastMinuteDeliveries
.filter(d => d.durationMs !== undefined);
if (withLatency.length === 0) return 0;
const total = withLatency.reduce((sum, d) => sum + d.durationMs, 0);
return Math.round(total / withLatency.length);
}
getSummary() {
return {
uptime: Math.floor((Date.now() - this.metrics.startTime) / 1000),
currentRate: this.getCurrentRate(),
successRate: this.getSuccessRate(),
averageLatency: this.getAverageLatency(),
totalDelivered: this.metrics.totalDelivered,
totalFailed: this.metrics.totalFailed,
totalRetried: this.metrics.totalRetried,
totalDeadLettered: this.metrics.totalDeadLettered,
};
}
}
Expected output: Metrics track key webhook delivery statistics with a 1-minute sliding window for real-time rates. Summary provides overall health at a glance.
Prometheus Metrics
const prometheus = require('prom-client');
// Create Prometheus metrics
const webhookDeliveriesTotal = new prometheus.Counter({
name: 'webhook_deliveries_total',
help: 'Total webhook deliveries',
labelNames: ['provider', 'event_type', 'status'],
});
const webhookDeliveryDuration = new prometheus.Histogram({
name: 'webhook_delivery_duration_ms',
help: 'Webhook delivery duration in milliseconds',
labelNames: ['provider'],
buckets: [50, 100, 200, 500, 1000, 2000, 5000, 10000],
});
const webhookDeliveriesInFlight = new prometheus.Gauge({
name: 'webhook_deliveries_in_flight',
help: 'Webhook deliveries currently in flight',
});
const webhookQueueDepth = new prometheus.Gauge({
name: 'webhook_queue_depth',
help: 'Current webhook delivery queue depth',
labelNames: ['subscriber_id'],
});
const webhookDeadLetterCount = new prometheus.Gauge({
name: 'webhook_dead_letter_count',
help: 'Number of webhooks in dead letter queue',
labelNames: ['event_type'],
});
// Record metrics during delivery
function recordDeliveryMetrics(provider, eventType, status, durationMs) {
webhookDeliveriesTotal.labels(provider, eventType, status).inc();
if (durationMs !== undefined) {
webhookDeliveryDuration.labels(provider).observe(durationMs);
}
}
Expected output: Prometheus metrics expose webhook delivery data for Grafana dashboards. Histograms show latency distribution. Counters show delivery volume. Gauges show current queue depth.
Health Check Endpoints
const express = require('express');
const app = express();
// Provider health check
app.get('/health/providers', async (req, res) => {
const results = await Promise.allSettled(
Object.entries(providers).map(async ([name, config]) => {
const response = await fetch(`${config.baseUrl}/health`, {
signal: AbortSignal.timeout(5000),
});
return { provider: name, status: response.ok ? 'healthy' : 'degraded' };
})
);
const providerStatus = results.map(r =>
r.status === 'fulfilled' ? r.value : { status: 'unreachable' }
);
const allHealthy = providerStatus.every(p => p.status === 'healthy');
res.status(allHealthy ? 200 : 503).json({
status: allHealthy ? 'healthy' : 'degraded',
providers: providerStatus,
});
});
// Subscriber health check
app.get('/health/subscribers', async (req, res) => {
const subscribers = await getSubscribers();
const results = [];
for (const sub of subscribers) {
const lastHourDeliveries = await getDeliveries(sub.id, { hours: 1 });
const successRate = calculateSuccessRate(lastHourDeliveries);
results.push({
id: sub.id,
url: sub.url,
status: successRate >= 95 ? 'healthy' : 'degraded',
successRate: `${successRate.toFixed(1)}%`,
deliveriesLastHour: lastHourDeliveries.length,
});
}
const degraded = results.filter(r => r.status === 'degraded');
res.status(degraded.length === 0 ? 200 : 503).json({
status: degraded.length === 0 ? 'healthy' : 'degraded',
totalSubscribers: results.length,
degradedSubscribers: degraded.length,
subscribers: results,
});
});
// Overall system health
app.get('/health', async (req, res) => {
const metrics = webhookMetrics.getSummary();
const dlqCount = await getDLQCount();
const queueDepth = await getQueueDepth();
const healthy = metrics.successRate >= 99
&& dlqCount < 100
&& queueDepth < 1000;
res.status(healthy ? 200 : 503).json({
status: healthy ? 'healthy' : 'degraded',
...metrics,
dlqCount,
queueDepth,
timestamp: new Date().toISOString(),
});
});
Expected output: Health endpoints check provider connectivity, subscriber delivery success rates, and overall system health. Degraded endpoints return 503 for load balancer integration.
Alerting Rules
// Alert manager configuration
const alertRules = [
{
name: 'HighDeliveryFailureRate',
condition: (metrics) => metrics.getSuccessRate() < 95,
severity: 'critical',
message: 'Webhook delivery success rate below 95%',
},
{
name: 'DLQGrowth',
condition: async (dlq) => {
const count = await dlq.getDLQCount();
return count > 100;
},
severity: 'warning',
message: 'Dead letter queue has more than 100 entries',
},
{
name: 'SubscriberDown',
condition: async (subscribers) => {
const results = [];
for (const sub of subscribers) {
const recentDeliveries = await getDeliveries(sub.id, { minutes: 5 });
if (recentDeliveries.length > 0) {
const successCount = recentDeliveries.filter(d => d.success).length;
if (successCount === 0) {
results.push(sub.url);
}
}
}
return results;
},
severity: 'critical',
message: (downSubscribers) =>
`Subscribers not receiving webhooks: ${downSubscribers.join(', ')}`,
},
{
name: 'HighLatency',
condition: (metrics) => metrics.getAverageLatency() > 5000,
severity: 'warning',
message: 'Average webhook delivery latency above 5 seconds',
},
];
async function evaluateAlerts() {
const firedAlerts = [];
for (const rule of alertRules) {
try {
const result = await rule.condition(webhookMetrics, dlq, subscribers);
if (result) {
firedAlerts.push({
name: rule.name,
severity: rule.severity,
message: typeof rule.message === 'function'
? rule.message(result)
: rule.message,
timestamp: new Date().toISOString(),
});
}
} catch (err) {
console.error(`Alert evaluation failed: ${rule.name}`, err);
}
}
return firedAlerts;
}
Expected output: Alert rules evaluate periodically. Fired alerts are sent to Slack, PagerDuty, or email. Each alert has severity (warning/critical) for appropriate response.
Common Mistakes
1. No Monitoring for Silent Failures
A subscriber that changed their URL silently fails all deliveries. Monitor delivery success rate per subscriber. Alert on zero success rate. Implement URL verification on changes.
2. Metrics Without Labels
Without provider/event-type/subscriber labels, you cannot identify which part of the system is failing. Add labels to all metrics. Use them to filter dashboards and alerts.
3. No Health Check Endpoints
Without health endpoints, load balancers cannot detect unhealthy instances. Implement /health that verifies database connectivity, queue depth, and recent delivery success rate.
4. Alert Fatigue
Too many alerts desensitize the team. Set meaningful thresholds. Use warning vs critical severity. Implement alert deduplication. Only alert on actionable conditions.
5. No Delivery Log Correlation
Metrics show the problem but not the cause. Correlate delivery logs with metrics. Each failed delivery should include: webhook ID, subscriber ID, error message, response body, and timing.
Practice Questions
1. What metrics are essential for webhook monitoring?
Delivery rate (webhooks/minute), success rate (%), average latency, queue depth, dead letter count, and per-subscriber success rate. These provide a complete picture of delivery health.
2. How do you monitor subscriber health?
Track delivery success rate per subscriber over recent Windows (5 min, 1 hour, 24 hours). Alert when success rate drops below 95%. Probe subscriber endpoints periodically.
3. What is the difference between health check and monitoring?
Health check is a point-in-time status (is the system running?). Monitoring is continuous data collection and analysis over time (how is the system performing?).
4. How do you set up alerting for webhook delivery failures?
Alert on: success rate below 95%, dead letter queue above 100 entries, any subscriber with 0% success in 5 minutes, queue depth above 1000, and average latency above 5 seconds.
Challenge
Build a complete webhook monitoring system: Prometheus metrics for delivery rates, success rates, and latency by provider/event/subscriber, Grafana dashboard with real-time charts, health endpoints for load balancer integration, alerting rules for failures and anomalies, and delivery log correlation for debugging.
FAQ
Mini Project: Webhook Monitoring Dashboard
Build a Grafana dashboard for webhook monitoring: delivery rate chart (1 min resolution), success rate gauge (per provider), latency heatmap (provider x duration), subscriber health table (success rate, last delivery, queue depth), DLQ growth trend, and alert history timeline.
What's Next
Now that you can monitor webhooks, learn about Logging Webhooks for detailed delivery debugging and audit trails.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro