Monitoring Delivery

title: "Monitoring Webhook Delivery" description: "Learn how to monitor webhook delivery with metrics, logging, and alerting to ensure reliable event distribution and quick incident response." weight: 28 date: 2026-06-28 lastmod: 2026-06-28 tags: ["apis", "webhooks"]

Monitoring webhook delivery is essential for maintaining integration health. Without visibility into delivery success rates, latency, and error patterns, webhook failures go unnoticed until users report data inconsistencies. This lesson covers observability patterns for both webhook providers and consumers.

## What You'll Learn
- Define key webhook delivery metrics and SLIs
- Implement structured logging for webhook events and deliveries
- Set up dashboards for real-time delivery visibility
- Configure alerting rules for common failure scenarios

## Why It Matters
Webhooks are the backbone of event-driven integrations. When webhook delivery fails, data becomes inconsistent across systems. Monitoring provides early warning of integration issues, helps debug delivery failures, and enables data-driven capacity planning.

## Real-World Use
- Stripe provides a webhook delivery health dashboard showing success rates and latency
- GitHub has a delivery log for each webhook with request/response details
- PagerDuty alerts on webhook delivery failure rates exceeding thresholds
- Datadog and New Relic offer webhook monitoring integrations

## Mermaid Flow
```mermaid
graph TD
    A[Webhook Delivery] --> B{Emit Metrics}
    B --> C[Delivery Count]
    B --> D[Success Rate]
    B --> E[Latency]
    B --> F[Retry Count]
    C --> G[Prometheus]
    D --> G
    E --> G
    F --> G
    G --> H[Grafana Dashboard]
    G --> I[Alert Manager]
    I --> J[PagerDuty / Slack]

Teacher's Corner

Explain the difference between provider-side monitoring (did we send it?) and consumer-side monitoring (did we process it?). Both are needed for full visibility. Teach the RED method: Rate (requests per second), Errors (failed requests), Duration (latency). Apply RED to webhook delivery.

Code Examples

Example 1: Prometheus Metrics for Webhook Delivery

from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time
import random

webhook_delivery_total = Counter(
    "webhook_delivery_total",
    "Total webhook deliveries",
    ["consumer", "status"]
)

webhook_delivery_latency = Histogram(
    "webhook_delivery_latency_seconds",
    "Webhook delivery latency",
    ["consumer"],
    buckets=(0.1, 0.5, 1.0, 2.0, 5.0, 10.0)
)

webhook_queue_depth = Gauge(
    "webhook_queue_depth",
    "Current webhook delivery queue depth",
    ["consumer"]
)

def deliver_webhook(consumer, event):
    start = time.time()
    try:
        time.sleep(random.uniform(0.1, 2.0))
        if random.random() < 0.1:
            raise Exception("simulated failure")
        webhook_delivery_total.labels(consumer=consumer, status="success").inc()
    except Exception:
        webhook_delivery_total.labels(consumer=consumer, status="failed").inc()
    finally:
        webhook_delivery_latency.labels(consumer=consumer).observe(
            time.time() - start
        )

start_http_server(8000)

for i in range(100):
    deliver_webhook("consumer-a", f"event-{i}")
    time.sleep(0.1)

Expected Output: Metrics available at http://localhost:8000/metrics showing delivery counts, success rates, and latency histograms.

Example 2: Structured Logging for Webhook Events

import json
import logging
import sys
from datetime import datetime

class WebhookLogger:
    def __init__(self):
        self.logger = logging.getLogger("webhooks")
        self.logger.setLevel(logging.INFO)
        handler = logging.StreamHandler(sys.stdout)
        handler.setFormatter(logging.Formatter(
            '%(message)s'
        ))
        self.logger.addHandler(handler)

    def log_delivery(self, event_id, consumer_url, status, latency_ms,
                     attempt, error=None):
        entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "event": "webhook_delivery",
            "event_id": event_id,
            "consumer_url": consumer_url,
            "status": status,
            "latency_ms": latency_ms,
            "attempt": attempt,
            "error": str(error) if error else None
        }
        self.logger.info(json.dumps(entry))

    def log_receipt(self, event_id, source, event_type, status):
        entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "event": "webhook_receipt",
            "event_id": event_id,
            "source": source,
            "event_type": event_type,
            "status": status
        }
        self.logger.info(json.dumps(entry))

wl = WebhookLogger()
wl.log_delivery("evt-001", "https://example.com/hooks",
                "success", 245, 1)
wl.log_delivery("evt-002", "https://example.com/hooks",
                "failed", 10000, 3, "Connection timeout")

Expected Output: Two JSON log lines with structured fields for consumption by log aggregators (ELK, Loki, Datadog).

Example 3: Health Check Endpoint with Delivery Status

import time
from flask import Flask, jsonify
from collections import defaultdict

app = Flask(__name__)

class DeliveryHealth:
    def __init__(self):
        self.recent = defaultdict(list)
        self.window = 300

    def record(self, consumer, status, latency):
        now = time.time()
        self.recent[consumer].append({
            "time": now, "status": status, "latency": latency
        })
        self.recent[consumer] = [
            r for r in self.recent[consumer]
            if now - r["time"] < self.window
        ]

    def get_health(self):
        results = {}
        for consumer, records in self.recent.items():
            total = len(records)
            successes = sum(1 for r in records if r["status"] == "success")
            avg_latency = sum(r["latency"] for r in records) / total if total else 0
            results[consumer] = {
                "total_deliveries": total,
                "success_rate": round(successes / total * 100, 1) if total else 0,
                "avg_latency_ms": round(avg_latency, 0),
                "healthy": (successes / total) > 0.95 if total else True
            }
        return results

health = DeliveryHealth()

@app.route("/health/webhooks")
def webhook_health():
    return jsonify(health.get_health())

for i in range(100):
    status = "success" if i % 20 != 0 else "failed"
    health.record("consumer-a", status, 150 + i)

Expected Output: GET /health/webhooks returns JSON with per-consumer success rates, latency, and health status.

Common Mistakes

1. Not monitoring webhook delivery at all, relying on users to report failures
2. Only monitoring from the provider side, missing consumer processing failures
3. Using unstructured log messages that cannot be parsed by log aggregators
4. Not setting up alerts for delivery failure rate thresholds
5. Measuring latency from the provider only, ignoring consumer processing time
6. Not tracking retry attempts, making it hard to distinguish first-attempt vs. eventual delivery
7. Monitoring only aggregate metrics without the ability to drill into specific consumer or event type issues

Practice Questions

1. What are the three key metrics in the RED method and how do they apply to webhooks?
2. Why is structured logging important for webhook monitoring?
3. How would you alert on a silent webhook failure (provider stops sending)?
4. What is the difference between provider-side and consumer-side monitoring?
5. Challenge: Design a monitoring system for a webhook provider that tracks delivery success rate, latency P50/P95/P99, retry distribution, queue depth, and consumer health. Implement with Prometheus metrics, structured JSON logging, and alerting rules for success rate below 99% and latency P95 above 5 seconds.

FAQ

Mini Project

Build a webhook monitoring dashboard. Create a Python application that: (1) simulates a webhook provider sending events to multiple consumers, (2) emits Prometheus metrics for delivery count, success rate, latency, and retry count, (3) writes structured JSON logs to a file, (4) exposes a /metrics endpoint for Prometheus scraping, and (5) sets up a Grafana dashboard (via provisioning JSON) showing delivery success rate (panel), latency heatmap, top failing consumers table, and event volume time series.

What's Next

Now that you can monitor webhook delivery, apply webhook security best practices to protect your webhook infrastructure from attacks.