Building a Webhook Provider — Complete Guide

In this tutorial, you will learn about Building a Webhook Provider. We cover key concepts, practical examples, and best practices to help you master this topic.

A webhook provider is a system that sends HTTP requests to consumer-registered endpoints when specific events occur. Building a reliable provider requires careful design around event delivery, retry logic, consumer registration, and security. This lesson walks through the architecture and implementation of a webhook provider.

What You'll Learn

• Design a webhook provider architecture with registration and delivery components
• Implement consumer subscription management with endpoint validation
• Build a reliable delivery engine with retry and backoff
• Handle provider-side concerns like rate limiting and event filtering

Why It Matters

Well-designed webhook providers reduce consumer integration effort, improve reliability, and prevent abuse. A poorly built provider leads to consumer churn, support overhead, and fragile integrations. Mastering provider design lets you build the webhook infrastructure that modern platforms require.

Real-World Use

• GitHub provides per-repository webhook configuration with secret tokens and event type filtering
• Stripe sends Webhooks for all account events with automatic retries for up to 3 days
• Slack allows consumers to register multiple URLs per app with different event subscriptions
• Shopify lets merchants configure webhook topics and API version per subscription

Mermaid Flow

graph TD
    A[Consumer Registers URL] --> B{Validate Endpoint}
    B -->|Challenge-Response| C[Store Subscription]
    C --> D[Event Occurs in System]
    D --> E[Match Event to Subscriptions]
    E --> F[Queue Delivery Task]
    F --> G[HTTP POST to Consumer URL]
    G --> H{Response 2xx?}
    H -->|Yes| I[Mark Delivered]
    H -->|No| J[Retry with Backoff]
    J --> K{Max Retries?}
    K -->|No| G
    K -->|Yes| L[Dead Letter Queue]

Teacher's Corner

Emphasize the Separation Of Concerns between event detection, subscription matching, and delivery execution. Each component should be independently scalable. Explain that the challenge-response verification step at registration prevents misconfigured URLs from entering the system. Highlight that the delivery engine must be idempotent so events can be retried without side effects.

Code Examples

Example 1: Subscription Registration with Challenge Verification

import hashlib
import hmac
import json
import secrets
from flask import Flask, request, jsonify

app = Flask(__name__)
subscriptions = []

@app.route("/api/subscribe", methods=["POST"])
def subscribe():
    data = request.get_json()
    url = data["url"]
    events = data.get("events", ["*"])
    secret = secrets.token_hex(32)

    challenge = secrets.token_hex(16)
    sub = {
        "id": secrets.token_hex(8),
        "url": url,
        "events": events,
        "secret": secret,
        "verified": False,
        "challenge": challenge
    }
    subscriptions.append(sub)

    return jsonify({
        "subscription_id": sub["id"],
        "challenge": challenge,
        "verify_url": f"{url}/verify"
    }), 201

@app.route("/api/subscribe/<sub_id>/verify", methods=["POST"])
def verify(sub_id):
    data = request.get_json()
    for sub in subscriptions:
        if sub["id"] == sub_id and data.get("challenge") == sub["challenge"]:
            sub["verified"] = True
            del sub["challenge"]
            return jsonify({"status": "verified"}), 200
    return jsonify({"error": "invalid challenge"}), 400

if __name__ == "__main__":
    app.run(port=5000)

Expected Output: POST /api/subscribe with {"url": "https://example.com/hooks", "events": ["order.created"]} returns 201 with subscription_id and challenge.

Example 2: Event Delivery Engine with Retry

import time
import requests
import threading

class DeliveryEngine:
    def __init__(self):
        self.queue = []
        self.max_retries = 5
        self.base_delay = 60

    def enqueue(self, subscription, event):
        task = {
            "sub": subscription,
            "event": event,
            "attempts": 0,
            "next_attempt": time.time()
        }
        self.queue.append(task)

    def run(self):
        while True:
            now = time.time()
            for task in self.queue[:]:
                if task["next_attempt"] > now:
                    continue
                try:
                    resp = requests.post(
                        task["sub"]["url"],
                        json=task["event"],
                        headers={"Content-Type": "application/json"},
                        timeout=10
                    )
                    if resp.status_code // 100 == 2:
                        self.queue.remove(task)
                        print(f"Delivered {task['event']['id']}")
                    else:
                        task["attempts"] += 1
                        if task["attempts"] >= self.max_retries:
                            self.queue.remove(task)
                            print(f"Dead letter: {task['event']['id']}")
                        else:
                            delay = self.base_delay * (2 ** (task["attempts"] - 1))
                            task["next_attempt"] = time.time() + delay
                except requests.exceptions.RequestException:
                    task["attempts"] += 1
                    if task["attempts"] >= self.max_retries:
                        self.queue.remove(task)
                    else:
                        delay = self.base_delay * (2 ** (task["attempts"] - 1))
                        task["next_attempt"] = time.time() + delay
            time.sleep(1)

engine = DeliveryEngine()
sub = {"url": "https://httpbin.org/post"}
engine.enqueue(sub, {"id": "evt-1", "type": "order.created"})
thread = threading.Thread(target=engine.run, daemon=True)
thread.start()
time.sleep(2)

Expected Output: Delivery attempt logged, with retry delays visible if the endpoint returns non-2xx.

Example 3: Event Type Filtering and Subscription Matching

def match_subscriptions(subscriptions, event_type):
    matched = []
    for sub in subscriptions:
        if not sub.get("verified", False):
            continue
        if "*" in sub["events"]:
            matched.append(sub)
        elif event_type in sub["events"]:
            matched.append(sub)
        elif any(event_type.startswith(prefix.rstrip("*")) for prefix in sub["events"] if prefix.endswith("*")):
            matched.append(sub)
    return matched

subs = [
    {"id": "1", "events": ["*"], "url": "https://all-events.example.com", "verified": True},
    {"id": "2", "events": ["order.created", "order.updated"], "url": "https://orders.example.com", "verified": True},
    {"id": "3", "events": ["user.*"], "url": "https://users.example.com", "verified": True},
    {"id": "4", "events": ["*"], "url": "https://unverified.example.com", "verified": False},
]

print(match_subscriptions(subs, "order.created"))
print(match_subscriptions(subs, "user.deleted"))
print(match_subscriptions(subs, "payment.received"))

Expected Output: [sub1, sub2]; [sub1, sub3]; [sub1] (sub4 is skipped because it is unverified).

Common Mistakes

1. Not validating consumer endpoints during registration, leading to spam or broken subscriptions
2. Using synchronous delivery that blocks the event processing pipeline
3. Implementing infinite retries without a dead-letter mechanism
4. Sending the same event payload to all subscribers without filtering
5. Not including delivery metadata (event ID, timestamp, signature) in the request headers
6. Storing secrets in plaintext in the subscription database
7. Allowing consumers to register invalid or internal network URLs without validation

Practice Questions

1. What is the purpose of a challenge-response verification during subscription?
2. How does exponential backoff help both the provider and consumer systems?
3. Why should delivery tasks be queued instead of sent synchronously?
4. How would you prevent a consumer from subscribing to events they should not receive?
5. Challenge: Design a webhook provider that supports URL-based routing, where different consumer endpoints can receive different event types with independent retry policies, and implements a circuit breaker per consumer URL.

FAQ

Mini Project

Build a complete webhook provider with a Python Flask application. Implement: (1) a subscription API with challenge-response verification, (2) an event simulation endpoint that generates test events, (3) a delivery engine with exponential backoff retry (max 5 retries), (4) a dead-letter queue endpoint that stores failed events, and (5) a dashboard endpoint that shows delivery statistics. Write unit tests for the delivery engine.

What's Next

After building a provider, learn how to be a good webhook consumer and handle incoming webhooks reliably.