Building a Webhook Consumer — Complete Guide
In this tutorial, you will learn about Building a Webhook Consumer. We cover key concepts, practical examples, and best practices to help you master this topic.
Learn to build a webhook consumer: receive webhooks, verify signatures, handle idempotency, process asynchronously, log deliveries, and build a robust webhook endpoint with Express.
What You Learn
You will build a production-ready webhook consumer that receives incoming webhooks, verifies authenticity with HMAC, prevents duplicate processing with idempotency, processes events asynchronously, and provides monitoring endpoints.
Why It Matters
Every webhook integration needs a consumer. A well-built consumer prevents security issues, duplicate processing, and data loss. A poorly built consumer misses events, processes duplicates, and is vulnerable to fake webhooks.
Real-World Use
DodaTech's billing system consumes webhooks from Stripe, Paddle, and PayPal. The consumer processes 50K payment events daily, with idempotency preventing duplicate charges and signature verification blocking fake payment notifications.
Consumer Architecture
graph LR
Internet[Internet / Provider] -->|POST /webhook| Receiver[Webhook Receiver]
Receiver --> Verifier[Signature Verifier]
Verifier --> Idempotency[Idempotency Check]
Idempotency --> Queue[Task Queue]
Queue --> Worker1[Worker 1]
Queue --> Worker2[Worker 2]
Queue --> WorkerN[Worker N]
Idempotency --> Store[(Idempotency Store)]
Receiver --> Logger[Request Logger]
Logger --> LogStore[(Delivery Log)]
The consumer receives the POST, verifies the signature, checks idempotency, queues the work asynchronously, and logs the delivery.
Basic Webhook Consumer
const express = require('express');
const crypto = require('crypto');
const app = express();
const PORT = process.env.PORT || 3000;
const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET || 'whsec_your_secret_here';
// Capture raw body for signature verification
app.use(express.json({
verify: (req, res, buf) => {
req.rawBody = buf.toString();
},
}));
// Webhook endpoint
app.post('/webhook', async (req, res) => {
const startTime = Date.now();
try {
// 1. Verify signature
const signature = req.headers['x-webhook-signature'];
if (!verifySignature(req.rawBody, signature, WEBHOOK_SECRET)) {
console.error('Invalid webhook signature');
return res.status(401).send('Invalid signature');
}
// 2. Verify timestamp (prevent replay)
const timestamp = req.headers['x-webhook-timestamp'];
if (timestamp && isWebhookStale(timestamp)) {
console.error('Stale webhook');
return res.status(400).send('Stale webhook');
}
// 3. Check idempotency
const webhookId = req.headers['x-webhook-id'] || req.body.id;
if (webhookId) {
const processed = await checkIdempotency(webhookId);
if (processed) {
console.log(`Duplicate webhook ${webhookId}`);
return res.status(200).send('Already processed');
}
await markProcessing(webhookId);
}
// 4. Acknowledge immediately
res.status(200).send('OK');
// 5. Process asynchronously
setImmediate(() => {
processWebhookAsync(req.body)
.then(() => markCompleted(webhookId))
.catch(err => {
console.error(`Webhook processing failed:`, err);
markFailed(webhookId);
});
});
// 6. Log delivery
logDelivery({
webhookId,
event: req.body.event || req.body.type,
status: 'accepted',
duration: Date.now() - startTime,
});
} catch (err) {
console.error('Webhook handler error:', err);
res.status(500).send('Internal error');
}
});
function verifySignature(rawBody, signatureHeader, secret) {
if (!signatureHeader || !rawBody) return false;
const expectedSig = signatureHeader.replace('sha256=', '');
const computedSig = crypto
.createHmac('sha256', secret)
.update(rawBody, 'utf8')
.digest('hex');
return crypto.timingSafeEqual(
Buffer.from(computedSig),
Buffer.from(expectedSig)
);
}
function isWebhookStale(timestamp) {
const age = Date.now() - new Date(timestamp).getTime();
return age > 300000; // 5 minutes
}
app.listen(PORT, () => {
console.log(`Webhook consumer on port ${PORT}`);
});
Expected output: Consumer verifies every webhook, rejects invalid signatures with 401, rejects stale webhooks with 400, acknowledges duplicates with 200, and processes valid webhooks asynchronously.
Idempotency Store
const redis = require('redis');
class IdempotencyStore {
constructor() {
this.client = redis.createClient({ url: process.env.REDIS_URL });
this.TTL = 86400; // 24 hours
}
async checkAndSet(webhookId) {
const result = await this.client.set(
`wh:${webhookId}`,
'processing',
{ EX: this.TTL, NX: true }
);
return result !== null; // true if first time
}
async markCompleted(webhookId) {
await this.client.set(
`wh:${webhookId}`,
'completed',
{ EX: this.TTL }
);
}
async markFailed(webhookId) {
await this.client.del(`wh:${webhookId}`);
}
async isProcessed(webhookId) {
const status = await this.client.get(`wh:${webhookId}`);
return status === 'completed';
}
}
Expected output: Redis-backed idempotency store with atomic check-and-set, TTL-based expiration, and status tracking for processing lifecycle.
Processing Events
// Event type handlers
const eventHandlers = {
'payment.succeeded': async (data) => {
console.log(`Payment succeeded: ${data.id}`);
await updateOrderStatus(data.orderId, 'paid');
await sendReceipt(data.customerEmail, data.amount);
},
'payment.failed': async (data) => {
console.log(`Payment failed: ${data.id}`);
await updateOrderStatus(data.orderId, 'failed');
await notifyCustomer(data.customerId, 'Payment failed');
},
'customer.subscription.updated': async (data) => {
console.log(`Subscription updated: ${data.id}`);
await updateSubscription(data.id, data.status, data.plan);
await syncWithBilling(data);
},
'charge.refunded': async (data) => {
console.log(`Charge refunded: ${data.id}`);
await processRefund(data.chargeId, data.amount);
await updateOrderStatus(data.orderId, 'refunded');
},
};
async function processWebhookAsync(payload) {
const eventType = payload.type || payload.event;
const handler = eventHandlers[eventType];
if (!handler) {
console.log(`No handler for event type: ${eventType}`);
return;
}
console.log(`Processing ${eventType}...`);
await handler(payload.data);
console.log(`Completed ${eventType}`);
}
Expected output: Each event type has a dedicated handler. Unknown event types are logged and skipped. Handlers process data and call external services as needed.
Monitoring and Health
// Consumer health and metrics
class ConsumerMetrics {
constructor() {
this.metrics = {
totalReceived: 0,
accepted: 0,
rejected: 0,
duplicate: 0,
failed: 0,
processingDurations: [],
};
this.startTime = Date.now();
}
record(status, duration) {
this.metrics.totalReceived++;
this.metrics[status]++;
if (duration) {
this.metrics.processingDurations.push(duration);
if (this.metrics.processingDurations.length > 1000) {
this.metrics.processingDurations.shift();
}
}
}
getStats() {
const durations = this.metrics.processingDurations;
const avgDuration = durations.length > 0
? durations.reduce((a, b) => a + b, 0) / durations.length
: 0;
return {
uptime: Math.floor((Date.now() - this.startTime) / 1000),
totalReceived: this.metrics.totalReceived,
accepted: this.metrics.accepted,
rejected: this.metrics.rejected,
duplicate: this.metrics.duplicate,
failed: this.metrics.failed,
avgProcessingDuration: Math.round(avgDuration),
successRate: this.metrics.totalReceived > 0
? ((this.metrics.accepted / this.metrics.totalReceived) * 100).toFixed(2)
: '100.00',
};
}
}
const metrics = new ConsumerMetrics();
// Metrics endpoint
app.get('/webhook/metrics', (req, res) => {
res.json(metrics.getStats());
});
// Health check
app.get('/health', (req, res) => {
const stats = metrics.getStats();
const healthy = stats.successRate > 90;
res.status(healthy ? 200 : 503).json({
status: healthy ? 'healthy' : 'degraded',
...stats,
});
});
Expected output: Metrics endpoint returns consumer statistics including total received, accepted, rejected, and processing duration. Health endpoint returns 503 if success rate drops below 90%.
Common Mistakes
1. Blocking Response on Processing
Returning the response only after processing completes causes timeouts and retries. Acknowledge immediately (200 OK). Process asynchronously using a task queue or setImmediate.
2. Missing Idempotency
Without idempotency, retried webhooks process multiple times. Implement idempotency storage. Check before processing. Return 200 for duplicates without re-processing.
3. No Signature Verification
Without signature verification, anyone can send fake webhooks. Always verify HMAC signatures. Use timing-safe comparison. Reject invalid signatures with 401.
4. Not Logging Raw Requests
Without raw request logs, debugging delivery issues is impossible. Log headers, raw body, response status, and processing result. Include timestamps for audit trail.
5. No Error Handling in Async Processing
Unhandled promise rejections in async processing crash the process silently. Wrap async processing in try/catch. Log errors. Implement retry logic for transient failures.
Practice Questions
1. Why must the webhook consumer acknowledge immediately?
Providers have timeouts (5-30 seconds). If the consumer takes too long, the provider retries. Acknowledge with 200 OK within seconds. Process the work asynchronously.
2. How do you handle unknown event types?
Log the unknown event type and acknowledge receipt. Do not crash or return an error. The provider may have added new event types. Forward the event to a dead letter queue for inspection.
3. What is the purpose of the verify middleware in Express?
express.json() with verify callback captures the raw request body before JSON Parsing. HMAC signature must be computed on the raw bytes. Parsed JSON may have different whitespace or key ordering.
4. How do you test webhook consumers locally?
Use ngrok to expose your local server. Most providers have test mode or test secrets. Send test webhooks from the provider dashboard. Verify signature, idempotency, and processing logic.
Challenge
Build a webhook consumer that handles webhooks from Stripe, GitHub, and a custom provider simultaneously. Each provider has different signature schemes, headers, and payload formats. Implement unified idempotency and logging across all providers.
FAQ
Mini Project: Multi-Provider Webhook Consumer
Build a consumer that receives webhooks from 3 providers (Stripe, GitHub, SendGrid). Each endpoint has provider-specific signature verification. Unified idempotency with Redis. Async processing with a task queue. Dashboard showing per-provider metrics and delivery logs.
What's Next
Now that you can consume webhooks, implement provider-specific integrations like Webhook with Express for framework-specific patterns.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro