Webhook Ordering — Complete Guide
In this tutorial, you will learn about Webhook Ordering. We cover key concepts, practical examples, and best practices to help you master this topic.
Learn webhook ordering challenges: out-of-order delivery, sequence tracking, ordering strategies, idempotency with ordering, and designing systems that handle unordered webhook delivery.
What You Learn
You will learn why Webhooks arrive out of order, how to detect out-of-order delivery, strategies for reordering webhooks, and how to design idempotent processing that handles unordered events without data corruption.
Why It Matters
Most webhook providers do not guarantee delivery order. Network delays, retries, and multi-server architectures deliver event 3 before event 2. Without ordering handling, out-of-order webhooks create inconsistent state, missed updates, and data corruption.
Real-World Use
DodaTech's order processing system receives webhooks for order.created, order.shipped, and order.delivered. A retry of order.shipped can arrive after order.delivered, causing the system to revert to shipped status. Sequence tracking prevents this regression.
Why Webhooks Arrive Out of Order
sequenceDiagram
participant Provider as Webhook Provider
participant Server1 as Server 1
participant Server2 as Server 2
participant Consumer as Consumer
Provider->>Provider: Event A (order.created)
Provider->>Server1: Deliver A
Provider->>Provider: Event B (order.updated)
Provider->>Server2: Deliver B
Server1-->>Consumer: A arrives (fast server)
Server2-->>Consumer: B arrives (slow server, arrives after A)
Note over Consumer: Order: A then B ✓
Provider->>Provider: Event C (order.shipped)
Provider->>Consumer: Deliver C
Provider->>Provider: Retry D (order.updated, previously failed)
Provider->>Consumer: Deliver D (arrives after C)
Note over Consumer: Order: C before D ✗
Network latency, server load differences, and retries all cause out-of-order delivery. A webhook that fired later can arrive before an earlier one.
Sequence Number Tracking
// Provider includes sequence numbers
class ProviderWithSequencing {
constructor() {
this.counter = 0;
this.sequences = new Map(); // resourceId -> sequence number
}
buildWebhook(resourceId, eventType, data) {
this.counter++;
const seq = this.sequences.get(resourceId) || 0;
const newSeq = seq + 1;
this.sequences.set(resourceId, newSeq);
return {
id: `wh_${this.counter}`,
event: eventType,
resourceId,
sequence: newSeq,
timestamp: Date.now(),
data,
};
}
}
// Consumer tracks sequences
class SequenceTracker {
constructor() {
this.sequences = new Map(); // resourceId -> last processed sequence
}
shouldProcess(webhook) {
const { resourceId, sequence } = webhook;
const lastSeq = this.sequences.get(resourceId) || 0;
if (sequence <= lastSeq) {
console.log(`Out of order: ${resourceId} seq ${sequence} <= ${lastSeq}`);
return {
process: false,
reason: 'out_of_order',
lastSeq,
currentSeq: sequence,
};
}
return { process: true };
}
markProcessed(resourceId, sequence) {
this.sequences.set(resourceId, sequence);
}
}
Expected output: Sequence numbers per resource allow the consumer to detect gaps and out-of-order delivery. Events with sequence <= last processed are rejected.
Reordering with Buffer
// Buffer for reordering webhooks by resource
class WebhookReorderer {
constructor(options = {}) {
this.buffers = new Map(); // resourceId -> sorted events[]
this.maxBufferSize = options.maxBufferSize || 50;
this.maxWaitMs = options.maxWaitMs || 5000;
this.processor = options.processor;
}
addEvent(webhook) {
const { resourceId, sequence } = webhook;
let buffer = this.buffers.get(resourceId);
if (!buffer) {
buffer = {
events: [],
lastProcessed: 0,
timer: null,
};
this.buffers.set(resourceId, buffer);
}
buffer.events.push(webhook);
buffer.events.sort((a, b) => a.sequence - b.sequence);
// Trim excess events
if (buffer.events.length > this.maxBufferSize) {
buffer.events = buffer.events.slice(-this.maxBufferSize);
}
// Try to process consecutive events
this.tryProcess(resourceId);
// Set timeout to flush remaining
if (!buffer.timer) {
buffer.timer = setTimeout(() => {
this.flushBuffer(resourceId);
}, this.maxWaitMs);
}
}
tryProcess(resourceId) {
const buffer = this.buffers.get(resourceId);
if (!buffer || buffer.events.length === 0) return;
const expected = buffer.lastProcessed + 1;
while (buffer.events.length > 0) {
const next = buffer.events[0];
if (next.sequence === expected) {
buffer.events.shift();
this.processor(next);
buffer.lastProcessed = expected;
expected++;
} else if (next.sequence < expected) {
// Duplicate or old event, discard
buffer.events.shift();
} else {
break; // Gap - wait for missing event
}
}
}
flushBuffer(resourceId) {
const buffer = this.buffers.get(resourceId);
if (!buffer) return;
// Process remaining events in order despite gaps
buffer.events.sort((a, b) => a.sequence - b.sequence);
for (const event of buffer.events) {
if (event.sequence > buffer.lastProcessed) {
this.processor(event);
buffer.lastProcessed = event.sequence;
}
}
buffer.events = [];
buffer.timer = null;
}
}
Expected output: Events are buffered per resource, sorted by sequence number. Consecutive events are processed immediately. Gaps wait for up to 5 seconds before flushing. Old events are discarded.
Idempotent Ordering
// Combine ordering with idempotency
class OrderedIdempotentProcessor {
constructor(options = {}) {
this.store = options.store; // Redis or DB
this.processor = options.processor;
}
async handleWebhook(req, res) {
const { id: webhookId, resourceId, sequence, event, data } = req.body;
// Check idempotency first
const processed = await this.store.get(`wh:order:${webhookId}`);
if (processed) {
return res.status(200).json({ status: 'duplicate' });
}
// Check sequence
const lastSequence = await this.store.get(`wh:seq:${resourceId}`) || 0;
if (sequence <= lastSequence) {
// Out of order - store for later processing
await this.store.zadd(`wh:pending:${resourceId}`, sequence, webhookId);
await this.store.set(`wh:pending:${webhookId}`, JSON.stringify(req.body), { EX: 3600 });
res.status(202).json({
status: 'queued',
message: 'Out of order, queued for reprocessing',
});
return;
}
// Process if sequence is next
if (sequence === lastSequence + 1) {
await this.processAndAdvance(resourceId, webhookId, req.body);
// Check for queued events
await this.processQueued(resourceId);
return res.status(200).json({ status: 'processed' });
}
// Gap detected - store for later
await this.store.zadd(`wh:pending:${resourceId}`, sequence, webhookId);
await this.store.set(`wh:pending:${webhookId}`, JSON.stringify(req.body), { EX: 3600 });
res.status(202).json({
status: 'queued',
message: 'Sequence gap, queued for later',
});
}
async processAndAdvance(resourceId, webhookId, body) {
await this.processor(body);
await this.store.set(`wh:order:${webhookId}`, 'processed', { EX: 86400 });
await this.store.incr(`wh:seq:${resourceId}`);
}
async processQueued(resourceId) {
while (true) {
const currentSeq = await this.store.get(`wh:seq:${resourceId}`) || 0;
const nextExpected = currentSeq + 1;
const next = await this.store.zrangebyscore(
`wh:pending:${resourceId}`,
nextExpected,
nextExpected
);
if (next.length === 0) break;
const webhookId = next[0];
const bodyStr = await this.store.get(`wh:pending:${webhookId}`);
if (!bodyStr) break;
await this.processAndAdvance(resourceId, webhookId, JSON.parse(bodyStr));
await this.store.zrem(`wh:pending:${resourceId}`, webhookId);
await this.store.del(`wh:pending:${webhookId}`);
}
}
}
Expected output: Webhooks are checked for idempotency, then sequence. In-order events are processed immediately. Out-of-order events are queued in a sorted set. When the missing event arrives, queued events are processed in sequence.
Common Mistakes
1. Assuming Order Is Guaranteed
No major provider guarantees webhook delivery order. Always assume out-of-order delivery. Design your system to reorder or tolerate unordered events.
2. Ignoring Gaps
A missing event does not mean it will never arrive. It may be delayed by retries. Buffer events and wait a reasonable time before processing gaps. Set a maximum wait window.
3. Processing Events Without Context
If order.shipped arrives before order.created, you cannot Process it without the order context. Store pending events and process them when the missing context arrives.
4. No Maximum Buffer Size
Buffered events consume memory indefinitely. Set a maximum buffer size (50-100 events per resource). Flush old events or reject them beyond the limit.
5. Treating All Resources the Same
Different resources have different ordering requirements. Order events need strict ordering. Analytics events can tolerate out-of-order delivery. Apply ordering only where needed.
Practice Questions
1. Why do webhooks arrive out of order?
Network latency differences, retries, and multi-server architectures cause out-of-order delivery. A webhook that fired later may take a faster network path or a retry may deliver after a newer event.
2. How does sequence tracking help with ordering?
Each event carries a sequence number per resource. The consumer tracks the last processed sequence number. Events with sequence <= processed are rejected as out of order or duplicate.
3. What is the purpose of the reorder buffer?
It holds out-of-order events temporarily while waiting for missing events. Events are sorted by sequence and processed when gaps are filled. A timeout flushes remaining events to prevent infinite waiting.
4. How do you handle events that never arrive (permanently missing)?
Set a maximum wait time (5-60 seconds). After the timeout, flush the buffer and process events despite gaps. If the missing event arrives later, skip it as duplicate.
Challenge
Build a webhook ordering system for an e-commerce platform. Events: order.created (seq 1), order.payment_received (seq 2), order.shipped (seq 3), order.delivered (seq 4). Simulate out-of-order delivery. Reorder and process in sequence. Handle gaps with 10-second timeout. Log reordering statistics.
FAQ
Mini Project: Webhook Order Monitor
Build a monitor that tracks webhook ordering for a system. Show: events received per resource, sequence gaps, buffered events, out-of-order rate (%), average reorder delay, and events discarded as duplicates. Alert when out-of-order rate exceeds 5%.
What's Next
Now that you understand ordering, learn how to Build a Webhook Provider that handles signing, retries, ordering, and delivery.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro