Idempotency for Retries — Complete Implementation Guide
In this tutorial, you will learn about Idempotency for Retries. We cover key concepts, practical examples, and best practices to help you master this topic.
Idempotency ensures that retrying an operation produces the same result as executing it once, preventing duplicate charges, duplicate records, and data corruption in unreliable networks.
What You'll Learn
By the end of this tutorial, you will implement idempotency keys, design idempotent APIs, handle duplicate requests safely, and prevent data corruption during retries.
Why It Matters
Without idempotency, retrying a payment or order creation charges the customer twice. DodaTech's payment and order APIs use idempotency keys to ensure safe retries.
Real-World Use
DodaZIP's subscription API uses idempotency keys on payment endpoints. If the response is lost, the client retries with the same key and receives the previous result without double-charging.
Idempotency Learning Path
flowchart LR
A[Circuit Breaker] --> B[Idempotency]
B --> C[Idempotency Keys]
C --> D[Safe APIs]
B --> E{You Are Here}
style E fill:#f90,color:#fff
The Problem: Duplicate Operations
When a network failure occurs after the server processes a request but before the client receives the response, the client retries and creates a duplicate.
// Problem scenario:
// 1. Client sends POST /orders
// 2. Server creates order (id: 42)
// 3. Network fails before response reaches client
// 4. Client retries - server creates order (id: 43) - DUPLICATE!
Idempotency Key Solution
An idempotency key is a unique identifier the client sends with the request. The server stores the result keyed by this identifier. On retry with the same key, the server returns the stored result.
const express = require("express");
const crypto = require("crypto");
const app = express();
app.use(express.json());
const idempotencyStore = new Map();
const CACHE_TTL = 86400000;
function idempotencyMiddleware(req, res, next) {
if (req.method !== "POST") return next();
const key = req.headers["idempotency-key"];
if (!key) {
return res.status(400).json({ error: "Idempotency-Key header required" });
}
const cached = idempotencyStore.get(key);
if (cached) {
console.log(`Idempotency hit for key ${key}`);
return res.status(cached.status).json(cached.body);
}
const originalJson = res.json.bind(res);
res.json = function (body) {
idempotencyStore.set(key, {
status: res.statusCode,
body,
timestamp: Date.now()
});
return originalJson(body);
};
next();
}
app.post("/orders", idempotencyMiddleware, (req, res) => {
const order = { id: Date.now(), product: req.body.product };
console.log(`Created order ${order.id}`);
res.status(201).json({ order });
});
// Cleanup old entries
setInterval(() => {
const cutoff = Date.now() - CACHE_TTL;
for (const [key, value] of idempotencyStore) {
if (value.timestamp < cutoff) idempotencyStore.delete(key);
}
}, 3600000);
app.listen(3000);
Designing Idempotent APIs
Certain HTTP methods are naturally idempotent. Others require explicit idempotency support.
| Method | Naturally Idempotent | Notes |
|---|---|---|
| GET | Yes | Reading is always safe |
| PUT | Yes | Same PUT produces same state |
| DELETE | Yes | Deleting a deleted resource is no-op |
| POST | No | Creates new resources - needs idempotency key |
| PATCH | No | May not be idempotent - needs care |
Idempotent Database Operations
Database operations can be made idempotent using unique constraints and upsert patterns.
// Non-idempotent: INSERT creates duplicate on retry
async function createUserNonIdempotent(email, name) {
await db.query("INSERT INTO users (email, name) VALUES ($1, $2)", [email, name]);
}
// Idempotent: ON CONFLICT DO NOTHING prevents duplicates
async function createUserIdempotent(email, name) {
await db.query(`
INSERT INTO users (email, name) VALUES ($1, $2)
ON CONFLICT (email) DO NOTHING
`, [email, name]);
}
// Idempotent: upsert pattern for updates
async function updateUserIdempotent(id, data) {
await db.query(`
INSERT INTO users (id, name, email) VALUES ($1, $2, $3)
ON CONFLICT (id) DO UPDATE SET name = $2, email = $3
`, [id, data.name, data.email]);
}
Common Mistakes
Not requiring idempotency keys for POST -- Without keys, retried POST requests create duplicates. Always require them for write operations.
Storing idempotency results forever -- Storage grows unbounded. Set a TTL (24 hours is standard) and clean up old entries.
Returning 200 on idempotent replay of 201 -- Return the original status code. A retried creation should still return 201.
Not validating idempotency key format -- Accept UUIDs only. Reject malformed keys to prevent collisions.
Forgetting to include idempotency in error responses -- If the first attempt failed with 500, the retry with the same key should reprocess, not return the cached error.
Practice Questions
What is an idempotency key? A unique identifier sent by the client that allows the server to detect and safely handle duplicate requests.
Why are PUT and DELETE naturally idempotent? PUT replaces the resource state, so the same PUT applied twice produces the same state. DELETE removes the resource, and deleting it again is a no-op.
How long should idempotency keys be stored? 24 hours is standard. Some APIs use 7 days for payment operations.
Challenge: Implement an idempotency middleware that removes the key after the first successful response but stores it temporarily for retry detection.
function idempotent(duration = 86400000) {
const store = new Map();
return (req, res, next) => {
const key = req.headers["idempotency-key"];
if (!key && req.method === "POST") {
return res.status(400).json({ error: "Missing Idempotency-Key" });
}
if (store.has(key)) return res.json(store.get(key).body);
// ... intercept res.json to cache
};
}
FAQ
Mini Project
Build an idempotent API with middleware, proper status code preservation, TTL-based cleanup, and integration with a retry-aware HTTP client.
class IdempotencyMiddleware {
constructor(ttl = 86400000) {
this.store = new Map();
this.ttl = ttl;
this.startCleanup();
}
middleware() {
return (req, res, next) => {
if (!["POST", "PATCH", "PUT"].includes(req.method)) return next();
const key = req.headers["idempotency-key"];
if (req.method === "POST" && !key) {
return res.status(400).json({ error: "Idempotency-Key required for POST" });
}
if (key && this.store.has(key)) {
const cached = this.store.get(key);
return res.status(cached.status).json(cached.body);
}
if (key) {
const originalJson = res.json.bind(res);
res.json = (body) => {
this.store.set(key, { status: res.statusCode, body, timestamp: Date.now() });
return originalJson(body);
};
}
next();
};
}
startCleanup() {
setInterval(() => {
const cutoff = Date.now() - this.ttl;
for (const [key, val] of this.store) {
if (val.timestamp < cutoff) this.store.delete(key);
}
}, 3600000);
}
}
What's Next
Now that you understand idempotency, explore retry strategies for HTTP clients. Then learn about retrying database operations.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro