Async Messaging Retry Patterns — Complete Implementation Guide
In this tutorial, you will learn about Async Messaging Retry Patterns. We cover key concepts, practical examples, and best practices to help you master this topic.
Async messaging retry patterns handle message processing failures by retrying with backoff, routing to dead letter queues, and ensuring no message is lost due to transient processing errors.
What You'll Learn
By the end of this tutorial, you will implement retry logic for message queues, configure dead letter queues, handle poison messages, and build reliable event processors.
Why It Matters
Messages in a queue represent work that must be done. Losing a message means losing work. DodaTech's Event-Driven Architecture uses retry queues to guarantee message processing.
Real-World Use
DodaZIP uses RabbitMQ for file conversion jobs. If conversion fails, the message is retried 3 times with backoff before being moved to a dead letter queue for manual inspection.
Async Messaging Retry Learning Path
flowchart LR
A[Database Retry] --> B[Async Messaging Retry]
B --> C[Dead Letter Queues]
B --> D[Retry Backoff]
B --> E{You Are Here}
style E fill:#f90,color:#fff
Basic Message Retry
When a message consumer fails to Process a message, the simplest retry is to reject the message so the queue re-delivers it.
const amqp = require("amqplib");
async function startConsumer() {
const connection = await amqp.connect("amqp://localhost");
const channel = await connection.createChannel();
const queue = "file-conversions";
await channel.assertQueue(queue, { durable: true });
channel.consume(queue, async (msg) => {
try {
const job = JSON.parse(msg.content.toString());
await processJob(job);
channel.ack(msg);
} catch (err) {
console.log(`Processing failed: ${err.message}`);
// Reject and requeue
channel.nack(msg, false, true);
}
});
}
Expected behavior: If processing fails, the message is requeued and redelivered. Without limits, this retries forever.
Retry with Dead Letter Queue
A dead letter queue (DLQ) receives messages that exceed the maximum retry count. This prevents infinite retry loops.
async function setupRetryQueues(channel) {
// Dead letter queue
await channel.assertQueue("file-conversions-dlq", { durable: true });
// Main queue with dead letter exchange
await channel.assertQueue("file-conversions", {
durable: true,
arguments: {
"x-dead-letter-exchange": "",
"x-dead-letter-routing-key": "file-conversions-dlq",
"x-message-ttl": 30000,
"x-max-retries": 3
}
});
channel.consume("file-conversions", async (msg) => {
const retryCount = (msg.properties.headers["x-retry-count"] || 0);
try {
const job = JSON.parse(msg.content.toString());
await processJob(job);
channel.ack(msg);
} catch (err) {
if (retryCount >= 3) {
console.log(`Moving to DLQ after ${retryCount} retries`);
channel.nack(msg, false, false);
} else {
console.log(`Retry ${retryCount + 1}/3`);
const retryMsg = {
...msg,
properties: {
...msg.properties,
headers: {
...msg.properties.headers,
"x-retry-count": retryCount + 1
}
}
};
channel.nack(msg, false, true);
}
}
});
}
Programmatic Retry with Backoff
For more control, implement retry with exponential backoff by publishing to a delayed retry queue.
async function processWithBackoffRetry(channel, msg) {
const retryCount = msg.properties.headers["x-retry-count"] || 0;
const maxRetries = 5;
try {
const job = JSON.parse(msg.content.toString());
await processJob(job);
channel.ack(msg);
} catch (err) {
if (retryCount >= maxRetries) {
console.log(`Failed after ${maxRetries} retries, sending to DLQ`);
channel.nack(msg, false, false);
return;
}
const delay = Math.min(1000 * Math.pow(2, retryCount), 60000);
const retryQueue = `retry-${delay}ms`;
// Create a delayed retry queue
await channel.assertQueue(retryQueue, {
durable: true,
deadLetterExchange: "",
deadLetterRoutingKey: "file-conversions",
messageTtl: delay
});
// Publish to retry queue with incremented retry count
channel.sendToQueue(retryQueue, msg.content, {
persistent: true,
headers: {
"x-retry-count": retryCount + 1,
"x-original-queue": "file-conversions"
}
});
channel.ack(msg);
}
}
Common Mistakes
Not setting a maximum retry count -- Without limits, a poison message retries forever, consuming resources and blocking other messages.
Requeuing without backoff -- Immediate requeue creates a tight loop. Use TTL-based retry queues for delayed retries.
Losing message content in retries -- Ensure the full original message is preserved through retry cycles.
Not monitoring dead letter queues -- A growing DLQ indicates systemic issues. Monitor and alert on DLQ depth.
Ignoring poison messages -- Messages that always fail (malformed data) need human inspection. Alert when messages enter the DLQ.
Practice Questions
What is a dead letter queue? A queue that receives messages that cannot be processed after exhausting retries. It stores failed messages for inspection.
How do you implement delayed retry in RabbitMQ? Create a queue with a TTL that routes back to the original queue after the TTL expires.
What is a poison message? A message that always fails processing due to malformed content or invalid data. It poisons any consumer that receives it.
Challenge: Implement a retry system that uses separate queues for each retry delay level.
// retry-1s-queue -> after 1s -> main-queue
// retry-2s-queue -> after 2s -> main-queue
// retry-4s-queue -> after 4s -> main-queue
// Each retry queue has a different TTL and routes back to main
FAQ
Mini Project
Build a complete message processing system with retry queues, exponential backoff, dead letter queue, and monitoring.
class ReliableMessageProcessor {
constructor(channel) {
this.channel = channel;
this.mainQueue = "jobs";
this.dlq = "jobs-dlq";
this.maxRetries = 5;
}
async setup() {
await this.channel.assertQueue(this.dlq, { durable: true });
await this.channel.assertQueue(this.mainQueue, {
durable: true,
arguments: {
"x-dead-letter-exchange": "",
"x-dead-letter-routing-key": this.dlq
}
});
await this.channel.consume(this.mainQueue, (msg) => this.handleMessage(msg));
}
async handleMessage(msg) {
const retryCount = msg.properties.headers["x-retry-count"] || 0;
try {
await this.process(JSON.parse(msg.content.toString()));
this.channel.ack(msg);
} catch (err) {
if (retryCount >= this.maxRetries) {
console.error(`Failed after ${this.maxRetries} retries`);
this.channel.nack(msg, false, false);
return;
}
const delay = Math.min(1000 * Math.pow(2, retryCount), 60000);
const delayQueue = `delay-${delay}ms`;
await this.channel.assertQueue(delayQueue, {
durable: true,
messageTtl: delay,
deadLetterExchange: "",
deadLetterRoutingKey: this.mainQueue
});
this.channel.sendToQueue(delayQueue, msg.content, {
persistent: true,
headers: {
"x-retry-count": retryCount + 1,
"x-original-queue": this.mainQueue
}
});
this.channel.ack(msg);
}
}
async process(job) {
// Implement actual processing logic
if (Math.random() < 0.5) {
throw new Error("Transient processing failure");
}
console.log(`Processed job: ${job.id}`);
}
}
What's Next
Now that you understand async messaging retry, explore defining retry policies for different operations. Then learn about testing retry logic.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro