Chaos Engineering for Retry -- Resilience Testing
In this tutorial, you will learn about Chaos Engineering for Retry. We cover key concepts, practical examples, and best practices to help you master this topic.
Apply chaos engineering to validate retry strategies by injecting failures and measuring recovery.
What You Will Learn
By the end of this tutorial you will understand the chaos engineering pattern, implement it in production code, avoid common pitfalls, and integrate it with other backend components.
Why It Matters
Chaos Engineering is a critical building block in backend systems. Getting it wrong causes security vulnerabilities, data loss, or poor performance. DodaTech uses this pattern across all production services.
Real-World Use
Doda Browser's sync service implements chaos engineering to handle thousands of concurrent requests. Durga Antivirus Pro uses this pattern for reliable scanning pipeline integrity.
Learning Path
flowchart LR
A[Backend Fundamentals] --> B[Chaos Engineering]
B --> C[Advanced Patterns]
B --> D{"You Are Here"}
style D fill:#f90,color:#fff
Core Implementation
The standard implementation follows a clear pattern that separates concerns and enables testing.
// chaos-engineering -- basic implementation
// This shows the core pattern for chaos engineering
class ChaosEngineeringHandler {
constructor(options) {
this.options = options || {};
this.metrics = { count: 0, errors: 0 };
}
async handle(request) {
this.metrics.count++;
try {
const result = await this.process(request);
return result;
} catch (err) {
this.metrics.errors++;
throw err;
}
}
async process(request) {
// Simulated processing
return { status: "ok", request };
}
getMetrics() {
return this.metrics;
}
}
const handler = new ChaosEngineeringHandler({});
handler.handle("test").then(console.log);
Expected output:
{"status": "ok", "request": "test"}
Advanced Configuration
Production systems need configurable parameters for chaos engineering.
// configurable-chaos-engineering.js
const config = {
enabled: process.env.FEATURE_ENABLED !== "false",
timeout: parseInt(process.env.TIMEOUT_MS || "5000"),
maxRetries: parseInt(process.env.MAX_RETRIES || "3"),
logLevel: process.env.LOG_LEVEL || "info",
whitelist: (process.env.IP_WHITELIST || "").split(",").filter(Boolean),
};
class ConfigurableChaosEngineeringHandler {
constructor(config) {
this.config = config;
this.validateConfig();
}
validateConfig() {
if (this.config.timeout < 100) {
throw new Error("timeout must be at least 100ms");
}
if (this.config.maxRetries < 0 || this.config.maxRetries > 10) {
throw new Error("maxRetries must be 0-10");
}
}
async run(request) {
const start = Date.now();
try {
const result = await this.processWithTimeout(request);
return result;
} finally {
const elapsed = Date.now() - start;
if (elapsed > this.config.timeout) {
console.warn("Request exceeded timeout: " + elapsed + "ms");
}
}
}
async processWithTimeout(request) {
return Promise.race([
this.process(request),
new Promise((_, reject) =>
setTimeout(() => reject(new Error("timeout")), this.config.timeout)
),
]);
}
async process(request) {
return { config: this.config, result: request };
}
}
const h = new ConfigurableChaosEngineeringHandler(config);
h.run("test").then(r => console.log(JSON.stringify(r)));
Expected output:
{"config": {"enabled": true, "timeout": 5000, "maxRetries": 3}, "result": "test"}
Error Handling
Proper error handling prevents crashes and ensures consistent failure responses.
// error-handling-chaos-engineering.js
class AppError extends Error {
constructor(message, statusCode = 500, code = "INTERNAL_ERROR") {
super(message);
this.statusCode = statusCode;
this.code = code;
this.timestamp = new Date().toISOString();
this.requestId = crypto.randomUUID();
}
toJSON() {
return {
error: this.code,
message: this.message,
requestId: this.requestId,
timestamp: this.timestamp,
};
}
}
class ValidationError extends AppError {
constructor(message) {
super(message, 400, "VALIDATION_ERROR");
}
}
class AuthError extends AppError {
constructor(message) {
super(message, 401, "AUTH_ERROR");
}
}
const errorHandler = {
handle(err, req) {
if (err instanceof AppError) {
return err.toJSON();
}
return new AppError("Internal server error", 500).toJSON();
},
};
const testErr = new ValidationError("Invalid input");
console.log(JSON.stringify(errorHandler.handle(testErr), null, 2));
Expected output:
{
"error": "VALIDATION_ERROR",
"message": "Invalid input",
"requestId": "...",
"timestamp": "..."
}
Common Mistakes
Not handling edge cases -- Empty inputs, null values, and unexpected data types cause runtime errors. Always validate and handle edge cases in your chaos engineering implementation.
Ignoring timeout configuration -- Default timeouts that are too long cause resource exhaustion. Too short causes false failures. Tune timeouts based on real-world measurements.
Missing Observability -- Without logging, metrics, and tracing, debugging chaos engineering issues in production requires guesswork. Add structured logging from day one.
Hardcoding configuration -- Environment-specific values like endpoints, limits, and credentials must come from configuration, not code. Use environment variables or a config service.
Forgetting cleanup on errors -- Resources like database connections, file handles, and network sockets must be released even when errors occur. Use try/finally or context managers.
Practice Questions
What is the primary purpose of chaos engineering in backend applications? It provides a standardized pattern for handling chaos engineering across your application, ensuring consistency and reducing code duplication.
When should you avoid using this pattern? When the overhead of the abstraction exceeds its benefits, such as in very simple operations or extreme performance-critical paths.
How do you test implementations of this pattern? Unit test the core logic in isolation, integration test with real dependencies, and use contract tests for distributed components.
How does this pattern interact with error handling middleware? Errors should be caught at each layer, logged with context, and propagated to a centralized error handler that returns consistent responses.
Challenge: Build a minimal implementation that handles concurrent requests, logs each operation, includes timeout protection, and has configurable retry logic.
const crypto = require("crypto");
class MiniHandler {
constructor(opts = {}) {
this.timeout = opts.timeout || 5000;
this.maxRetries = opts.maxRetries || 3;
this._metrics = { handled: 0, timedOut: 0, errors: 0 };
}
async handle(request) {
for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
try {
this._metrics.handled++;
return await this._execute(request);
} catch (err) {
this._metrics.errors++;
if (attempt === this.maxRetries) throw err;
await new Promise(r => setTimeout(r, attempt * 100));
}
}
}
async _execute(request) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), this.timeout);
try {
const response = await this._work(request, controller.signal);
return response;
} finally {
clearTimeout(timer);
}
}
async _work(request, signal) {
return { id: crypto.randomUUID(), result: request };
}
metrics() { return this._metrics; }
}
async function demo() {
const h = new MiniHandler({ timeout: 1000 });
const r = await h.handle("demo");
console.log(JSON.stringify(r, null, 2));
console.log("Metrics:", JSON.stringify(h.metrics()));
}
demo();
FAQ
Mini Project
Build a complete chaos engineering system with configuration, error handling, logging, monitoring, and testing. Include unit tests and a demo script.
const crypto = require("crypto");
class Logger {
info(msg, ctx = {}) { console.log("INFO: " + msg, JSON.stringify(ctx)); }
error(msg, ctx = {}) { console.error("ERROR: " + msg, JSON.stringify(ctx)); }
}
class Metrics {
constructor() {
this._data = { operations: 0, errors: 0, totalTime: 0 };
}
record(duration, error = false) {
this._data.operations++;
this._data.totalTime += duration;
if (error) this._data.errors++;
}
report() {
return {
...this._data,
avgTime: this._data.operations > 0
? (this._data.totalTime / this._data.operations).toFixed(2)
: 0,
};
}
}
class ChaosEngineeringSystem {
constructor(config = {}) {
this.config = config;
this.logger = new Logger();
this.metrics = new Metrics();
}
async execute(task) {
const start = Date.now();
const traceId = crypto.randomUUID();
this.logger.info("executing", { task, traceId });
try {
const result = await this.run(task);
this.metrics.record(Date.now() - start);
this.logger.info("completed", { task, traceId });
return result;
} catch (err) {
this.metrics.record(Date.now() - start, true);
this.logger.error("failed", { task, traceId, error: err.message });
throw err;
}
}
async run(task) {
return { task, status: "ok", timestamp: new Date().toISOString() };
}
}
async function main() {
const system = new ChaosEngineeringSystem();
for (const t of ["task-1", "task-2", "task-3"]) {
const r = await system.execute(t);
console.log(JSON.stringify(r));
}
console.log("Metrics: " + JSON.stringify(system.metrics.report()));
}
main();
What is Next
Now that you understand chaos engineering, explore related patterns and advanced implementations. Check out more backend development patterns and best practices for building production-ready applications.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro