API Chaos Testing — Latency Injection, Failure Simulation, and Resilience Validation

In this tutorial, you will learn about API Chaos Testing. We cover key concepts, practical examples, and best practices to help you master this topic.

API chaos testing intentionally injects failures into your system to validate that your application handles network latency, service outages, resource exhaustion, and unexpected errors gracefully.

What You'll Learn

• How to inject latency and failures into API dependencies
• Testing circuit breaker and retry behavior
• Validating fallback responses and degraded mode

Why It Matters

Systems that work perfectly in test environments often fail under real-world conditions. Chaos testing exposes weaknesses before they cause incidents, building confidence in production resilience.

Real-World Use

A ride-sharing platform uses chaos testing to simulate its payment gateway going down. Tests reveal that the ride completion flow crashes without a payment response. The team adds a fallback that queues payments for retry, preventing ride loss.

flowchart LR
    A[Chaos Test] --> B[Inject Failure]
    B --> C[System Under Test]
    C --> D[Observe Behavior]
    D --> E{Resilient?}
    E -->|Yes| F[Pass]
    E -->|No| G[Identify Weakness]
    G --> H[Fix and Retest]

Injecting Latency with Toxiproxy

Toxiproxy proxies network connections and can inject delays.

import toxiproxy
from toxiproxy import Toxiproxy

# Start Toxiproxy and create a proxy
api = Toxiproxy("http://localhost:8474")
proxy = api.create("payment_api", "localhost:8080", "payment:9090")

# Inject 3-second latency
proxy.toxics().add_latency("latency_toxic", 3000)

# Test that the system still works (with timeout)
import requests
try:
    resp = requests.get("http://localhost:8080/health", timeout=5)
    print(f"System responded in {resp.elapsed.total_seconds():.1f}s")
except requests.Timeout:
    print("System timed out - need longer timeout or retry logic")

Expected output: The system either responds after 3+ seconds or times out.

Simulating Service Failure

Use Toxiproxy to disconnect a downstream service.

# Disconnect the downstream service
proxy.toxics().add_timeout("disconnect", 0)

# Test the fallback
resp = requests.get("http://localhost:8080/api/orders")
data = resp.json()

if resp.status_code == 200 and "fallback" in data:
    print(f"Fallback response: {data['fallback']}")
elif resp.status_code == 503:
    print("System returned 503 - no fallback implemented")
else:
    print("Unexpected response - needs review")

Expected output: The system returns a fallback response or a 503 status.

Testing Circuit Breaker

Simulate repeated failures to trigger the circuit breaker.

import time

def simulate_repeated_failures():
    # Inject timeout toxic
    proxy.toxics().add_timeout("circuit_breaker_toxic", 0)
    results = []

    for i in range(10):
        try:
            resp = requests.get("http://localhost:8080/api/orders", timeout=2)
            results.append(resp.status_code)
        except requests.Timeout:
            results.append("timeout")
        except requests.ConnectionError:
            results.append("connection_error")

        time.sleep(0.5)

    print(f"Request results: {results}")

simulate_repeated_failures()

Expected output: After N consecutive failures, the circuit breaker opens and immediate fallback responses appear.

Common Mistakes

Practice Questions

1. What is chaos engineering? A: The practice of intentionally injecting failures into a system to uncover weaknesses before they cause incidents.
2. What is Toxiproxy? A: A proxy that can inject network failures like latency, disconnects, and bandwidth limits.
3. What is a circuit breaker pattern? A: A design pattern that detects repeated failures and opens the circuit to prevent cascading failures.
4. What is the difference between chaos testing and load testing? A: Load testing measures performance under scale; chaos testing measures behavior under failure.
5. What is the blast radius in chaos testing? A: The scope of impact a failure can cause. Minimizing blast radius is a key principle.

Challenge

Design and run a chaos experiment for an e-commerce checkout flow. Inject 5 seconds of latency to the payment service, verify the checkout shows a loading state and eventually a timeout message. Then simulate a complete payment service outage and verify the checkout returns a fallback response with queued order. Measure recovery time after the outage ends.

FAQ

What tools support API chaos testing?

Toxiproxy, Chaos Monkey, Gremlin, Litmus, and custom scripts with proxy libraries all support chaos testing.

How do you run chaos tests in CI/CD?

Run chaos tests in a staging environment that mirrors production, not in CI where tests must be fast.

What is the difference between Toxiproxy and Chaos Monkey?

Toxiproxy injects network-level failures (latency, disconnects); Chaos Monkey terminates instances.

How do you measure resilience?

Track metrics like error rate, p95 response time, recovery time, and whether fallbacks activate correctly.

What is a Steady State in chaos testing?

The normal behavior of the system (metrics within thresholds) before chaos is injected.

How often should chaos tests run?

Continuously in staging environments and during game days. Production chaos should be scheduled and careful.

What is a chaos Game Day?

A scheduled event where the team runs chaos experiments together and practices Incident Response.

Mini Project

Build a chaos test suite for a microservice that depends on three downstream APIs (auth, payment, inventory). Use Toxiproxy to: inject 2s latency on auth (verify login still works with longer timeout), disconnect payment (verify fallback to queued payments), limit bandwidth on inventory to 1KB/s (verify image loading degrades gracefully), and simulate all three failures simultaneously (verify system doesn't crash). Document the blast radius for each experiment.

What's Next

Next, explore API fuzz testing to find unexpected vulnerabilities through malformed inputs.