Latency Injection — Simulating Network & Service Delays

In this tutorial, you'll learn about Latency Injection. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Latency Injection is a Chaos Engineering technique that adds artificial delays to network traffic or service calls. It is one of the most effective ways to test how your application handles timeouts, retries, and degraded performance from upstream dependencies.

What You Will Learn

This tutorial teaches you multiple techniques for injecting latency: using Linux tc, application-level proxies, and chaos platforms. You will learn when to use each approach.

Why It Matters

Network latency is the most common failure mode in Distributed Systems. A slow database, a congested network link, or a busy upstream API can cause cascading failures. Latency Injection helps you verify that your timeouts and retries work correctly before a real slowdown affects users.

Real-World Use

DodaTech uses Latency Injection to test the Durga Antivirus Pro update service. By simulating a 5-second delay on the update manifest endpoint the team discovered that the client had no timeout configured and would hang indefinitely. This was fixed before it caused a real production issue.

Prerequisites

Before starting you should understand:

• Basic Chaos Engineering concepts (hypothesis, steady state, Blast Radius)
• Linux networking basics (interfaces, ports, routing)
• How timeouts and retries work in application code
• Access to a Linux host or Docker container for experiments

Step 1: Use tc for Network Latency

The Linux tc (traffic control) command adds latency at the network interface level:

# Add 200ms latency to the eth0 interface
sudo tc qdisc add dev eth0 root netem delay 200ms

# Verify the rule
sudo tc qdisc show dev eth0
# Expected output:
# qdisc netem 8001: root refcnt 2 limit 1000 delay 200.0ms

# Test the latency from another host
ping target-host
# Expected output:
# rtt min/avg/max/mdev = 200.345/201.123/202.456/0.567 ms

Step 2: Remove Latency and Add Jitter

Clean up and add latency with variation (jitter):

# Remove the existing rule
sudo tc qdisc del dev eth0 root

# Add latency with 50ms jitter
sudo tc qdisc add dev eth0 root netem delay 200ms 50ms

# Test - notice the variation
ping -c 5 target-host
# Expected output (jitter causes variation):
# rtt min/avg/max/mdev = 152.345/184.567/251.234/15.678 ms

Step 3: Use a Proxy for Application-Level Latency

For service-to-service calls use a proxy like Toxiproxy:

# toxiproxy-config.yaml
proxies:
  - name: payment-db-proxy
    listen: "0.0.0.0:15432"
    upstream: "postgres:5432"
    toxicity:
      - type: latency
        stream: downstream
        toxicity: 0.5
        attributes:
          latency: 500
          jitter: 100

# Start Toxiproxy with the configuration
toxiproxy-server --config toxiproxy-config.yaml
# Expected output:
# [INFO] Starting toxiproxy on 0.0.0.0:8474
# [INFO] Proxy payment-db-proxy listening on 0.0.0.0:15432

Step 4: Use Chaos Mesh for Kubernetes Latency

In Kubernetes environments use Chaos Mesh for Latency Injection:

# mesh-latency.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: app-latency-test
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: payment-service
  delay:
    latency: 500ms
    jitter: 100ms
  duration: 120s

kubectl apply -f mesh-latency.yaml
# Expected output:
# networkchaos.chaos-mesh.org/app-latency-test created

Step 5: Verify Timeout and Retry Behavior

While latency is active test your applications timeout handling:

# Test the service while latency is injected
time curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://service/api/health
# Expected output with 500ms latency:
# 200 1.234s
# The total time includes the 500ms network latency plus processing time

# Test with a timeout shorter than the injected latency
time curl --connect-timeout 0.3 -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://service/api/health
# Expected output:
# 000 0.301s
# Connection timed out because 300ms < 500ms latency

Learning Path

flowchart LR
  A[Azure Chaos Studio] --> B[Latency Injection]
  B --> C[Fault Injection Proxy]
  C --> D[Dependency Testing]
  D --> E[Resilience Testing]
  style B fill:#f90,color:#fff

Common Errors

1. Forgetting to remove tc rules after testing: tc rules persist until explicitly removed. Always run tc qdisc del after tests.
2. Adding latency to the wrong interface: Check which interface handles the traffic you want to affect. Loopback needs dev lo.
3. Setting latency too high for the timeout configuration: If latency exceeds the timeout all requests will fail. Start with small values.
4. Confusing downstream and upstream in proxy configurations: Downstream is from proxy to client. Upstream is from proxy to server. Latency on the wrong stream may not affect the target path.
5. Running latency experiments without monitoring: Without real-time latency metrics you cannot correlate the injected delay with observed behavior.

Practice Questions

1. How does the tc command add network latency to a Linux interface?
2. What is jitter and why would you add it to latency experiments?
3. How does Toxiproxy differ from tc for Latency Injection?
4. Why should you test with a timeout value lower than the injected latency?
5. How do you clean up tc rules after an experiment?

Challenge

Set up a three-tier application (client, app server, database) and inject increasing latency from 100ms to 2000ms between the app server and database. Document at what latency point the application starts returning errors and compare this to the configured timeout value.

FAQ