Dependency Testing — Testing External Service Failures

In this tutorial, you'll learn about Dependency Testing. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Dependency testing in Chaos Engineering focuses on how your application behaves when external services — APIs, databases, Message Queues, or third-party providers — become unavailable or degraded. Modern applications typically depend on dozens of external services, and any one of them can fail.

What You Will Learn

This tutorial teaches you how to test external service dependencies using mock servers, proxy-based Fault Injection, and network-level blocking to simulate provider outages.

Why It Matters

A single external dependency failure can cascade into a full application outage. The 2023 Fastly CDN outage took down 15 percent of the internet because applications did not handle CDN failures gracefully. Dependency testing reveals these single points of failure before they cause incidents.

Real-World Use

DodaTech maintains a dependency failure matrix for each microservice. The matrix lists every external dependency and the expected behavior when that dependency fails. Chaos experiments validate that the actual behavior matches the documented expectations.

Prerequisites

Before starting you should understand:

• Chaos Engineering fundamentals
• How HTTP APIs and database connections work
• Basic Docker and networking concepts
• Your own applications external dependencies (third-party APIs, databases, caches)

Step 1: Identify External Dependencies

Map all external dependencies and classify them by criticality:

# List all outbound connections from running containers
docker run --rm --network host nicolaka/netshoot:latest \
  ss -tpn | grep ESTAB | awk '{print $4}'
# Expected output:
# 192.168.1.1:5432     (PostgreSQL)
# 3.22.123.45:443      (External API)
# 127.0.0.1:6379       (Redis)

Step 2: Block a Dependency Using iptables

Simulate a complete dependency outage by blocking traffic at the network level:

# Block all traffic to the external API server
sudo iptables -A OUTPUT \
  -d 3.22.123.45 \
  -j DROP

# Verify the block
curl -s -o /dev/null -w "%{http_code}" https://api.external-service.com/v1/status
# Expected output: No output (connection hangs until timeout)

# Remove the block after testing
sudo iptables -D OUTPUT -d 3.22.123.45 -j DROP

Step 3: Test Degraded Mode with Toxiproxy

Use Toxiproxy to simulate partial failures like high latency:

# dependency-test.yaml
proxies:
  - name: external-api
    listen: "0.0.0.0:8081"
    upstream: "api.external-service.com:443"
    toxic:
      - type: latency
        stream: downstream
        attributes:
          latency: 3000
      - type: timeout
        stream: downstream
        attributes:
          timeout: 5000

# Apply the configuration
toxiproxy-cli configure dependency-test.yaml
# Expected output:
# Configured proxy external-api with 2 toxics

Step 4: Test Graceful Degradation

Run your application and verify it handles the failure gracefully:

# Start the application with the proxy
DEPENDENCY_URL=http://localhost:8081 python app.py

# Make requests and check for graceful handling
curl -s http://localhost:5000/api/orders | jq .
# Expected output (graceful degradation):
# {
#   "orders": [],
#   "metadata": {
#     "payment_status": "unavailable",
#     "cache_hit": true
#   }
# }
# The API returns cached data instead of failing

Step 5: Automate Dependency Tests

Integrate dependency testing into your CI/CD pipeline:

# .github/workflows/dependency-test.yml
name: Dependency Resilience Tests
on: [deployment]
jobs:
  test-dependency-failures:
    runs-on: ubuntu-latest
    steps:
      - name: Start Toxiproxy
        run: docker run -d -p 8474:8474 shopify/toxiproxy
      - name: Block External API
        run: sudo iptables -A OUTPUT -d api.external-service.com -j DROP
      - name: Run Integration Tests
        run: pytest tests/integration/
      - name: Clean Up
        run: sudo iptables -D OUTPUT -d api.external-service.com -j DROP

Learning Path

flowchart LR
  A[Fault Injection Proxy] --> B[Dependency Testing]
  B --> C[Resilience Testing]
  C --> D[Database Faults]
  D --> E[Network Partitioning]
  style B fill:#f90,color:#fff

Common Errors

1. Testing dependencies in isolation only: Real failures cascade. When one API fails the retries may overwhelm another service. Test combined scenarios.
2. Not distinguishing between hard and soft dependencies: Hard dependencies (cannot function without them) need different handling than soft dependencies (degraded mode is acceptable).
3. Forgetting to restore iptables rules: A forgotten DROP rule will cause hours of debugging. Automate cleanup with trap commands or CI teardown steps.
4. Testing with artificial timeouts that differ from production: Ensure the timeout values in tests match real production configuration.
5. Ignoring DNS failures: DNS resolution failures are a common real-world outage. Test what happens when DNS cannot resolve the dependency hostname.

Practice Questions

1. What is the difference between a hard dependency and a soft dependency?
2. How do you simulate a complete external API outage using iptables?
3. What does graceful degradation look like in an API response?
4. How can Toxiproxy simulate partial dependency failures?
5. Why is it important to test cascading dependency failures?

Challenge

Create a dependency failure matrix for an application that uses PostgreSQL, Redis, and two external REST APIs. Write tests for each dependency failing individually and for the worst case where all dependencies fail simultaneously. Verify the application returns meaningful errors and does not crash.

FAQ