Caching in API Gateway — Improve Performance and Reduce Backend Load

In this tutorial, you will learn about Caching in API Gateway. We cover key concepts, practical examples, and best practices to help you master this topic.

Caching in an API gateway stores responses from backend services and serves them directly to clients for repeat requests, reducing backend load, lowering latency, and improving overall system throughput.

What You'll Learn

• In-memory and Redis-based caching strategies in the gateway
• Cache key design including query parameters and headers
• Cache invalidation: TTL, purge, and write-through patterns

Why It Matters

Many API responses are identical for a given request. Product catalogs, public documentation, and configuration data change infrequently. Caching at the gateway serves these responses without hitting backend services, reducing latency from 100ms to 1ms and backend load by 80%.

Real-World Use

Durga Antivirus Pro's threat definition database updates every 6 hours. The gateway caches the latest definitions endpoint for 6 hours. During a malware outbreak, millions of clients request definitions. The gateway serves 99.9% from cache, and the backend handles only one request every 6 hours.

flowchart LR
    Client["Client"] --> GW["Gateway\nCache Check"]
    GW -->|"Cache HIT"| Cache["In-Memory / Redis"]
    GW -->|"Cache MISS"| Backend["Backend\nService"]
    Backend --> GW
    GW --> Client
    style GW fill:#dbeafe,stroke:#2563eb
    style Cache fill:#bbf7d0,stroke:#16a34a

In-Memory Cache with TTL

import time
import threading

class TTLCache:
    def __init__(self, default_ttl=300):
        self.cache = {}
        self.default_ttl = default_ttl
        self.lock = threading.Lock()

    def get(self, key):
        with self.lock:
            if key in self.cache:
                value, expiry = self.cache[key]
                if time.time() < expiry:
                    return value
                del self.cache[key]
        return None

    def set(self, key, value, ttl=None):
        ttl = ttl or self.default_ttl
        with self.lock:
            self.cache[key] = (value, time.time() + ttl)

cache = TTLCache(default_ttl=60)

@app.route("/api/threats/latest")
def get_latest_threats():
    cached = cache.get("latest_threats")
    if cached:
        return cached
    resp = requests.get("http://threat-service:8080/latest")
    data = resp.json()
    cache.set("latest_threats", data, ttl=3600)
    return jsonify(data)

Redis Cache for Distributed Gateways

For multiple gateway instances, Redis provides a shared cache:

import redis
import json

redis_client = redis.Redis(host="redis", port=6379, db=0)

@app.route("/api/products/<product_id>")
def get_product(product_id):
    cache_key = f"product:{product_id}"
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)
    resp = requests.get(f"http://product-service:8080/products/{product_id}")
    data = resp.json()
    redis_client.setex(cache_key, 3600, json.dumps(data))
    return jsonify(data)

Cache Key Design

Include request parameters that affect the response in the cache key:

def make_cache_key(path, args, headers):
    key_parts = [path]
    for param, value in sorted(args.items()):
        key_parts.append(f"{param}={value}")
    accept = headers.get("Accept-Language", "en")
    key_parts.append(f"lang={accept}")
    return ":".join(key_parts)

cache_key = make_cache_key("/api/products", request.args, request.headers)

Cache Invalidation

Invalidate cache entries when data changes:

@app.route("/api/products/<product_id>", methods=["PUT"])
def update_product(product_id):
    resp = requests.put(
        f"http://product-service:8080/products/{product_id}",
        json=request.json
    )
    if resp.status_code == 200:
        redis_client.delete(f"product:{product_id}")
    return resp.content, resp.status_code

Common Mistakes

1. Caching Authenticated Responses

Never cache responses containing user-specific data unless the cache key includes the user ID. Otherwise, user A sees user B's data.

2. Cache Stampede

When a popular cache key expires, multiple requests simultaneously hit the backend. Use lock-based refresh or probabilistic early expiration.

3. Stale Data Serving

Without invalidation, clients see outdated data. Set appropriate TTLs and implement invalidation for write operations.

4. Unlimited Cache Growth

In-memory caches without eviction policies consume all available RAM. Use LRU eviction and set memory limits.

5. Caching Error Responses

If a backend returns 500, the gateway might cache the error response. Never cache non-2xx responses.

Practice Questions

1. How does caching at the gateway differ from caching at the client or backend?
2. What should be included in a cache key?
3. What is a cache stampede and how can you prevent it?
4. Why should authenticated responses not be cached without user-specific keys?
5. What is the difference between TTL-based and write-through cache invalidation?

Answers:

1. Gateway caches serve all clients, reducing backend load. Client caches serve one user. Backend caches help but don't reduce network requests.
2. URL path, query parameters, request headers that affect the response (Accept-Language), and for authenticated endpoints, the user ID.
3. A cache stampede occurs when many requests hit the backend simultaneously after a key expires. Prevent with locking, early expiration, or request collapsing.
4. Without user-specific keys, one user's private data is served to another user, causing a severe privacy breach.
5. TTL invalidation automatically expires entries after a time period. Write-through invalidation actively removes entries when data changes.

Challenge: Design a caching Strategy for a news API where articles rarely change but comments change frequently. The homepage should never be more than 5 minutes stale, but individual articles can be cached for 1 hour.

FAQ

Mini Project

Build a Flask gateway with Redis caching for three endpoints. Use different TTLs per endpoint, implement cache invalidation on PUT/DELETE requests, add X-Cache: HIT/MISS headers to responses, and handle cache stampede with a lock mechanism.

What's Next

Continue with IP Whitelisting in API Gateway for access control, or explore Logging and Monitoring in Gateway for Observability.