Cache Rate Limiting: Protecting Cache Infrastructure from Abuse
In this tutorial, you will learn about Cache Rate Limiting: Protecting Cache Infrastructure from Abuse. We cover key concepts, practical examples, and best practices to help you master this topic.
Cache rate limiting uses Redis as a high-performance rate limiter to control request rates, prevent cache abuse, and ensure fair resource allocation across users and services using sliding windows, token buckets, and distributed counters.
flowchart TD
Request[Incoming Request] --> RateLimit{Rate Limit Check}
RateLimit -->|Under Limit| Cache[Process Request]
RateLimit -->|Over Limit| Reject[429 Too Many Requests]
Cache --> Redis[Redis Counter]
Redis -->|Increment| Count[Current Count]
Count --> RateLimit
Reject --> Headers[RateLimit Headers]
Headers --> Client[Client]
What You'll Learn
- Sliding window rate limiting with Redis sorted sets
- Token bucket algorithm for burst-tolerant rate limiting
- Distributed rate limiting across application instances
- Rate limit response headers and backoff strategies
Why It Matters
Without rate limiting, a single abusive client can overwhelm your cache infrastructure, causing increased latency and eviction for all other users. Redis-based rate limiters handle 100,000+ checks per second with sub-millisecond latency, making them ideal for high-traffic cache protection.
Real-World Use
DodaTech's API Gateway uses Redis-based sliding window rate limiters per API key. Each key is limited to 1000 requests per minute with a burst allowance of 50. When a key exceeds the limit, the gateway returns 429 with Retry-After headers. This protects the Redis cache from being overwhelmed by misconfigured or malicious clients.
Sliding Window Rate Limiter
Track request counts in a sliding time window:
import redis
import time
import json
r = redis.Redis(decode_responses=True)
class SlidingWindowRateLimiter:
def __init__(self, redis_client):
self.r = redis_client
def check_rate_limit(self, key, max_requests, window_seconds):
"""Check if a request is within the rate limit using a sliding window."""
now = time.time()
window_start = now - window_seconds
sorted_set_key = f"ratelimit:{key}"
count_key = f"ratelimit:{key}:count"
pipe = self.r.pipeline()
pipe.zremrangebyscore(sorted_set_key, 0, window_start)
pipe.zcard(sorted_set_key)
pipe.zadd(sorted_set_key, {f"{now}:{id(self)}": now})
pipe.expire(sorted_set_key, window_seconds * 2)
results = pipe.execute()
current_count = results[1]
if current_count > max_requests:
oldest = self.r.zrange(sorted_set_key, 0, 0, withscores=True)
retry_after = 0
if oldest:
retry_after = max(0, int(window_seconds - (now - oldest[0][1])))
return {
"allowed": False,
"current": current_count,
"limit": max_requests,
"remaining": 0,
"retry_after": retry_after,
"reset_at": now + retry_after,
}
return {
"allowed": True,
"current": current_count,
"limit": max_requests,
"remaining": max_requests - current_count,
"retry_after": 0,
}
def get_headers(self, result):
"""Generate standard rate limit response headers."""
return {
"X-RateLimit-Limit": result["limit"],
"X-RateLimit-Remaining": result["remaining"],
"X-RateLimit-Reset": int(result.get("reset_at", time.time())),
}
limiter = SlidingWindowRateLimiter(r)
for i in range(12):
result = limiter.check_rate_limit("api:user_42", 10, 60)
if result["allowed"]:
print(f"Request {i+1:2d}: ALLOWED (remaining: {result['remaining']})")
else:
print(f"Request {i+1:2d}: DENIED (retry after {result['retry_after']}s)")
break
headers = limiter.get_headers(result)
print(f"\nRate limit headers: {headers}")
Expected output:
Request 1: ALLOWED (remaining: 9)
Request 2: ALLOWED (remaining: 8)
...
Request 10: ALLOWED (remaining: 0)
Request 11: DENIED (retry after 60s)
Rate limit headers: {'X-RateLimit-Limit': 10, 'X-RateLimit-Remaining': 0, 'X-RateLimit-Reset': 1719580860}
Token Bucket Rate Limiter
Allow bursts while limiting average rate:
import redis
import time
import json
r = redis.Redis(decode_responses=True)
class TokenBucketRateLimiter:
def __init__(self, redis_client):
self.r = redis_client
def check(self, key, capacity, refill_rate, refill_time=1):
"""Check if a token is available using the token bucket algorithm."""
tokens_key = f"tokenbucket:{key}:tokens"
timestamp_key = f"tokenbucket:{key}:ts"
now = time.time()
pipe = self.r.pipeline()
pipe.get(tokens_key)
pipe.get(timestamp_key)
tokens_str, last_refill_str = pipe.execute()
tokens = float(tokens_str) if tokens_str else capacity
last_refill = float(last_refill_str) if last_refill_str else now
elapsed = now - last_refill
new_tokens = min(capacity, tokens + elapsed * (refill_rate / refill_time))
if new_tokens >= 1:
new_tokens -= 1
pipe.multi()
pipe.set(tokens_key, new_tokens)
pipe.set(timestamp_key, now)
pipe.execute()
return {
"allowed": True,
"tokens_remaining": round(new_tokens, 2),
"capacity": capacity,
}
else:
wait_time = (1 - new_tokens) * (refill_time / refill_rate)
return {
"allowed": False,
"tokens_remaining": round(new_tokens, 2),
"capacity": capacity,
"retry_after": round(wait_time, 2),
}
bucket = TokenBucketRateLimiter(r)
for i in range(8):
result = bucket.check("api:burst", capacity=5, refill_rate=2, refill_time=10)
status = "ALLOWED" if result["allowed"] else "DENIED"
print(f"Request {i+1}: {status} (tokens: {result['tokens_remaining']}/{result['capacity']})")
if not result["allowed"]:
print(f" Retry after: {result['retry_after']}s")
Expected output:
Request 1: ALLOWED (tokens: 4.0/5)
Request 2: ALLOWED (tokens: 3.0/5)
...
Request 5: ALLOWED (tokens: 0.0/5)
Request 6: DENIED (tokens: 0.0/5)
Retry after: 5.0s
Distributed Rate Limiting
Coordinate rate limits across multiple application instances:
import redis
import time
import json
import threading
r = redis.Redis(decode_responses=True)
class DistributedRateLimiter:
def __init__(self, redis_client):
self.r = redis_client
def check(self, key, limit, window_seconds, cost=1):
"""Distributed rate limit check using a Lua script for atomicity."""
script = """
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local cost = tonumber(ARGV[3])
local now = redis.call('TIME')[1]
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local count = redis.call('ZCARD', key)
if count + cost > limit then
return {0, count, limit}
end
for i = 1, cost do
redis.call('ZADD', key, now, now .. ':' .. math.random())
end
redis.call('EXPIRE', key, window * 2)
return {1, count + cost, limit}
"""
result = self.r.eval(script, 1, key, limit, window_seconds, cost)
allowed = result[0] == 1
return {
"allowed": allowed,
"current": result[1],
"limit": result[2],
"remaining": max(0, result[2] - result[1]),
}
def test_concurrent_access(self, key, limit, window, num_threads, requests_per_thread):
"""Test rate limiting under concurrent access."""
results = []
def worker():
for _ in range(requests_per_thread):
result = self.check(key, limit, window)
results.append(result["allowed"])
threads = [threading.Thread(target=worker) for _ in range(num_threads)]
for t in threads:
t.start()
for t in threads:
t.join()
total = len(results)
allowed = sum(results)
return {
"total_requests": total,
"allowed": allowed,
"denied": total - allowed,
}
dist_limiter = DistributedRateLimiter(r)
for i in range(6):
result = dist_limiter.check("api:distributed", 5, 60)
print(f"Request {i+1}: {'ALLOWED' if result['allowed'] else 'DENIED'} "
f"({result['current']}/{result['limit']})")
print("\nConcurrent access test:")
concurrent = dist_limiter.test_concurrent_access("api:concurrent", 10, 60, 5, 5)
print(f" Total: {concurrent['total_requests']}, "
f"Allowed: {concurrent['allowed']}, "
f"Denied: {concurrent['denied']}")
Expected output:
Request 1: ALLOWED (1/5)
Request 2: ALLOWED (2/5)
...
Request 5: ALLOWED (5/5)
Request 6: DENIED (5/5)
Concurrent access test:
Total: 25, Allowed: 10, Denied: 15
Common Mistakes
- Using fixed window counters (INCR + EXPIRE) — fixed windows allow 2x the limit at window boundaries. A client can send 100 requests at 00:00:59 and another 100 at 00:01:01 (200 in 2 seconds). Use sliding windows.
- Forgetting to expire rate limit keys — without expiry, rate limit keys accumulate in Redis, consuming memory. Always set EXPIRE on rate limit keys with a TTL equal to 2x the window.
- Using separate commands for read and write (race conditions) — checking the counter and then incrementing is not atomic. Use Lua scripts or transactions for atomic rate limit operations.
- Not accounting for distributed system clock drift — if your application servers have different wall clocks, a client could be rate-limited by one server just after being allowed by another. Use Redis TIME command.
- Setting rate limits too low for legitimate traffic spikes — burst traffic (flash sales, news events) triggers false rate limiting. Use token buckets with burst allowance to handle spikes.
Practice Questions
- What is the advantage of a sliding window over a fixed window for rate limiting?
- How does the token bucket algorithm allow bursts while limiting average rate?
- Why must rate limit operations be atomic in Distributed Systems?
- What is the thundering herd problem in rate limiting and how do you prevent it?
- How do standard rate limit headers help clients adjust their request rate?
Challenge
Build a multi-tier rate limiter for an API that supports: (1) per-user limit (100 req/min), (2) per-IP limit (1000 req/min), (3) global limit (10000 req/min), (4) burst allowance of 20 requests, and (5) different limits for different endpoints (GET vs POST). Use a Lua script that checks all three limits atomically and returns the most restrictive result. Test with concurrent clients.
FAQ
Mini Project
Build a rate limiter middleware for a web framework that: (1) supports configurable rate limits per route, per user, and per IP, (2) uses sliding windows with Redis sorted sets, (3) returns standard rate limit headers on every response, (4) handles 429 responses with Retry-After, (5) supports rate limit exemption lists for internal services, and (6) exports metrics to Prometheus for monitoring rate limit hit rates and denial rates.
What's Next
Continue with Cache Circuit Breaker to learn about fault tolerance patterns for cache dependencies. Then explore Cache Fallback for graceful degradation when the cache is unavailable.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro