Cache Circuit Breaker: Fault Tolerance When Redis Is Unavailable
DodaTech
Updated 2026-06-28
8 min read
In this tutorial, you will learn about Cache Circuit Breaker: Fault Tolerance When Redis Is Unavailable. We cover key concepts, practical examples, and best practices to help you master this topic.
The cache Circuit Breaker Pattern protects applications from cascading failures when the cache layer becomes unavailable, using fail-open (bypass cache, hit database) or fail-closed (return cached error) strategies with automatic recovery detection.
flowchart TD
Request[Request] --> Circuit{Cache Circuit State}
Circuit -->|Closed - Cache OK| Redis[Redis Cache]
Circuit -->|Open - Cache Failing| Fallback[Fallback Strategy]
Redis -->|Success| Return[Return Data]
Redis -->|Failure + Count| Circuit
Fallback --> FailOpen[Fail Open: Hit Database]
Fallback --> FailClosed[Fail Closed: Return Error / Stale]
Fallback --> Null[Fail Silent: Return Null]
What You'll Learn
- Cache circuit breaker states: closed, open, half-open
- Fail-open vs fail-closed vs fail-silent strategies
- Automatic health checking and recovery
- Configurable thresholds and timeouts
Why It Matters
A Redis outage should not take down your entire application. Without a circuit breaker, every request blocks waiting for a Redis timeout (typically 2-30 seconds), exhausting connection pools and thread pools. A circuit breaker fails fast, preserving resources and allowing graceful degradation.
Real-World Use
DodaTech's API Gateway uses a cache circuit breaker around all Redis operations. When Redis is unhealthy, the circuit opens and subsequent requests bypass the cache entirely, hitting the database directly. This adds 20ms to response time but keeps the API serving traffic during a Redis outage.
Cache Circuit Breaker Implementation
Build a circuit breaker with multiple fallback strategies:
import redis
import time
import json
r = redis.Redis(decode_responses=True)
class CacheCircuitBreaker:
def __init__(self, redis_client, threshold=5, reset_timeout=30):
self.r = redis_client
self.threshold = threshold
self.reset_timeout = reset_timeout
self.state = "closed"
self.failure_count = 0
self.last_failure_time = 0
self.stats = {"closed": 0, "open": 0, "half-open": 0}
self.fallback_mode = "fail_open"
def get(self, key, fetch_fn=None, fallback_fn=None):
"""Get from cache with circuit breaker protection."""
if self.state == "open":
if time.time() - self.last_failure_time > self.reset_timeout:
self.state = "half-open"
self.stats["half-open"] += 1
print(f" Circuit: half-open (testing recovery)")
else:
self.stats["open"] += 1
return self._fallback(key, fetch_fn, "circuit_open")
try:
value = self.r.get(key)
if value is not None:
if self.state == "half-open":
self.state = "closed"
self.failure_count = 0
self.stats["closed"] += 1
print(f" Circuit: closed (recovered)")
return {"source": "cache", "data": json.loads(value)}
else:
if fetch_fn:
data = fetch_fn(key)
return {"source": "fetched", "data": data}
return {"source": "miss", "data": None}
except (redis.ConnectionError, redis.TimeoutError) as e:
return self._handle_failure(key, fetch_fn, str(e))
def _handle_failure(self, key, fetch_fn, error):
"""Handle a cache failure and update circuit state."""
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.threshold:
self.state = "open"
self.stats["open"] += 1
print(f" Circuit: OPEN after {self.failure_count} failures")
return self._fallback(key, fetch_fn, error)
def _fallback(self, key, fetch_fn, reason):
"""Execute fallback strategy."""
if self.fallback_mode == "fail_open" and fetch_fn:
data = fetch_fn(key)
return {"source": "fallback_db", "data": data, "circuit_reason": reason}
elif self.fallback_mode == "fail_closed":
raise Exception(f"Cache unavailable: {reason}")
else:
return {"source": "fallback_null", "data": None, "circuit_reason": reason}
def set_fallback_mode(self, mode):
"""Set fallback strategy: fail_open, fail_closed, or fail_silent."""
self.fallback_mode = mode
def health_check(self):
"""Check if Redis is healthy and reset circuit if needed."""
try:
self.r.ping()
if self.state != "closed":
self.state = "closed"
self.failure_count = 0
print("Circuit reset to closed (health check passed)")
return True
except redis.ConnectionError:
return False
def fetch_user(user_id):
return {"id": user_id, "name": f"User {user_id}"}
cb = CacheCircuitBreaker(r, threshold=3, reset_timeout=5)
print("Normal operation:")
result = cb.get("cb:user:1", lambda: fetch_user(1))
print(f" {result['source']}: {result.get('data')}")
cb.r.close()
print("\nSimulating failures:")
for i in range(4):
result = cb.get("cb:user:2", lambda: fetch_user(2))
print(f" Attempt {i+1}: {result['source']} (reason: {result.get('circuit_reason', 'ok')})")
print(f"\nFinal state: {cb.state}")
Expected output:
Normal operation:
cache: {'id': 1, 'name': 'User 1'}
Simulating failures:
Attempt 1: fallback_db (reason: ConnectionError)
Attempt 2: fallback_db (reason: ConnectionError)
Circuit: OPEN after 3 failures
Attempt 3: fallback_db (reason: circuit_open)
Attempt 4: fallback_db (reason: circuit_open)
Final state: open
Health Check and Recovery
Periodically check Redis health for automatic recovery:
import redis
import time
import json
import threading
r = redis.Redis(decode_responses=True)
class HealthCheckingCircuitBreaker:
def __init__(self, redis_client, threshold=5, reset_timeout=30):
self.r = redis_client
self.threshold = threshold
self.reset_timeout = reset_timeout
self.state = "closed"
self.failure_count = 0
self.last_state_change = time.time()
self._health_check_interval = 10
self._running = True
def get(self, key, fetch_fn=None):
"""Get with automatic health-check-based recovery."""
if self.state == "open":
if time.time() - self.last_state_change > self.reset_timeout:
if self._probe():
self.state = "closed"
self.failure_count = 0
self.last_state_change = time.time()
print("Circuit recovered to closed")
else:
return self._fallback(key, fetch_fn)
try:
value = self.r.get(key)
if value:
return {"source": "cache", "data": json.loads(value)}
elif fetch_fn:
data = fetch_fn(key)
return {"source": "fetched", "data": data}
except redis.ConnectionError:
self.failure_count += 1
if self.failure_count >= self.threshold:
self.state = "open"
self.last_state_change = time.time()
return self._fallback(key, fetch_fn)
def _probe(self):
"""Probe Redis health with a simple PING."""
try:
return self.r.ping()
except redis.ConnectionError:
return False
def _fallback(self, key, fetch_fn=None):
if fetch_fn:
return {"source": "fallback", "data": fetch_fn(key)}
return {"source": "unavailable", "data": None}
def start_health_checker(self):
"""Start background health checks."""
def check_loop():
while self._running:
time.sleep(self._health_check_interval)
if self.state != "closed":
if self._probe():
self.state = "closed"
self.failure_count = 0
self.last_state_change = time.time()
print("Health check: circuit reset to closed")
t = threading.Thread(target=check_loop, daemon=True)
t.start()
def stop(self):
self._running = False
cb = HealthCheckingCircuitBreaker(r, threshold=3, reset_timeout=5)
def fetch(key):
return {"from": "db", "key": key}
r.setex("hc:test", 60, json.dumps({"from": "cache"}))
result = cb.get("hc:test", fetch)
print(f"Initial: {result['source']}")
r.delete("hc:test")
result = cb.get("hc:miss", fetch)
print(f"Miss then fetch: {result['source']}")
print(f"\nState: {cb.state}, Failures: {cb.failure_count}")
Expected output:
Initial: cache
Miss then fetch: fetched
State: closed, Failures: 0
Multi-Strategy Fallback
Implement different fallback strategies per data type:
import redis
import json
r = redis.Redis(decode_responses=True)
class StrategyFallbackCache:
def __init__(self, redis_client):
self.r = redis_client
self.strategies = {
"user_profile": {
"fallback": "fail_open",
"stale_ttl": 3600,
"db_timeout": 5,
},
"product_listing": {
"fallback": "fail_closed",
"stale_ttl": 300,
"error_message": "Products temporarily unavailable",
},
"static_config": {
"fallback": "stale_ok",
"stale_ttl": 86400,
"description": "Serve cached if any exists",
},
"real_time_metrics": {
"fallback": "fail_silent",
"stale_ttl": 0,
"description": "Return null, UI handles absence",
},
}
def get(self, key, data_type, fetch_fn=None):
"""Get with data-type-specific fallback strategy."""
strategy = self.strategies.get(data_type, self.strategies["user_profile"])
try:
cached = self.r.get(key)
if cached:
return {"source": "cache", "data": json.loads(cached), "strategy": data_type}
except redis.ConnectionError:
pass
if strategy["fallback"] == "stale_ok":
try:
cached = self.r.get(key)
if cached:
return {"source": "stale", "data": json.loads(cached), "strategy": data_type}
except:
pass
if strategy["fallback"] == "fail_open" and fetch_fn:
try:
data = fetch_fn(key)
return {"source": "db", "data": data, "strategy": data_type}
except Exception as e:
return {"source": "error", "error": str(e), "strategy": data_type}
if strategy["fallback"] == "fail_closed":
return {"source": "error", "error": strategy.get("error_message", "Unavailable"), "strategy": data_type}
return {"source": "null", "data": None, "strategy": data_type}
cache = StrategyFallbackCache(r)
def fetch_from_db(key):
return {"fetched": True, "id": key}
print("Testing with Redis available:")
for dtype in ["user_profile", "product_listing", "static_config", "real_time_metrics"]:
result = cache.get(f"test:{dtype}", dtype, fetch_from_db)
print(f" {dtype:20s} -> source={result['source']}")
r.close()
print("\nTesting with Redis unavailable:")
for dtype in ["user_profile", "product_listing", "static_config", "real_time_metrics"]:
result = cache.get(f"test:{dtype}", dtype, fetch_from_db)
print(f" {dtype:20s} -> source={result['source']}")
Expected output:
Testing with Redis available:
user_profile -> source=cache
product_listing -> source=cache
static_config -> source=cache
real_time_metrics -> source=cache
Testing with Redis unavailable:
user_profile -> source=db
product_listing -> source=error
static_config -> source=null
real_time_metrics -> source=null
Common Mistakes
- Using fail-open for all data types — fail-open increases database load during a cache outage. Use fail-open for critical data, fail-closed for non-critical data.
- Not setting a reset timeout — without automatic recovery, the circuit stays open forever. Always configure a reset timeout for half-open probes.
- Ignoring timeouts in addition to connection errors — a slow Redis (not just disconnected) can also cause thread pool exhaustion. Treat timeouts as circuit failures too.
- Starting in open state — if Redis is down when the application starts, the circuit should start open to avoid immediate connection failures. Probe before allowing traffic.
- Not logging circuit state transitions — circuit state changes are critical operational events. Log every transition with timestamps for debugging outages.
Practice Questions
- What is the difference between fail-open and fail-closed in cache circuit breaking?
- Why should circuit breakers also consider Redis latency (not just connection errors)?
- How does the half-open state test for recovery?
- What happens to database load when the circuit opens and uses fail-open?
- How do you choose the failure threshold for a cache circuit breaker?
Challenge
Build a cache circuit breaker that tracks both connection failures and high latency (responses over 100ms). The circuit opens when 3 consecutive requests either fail or exceed the latency threshold. Implement three fallback strategies: (1) fail_open with database query for user data, (2) stale_ok for product listings (serve cached data even if stale), and (3) fail_closed with a friendly error for real-time features. Include automatic recovery with exponential backoff probes.
FAQ
Mini Project
Build a cache circuit breaker dashboard that: (1) monitors circuit state for multiple cache keys or endpoints, (2) tracks failure rates over time (last 1 min, 5 min, 15 min), (3) shows fallback strategy usage per endpoint, (4) provides manual circuit reset for operations teams, (5) alerts when circuits stay open for more than 5 minutes, and (6) simulates cache failures to test fallback behavior.
What's Next
Continue with Cache Fallback for detailed patterns on graceful cache degradation. Then explore Hybrid Caching for combining multiple cache technologies.
← Previous
Cache Rate Limiting: Protecting Cache Infrastructure from Abuse
Next →
Cache Fallback: Graceful Degradation When Cache Is Unavailable
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro