Cache Observability: Monitoring, Metrics, and Distributed Tracing for Caches
In this tutorial, you will learn about Cache Observability: Monitoring, Metrics, and Distributed Tracing for Caches. We cover key concepts, practical examples, and best practices to help you master this topic.
Cache observability provides visibility into cache health and performance through Prometheus metrics, Redis INFO command monitoring, distributed tracing for cache operations, and dashboards that reveal hit rates, latency, eviction rates, and memory usage at a glance.
flowchart LR
Cache[Redis Cache] --> Metrics[Prometheus Metrics]
Cache --> Events[Redis INFO / SLOWLOG]
App[Application] --> Tracing[Distributed Tracing]
Metrics --> Dashboard[Grafana Dashboard]
Events --> Dashboard
Tracing --> Dashboard
Dashboard --> Alert[Alerting Rules]
Dashboard --> Analysis[Performance Analysis]
What You'll Learn
- Key Redis metrics to monitor: hit rate, memory, evictions, latency
- Prometheus exporter setup for Redis
- Distributed tracing for cache operations with OpenTelemetry
- Building Grafana dashboards for cache observability
Why It Matters
Without observability, you are flying blind. A slow cache eviction policy, memory leak, or network issue silently degrades performance. Observability catches problems before they cause outages — a dropping hit rate warns you a day before memory fills up and cache thrashing begins.
Real-World Use
DodaTech's cache observability stack monitors 12 Redis clusters across 3 regions. A Grafana dashboard tracks hit rate, memory usage, eviction rate, and command latency per cluster. When hit rate drops below 80% or evictions exceed 100/s, an alert fires. This has caught three memory leaks and two misconfigured eviction policies before they caused production incidents.
Key Redis Metrics
Collect and analyze essential Redis metrics:
import redis
import time
import json
r = redis.Redis(decode_responses=True)
class RedisMetricsCollector:
def __init__(self, redis_client):
self.r = redis_client
self.history = {"hit_rate": [], "memory": [], "evictions": [], "latency": []}
def collect(self):
"""Collect current Redis metrics."""
info = self.r.info()
keyspace_hits = info.get("keyspace_hits", 0)
keyspace_misses = info.get("keyspace_misses", 0)
total_ops = keyspace_hits + keyspace_misses
hit_rate = (keyspace_hits / total_ops * 100) if total_ops > 0 else 0
metrics = {
"hit_rate": round(hit_rate, 1),
"used_memory": info.get("used_memory", 0),
"used_memory_rss": info.get("used_memory_rss", 0),
"mem_fragmentation_ratio": info.get("mem_fragmentation_ratio", 0),
"evicted_keys": info.get("evicted_keys", 0),
"total_commands_processed": info.get("total_commands_processed", 0),
"connected_clients": info.get("connected_clients", 0),
"blocked_clients": info.get("blocked_clients", 0),
"rejected_connections": info.get("rejected_connections", 0),
"uptime_in_seconds": info.get("uptime_in_seconds", 0),
"keyspace": self._parse_keyspace(info),
}
self.history["hit_rate"].append(metrics["hit_rate"])
self.history["memory"].append(metrics["used_memory"])
self.history["evictions"].append(metrics["evicted_keys"])
return metrics
def _parse_keyspace(self, info):
"""Parse keyspace information."""
keyspace = {}
for key, value in info.items():
if key.startswith("db"):
parts = str(value).split(",")
parsed = {}
for part in parts:
kv = part.split("=")
if len(kv) == 2:
parsed[kv[0]] = kv[1]
keyspace[key] = parsed
return keyspace
def health_score(self):
"""Calculate a cache health score (0-100)."""
metrics = self.collect()
score = 100
if metrics["hit_rate"] < 80:
score -= 20
if metrics["mem_fragmentation_ratio"] > 2.0:
score -= 15
if metrics["evicted_keys"] > 1000:
score -= 15
if metrics["blocked_clients"] > 0:
score -= 10
if metrics["rejected_connections"] > 0:
score -= 10
return {"health_score": max(0, score), "metrics": metrics}
collector = RedisMetricsCollector(r)
metrics = collector.collect()
print("Current Redis Metrics:")
print(f" Hit Rate: {metrics['hit_rate']}%")
print(f" Memory: {metrics['used_memory'] / 1024/1024:.1f} MB")
print(f" RSS: {metrics['used_memory_rss'] / 1024/1024:.1f} MB")
print(f" Fragmentation: {metrics['mem_fragmentation_ratio']:.2f}")
print(f" Evictions: {metrics['evicted_keys']}")
print(f" Clients: {metrics['connected_clients']}")
print(f" Uptime: {metrics['uptime_in_seconds'] // 3600}h")
health = collector.health_score()
print(f"\nHealth Score: {health['health_score']}/100")
Expected output:
Current Redis Metrics:
Hit Rate: 94.3%
Memory: 45.2 MB
RSS: 52.1 MB
Fragmentation: 1.15
Evictions: 0
Clients: 5
Uptime: 72h
Health Score: 100/100
Cache Operation Tracing
Trace individual cache operations for latency analysis:
import redis
import time
import json
import uuid
r = redis.Redis(decode_responses=True)
class TracedCache:
def __init__(self, redis_client):
self.r = redis_client
self.traces = []
def _trace(self, operation, key, start_time, success, extra=None):
"""Record a trace for a cache operation."""
trace = {
"trace_id": str(uuid.uuid4())[:8],
"operation": operation,
"key": key,
"duration_ms": round((time.time() - start_time) * 1000, 2),
"success": success,
"timestamp": time.time(),
}
if extra:
trace.update(extra)
self.traces.append(trace)
if trace["duration_ms"] > 5:
print(f" SLOW: {operation} {key} took {trace['duration_ms']}ms")
return trace
def get(self, key, fetch_fn=None):
"""Traced cache get with slow operation logging."""
start = time.time()
try:
value = self.r.get(key)
if value is not None:
self._trace("GET_HIT", key, start, True, {"size_bytes": len(value)})
return {"source": "cache", "data": json.loads(value)}
self._trace("GET_MISS", key, start, True)
if fetch_fn:
db_start = time.time()
data = fetch_fn(key)
self._trace("DB_FETCH", key, db_start, True)
self.set(key, data)
return {"source": "db", "data": data}
return None
except Exception as e:
self._trace("GET_ERROR", key, start, False, {"error": str(e)})
raise
def set(self, key, data, ttl=3600):
"""Traced cache set."""
start = time.time()
try:
self.r.setex(key, ttl, json.dumps(data))
self._trace("SET", key, start, True, {"ttl": ttl})
except Exception as e:
self._trace("SET_ERROR", key, start, False, {"error": str(e)})
raise
def get_slow_traces(self, threshold_ms=10):
"""Get all traces that exceeded the threshold."""
return [t for t in self.traces if t["duration_ms"] > threshold_ms]
def latency_report(self):
"""Generate a latency percentiles report."""
durations = [t["duration_ms"] for t in self.traces]
if not durations:
return {}
durations.sort()
return {
"total_operations": len(durations),
"p50": durations[len(durations) // 2],
"p95": durations[int(len(durations) * 0.95)],
"p99": durations[int(len(durations) * 0.99)],
"max": max(durations),
"avg": sum(durations) / len(durations),
}
cache = TracedCache(r)
def fetch(key):
return {"id": key, "data": "from_db"}
cache.set("trace:test", {"hello": "world"})
result = cache.get("trace:test")
print(f"Get cached: {result['source']}")
result = cache.get("trace:nonexistent", lambda: fetch("42"))
print(f"Get missing: {result['source']}")
report = cache.latency_report()
print(f"\nLatency report: {json.dumps(report, indent=2)}")
slow = cache.get_slow_traces(threshold_ms=1)
print(f"\nSlow operations (>1ms): {len(slow)}")
Expected output:
Get cached: cache
Get missing: db
Latency report: {
"total_operations": 4,
"p50": 0.42,
"p95": 1.23,
"p99": 1.56,
"max": 2.1,
"avg": 0.85
}
Slow operations (>1ms): 1
Common Mistakes
- Only monitoring cache hit rate — high hit rate can mask high memory usage or eviction rate. A cache with 99% hit rate that evicts 10,000 keys/s is unhealthy. Monitor evictions, memory, and latency too.
- Not tracking cache command latency — slow Redis commands (KEYS, SMEMBERS on large sets) can block Redis for seconds. Monitor slowlog and p99 latency per command type.
- Ignoring memory fragmentation — fragmentation over 1.5x wastes 33% of your RAM. Monitor and set alerts. Activate auto-defrag or schedule maintenance.
- Not instrumenting cache miss code paths — if cache misses are slow because of database query issues, you won't see it in cache metrics. Trace the full request path including fallback behavior.
- Setting alerts without proper thresholds — hit rate naturally dips during deployments (cold cache). Set alerts that trigger only on sustained (5+ minute) degradation, not momentary dips.
Practice Questions
- What are the five most important Redis metrics to monitor?
- How does fragmentation ratio affect effective cache capacity?
- Why should you monitor both hit rate AND eviction rate?
- How does distributed tracing help debug cache performance issues?
- What is Redis SLOWLOG and how do you use it?
Challenge
Build a cache observability system that: (1) collects Redis metrics every 10 seconds using INFO, (2) tracks hit rate, memory, evictions, command latency, and connected clients over time, (3) emits Prometheus metrics for these values, (4) creates a Grafana dashboard with panels for each metric, (5) sets up alerts for: hit rate below 80% for 5 minutes, eviction rate above 100/s for 1 minute, fragmentation above 2.0, and p99 latency above 50ms, and (6) provides a health check endpoint returning a 0-100 health score.
FAQ
Mini Project
Build a cache monitoring CLI tool that: (1) connects to Redis and runs INFO, (2) parses and displays key metrics in a formatted table, (3) computes trends by comparing current values to a previous snapshot, (4) highlights metrics that exceed configurable thresholds, (5) supports following mode (refresh every 5 seconds), (6) outputs JSON for integration with monitoring systems, and (7) color-codes health status (green/yellow/red) for each metric.
What's Next
Continue with Cache Alerting to learn about alerting strategies for cache health issues. Then explore Cache Cost Optimization for reducing Redis infrastructure costs.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro