Cache Alerting: Setting Up Alerts for Redis Health and Performance Issues
DodaTech
Updated 2026-06-28
9 min read
In this tutorial, you will learn about Cache Alerting: Setting Up Alerts for Redis Health and Performance Issues. We cover key concepts, practical examples, and best practices to help you master this topic.
Cache alerting monitors key performance indicators like hit rate, memory usage, eviction rate, and command latency, triggering notifications when metrics cross thresholds that indicate cache health problems or impending failures.
flowchart TD
Redis[Redis Instance] --> Prometheus[Prometheus Metrics]
Prometheus --> AlertManager[AlertManager]
AlertManager --> Rules{Alert Rules}
Rules -->|Hit Rate < 80%| HitRate[Critical: Hit Rate Drop]
Rules -->|Evictions > 100/s| Evictions[Warning: High Evictions]
Rules -->|Memory > 80%| Memory[Warning: Memory Pressure]
Rules -->|Latency > 50ms| Latency[Critical: High Latency]
Rules -->|Frag > 2.0| Frag[Warning: Fragmentation]
HitRate --> Notify[PagerDuty / Slack / Email]
Evictions --> Notify
Memory --> Notify
Latency --> Notify
Frag --> Notify
What You'll Learn
- Key alert rules for Redis cache health
- Threshold tuning to avoid false positives
- Alert severity levels and notification routing
- Runbook integration for common alert scenarios
Why It Matters
Without alerts, a slow memory leak in Redis will eventually fill the instance, trigger OOM kills, and take down your application. Early alerts (memory at 70%) give you hours to respond. Late alerts (memory at 95%) mean scramble-mode engineering at 2 AM.
Real-World Use
DodaTech's cache alerting system monitors 12 Redis clusters with 15 alert rules. Each alert has a severity, runbook link, and auto-remediation script. When memory exceeds 75%, an auto-scaling action adds more Redis nodes. When hit rate drops below 80% for 10 minutes, an engineer investigates the access pattern change.
Alert Rule Configuration
Define and evaluate cache alert rules:
import redis
import time
import json
r = redis.Redis(decode_responses=True)
class CacheAlertManager:
def __init__(self, redis_client):
self.r = redis_client
self.alert_history = []
self.rules = self._default_rules()
def _default_rules(self):
"""Define default alert rules with thresholds."""
return [
{
"name": "hit_rate_low",
"description": "Cache hit rate below threshold",
"severity": "critical",
"expr": lambda i: i["hit_rate"] < 80,
"duration": 300,
"message": "Hit rate is {hit_rate}% (threshold: 80%)",
},
{
"name": "memory_high",
"description": "Redis memory usage above threshold",
"severity": "warning",
"expr": lambda i: i["memory_percent"] > 80,
"duration": 120,
"message": "Memory at {memory_percent}% of max (threshold: 80%)",
},
{
"name": "eviction_rate_high",
"description": "Cache eviction rate above threshold",
"severity": "warning",
"expr": lambda i: i["eviction_rate"] > 100,
"duration": 60,
"message": "Eviction rate {eviction_rate}/s (threshold: 100/s)",
},
{
"name": "fragmentation_high",
"description": "Memory fragmentation above threshold",
"severity": "warning",
"expr": lambda i: i["fragmentation"] > 2.0,
"duration": 300,
"message": "Fragmentation ratio {fragmentation} (threshold: 2.0)",
},
{
"name": "latency_high",
"description": "Redis command latency above threshold",
"severity": "critical",
"expr": lambda i: i["p99_latency_ms"] > 50,
"duration": 60,
"message": "p99 latency {p99_latency_ms}ms (threshold: 50ms)",
},
{
"name": "oom_risk",
"description": "Redis near maxmemory, OOM risk",
"severity": "critical",
"expr": lambda i: i["memory_percent"] > 95,
"duration": 30,
"message": "CRITICAL: Memory at {memory_percent}% - OOM imminent",
},
{
"name": "connections_high",
"description": "Too many connected clients",
"severity": "warning",
"expr": lambda i: i["connected_clients"] > i.get("max_clients", 10000) * 0.8,
"duration": 120,
"message": "Connections at {connected_clients} (80% of max)",
},
]
def evaluate(self):
"""Evaluate all alert rules against current metrics."""
info = self.r.info()
keyspace_hits = info.get("keyspace_hits", 0)
keyspace_misses = info.get("keyspace_misses", 0)
total_ops = keyspace_hits + keyspace_misses
hit_rate = (keyspace_hits / total_ops * 100) if total_ops > 0 else 100
maxmemory = info.get("maxmemory", 0)
used_memory = info.get("used_memory", 0)
memory_percent = (used_memory / maxmemory * 100) if maxmemory > 0 else 0
indicators = {
"hit_rate": round(hit_rate, 1),
"memory_percent": round(memory_percent, 1),
"eviction_rate": info.get("evicted_keys", 0),
"fragmentation": info.get("mem_fragmentation_ratio", 1.0),
"p99_latency_ms": info.get("latency", 0),
"connected_clients": info.get("connected_clients", 0),
"max_clients": int(info.get("maxclients", 10000)),
}
fired_alerts = []
for rule in self.rules:
try:
if rule["expr"](indicators):
alert = {
"rule": rule["name"],
"severity": rule["severity"],
"message": rule["message"].format(**indicators),
"timestamp": time.time(),
"indicators": indicators,
}
fired_alerts.append(alert)
self.alert_history.append(alert)
except Exception:
continue
return {"fired": fired_alerts, "count": len(fired_alerts), "indicators": indicators}
def get_firing(self, min_severity="warning"):
"""Get currently firing alerts filtered by severity."""
severities = {"info": 0, "warning": 1, "critical": 2}
min_level = severities.get(min_severity, 0)
return [a for a in self.alert_history if severities.get(a["severity"], 0) >= min_level]
alert = CacheAlertManager(r)
result = alert.evaluate()
print(f"Alert evaluation: {result['count']} alerts firing")
print(f"Indicators: {json.dumps(result['indicators'], indent=2)}")
for alert_info in result["fired"]:
print(f" [{alert_info['severity'].upper()}] {alert_info['message']}")
Expected output:
Alert evaluation: 0 alerts firing
Indicators: {
"hit_rate": 94.3,
"memory_percent": 25.6,
"eviction_rate": 0,
"fragmentation": 1.15,
"p99_latency_ms": 5,
"connected_clients": 5,
"max_clients": 10000
}
Runbook Integration
Create runbooks for common cache alerts:
import redis
import json
r = redis.Redis(decode_responses=True)
class CacheRunbook:
def __init__(self, redis_client):
self.r = redis_client
self.runbooks = {
"hit_rate_low": {
"title": "Low Cache Hit Rate",
"severity": "critical",
"symptoms": ["Increased database load", "Higher response latency", "Potential cost increase"],
"checks": [
"Check if a deployment changed cache keys or TTLs",
"Check if new feature is accessing different data patterns",
"Check Redis memory - is the cache full and evicting aggressively?",
"Check if cache was recently flushed or restarted",
"Verify eviction policy is appropriate for current access pattern",
],
"actions": [
"If cache was flushed: wait for warm-up (5-15 minutes)",
"If TTL changed: revert TTL to previous value",
"If memory full: increase maxmemory or add cluster node",
"If access pattern changed: consider warming the new hot keys",
"If eviction policy wrong: change maxmemory-policy to allkeys-lfu",
],
},
"memory_high": {
"title": "Redis Memory Usage High",
"severity": "warning",
"symptoms": ["Eviction rate increases", "Hit rate drops", "Potential OOM risk"],
"checks": [
"Run MEMORY STATS to see where memory is allocated",
"Run MEMORY DOCTOR for recommendations",
"Check for memory leaks (keys accumulating without TTL)",
"Check fragmentation ratio",
"Review maxmemory setting vs available RAM",
],
"actions": [
"If near maxmemory: increase maxmemory if RAM available",
"If high fragmentation: enable ACTIVE_DEFRAG or restart during maintenance",
"If keys without TTL: set TTLs or review application code",
"If sustained: add more Redis nodes (Cluster) or increase instance size",
],
},
"eviction_rate_high": {
"title": "High Cache Eviction Rate",
"severity": "warning",
"symptoms": ["Hit rate dropping", "Increased database load", "Cache thrashing"],
"checks": [
"Check maxmemory vs used_memory",
"Check eviction policy - is it appropriate?",
"Check if TTLs are too long (data not expiring naturally)",
"Check for sudden traffic increase",
],
"actions": [
"If memory full: increase maxmemory or add nodes",
"If TTLs too long: reduce TTLs for less important data",
"If policy wrong: switch to allkeys-lfu for skewed access patterns",
"If traffic spike: wait it out or scale up temporarily",
],
},
}
def get_runbook(self, alert_name):
"""Get the runbook for a specific alert."""
return self.runbooks.get(alert_name, {
"title": "Unknown Alert",
"actions": ["Check Redis logs", "Check application logs", "Contact on-call engineer"],
})
def generate_report(self, firing_alerts):
"""Generate a remediation report for firing alerts."""
report = []
for alert_info in firing_alerts:
runbook = self.get_runbook(alert_info["rule"])
report.append({
"alert": alert_info["rule"],
"message": alert_info["message"],
"severity": alert_info["severity"],
"runbook": runbook,
})
return report
runbooks = CacheRunbook(r)
firing = [
{"rule": "hit_rate_low", "message": "Hit rate is 65%", "severity": "critical"},
{"rule": "memory_high", "message": "Memory at 82%", "severity": "warning"},
]
report = runbooks.generate_report(firing)
for entry in report:
print(f"\n[{entry['severity'].upper()}] {entry['alert']}")
print(f" Message: {entry['message']}")
print(f" Title: {entry['runbook']['title']}")
print(f" First check: {entry['runbook']['checks'][0]}")
Expected output:
[CRITICAL] hit_rate_low
Message: Hit rate is 65%
Title: Low Cache Hit Rate
First check: Check if a deployment changed cache keys or TTLs
[WARNING] memory_high
Message: Memory at 82%
Title: Redis Memory Usage High
First check: Run MEMORY STATS to see where memory is allocated
Common Mistakes
- Setting thresholds too tight — a hit rate alert at 95% fires constantly during deployments (cache cold start). Set thresholds that account for normal operations: 80% for hit rate, 80% for memory.
- Alert fatigue from eviction rate spikes — eviction rate spikes during traffic bursts (flash sales). Use rate-of-change alerts instead of absolute thresholds: sustained evictions >100/s for 5 minutes.
- Not differentiating between alert causes — a hit rate drop from cache flush vs a hit rate drop from memory pressure requires different responses. Add context to alerts.
- Forgetting to set up notification routing — page the right team at the right time. Send warning-level alerts to Slack during business hours, critical alerts to PagerDuty 24/7.
- Not testing alert rules — test each rule by simulating the condition (flush cache for hit rate, fill memory for memory pressure). Verify the alert fires and the right people get notified.
Practice Questions
- What are the five most important Redis alert rules for production?
- Why should you add duration (time before firing) to alert rules?
- What is the difference between absolute and rate-of-change alerts for eviction rate?
- How do you prevent alert fatigue for cache monitoring?
- What information should a cache alert runbook contain?
Challenge
Design a comprehensive alerting strategy for a Redis Cluster with 6 nodes. Define alert rules for: (1) per-node hit rate below 80%, (2) per-node memory above 75%, (3) cluster-wide eviction rate above 500/s, (4) any node with fragmentation above 2.0, (5) any node with p99 latency above 50ms, (6) any node down for more than 30 seconds, (7) cluster state not OK, and (8) Replication lag above 10 seconds on any replica. Define severity, duration, notification channel, and runbook action for each.
FAQ
Mini Project
Build a cache alert rule generator that: (1) observes Redis metrics for 24 hours to establish baselines, (2) suggests threshold values for each alert rule based on observed percentiles, (3) generates Prometheus alert rules in YAML format, (4) creates Grafana alert rules for UI-based alerting, (5) generates runbook documentation for each alert, and (6) exports notification routing configuration for PagerDuty and Slack.
What's Next
Continue with Cache Cost Optimization to learn strategies for reducing Redis infrastructure costs. Then explore Cloud Cache Services for managed Redis on AWS, GCP, and Azure.
← Previous
Cache Observability: Monitoring, Metrics, and Distributed Tracing for Caches
Next →
Cache Cost Optimization: Reducing Redis Infrastructure Costs
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro