Caching Monitoring: Observability, Alerting, and Incident Response

In this tutorial, you will learn about Caching Monitoring: Observability, Alerting, and Incident Response. We cover key concepts, practical examples, and best practices to help you master this topic.

Monitoring a cache system is critical for detecting performance degradation, capacity issues, and failures before they impact users. Effective cache monitoring combines application-level metrics (hit rate, latency), infrastructure metrics (memory, connections), and business-level signals (stale data incidents).

flowchart TB
    subgraph Monitoring Stack
        Metrics[Prometheus Metrics]
        Logs[Cache Access Logs]
        Events[Invalidation Events]
    end
    Metrics --> Grafana[Grafana Dashboard]
    Logs --> Loki[Loki / Elasticsearch]
    Events --> Alert[AlertManager]
    Alert --> Pager[PagerDuty / Slack]
    Grafana --> Runbook[Cache Runbook]

What You'll Learn

• Key Redis and cache metrics to monitor in production
• Setting up Grafana dashboards for cache performance
• Alerting rules for common cache failure modes
• Cache incident runbooks: memory pressure, hot keys, network partitions

Why It Matters

A cache that silently degrades (e.g., increasing miss rate due to memory pressure) can cause a gradual performance decline that is hard to diagnose without monitoring. Proactive monitoring alerts you before users notice slowness.

Real-World Use

An e-commerce platform's Redis cache hit rate dropped from 95% to 60% over 24 hours. Monitoring alerted the team. They discovered a new deployment had added 500,000 new cache keys without increasing maxmemory, causing aggressive eviction. They fixed the TTL and increased memory.

Cache Monitoring Implementation

Prometheus Cache Metrics Exporter

const promClient = require('prom-client');

const cacheHitsTotal = new promClient.Counter({
  name: 'cache_hits_total',
  help: 'Total number of cache hits',
  labelNames: ['cache_name', 'key_pattern']
});

const cacheMissesTotal = new promClient.Counter({
  name: 'cache_misses_total',
  help: 'Total number of cache misses',
  labelNames: ['cache_name', 'key_pattern']
});

const cacheLatency = new promClient.Histogram({
  name: 'cache_latency_seconds',
  help: 'Cache operation latency in seconds',
  labelNames: ['operation', 'cache_name'],
  buckets: [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1]
});

const cacheSize = new promClient.Gauge({
  name: 'cache_size_bytes',
  help: 'Current cache memory usage in bytes',
  labelNames: ['cache_name']
});

async function trackCacheOperation(cacheName, operation, fn) {
  const end = cacheLatency.startTimer({ operation, cacheName });
  try {
    const result = await fn();
    return result;
  } finally {
    end();
  }
}

Expected output:

Prometheus scrapes metrics every 15s. Grafana dashboard shows hit rate trend, latency heatmap, and memory usage over time.

Redis INFO Monitoring Integration

const Redis = require('ioredis');

async function collectRedisMetrics(redis) {
  const info = await redis.info();
  const metrics = {};

  for (const line of info.split('\n')) {
    if (line.startsWith('used_memory:')) {
      metrics.usedMemoryBytes = parseInt(line.split(':')[1]);
    }
    if (line.startsWith('keyspace_hits:')) {
      metrics.keyspaceHits = parseInt(line.split(':')[1]);
    }
    if (line.startsWith('keyspace_misses:')) {
      metrics.keyspaceMisses = parseInt(line.split(':')[1]);
    }
    if (line.startsWith('evicted_keys:')) {
      metrics.evictedKeys = parseInt(line.split(':')[1]);
    }
    if (line.startsWith('connected_clients:')) {
      metrics.connectedClients = parseInt(line.split(':')[1]);
    }
    if (line.startsWith('used_cpu_sys:')) {
      metrics.cpuSys = parseFloat(line.split(':')[1]);
    }
  }

  const totalOps = metrics.keyspaceHits + metrics.keyspaceMisses;
  metrics.hitRate = totalOps > 0 ? (metrics.keyspaceHits / totalOps) * 100 : 0;

  return metrics;
}

Expected output:

{ usedMemoryBytes: 2147483648, keyspaceHits: 500000, keyspaceMisses: 25000, hitRate: 95.24, connectedClients: 42, evictedKeys: 150 }

Alerting Rules for Cache

const alertingRules = [
  {
    name: 'LowCacheHitRate',
    condition: (metrics) => metrics.hitRate < 80,
    message: 'Cache hit rate below 80% - check TTL, memory, or eviction policy',
    severity: 'warning'
  },
  {
    name: 'HighEvictionRate',
    condition: (metrics) => metrics.evictedKeys > 1000,
    message: 'High eviction rate - cache may be undersized',
    severity: 'warning'
  },
  {
    name: 'CacheLatencySpike',
    condition: (metrics) => metrics.p99Latency > 0.05,
    message: 'Cache P99 latency above 50ms - possible network or overload issue',
    severity: 'critical'
  },
  {
    name: 'RedisMemoryPressure',
    condition: (metrics) => metrics.usedMemoryPercent > 85,
    message: 'Redis memory usage above 85% - scale up or reduce cache size',
    severity: 'critical'
  },
  {
    name: 'CacheDisconnected',
    condition: (metrics) => metrics.connectedClients === 0,
    message: 'No connected clients to cache - service may be down',
    severity: 'critical'
  }
];

function evaluateAlerts(metrics) {
  return alertingRules
    .filter(rule => rule.condition(metrics))
    .map(rule => ({ alert: rule.name, message: rule.message, severity: rule.severity }));
}

Expected output:

When hit rate drops below 80%, an alert fires with severity 'warning'. When P99 latency exceeds 50ms, a 'critical' alert fires.

Common Mistakes

• Only monitoring aggregate hit rate without per-key-pattern breakdown — a critical endpoint may have low hit rate while average looks healthy.
• Not setting up alerts for cache failure modes — a Redis outage can go unnoticed if the application degrades gracefully but slowly.
• Monitoring cache metrics without correlating with application metrics — a cache miss spike may coincide with a deployment.
• Ignoring eviction rate — a low hit rate with high eviction means the cache is too small.
• Not monitoring cache connection pools — exhausted connection pools cause request queuing and timeouts.

Practice Questions

1. What are the three most important cache metrics to monitor?
2. How do you distinguish between a cache size issue and a TTL issue from metrics?
3. What metrics would you alert on for a Redis cache?
4. How does cache monitoring differ between in-memory and distributed caches?
5. What is a cache runbook and what should it contain?

Challenge

Design a monitoring system for a multi-node Redis Cluster. Collect per-node metrics, aggregate them, and create a Grafana dashboard showing: cluster hit rate, per-node memory usage, network latency between nodes, and eviction rate. Set up alerts for node failure and memory pressure.

FAQ

Mini Project

Add comprehensive monitoring to your cache layer. Implement Prometheus metrics for hit rate, latency, and memory. Create a Grafana dashboard. Set up alert rules for low hit rate, high eviction, and latency spikes. Simulate a cache failure and verify the alert fires.

What's Next

Continue with HTTP Caching to learn about HTTP-level caching for APIs and web pages.