Caching Performance: Measuring Hit Ratios, Latency, and Throughput

In this tutorial, you will learn about Caching Performance: Measuring Hit Ratios, Latency, and Throughput. We cover key concepts, practical examples, and best practices to help you master this topic.

Measuring cache performance is essential to verify that your caching strategy is effective. Key metrics include cache hit ratio, read/write latency, miss penalty, throughput, and eviction rate. Without measurement, you cannot tune your cache or justify its cost.

flowchart LR
    Request[Incoming Request] --> StatsCollector[Cache Stats Collector]
    StatsCollector -->|Hit| HitCounter[Hits + 1]
    StatsCollector -->|Miss| MissCounter[Misses + 1]
    StatsCollector -->|Latency| LatencyHistogram[Latency Distribution]
    HitCounter --> Dashboard[Prometheus / Grafana]
    MissCounter --> Dashboard
    LatencyHistogram --> Dashboard
    Dashboard --> Alert[Alert on Low Hit Rate]

What You'll Learn

• Key cache performance metrics and how to measure them
• Cache profiling: finding hot keys, cold keys, and wasted memory
• Benchmarking cache backends (Redis, Memcached, local)
• Tuning cache size, TTL, and eviction policy based on data

Why It Matters

A cache with a 50% hit rate is often worse than no cache — it adds latency for cache lookups while still hitting the database for half of requests. Measuring hit rate and miss penalty tells you whether the cache is actually helping.

Real-World Use

A microservice team noticed high database CPU despite having Redis caching. They added hit-ratio monitoring and discovered the hit rate was only 45%. The issue: TTL was too short (30 seconds) for the access pattern. Increasing TTL to 300 seconds raised the hit rate to 92% and reduced DB CPU by 70%.

Cache Performance Measurement

Cache Statistics Middleware

class CacheStats {
  constructor() {
    this.hits = 0;
    this.misses = 0;
    this.totalLatency = 0;
    this.callCount = 0;
    this.histogram = { p50: 0, p90: 0, p99: 0 };
    this.latencies = [];
  }

  record(isHit, latencyMs) {
    if (isHit) this.hits++;
    else this.misses++;
    this.totalLatency += latencyMs;
    this.callCount++;
    this.latencies.push(latencyMs);
  }

  hitRate() {
    if (this.callCount === 0) return 0;
    return (this.hits / this.callCount) * 100;
  }

  avgLatency() {
    if (this.callCount === 0) return 0;
    return this.totalLatency / this.callCount;
  }

  percentile(p) {
    if (this.latencies.length === 0) return 0;
    const sorted = [...this.latencies].sort((a, b) => a - b);
    const index = Math.ceil((p / 100) * sorted.length) - 1;
    return sorted[index];
  }

  metrics() {
    return {
      hitRate: this.hitRate().toFixed(2) + '%',
      avgLatencyMs: this.avgLatency().toFixed(2),
      p50Ms: this.percentile(50).toFixed(2),
      p90Ms: this.percentile(90).toFixed(2),
      p99Ms: this.percentile(99).toFixed(2),
      totalCalls: this.callCount
    };
  }

  reset() {
    this.hits = 0;
    this.misses = 0;
    this.totalLatency = 0;
    this.callCount = 0;
    this.latencies = [];
  }
}

const cacheStats = new CacheStats();

// Usage in cache layer
const start = Date.now();
const result = await cache.get(key);
cacheStats.record(result !== null, Date.now() - start);

Expected output:

After 1000 requests: { hitRate: '92.50%', avgLatencyMs: '1.23', p50Ms: '0.50', p90Ms: '2.10', p99Ms: '5.80', totalCalls: 1000 }

Cache Key Access Frequency Profiling

class AccessProfiler {
  constructor() {
    this.accessCount = new Map();
    this.lastAccess = new Map();
  }

  record(key) {
    this.accessCount.set(key, (this.accessCount.get(key) || 0) + 1);
    this.lastAccess.set(key, Date.now());
  }

  topHotKeys(n = 10) {
    return Array.from(this.accessCount.entries())
      .sort((a, b) => b[1] - a[1])
      .slice(0, n)
      .map(([key, count]) => ({ key, count }));
  }

  coldKeys(threshold = 2) {
    return Array.from(this.accessCount.entries())
      .filter(([key, count]) => count < threshold)
      .map(([key, count]) => ({ key, count }));
  }

  wastedSpace() {
    const now = Date.now();
    let wasted = 0;
    for (const [key, lastAccess] of this.lastAccess) {
      if (now - lastAccess > 3600000) wasted++;
    }
    return wasted;
  }
}

Expected output:

Top 10 hot keys show which entries are accessed most frequently. Cold keys (accessed < 2 times) indicate cache entries that are wasting memory.

Benchmarking Cache Backend

const crypto = require('crypto');

async function benchmarkCache(cacheClient, keyCount = 1000, iterations = 10000) {
  const keys = Array.from({ length: keyCount }, () =>
    crypto.randomBytes(8).toString('hex')
  );

  // Pre-populate
  for (const key of keys) {
    await cacheClient.setEx(key, 3600, JSON.stringify({ data: key }));
  }

  // Benchmark reads
  const readTimes = [];
  for (let i = 0; i < iterations; i++) {
    const key = keys[Math.floor(Math.random() * keys.length)];
    const start = process.hrtime.bigint();
    await cacheClient.get(key);
    readTimes.push(Number(process.hrtime.bigint() - start) / 1000);
  }

  // Benchmark writes
  const writeTimes = [];
  for (let i = 0; i < iterations; i++) {
    const key = keys[Math.floor(Math.random() * keys.length)];
    const start = process.hrtime.bigint();
    await cacheClient.setEx(key, 3600, JSON.stringify({ data: key }));
    writeTimes.push(Number(process.hrtime.bigint() - start) / 1000);
  }

  const avg = (arr) => arr.reduce((a, b) => a + b, 0) / arr.length;

  return {
    read: { avgMicros: avg(readTimes).toFixed(2), p99Micros: percentile(readTimes, 99).toFixed(2) },
    write: { avgMicros: avg(writeTimes).toFixed(2), p99Micros: percentile(writeTimes, 99).toFixed(2) },
    throughput: (iterations * 1000000 / avg(readTimes)).toFixed(0) + ' ops/sec'
  };
}

Expected output:

{ read: { avgMicros: '45.23', p99Micros: '120.50' }, write: { avgMicros: '52.10', p99Micros: '150.80' }, throughput: '22123 ops/sec' }

Common Mistakes

• Measuring only hit rate without miss penalty — a 99% hit rate on a 100ms cache lookup might still be slower than a direct DB query if the cache is inefficient.
• Not measuring by key pattern — aggregate hit rate can look healthy while a critical endpoint has a 10% hit rate.
• Benchmarking with a single connection — real workloads have concurrent access, which affects latency distribution.
• Forgetting to warm up the cache before benchmarking — cold cache results are misleading.
• Using average latency instead of percentiles — the average hides tail latency that affects user experience.

Practice Questions

1. What is a good cache hit rate target?
2. Why is the P99 latency more important than average latency for cache performance?
3. How do you identify hot keys in a cache?
4. What is the difference between read throughput and read latency?
5. How do you calculate the effective speedup from caching?

Challenge

Set up a cache benchmarking suite for the blog API. Measure hit rate, P50/P90/P99 latency, and throughput for Redis vs. in-memory cache vs. no cache. Run the benchmark at 100, 1000, and 10000 concurrent requests. Identify the inflection point where each cache type starts to degrade.

FAQ

Mini Project

Add comprehensive cache metrics to the blog API. Expose a /cache/metrics endpoint returning hit rate, miss penalty, P50/P90/P99 latency, and top 10 hot keys. Set up Prometheus metrics and create a Grafana dashboard showing cache performance over time.

What's Next

Continue with Caching Monitoring to learn about cache monitoring, alerting, and Observability in production.