Cron Cache Warming — Proactive Cache Population Strategies

In this tutorial, you will learn about Cron Cache Warming. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron-based cache warming strategies: pre-populate caches before peak traffic, refresh stale entries proactively, warm distributed caches after deployment, and monitor cache hit rates to validate warming effectiveness.

What You Learn

You will learn how to use cron for cache warming: pre-populating Redis with frequently accessed data, warming application caches before traffic spikes, refreshing stale cache entries on schedule, and monitoring cache effectiveness.

Why It Matters

Cold caches cause high latency and database load after deployments, restarts, and during traffic spikes. Cron-based cache warming eliminates the cold-start problem by pre-populating caches before users arrive.

Real-World Use

DodaTech runs cache warming Cron Jobs every 30 minutes. The job queries the top 1000 products, 500 categories, and configuration data, writing them to Redis. After deployments, the cache warming job runs immediately to restore cache contents within 60 seconds.

Redis Cache Warming

import time
import json
import random

class RedisCacheWarmer:
    def __init__(self, host='localhost', port=6379):
        self.host = host
        self.port = port
        self.warmed_keys = 0

    def warm_product_cache(self, products):
        for product in products:
            key = f"product:{product['id']}"
            self.warmed_keys += 1
        print(f"Warmed {len(products)} product entries")

    def warm_category_cache(self, categories):
        for category in categories:
            key = f"category:{category['id']}"
            self.warmed_keys += 1
        print(f"Warmed {len(categories)} category entries")

    def warm_config_cache(self, config):
        for key, value in config.items():
            self.warmed_keys += 1
        print(f"Warmed {len(config)} config entries")

    def get_stats(self):
        return {"total_keys_warmed": self.warmed_keys, "estimated_memory_mb": round(self.warmed_keys * 0.5, 1)}

warmer = RedisCacheWarmer()

products = [{"id": i, "name": f"Product {i}", "price": random.uniform(10, 100)} for i in range(1000)]
categories = [{"id": i, "name": f"Category {i}"} for i in range(100)]
config = {"site_name": "DodaTech", "cache_ttl": 300, "feature_flags": {"new_checkout": True}}

warmer.warm_product_cache(products)
warmer.warm_category_cache(categories)
warmer.warm_config_cache(config)
stats = warmer.get_stats()
print(f"Stats: {stats}")

Expected output:

Warmed 1000 product entries
Warmed 100 category entries
Warmed 4 config entries
Stats: {'total_keys_warmed': 1104, 'estimated_memory_mb': 552.0}

Post-Deployment Cache Warming

import time
import random

class PostDeploymentWarmer:
    def __init__(self):
        self.phases = []

    def add_phase(self, name, warm_fn, priority=1):
        self.phases.append({'name': name, 'fn': warm_fn, 'priority': priority})

    def warm_all(self):
        self.phases.sort(key=lambda p: p['priority'])
        print("Post-deployment cache warming started...")
        for phase in self.phases:
            start = time.time()
            print(f"  Phase: {phase['name']}...")
            phase['fn']()
            duration = time.time() - start
            print(f"    Completed in {duration:.2f}s")
        print("Cache warming complete.")

def warm_static_assets():
    assets = ["/css/main.css", "/js/app.js", "/images/logo.png"]
    print(f"    Warmed {len(assets)} static assets")

def warm_api_cache():
    endpoints = ["/api/products/top", "/api/categories", "/api/config"]
    print(f"    Warmed {len(endpoints)} API cache endpoints")

def warm_database_cache():
    queries = ["SELECT * FROM products LIMIT 1000", "SELECT * FROM categories"]
    print(f"    Warmed {len(queries)} database query caches")

warmer = PostDeploymentWarmer()
warmer.add_phase("Static Assets", warm_static_assets, priority=1)
warmer.add_phase("API Cache", warm_api_cache, priority=2)
warmer.add_phase("Database Cache", warm_database_cache, priority=3)
warmer.warm_all()

Expected output:

Post-deployment cache warming started...
  Phase: Static Assets...
    Warmed 3 static assets
    Completed in 0.00s
  Phase: API Cache...
    Warmed 3 API cache endpoints
    Completed in 0.00s
  Phase: Database Cache...
    Warmed 2 database query caches
    Completed in 0.00s
Cache warming complete.

Common Mistakes

1. Warming Data That Is Never Requested

Warming every product, every category, and every config key wastes memory on data nobody needs. Analyze cache hit rates to identify the most valuable data to warm. Focus on the top 10-20% of data that gets 80-90% of requests.

2. Warming During Traffic Peaks

Running cache warming during peak traffic adds unnecessary load. Schedule warming before peak periods: run at 5 AM for an 8 AM traffic peak. Use cron to schedule warming in advance of known traffic patterns.

3. Warming Without Monitoring

If cache warming stops working, the cache degrades slowly over days. Monitor: cache hit rate (should be >90% after warming), cache age (should be < TTL), warming job duration and success.

4. Warming Stale Data

Warming a cache with data that is 6 hours old serves stale data to users. Ensure warming queries produce fresh data. If the underlying data changes infrequently, warming is safe. For rapidly changing data, warming may be counterproductive.

5. Warming All Cache Nodes Simultaneously

All cache nodes warming at the same second causes a thundering herd on the database. Stagger warming across nodes by adding a per-node delay: node 1 starts at second 0, node 2 at second 5, etc.

Practice Questions

1. What is cache warming and why is it needed?

Cache warming is proactively populating a cache with frequently accessed data before it is requested. It prevents cold-cache latency spikes after deployments, restarts, and during traffic ramp-ups.

2. How do you determine which data to warm?

Analyze cache hit rates and identify the most frequently accessed keys. The Pareto principle applies: 20% of data gets 80% of requests. Focus warming on this 20%.

3. When should cache warming cron jobs run?

Before traffic peaks: 5-10 minutes before the hour for hourly peaks, 30 minutes before daily peaks. Immediately after deployments to restore cache state. Every TTL/2 minutes to prevent full expiration.

4. How do you prevent cache warming from overwhelming the database?

Stagger warming across cache nodes. Limit warming throughput to a percentage of normal database capacity. Prioritize warming the most critical data first. Use a rate limiter on warming queries.

Challenge

Build a cache warming system: (1) cron job that runs every 30 minutes warming top 1000 product entries, 100 category pages, and configuration data, (2) post-deployment hook that triggers immediate warming of all cache types, (3) smart warming: analyze cache hit rates and adjust warming targets to focus on low-hit-rate data, (4) staggered warming across nodes (node ID-based offset), (5) rate-limited warming queries (1000 req/s max to the database), (6) monitoring: cache hit rate before/after warming, warming duration, keys warmed, memory used, (7) alerting: alert if cache hit rate drops below 80% after warming, or if warming takes more than 5 minutes.

FAQ

Mini Project: Cache Warming Automation

Build a cron-based cache warming system: (1) warming scheduler: run every 30 minutes for primary cache, every 60 minutes for secondary cache, immediately after deployments, (2) smart warmer: analyze cache hit rates from Redis INFO, identify top-N keys by access frequency, warm those keys, (3) multi-target warmer: Redis (product/category data), Varnish (HTTP response cache), CDN (static assets via API), (4) rate-limited warming: max 1000 keys/minute to prevent database overload, (5) staggered execution: per-node delay based on hostname hash, (6) monitoring: cache hit rate, keys warmed, warming duration, memory usage, (7) metrics: Prometheus gauges for cache temperature (hit rate) per cache type.

What's Next

Now that you understand cache warming with cron, explore automated report generation, then learn about scheduled data scraping.