Cache Replication: High Availability with Redis Sentinel
In this tutorial, you will learn about Cache Replication: High Availability with Redis Sentinel. We cover key concepts, practical examples, and best practices to help you master this topic.
Cache replication maintains copies of cached data across multiple Redis nodes, providing automatic failover when the master fails, read scalability through replica nodes, and data durability against node losses.
flowchart TD
Client[Application] --> Sentinel[Redis Sentinel Cluster]
Client -->|Writes| Master[Redis Master]
Client -->|Reads| Replica1[Replica 1]
Client -->|Reads| Replica2[Replica 2]
Master -->|Replication| Replica1
Master -->|Replication| Replica2
Sentinel -->|Monitor| Master
Sentinel -->|Monitor| Replica1
Sentinel -->|Monitor| Replica2
Sentinel -.->|Failover| Master
Replica1 -.->|Promoted| Master
What You'll Learn
- Master-replica replication architecture and configuration
- Redis Sentinel setup for automatic failover
- Read scaling with replica nodes
- Consistency trade-offs in replicated cache setups
Why It Matters
Without replication, a single Redis node failure causes a complete cache miss storm, where all requests hit the database simultaneously. This often cascades into a full database outage. Replication reduces the MTTR (mean time to recovery) from minutes to seconds.
Real-World Use
DodaTech's Redis Sentinel deployment monitors 3 master nodes (each with 2 replicas) across 3 availability zones. When a master in us-east-1a fails, Sentinel promotes its replica in us-east-1b within 12 seconds, keeping the cache online without manual intervention.
Sentinel Configuration
Set up Sentinel monitoring for automatic failover:
import redis
class SentinelCache:
def __init__(self, sentinel_hosts, master_name="mymaster"):
self.sentinel = redis.Sentinel(
sentinel_hosts,
socket_timeout=0.1,
sentinel_kwargs={"password": None}
)
self.master_name = master_name
self._master = None
self._replica = None
@property
def master(self):
"""Get the current master connection."""
if self._master is None:
self._master = self.sentinel.master_for(self.master_name)
return self._master
@property
def replica(self):
"""Get a replica connection for reads."""
if self._replica is None:
self._replica = self.sentinel.slave_for(self.master_name)
return self._replica
def set(self, key, value, ttl=3600):
"""Write to master."""
return self.master.setex(key, ttl, value)
def get(self, key, allow_stale=True):
"""Read from replica if allowed, otherwise from master."""
try:
if allow_stale:
return self.replica.get(key)
return self.master.get(key)
except redis.exceptions.ConnectionError:
return self.master.get(key)
def get_master_info(self):
"""Get info about the current master."""
try:
info = self.master.info()
return {
"role": info["role"],
"connected_slaves": info.get("connected_slaves", 0),
"used_memory_human": info.get("used_memory_human"),
"uptime_in_seconds": info.get("uptime_in_seconds"),
}
except Exception as e:
return {"error": str(e)}
cache = SentinelCache(
sentinel_hosts=[("127.0.0.1", 26379)],
master_name="cache-cluster"
)
info = cache.get_master_info()
print(f"Master role: {info.get('role', 'unknown')}")
print(f"Connected replicas: {info.get('connected_slaves', 0)}")
print(f"Memory: {info.get('used_memory_human', 'unknown')}")
cache.set("replication:test", "cache replication works", ttl=300)
value = cache.get("replication:test")
print(f"Read from replica: {value}")
Expected output:
Master role: master
Connected replicas: 2
Memory: 15.42M
Read from replica: cache replication works
Read Scaling with Replicas
Distribute read traffic across replicas to reduce master load:
import redis
import random
import time
class ReadScalingCache:
def __init__(self, replicas):
"""replicas is a list of (host, port) tuples."""
self.replicas = [redis.Redis(host=h, port=p, decode_responses=True)
for h, p in replicas]
def get(self, key):
"""Read from a random replica for load distribution."""
replica = random.choice(self.replicas)
start = time.perf_counter()
value = replica.get(key)
elapsed = time.perf_counter() - start
return {
"value": value,
"from": f"{replica.connection_pool.connection_kwargs['host']}:"
f"{replica.connection_pool.connection_kwargs['port']}",
"latency_ms": round(elapsed * 1000, 2)
}
def check_consistency(self, key, expected_value):
"""Check if all replicas have the same value for a key."""
results = {}
consistent = True
for replica in self.replicas:
value = replica.get(key)
host = replica.connection_pool.connection_kwargs['host']
port = replica.connection_pool.connection_kwargs['port']
results[f"{host}:{port}"] = value
if value != expected_value:
consistent = False
return {
"consistent": consistent,
"replicas": results,
}
replicas = [
("127.0.0.1", 6380),
("127.0.0.1", 6381),
("127.0.0.1", 6382),
]
cache = ReadScalingCache(replicas)
for _ in range(5):
result = cache.get("scaling:test")
print(f"Read from {result['from']} ({result['latency_ms']}ms): {result['value']}")
consistency = cache.check_consistency("scaling:test", "replicated_value")
print(f"\nReplicas consistent: {consistency['consistent']}")
for replica, value in consistency['replicas'].items():
print(f" {replica}: {value}")
Expected output:
Read from 127.0.0.1:6381 (0.42ms): replicated_value
Read from 127.0.0.1:6382 (0.38ms): replicated_value
Read from 127.0.0.1:6380 (0.41ms): replicated_value
Read from 127.0.0.1:6382 (0.39ms): replicated_value
Read from 127.0.0.1:6381 (0.40ms): replicated_value
Replicas consistent: True
127.0.0.1:6380: replicated_value
127.0.0.1:6381: replicated_value
127.0.0.1:6382: replicated_value
Replication Lag Monitoring
Track replication delay to detect issues:
import redis
import time
class LagMonitor:
def __init__(self, master, replicas):
self.master = redis.Redis(host=master[0], port=master[1])
self.replicas = [
redis.Redis(host=h, port=p) for h, p in replicas
]
def measure_lag(self):
"""Measure replication lag for each replica."""
master_time = self.master.time()
master_time_seconds = master_time[0] + master_time[1] / 1_000_000
results = []
for replica in self.replicas:
info = replica.info("replication")
lag = info.get("master_last_io_seconds_ago", -1)
replica_info = replica.info("server")
uptime = replica_info.get("uptime_in_seconds", 0)
results.append({
"host": replica.connection_pool.connection_kwargs['host'],
"port": replica.connection_pool.connection_kwargs['port'],
"lag_seconds": lag,
"master_link_status": info.get("master_link_status", "unknown"),
"uptime": uptime,
})
return results
def report(self):
"""Generate a replication health report."""
results = self.measure_lag()
print("Replication Lag Report:")
print(f"{'Host':20s} {'Lag(s)':8s} {'Status':12s} {'Uptime(s)':10s}")
print("-" * 50)
for r in results:
host = f"{r['host']}:{r['port']}"
status = "UP" if r['master_link_status'] == b"up" else "DOWN"
print(f"{host:20s} {r['lag_seconds']:8d} {status:12s} {r['uptime']:10d}")
return results
monitor = LagMonitor(
master=("127.0.0.1", 6379),
replicas=[("127.0.0.1", 6380), ("127.0.0.1", 6381)]
)
results = monitor.report()
all_healthy = all(r['lag_seconds'] < 5 and r['master_link_status'] == b"up" for r in results)
print(f"\nAll replicas healthy: {all_healthy}")
Expected output:
Replication Lag Report:
Host Lag(s) Status Uptime(s)
--------------------------------------------------
127.0.0.1:6380 0 UP 3600
127.0.0.1:6381 1 UP 3600
All replicas healthy: True
Common Mistakes
- Reading from replicas for write-after-read consistency — replication lag means a replica may not have the data you just wrote. Use master reads for consistency-critical paths.
- Not monitoring replication lag — high lag (over 5 seconds) means stale data is being served. Lag spikes often precede master failures.
- Using too many replicas without network capacity — each replica creates a full replication stream. 5+ replicas can overwhelm the master's network bandwidth.
- Assuming immediate failover — Sentinel takes 10-30 seconds to detect failure and promote a replica. During this window, the cache is unavailable.
- Not testing failover scenarios — the first failover is always the worst. Test monthly by killing the master Process and verifying applications reconnect properly.
Practice Questions
- What is the role of Redis Sentinel in cache replication?
- How does replication lag affect read consistency?
- What happens during a Sentinel failover and how long does it take?
- Why should consistency-critical reads go to the master?
- How many replicas should a master have for optimal availability?
Challenge
Design a multi-region cache replication topology with Redis Sentinel across 3 availability zones. Each zone has 1 master and 2 replicas. Sentinel is deployed with 5 instances (quorum=3). Write a failover test that: kills the master in zone A, measures downtime, verifies the replica in zone B is promoted, confirms applications reconnect, and measures the cache warm-up time after failover.
FAQ
Mini Project
Build a Redis Sentinel deployment toolkit that: (1) configures 1 master + 2 replicas, (2) sets up 3 Sentinel instances, (3) deploys a test application that reads from replicas and writes to the master, (4) simulates a master failure and measures failover time, and (5) verifies the application auto-reconnects to the new master without data loss.
What's Next
Continue with Cache Persistence to learn about RDB snapshots and AOF logs for data durability. Then explore Cache Transactions for atomic cache operations.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro