Cache Transactions: Atomic Operations with MULTI, EXEC, and WATCH
In this tutorial, you will learn about Cache Transactions: Atomic Operations with MULTI, EXEC, and WATCH. We cover key concepts, practical examples, and best practices to help you master this topic.
Cache transactions enable atomic execution of multiple cache operations using Redis MULTI/EXEC blocks, WATCH for optimistic concurrency control, and Lua scripts for complex conditional updates that prevent race conditions in concurrent cache access.
flowchart LR
Start[Begin Transaction] --> Queue[Queue Commands]
Queue -->|MULTI| Cmd1[INCR counter]
Queue --> Cmd2[SET key value]
Queue --> Cmd3[HSET user field data]
Cmd3 --> Execute[EXEC - Atomic Execution]
Execute --> Result[All or Nothing]
Execute -->|WATCH Conflict| Retry[Retry Transaction]
What You'll Learn
- MULTI/EXEC Transaction blocks for atomic batch operations
- WATCH for optimistic locking and conditional transactions
- Lua scripting for complex atomic cache updates
- Error handling and rollback in Redis transactions
Why It Matters
Without transactions, concurrent cache updates cause race conditions: two requests can read the same counter, increment it, and write back the same value, losing one increment. Transactions ensure cache state remains consistent under concurrent access from multiple application instances.
Real-World Use
DodaTech's rate limiter uses a Lua script to atomically check and increment a counter. Without atomicity, two simultaneous requests could both read 9/10, both increment to 10/10, and both pass the rate limit check. The Lua script ensures only one request passes at the limit boundary.
MULTI/EXEC Transaction
Execute multiple commands atomically:
import redis
import time
r = redis.Redis(decode_responses=True)
class CacheTransaction:
def __init__(self, redis_client):
self.r = redis_client
def atomic_counter_increment(self, key, amount=1):
"""Increment a counter within a transaction."""
pipe = self.r.pipeline()
pipe.multi()
pipe.get(key)
pipe.incrby(key, amount)
results = pipe.execute()
old_value = int(results[0]) if results[0] else 0
new_value = int(results[1])
return {"old_value": old_value, "new_value": new_value}
def atomic_batch_set(self, items, ttl=3600):
"""Atomically set multiple keys with expiry."""
pipe = self.r.pipeline(transaction=True)
pipe.multi()
for key, value in items:
pipe.setex(key, ttl, value)
results = pipe.execute()
return {"items_set": len(results), "success": all(results)}
def atomic_read_modify_write(self, key, modifier_fn):
"""Atomically read, modify, and write a value."""
pipe = self.r.pipeline()
while True:
try:
pipe.watch(key)
current = pipe.get(key)
new_value = modifier_fn(int(current) if current else 0)
pipe.multi()
pipe.set(key, new_value)
pipe.execute()
return {"old": int(current) if current else 0, "new": new_value}
except redis.WatchError:
continue
tx = CacheTransaction(r)
result = tx.atomic_counter_increment("tx:visitors")
print(f"Visitor count: {result['old']} -> {result['new']}")
items = [("tx:user:1", "Alice"), ("tx:user:2", "Bob"), ("tx:user:3", "Charlie")]
result = tx.atomic_batch_set(items, ttl=300)
print(f"Batch set: {result['items_set']} items, success: {result['success']}")
result = tx.atomic_read_modify_write("tx:score", lambda x: x + 10)
print(f"Score: {result['old']} -> {result['new']}")
Expected output:
Visitor count: 0 -> 1
Batch set: 3 items, success: True
Score: 0 -> 10
Optimistic Locking with WATCH
Prevent race conditions with conditional transactions:
import redis
import threading
import time
r = redis.Redis(decode_responses=True)
class OptimisticCacheLock:
def __init__(self, redis_client):
self.r = redis_client
def transfer_points(self, from_key, to_key, amount):
"""Atomically transfer points between two users."""
pipe = self.r.pipeline()
try:
pipe.watch(from_key, to_key)
from_balance = int(pipe.get(from_key) or 0)
to_balance = int(pipe.get(to_key) or 0)
if from_balance < amount:
pipe.unwatch()
return {"success": False, "reason": "insufficient_balance"}
pipe.multi()
pipe.decrby(from_key, amount)
pipe.incrby(to_key, amount)
pipe.execute()
return {
"success": True,
"from": from_balance - amount,
"to": to_balance + amount,
"transferred": amount,
}
except redis.WatchError:
return {"success": False, "reason": "conflict_retry"}
def concurrent_transfer_demo(self):
"""Demonstrate optimistic locking with concurrent transfers."""
self.r.set("points:alice", 100)
self.r.set("points:bob", 50)
results = []
def transfer():
result = self.transfer_points("points:alice", "points:bob", 30)
results.append(result)
threads = [threading.Thread(target=transfer) for _ in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
return results
lock = OptimisticCacheLock(r)
result = lock.transfer_points("points:alice", "points:bob", 30)
print(f"Transfer: {result}")
results = lock.concurrent_transfer_demo()
success_count = sum(1 for r in results if r.get("success"))
conflict_count = sum(1 for r in results if r.get("reason") == "conflict_retry")
print(f"Concurrent transfers: {success_count} succeeded, {conflict_count} conflicts")
Expected output:
Transfer: {'success': True, 'from': 70, 'to': 80, 'transferred': 30}
Concurrent transfers: 1 succeeded, 4 conflicts
Lua Scripting for Atomic Operations
Execute complex logic atomically on the server:
import redis
r = redis.Redis(decode_responses=True)
class LuaCacheAtomic:
def __init__(self, redis_client):
self.r = redis_client
def atomic_rate_limiter(self, key, max_requests, window_seconds):
"""Atomically check and increment a rate limit counter."""
script = """
local key = KEYS[1]
local max = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local now = redis.call('TIME')[1]
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local count = redis.call('ZCARD', key)
if count >= max then
return {0, count}
end
redis.call('ZADD', key, now, now .. ':' .. math.random())
redis.call('EXPIRE', key, window)
return {1, count + 1}
"""
return self.r.eval(script, 1, key, max_requests, window_seconds)
def atomic_cache_fill(self, key, fetch_fn, ttl=3600):
"""Atomically check cache, fetch if missing, store result."""
script = """
local key = KEYS[1]
local ttl = tonumber(ARGV[1])
local cached = redis.call('GET', key)
if cached then
return {1, cached}
end
return {0, 'miss'}
"""
result = self.r.eval(script, 1, key, ttl)
if result[0] == 1:
return {"source": "cache", "data": result[1]}
data = fetch_fn()
self.r.setex(key, ttl, data)
return {"source": "fetched", "data": data}
rate_limiter = LuaCacheAtomic(r)
for i in range(5):
allowed, count = rate_limiter.atomic_rate_limiter(
"ratelimit:api:user_42", 3, 60
)
status = "ALLOWED" if allowed == 1 else "DENIED"
print(f"Request {i+1}: {status} (count: {count})")
Expected output:
Request 1: ALLOWED (count: 1)
Request 2: ALLOWED (count: 2)
Request 3: ALLOWED (count: 3)
Request 4: DENIED (count: 3)
Request 5: DENIED (count: 3)
Common Mistakes
- Assuming MULTI/EXEC provides rollback — Redis transactions do not support rollback. If one command fails, the others still execute. The transaction guarantee is isolation, not atomicity in the ACID sense.
- Using WATCH without retry logic — WATCH only detects conflicts, it does not retry. Always wrap WATCH transactions in a retry loop.
- Mixing blocking commands (WATCH) with non-blocking — once WATCH is called, the connection is in a special state. Keep the pipeline for WATCH, MULTI, and EXEC in the same connection.
- Sending too many commands in one transaction — a transaction with 10,000 commands blocks Redis for the duration of execution. Keep transactions under 100 commands.
- Using Lua scripts for operations that could use built-in commands — INCR, HSETNX, and SETNX are atomic by themselves. Only use Lua when you need atomicity across multiple keys or complex conditional logic.
Practice Questions
- What does the MULTI command do in a Redis transaction?
- How does WATCH prevent race conditions in concurrent transactions?
- Why should you retry a transaction after a WatchError?
- What advantage do Lua scripts have over MULTI/EXEC for atomic operations?
- Can you roll back a Redis transaction if one command fails?
Challenge
Build a distributed rate limiter using Redis Lua scripting. Each IP address gets a Sliding Window of 100 requests per minute. The Lua script must: (1) remove expired entries, (2) count current entries, (3) add a new entry if under the limit, and (4) return remaining capacity and retry-after time. Test with concurrent requests from multiple threads.
FAQ
Mini Project
Build a distributed counter service that uses Redis transactions to maintain per-minute, per-hour, and per-day counters atomically. Use a Lua script that: (1) increments the current minute counter, (2) increments the current hour counter, (3) increments the daily counter, (4) sets appropriate TTLs on each counter, and (5) returns all three counts. Test with 50 concurrent threads writing simultaneously.
What's Next
Continue with Cache Locking to learn about distributed locks with Redlock and Redis. Then explore Cache Batch Operations for efficient bulk cache operations.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro