Skip to content

Cache Transactions: Atomic Operations with MULTI, EXEC, and WATCH

DodaTech Updated 2026-06-28 7 min read

In this tutorial, you will learn about Cache Transactions: Atomic Operations with MULTI, EXEC, and WATCH. We cover key concepts, practical examples, and best practices to help you master this topic.

Cache transactions enable atomic execution of multiple cache operations using Redis MULTI/EXEC blocks, WATCH for optimistic concurrency control, and Lua scripts for complex conditional updates that prevent race conditions in concurrent cache access.

flowchart LR
    Start[Begin Transaction] --> Queue[Queue Commands]
    Queue -->|MULTI| Cmd1[INCR counter]
    Queue --> Cmd2[SET key value]
    Queue --> Cmd3[HSET user field data]
    Cmd3 --> Execute[EXEC - Atomic Execution]
    Execute --> Result[All or Nothing]
    Execute -->|WATCH Conflict| Retry[Retry Transaction]

What You'll Learn

  • MULTI/EXEC Transaction blocks for atomic batch operations
  • WATCH for optimistic locking and conditional transactions
  • Lua scripting for complex atomic cache updates
  • Error handling and rollback in Redis transactions

Why It Matters

Without transactions, concurrent cache updates cause race conditions: two requests can read the same counter, increment it, and write back the same value, losing one increment. Transactions ensure cache state remains consistent under concurrent access from multiple application instances.

Real-World Use

DodaTech's rate limiter uses a Lua script to atomically check and increment a counter. Without atomicity, two simultaneous requests could both read 9/10, both increment to 10/10, and both pass the rate limit check. The Lua script ensures only one request passes at the limit boundary.

MULTI/EXEC Transaction

Execute multiple commands atomically:

import redis
import time

r = redis.Redis(decode_responses=True)

class CacheTransaction:
    def __init__(self, redis_client):
        self.r = redis_client

    def atomic_counter_increment(self, key, amount=1):
        """Increment a counter within a transaction."""
        pipe = self.r.pipeline()
        pipe.multi()
        pipe.get(key)
        pipe.incrby(key, amount)
        results = pipe.execute()

        old_value = int(results[0]) if results[0] else 0
        new_value = int(results[1])
        return {"old_value": old_value, "new_value": new_value}

    def atomic_batch_set(self, items, ttl=3600):
        """Atomically set multiple keys with expiry."""
        pipe = self.r.pipeline(transaction=True)
        pipe.multi()

        for key, value in items:
            pipe.setex(key, ttl, value)

        results = pipe.execute()
        return {"items_set": len(results), "success": all(results)}

    def atomic_read_modify_write(self, key, modifier_fn):
        """Atomically read, modify, and write a value."""
        pipe = self.r.pipeline()
        while True:
            try:
                pipe.watch(key)
                current = pipe.get(key)
                new_value = modifier_fn(int(current) if current else 0)

                pipe.multi()
                pipe.set(key, new_value)
                pipe.execute()
                return {"old": int(current) if current else 0, "new": new_value}
            except redis.WatchError:
                continue

tx = CacheTransaction(r)

result = tx.atomic_counter_increment("tx:visitors")
print(f"Visitor count: {result['old']} -> {result['new']}")

items = [("tx:user:1", "Alice"), ("tx:user:2", "Bob"), ("tx:user:3", "Charlie")]
result = tx.atomic_batch_set(items, ttl=300)
print(f"Batch set: {result['items_set']} items, success: {result['success']}")

result = tx.atomic_read_modify_write("tx:score", lambda x: x + 10)
print(f"Score: {result['old']} -> {result['new']}")

Expected output:

Visitor count: 0 -> 1
Batch set: 3 items, success: True
Score: 0 -> 10

Optimistic Locking with WATCH

Prevent race conditions with conditional transactions:

import redis
import threading
import time

r = redis.Redis(decode_responses=True)

class OptimisticCacheLock:
    def __init__(self, redis_client):
        self.r = redis_client

    def transfer_points(self, from_key, to_key, amount):
        """Atomically transfer points between two users."""
        pipe = self.r.pipeline()
        try:
            pipe.watch(from_key, to_key)

            from_balance = int(pipe.get(from_key) or 0)
            to_balance = int(pipe.get(to_key) or 0)

            if from_balance < amount:
                pipe.unwatch()
                return {"success": False, "reason": "insufficient_balance"}

            pipe.multi()
            pipe.decrby(from_key, amount)
            pipe.incrby(to_key, amount)
            pipe.execute()

            return {
                "success": True,
                "from": from_balance - amount,
                "to": to_balance + amount,
                "transferred": amount,
            }
        except redis.WatchError:
            return {"success": False, "reason": "conflict_retry"}

    def concurrent_transfer_demo(self):
        """Demonstrate optimistic locking with concurrent transfers."""
        self.r.set("points:alice", 100)
        self.r.set("points:bob", 50)

        results = []
        def transfer():
            result = self.transfer_points("points:alice", "points:bob", 30)
            results.append(result)

        threads = [threading.Thread(target=transfer) for _ in range(5)]
        for t in threads:
            t.start()
        for t in threads:
            t.join()

        return results

lock = OptimisticCacheLock(r)

result = lock.transfer_points("points:alice", "points:bob", 30)
print(f"Transfer: {result}")

results = lock.concurrent_transfer_demo()
success_count = sum(1 for r in results if r.get("success"))
conflict_count = sum(1 for r in results if r.get("reason") == "conflict_retry")
print(f"Concurrent transfers: {success_count} succeeded, {conflict_count} conflicts")

Expected output:

Transfer: {'success': True, 'from': 70, 'to': 80, 'transferred': 30}
Concurrent transfers: 1 succeeded, 4 conflicts

Lua Scripting for Atomic Operations

Execute complex logic atomically on the server:

import redis

r = redis.Redis(decode_responses=True)

class LuaCacheAtomic:
    def __init__(self, redis_client):
        self.r = redis_client

    def atomic_rate_limiter(self, key, max_requests, window_seconds):
        """Atomically check and increment a rate limit counter."""
        script = """
        local key = KEYS[1]
        local max = tonumber(ARGV[1])
        local window = tonumber(ARGV[2])
        local now = redis.call('TIME')[1]

        redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
        local count = redis.call('ZCARD', key)

        if count >= max then
            return {0, count}
        end

        redis.call('ZADD', key, now, now .. ':' .. math.random())
        redis.call('EXPIRE', key, window)
        return {1, count + 1}
        """
        return self.r.eval(script, 1, key, max_requests, window_seconds)

    def atomic_cache_fill(self, key, fetch_fn, ttl=3600):
        """Atomically check cache, fetch if missing, store result."""
        script = """
        local key = KEYS[1]
        local ttl = tonumber(ARGV[1])
        local cached = redis.call('GET', key)
        if cached then
            return {1, cached}
        end
        return {0, 'miss'}
        """
        result = self.r.eval(script, 1, key, ttl)
        if result[0] == 1:
            return {"source": "cache", "data": result[1]}

        data = fetch_fn()
        self.r.setex(key, ttl, data)
        return {"source": "fetched", "data": data}

rate_limiter = LuaCacheAtomic(r)

for i in range(5):
    allowed, count = rate_limiter.atomic_rate_limiter(
        "ratelimit:api:user_42", 3, 60
    )
    status = "ALLOWED" if allowed == 1 else "DENIED"
    print(f"Request {i+1}: {status} (count: {count})")

Expected output:

Request 1: ALLOWED (count: 1)
Request 2: ALLOWED (count: 2)
Request 3: ALLOWED (count: 3)
Request 4: DENIED (count: 3)
Request 5: DENIED (count: 3)

Common Mistakes

  • Assuming MULTI/EXEC provides rollback — Redis transactions do not support rollback. If one command fails, the others still execute. The transaction guarantee is isolation, not atomicity in the ACID sense.
  • Using WATCH without retry logic — WATCH only detects conflicts, it does not retry. Always wrap WATCH transactions in a retry loop.
  • Mixing blocking commands (WATCH) with non-blocking — once WATCH is called, the connection is in a special state. Keep the pipeline for WATCH, MULTI, and EXEC in the same connection.
  • Sending too many commands in one transaction — a transaction with 10,000 commands blocks Redis for the duration of execution. Keep transactions under 100 commands.
  • Using Lua scripts for operations that could use built-in commands — INCR, HSETNX, and SETNX are atomic by themselves. Only use Lua when you need atomicity across multiple keys or complex conditional logic.

Practice Questions

  1. What does the MULTI command do in a Redis transaction?
  2. How does WATCH prevent race conditions in concurrent transactions?
  3. Why should you retry a transaction after a WatchError?
  4. What advantage do Lua scripts have over MULTI/EXEC for atomic operations?
  5. Can you roll back a Redis transaction if one command fails?

Challenge

Build a distributed rate limiter using Redis Lua scripting. Each IP address gets a Sliding Window of 100 requests per minute. The Lua script must: (1) remove expired entries, (2) count current entries, (3) add a new entry if under the limit, and (4) return remaining capacity and retry-after time. Test with concurrent requests from multiple threads.

FAQ

Does Redis support ACID transactions?

Redis provides isolation (commands in a transaction execute sequentially without interleaving) and durability (with AOF). It does not support rollback — if a command fails, subsequent commands still execute.

When should I use MULTI/EXEC vs Lua scripting?

Use MULTI/EXEC for simple batched commands where you don't need conditional logic. Use Lua scripting when you need to make decisions based on current data (if-then logic during the transaction).

Can I use transactions in Redis Cluster?

Yes, but all keys in a transaction must belong to the same hash slot. Use hash tags to co-locate related keys. Lua scripts in Cluster require all keys to be in the same slot too.

What happens if a Redis transaction executes partially?

Redis transactions are atomic from an isolation perspective: no other client's commands interleave. But if Redis crashes during EXEC, some commands may be applied and others not. Check return values.

How do I debug a Lua script in Redis?

Use redis.log() in your script to write to the Redis log file. Test your script with EVAL before deploying. Use SCRIPT LOAD to register scripts and EVALSHA for efficient execution.

Mini Project

Build a distributed counter service that uses Redis transactions to maintain per-minute, per-hour, and per-day counters atomically. Use a Lua script that: (1) increments the current minute counter, (2) increments the current hour counter, (3) increments the daily counter, (4) sets appropriate TTLs on each counter, and (5) returns all three counts. Test with 50 concurrent threads writing simultaneously.

What's Next

Continue with Cache Locking to learn about distributed locks with Redlock and Redis. Then explore Cache Batch Operations for efficient bulk cache operations.

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro