Distributed Locking — Redis, ZooKeeper & Database Locks Guide

In this tutorial, you'll learn about Distributed Locking. We cover key concepts, practical examples, and best practices.

Distributed locking coordinates access to shared resources across multiple nodes using a mutually exclusive lease, typically implemented with Redis, ZooKeeper, or database advisory locks, ensuring only one process holds the lock at a time.

Why Distributed Locking Matters

In a single-process application, a mutex prevents two threads from accessing shared state simultaneously. In a distributed system, the same problem exists across machines — two cron jobs on different servers might both try to process the same file, or two instances of a payment service might double-charge a user. Distributed locks prevent this. Apache ZooKeeper is used by Kafka, HBase, and Hadoop for distributed coordination. Redis-based locks protect critical sections at millions of operations per second. Durga Antivirus Pro uses distributed locks to ensure only one scan worker processes a given file at a time, preventing duplicate scan results.

Plain-Language Explanation

Think of a single restroom with a key hanging on a hook. To use the restroom, you take the key. If it's not there, you wait. When you're done, you hang the key back. The key is the lock. The hook is the lock service. Distributed locking works the same way across machines: acquire a token (key), do your work, release the token. The challenge is handling the key holder crashing and never returning the key — that's why distributed locks have a timeout (lease).

graph TD
    P1[Process A] -->|1. ACQUIRE lock:job-1| LOCK[Lock Service
Redis / ZooKeeper]
    P2[Process B] -->|2. ACQUIRE lock:job-1| LOCK
    LOCK -->|3. LOCK HELD| P1
    LOCK -->|4. LOCK_ACQUIRED: false| P2
    P1 -->|5. RELEASE lock:job-1| LOCK
    P2 -->|6. ACQUIRE lock:job-1| LOCK
    LOCK -->|7. LOCK HELD| P2
    P2 -->|8. RELEASE| LOCK
    style LOCK fill:#e67e22,color:#fff
    style P1 fill:#27ae60,color:#fff
    style P2 fill:#3498db,color:#fff

Redis Redlock Algorithm

Redlock uses multiple Redis instances for fault-tolerant locking:

import time, uuid

class RedisLock:
    def __init__(self, redis_client, lock_key: str, ttl_ms: int = 10000):
        self.client = redis_client
        self.lock_key = f"lock:{lock_key}"
        self.ttl_ms = ttl_ms
        self.lock_value = str(uuid.uuid4())

    def acquire(self) -> bool:
        result = self.client.set(self.lock_key, self.lock_value,
                                 nx=True, px=self.ttl_ms)
        return result is not None

    def release(self) -> bool:
        script = """
        if redis.call("get", KEYS[1]) == ARGV[1] then
            return redis.call("del", KEYS[1])
        else
            return 0
        end
        """
        result = self.client.eval(script, 1, self.lock_key, self.lock_value)
        return result == 1

    def __enter__(self):
        acquired = self.acquire()
        if not acquired:
            raise Exception("Could not acquire lock")
        return self

    def __exit__(self, *args):
        self.release()

lock = RedisLock(redis.Redis(), "job:process-orders")
with lock:
    print("Processing orders...")
    time.sleep(2)
print("Lock released")

ZooKeeper Ephemeral Lock

ZooKeeper uses ephemeral sequential nodes for distributed locks:

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

lock = zk.Lock("/locks/job-process-orders")

def process_orders():
    with lock:
        print("Processing orders with ZooKeeper lock...")
        time.sleep(2)

threads = [threading.Thread(target=process_orders) for _ in range(3)]
for t in threads: t.start()
for t in threads: t.join()
zk.stop()

PostgreSQL Advisory Lock

PostgreSQL provides session-level advisory locks:

import psycopg2

conn = psycopg2.connect("dbname=mydb")
cur = conn.cursor()
cur.execute("SET lock_timeout = '5s'")

lock_id = hash("job:process-orders") % 2147483647

cur.execute("SELECT pg_advisory_lock(%s)", (lock_id,))
print("Acquired PostgreSQL advisory lock")
try:
    time.sleep(3)
finally:
    cur.execute("SELECT pg_advisory_unlock(%s)", (lock_id,))
    print("Released lock")

Lease-Based Lock with Heartbeat

A lock with automatic expiry and heartbeat renewal:

import threading, time

class LeaseLock:
    def __init__(self, store, key: str, lease_seconds: int = 30):
        self.store = store
        self.key = key
        self.lease_seconds = lease_seconds
        self.holder_id = str(uuid.uuid4())
        self._heartbeat_thread = None
        self._running = False

    def acquire(self) -> bool:
        acquired = self.store.add(self.key, self.holder_id, expire=self.lease_seconds)
        if acquired:
            self._running = True
            self._heartbeat_thread = threading.Thread(target=self._heartbeat, daemon=True)
            self._heartbeat_thread.start()
        return acquired

    def _heartbeat(self):
        while self._running:
            time.sleep(self.lease_seconds // 3)
            self.store.touch(self.key, self.lease_seconds)

    def release(self):
        self._running = False
        current = self.store.get(self.key)
        if current == self.holder_id:
            self.store.delete(self.key)

Common Mistakes

1. No lock timeout: Locked forever if the holder crashes. Always set a TTL/lease on every distributed lock.

2. Not using fencing tokens: A lock might expire while the holder is still working. Use fencing tokens (monotonically increasing) to prevent stale holders from mutating state.

3. Single Redis instance: A single Redis master can lose the lock on failover. Redlock uses N/2+1 instances for quorum-based locking.

4. Long GC pauses: A process holding a lock might pause for garbage collection, the lock expires, another process acquires it, and both modify the same resource. Use fencing or very short TTL.

5. Blocking indefinitely: A process waiting for a lock should have a timeout. Design for the case where the lock is never released.

Practice Questions

1. What problem does the Redlock algorithm solve? Reliable distributed locking across multiple Redis instances. It uses N/2+1 quorum to remain available even if some nodes fail.

2. How does ZooKeeper implement distributed locks? Clients create ephemeral sequential nodes. The client with the lowest sequence number holds the lock. When it disconnects, the ephemeral node is automatically deleted, releasing the lock.

3. What is a fencing token? A monotonically increasing number issued when a lock is acquired. The protected resource checks that the token is the most recently issued one, preventing stale lock holders from making modifications.

4. Why not use database locks for everything? Database locks scale poorly under high contention and create long-lived connections. Redis and ZooKeeper are optimized for high-throughput lock operations.

5. When should you avoid distributed locking? When the operation can be made idempotent. Instead of locking to prevent duplicate payment, design payments to be idempotent — processing twice produces the same result.

Mini Project

Build a distributed lock with Redis:

import redis, time, uuid, threading

r = redis.Redis(decode_responses=True)

def run_job(worker_id: str, job_name: str, duration: float):
    lock_key = f"lock:{job_name}"
    lock_val = str(uuid.uuid4())
    acquired = r.set(lock_key, lock_val, nx=True, px=5000)
    if not acquired:
        print(f"{worker_id}: could not acquire lock for {job_name}")
        return
    print(f"{worker_id}: acquired lock for {job_name}")
    time.sleep(duration)
    if r.get(lock_key) == lock_val:
        r.delete(lock_key)
        print(f"{worker_id}: released lock for {job_name}")

threads = []
for i in range(3):
    t = threading.Thread(target=run_job, args=(f"worker-{i}", "daily-report", 2))
    threads.append(t)
    t.start()
for t in threads:
    t.join()

Expected output:

worker-0: acquired lock for daily-report
worker-1: could not acquire lock for daily-report
worker-2: could not acquire lock for daily-report
worker-0: released lock for daily-report

Cross-References

• Distributed Counter
• Leader Election
• Consistent Hashing
• Distributed Queue
• Caching