Distributed Counter Design — Without Race Conditions Guide

In this tutorial, you'll learn about Distributed Counter Design. We cover key concepts, practical examples, and best practices.

Distributed counters must provide accurate counts across multiple nodes without race conditions, using techniques like atomic increments, CRDT-based counters, sharded counters, or eventually consistent counts depending on accuracy requirements.

Why Distributed Counter Design Matters

Every large-scale application has counters: YouTube counts billions of video views, Twitter tracks tweet impressions, e-commerce sites show product view counts, and rate limiters track request counts per user. A simple counter += 1 in a database fails under concurrent access — two concurrent reads get the same value and both write the same increment, losing one. At Facebook, the Like counter handles billions of increments daily across thousands of servers. Doda Browser's extension download counter uses a Redis-based sharded counter to display accurate download numbers without database contention.

Plain-Language Explanation

Imagine a room full of people counting how many times a word is said during a lecture. If everyone writes the count on their own notepad, you have 50 different numbers. If everyone tries to write on the same notepad, they bump elbows and some counts get lost. The solution: give each person their own notepad, let them count independently, and periodically ask each person "what's your number?" and add them up. Each person's count is always correct on their own notepad. The total might be slightly delayed — but it's accurate.

graph TD
    subgraph "Sharded Counter"
        C1[Client 1] --> S1[Counter Shard 1
Redis / Memcached]
        C2[Client 2] --> S2[Counter Shard 2]
        C3[Client 3] --> S3[Counter Shard N]
        C4[Client 4] --> S1
    end
    subgraph "Aggregation"
        S1 --> AGGREGATOR[Aggregator
Read All Shards]
        S2 --> AGGREGATOR
        S3 --> AGGREGATOR
        AGGREGATOR --> RESULT[Total Count]
    end
    AGGREGATOR --> DB[(Persistent Store
Snapshot)]
    style S1 fill:#e74c3c,color:#fff
    style S2 fill:#27ae60,color:#fff
    style S3 fill:#3498db,color:#fff
    style AGGREGATOR fill:#f39c12,color:#fff

Redis Atomic Counter

Redis INCR is atomic — no race condition:

import redis, threading, time

r = redis.Redis(decode_responses=True)

def increment_view_count(post_id: str, count: int = 100):
    for _ in range(count):
        r.incr(f"views:{post_id}")

threads = []
for _ in range(10):
    t = threading.Thread(target=increment_view_count, args=("post_42", 100))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

final = r.get("views:post_42")
print(f"Expected: 1000, Got: {final}")

Expected output:

Expected: 1000, Got: 1000

PostgreSQL Atomic Counter

PostgreSQL advisory locks or atomic UPDATE:

CREATE TABLE counters (
    counter_id VARCHAR(64) PRIMARY KEY,
    value BIGINT NOT NULL DEFAULT 0
);

INSERT INTO counters (counter_id, value) VALUES ('video_views:abc123', 0);

UPDATE counters SET value = value + 1 WHERE counter_id = 'video_views:abc123'
RETURNING value;

import psycopg2

conn = psycopg2.connect("dbname=counters")
cur = conn.cursor()

def increment_and_get(counter_id: str) -> int:
    cur.execute(
        "UPDATE counters SET value = value + 1 WHERE counter_id = %s RETURNING value",
        (counter_id,)
    )
    conn.commit()
    return cur.fetchone()[0]

print(f"Count: {increment_and_get('video_views:abc123')}")

Sharded Counter

Partition increment operations across Redis instances:

import hashlib

class ShardedCounter:
    def __init__(self, shards: list[redis.Redis]):
        self.shards = shards
        self.num_shards = len(shards)

    def _get_shard(self, counter_id: str) -> redis.Redis:
        shard_idx = int(hashlib.md5(counter_id.encode()).hexdigest(), 16) % self.num_shards
        return self.shards[shard_idx]

    def increment(self, counter_id: str, amount: int = 1) -> int:
        shard = self._get_shard(counter_id)
        return shard.incrby(f"counter:{counter_id}", amount)

    def get(self, counter_id: str) -> int:
        total = 0
        for shard in self.shards:
            val = shard.get(f"counter:{counter_id}")
            if val:
                total += int(val)
        return total

CRDT G-Counter

A grow-only counter using Conflict-free Replicated Data Types:

class GCounter:
    def __init__(self, node_id: str, num_nodes: int):
        self.node_id = node_id
        self.counts = [0] * num_nodes

    def increment(self, amount: int = 1):
        self.counts[self.node_id] += amount

    def value(self) -> int:
        return sum(self.counts)

    def merge(self, other: 'GCounter'):
        for i in range(len(self.counts)):
            self.counts[i] = max(self.counts[i], other.counts[i])

node_a, node_b = GCounter(0, 2), GCounter(1, 2)
node_a.increment(5)
node_b.increment(3)
node_a.increment(2)
node_a.merge(node_b)
node_b.merge(node_a)
print(f"Node A total: {node_a.value()}, Node B total: {node_b.value()}")

Expected output:

Node A total: 10, Node B total: 10

Common Mistakes

1. Read-modify-write race: Reading a counter, adding 1 in application code, and writing it back. Two concurrent readers see the same value and overwrite each other. Always use atomic operations.

2. Single-node bottleneck: A single counter that must be incremented on every request becomes a throughput bottleneck. Shard the counter across N nodes.

3. Strong consistency everywhere: Not every counter needs to be perfectly accurate at all times. View counts tolerate 5-second delays. Separate accuracy-critical counters (inventory) from approximate ones.

4. No crash recovery: In-memory counters lose data on restart. Periodically snapshot to a persistent store or use AOF with Redis.

5. Integer overflow: A PostgreSQL BIGINT maxes at 9.2 quintillion. Sounds huge, but Twitter's total tweets passed 1 trillion. Use BIGINT or switch to a logarithmic counter for astronomical scales.

Practice Questions

1. Why does a simple counter += 1 fail in distributed systems? Two processes read the same value, both increment to the same result, and the second write overwrites the first. One increment is lost. Atomic operations prevent this.

2. How does a CRDT counter guarantee accuracy? Each node tracks its own increments independently. Merging takes the max of each node's count. Since each node's count is monotonically increasing, the merge is commutative and associative.

3. When would you use a sharded counter? When the increment rate exceeds a single Redis instance's throughput (100K+/sec). Shard across N instances and sum on read.

4. How do you handle counter overflow at scale? Use BIGINT for counters up to 9.2 quintillion. For larger scales, use a logarithmic counter (stores order of magnitude) or switch to counting with probabilistic data structures like HyperLogLog.

5. What is the difference between exact and approximate counters? Exact counters track every increment precisely. Approximate counters (like HyperLogLog) use probabilistic counting with configurable error (1-2%). Approximate counters use less memory and tolerate the error for view counts.

Mini Project

Build a distributed counter with SQLite (simulating shards):

import sqlite3, threading, time, hashlib

class SQLiteShardedCounter:
    def __init__(self, num_shards: int = 4):
        self.shards = []
        for i in range(num_shards):
            conn = sqlite3.connect(f"/tmp/counter_shard_{i}.db", check_same_thread=False)
            conn.execute("CREATE TABLE IF NOT EXISTS counters (id TEXT PRIMARY KEY, val INTEGER)")
            self.shards.append(conn)

    def _shard(self, counter_id: str) -> sqlite3.Connection:
        idx = int(hashlib.md5(counter_id.encode()).hexdigest(), 16) % len(self.shards)
        return self.shards[idx]

    def incr(self, counter_id: str, amount: int = 1) -> int:
        conn = self._shard(counter_id)
        conn.execute(
            "INSERT INTO counters (id, val) VALUES (?, ?) "
            "ON CONFLICT(id) DO UPDATE SET val = val + ?",
            (counter_id, amount, amount),
        )
        conn.commit()
        cur = conn.execute("SELECT val FROM counters WHERE id = ?", (counter_id,))
        return cur.fetchone()[0]

    def get(self, counter_id: str) -> int:
        total = 0
        for conn in self.shards:
            cur = conn.execute("SELECT val FROM counters WHERE id = ?", (counter_id,))
            row = cur.fetchone()
            if row:
                total += row[0]
        return total

counter = SQLiteShardedCounter(4)

def worker(n):
    for _ in range(n):
        counter.incr("page_views")

threads = [threading.Thread(target=worker, args=(100,)) for _ in range(10)]
for t in threads: t.start()
for t in threads: t.join()

print(f"Total: {counter.get('page_views')}")

Cross-References

• Distributed Locking
• Key-Value Store Design
• Consistent Hashing
• Distributed Queue
• Rate Limiting