Leader Election — Raft, Paxos & ZooKeeper Guide
In this tutorial, you'll learn about Leader Election. We cover key concepts, practical examples, and best practices.
Leader election is the process by which distributed system nodes select one node as the leader to coordinate actions, replicate state, and handle failures — a fundamental building block of consensus algorithms like Raft and Paxos.
Why Leader Election Matters
Distributed systems need a single source of truth for coordination. Who decides what the next database log entry is? Who assigns work to workers? Who manages cluster membership? Without leader election, every node acts independently, leading to split-brain scenarios. Raft is used by etcd (Kubernetes), Consul, and MongoDB. ZooKeeper provides leader election for Kafka, HBase, and Solr. Doda Browser's sync service uses Raft-based leader election in etcd to ensure only one node processes queue-to-device push notifications at a time.
Plain-Language Explanation
Imagine a group of people stranded on an island. They need one person to make decisions — who gathers firewood today, who builds shelter. They could vote, but everyone has a different opinion. Instead, they agree: whoever's birthday is closest raises their hand first. If that person gets eaten by a crocodile, the next birthday steps up. This is leader election — a protocol to pick one coordinator, detect when it's gone, and select a replacement automatically.
graph TD
subgraph "Raft Cluster"
L((Leader))
F1((Follower))
F2((Follower))
F3((Follower))
end
Client -->|Write Request| L
L -->|Log Replication| F1
L -->|Log Replication| F2
L -->|Log Replication| F3
F1 -->|Heartbeat| L
F2 -->|Heartbeat| L
F3 -->|Heartbeat| L
subgraph "Leader Failure"
Lx((X Leader Down))
F1 -.->|Timeout & Vote Request| F2
F2 -.->|Vote Granted| F1
F1 -.->|New Leader| F3
end
style L fill:#e74c3c,color:#fff
style Lx fill:#cccccc,color:#666
style F1 fill:#3498db,color:#fff
style F2 fill:#27ae60,color:#fff
style F3 fill:#f39c12,color:#fff
Raft Leader Election (Simulated)
Raft's randomized timeouts prevent split votes:
import random, time, threading
class RaftNode:
def __init__(self, node_id: str, nodes: list[str]):
self.node_id = node_id
self.nodes = nodes
self.term = 0
self.state = "follower"
self.voted_for = None
self.leader_id = None
self.election_timeout = random.uniform(150, 300) / 1000
self.last_heartbeat = time.time()
def start_election(self):
self.state = "candidate"
self.term += 1
self.voted_for = self.node_id
votes = 1
for node in self.nodes:
if node != self.node_id:
if random.random() < 0.8:
votes += 1
majority = len(self.nodes) // 2 + 1
if votes >= majority:
self.state = "leader"
self.leader_id = self.node_id
print(f"{self.node_id} becomes leader for term {self.term}")
return True
return False
def tick(self):
if self.state == "follower" and time.time() - self.last_heartbeat > self.election_timeout:
return self.start_election()
return False
nodes = [RaftNode(f"node-{i}", [f"node-{j}" for j in range(5)]) for i in range(5)]
for n in nodes:
n.tick()
ZooKeeper Leader Election
ZooKeeper uses ephemeral sequential nodes:
from kazoo.client import KazooClient
from kazoo.recipe.election import Election
zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
def leader_callback():
print("I am the leader!")
def not_leader_callback():
print("I am a follower")
election = Election(zk, "/myapp/leader")
election.run(leader_callback, not_leader_callback)
Lower-level implementation:
class ZKLeaderElection:
def __init__(self, zk, path: str, node_id: str):
self.zk = zk
self.path = path
self.node_id = node_id
self.election_path = f"{path}/candidate-"
self.candidate_path = None
def elect(self) -> bool:
self.candidate_path = self.zk.create(
self.election_path,
value=self.node_id.encode(),
sequence=True,
ephemeral=True,
)
candidates = sorted(self.zk.get_children(self.path))
my_seq = self.candidate_path.split("-")[-1]
lowest_seq = candidates[0].split("-")[-1]
if my_seq == lowest_seq:
print(f"{self.node_id} is the leader")
return True
print(f"{self.node_id} is a follower")
return False
Bully Algorithm
Simpler approach: highest-ID node becomes leader:
class BullyElection:
def __init__(self, node_id: int, all_nodes: list[int]):
self.node_id = node_id
self.all_nodes = all_nodes
self.leader_id = None
self.alive = True
def start_election(self):
higher_nodes = [n for n in self.all_nodes if n > self.node_id]
if not higher_nodes:
self.leader_id = self.node_id
print(f"{self.node_id} is the leader (highest ID)")
return self.node_id
responses = []
for h in higher_nodes:
if random.random() < 0.7:
responses.append(h)
if not responses:
self.leader_id = self.node_id
return self.node_id
highest_responder = max(responses)
print(f"{self.node_id}: {highest_responder} responded, deferring")
self.leader_id = highest_responder
return highest_responder
etcd Raft-Based Election
etcd exposes leader election through the v3 API:
import etcd3
etcd = etcd3.client(host='127.0.0.1', port=2379)
def campaign():
election = etcd.election('my-service')
try:
election.campaign(b'worker-1')
print("Elected as leader!")
election.observe(lambda resp: print(f"Leader changed: {resp}"))
election.resign()
except Exception as e:
print(f"Not elected: {e}")
Common Mistakes
No heartbeat mechanism: Without heartbeats, followers can't detect leader failure. Implement periodic heartbeats with a configurable timeout.
Split brain: Two nodes both think they're leaders. Prevent with:
- Majority quorum (Raft requires N/2+1 votes)
- Fencing (term numbers in Raft)
- Lease-based leadership
Too short election timeout: Sub-100ms timeouts cause frequent unnecessary elections in high-latency networks. Set 150-300ms minimum.
Large cluster size: Raft's leader handles all writes. Beyond 7-9 nodes, the leader becomes a bottleneck. Use read replicas.
Ignoring clock skew: Election timeouts depend on clocks. Use monotonic clocks (time.monotonic()) not wall clocks which can jump backward.
Practice Questions
How does Raft ensure only one leader per term? Each server votes only once per term. A candidate needs a majority of votes. The term number is included in all messages — higher-term messages preempt lower-term leaders.
What is ZooKeeper's leader election mechanism? Each candidate creates an ephemeral sequential node. The candidate with the lowest sequence number is the leader. Ephemeral nodes are deleted on disconnect, triggering re-election.
How does leader election differ from consensus? Leader election picks one coordinator. Consensus (Paxos, Raft) gets multiple nodes to agree on a value. Raft combines both — leader election enables efficient log replication for consensus.
What happens during a network partition? The partition with the majority continues with a leader. The minority partition can't form quorum and enters a read-only state. When the partition heals, the minority catches up.
When would you use the Bully algorithm over Raft? Bully is simpler for small, stable clusters where node IDs are well-known. It doesn't handle network partitions well. Use Raft for production systems requiring fault tolerance.
Mini Project
Simulate Raft leader election:
import random, time
class SimRaft:
def __init__(self, n: int):
self.nodes = {i: {"term": 0, "state": "follower", "leader": None, "timeout": random.uniform(1, 3)} for i in range(n)}
self.clock = 0
def tick(self):
self.clock += 0.1
for nid, node in self.nodes.items():
if node["state"] == "follower" and self.clock > node["timeout"]:
self.election(nid, node)
def election(self, nid: int, node: dict):
node["state"] = "candidate"
node["term"] += 1
votes = 1
for other in self.nodes:
if other != nid and random.random() < 0.7:
votes += 1
majority = len(self.nodes) // 2 + 1
if votes >= majority:
node["state"] = "leader"
node["leader"] = nid
for n in self.nodes.values():
n["timeout"] = self.clock + random.uniform(1, 3)
print(f"[t={self.clock:.1f}] Node {nid} elected leader (term {node['term']}, votes {votes}/{len(self.nodes)})")
else:
node["state"] = "follower"
raft = SimRaft(5)
for _ in range(20):
raft.tick()
time.sleep(0.1)
Cross-References
- Distributed Locking
- Consensus Algorithms
- Distributed Systems
- Consistency Models
- Distributed Queue
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro