Distributed Operating Systems â Architecture & Design Guide
In this tutorial, you'll learn about Distributed Operating Systems. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
A distributed operating system manages multiple independent computers as a single unified system, providing shared resources, transparent access, fault tolerance, and scalability across a network of nodes.
What You'll Learn
In this tutorial, you'll learn the architecture of distributed operating systems: distributed file systems like NFS and Ceph, consensus algorithms (Paxos and Raft), distributed synchronization (mutual exclusion, leader election), clock synchronization with NTP, distributed shared memory, and fault tolerance patterns â with practical Python simulations.
Why It Matters
Modern applications are distributed by nature. From Google's Spanner to Amazon's DynamoDB to Kubernetes, Distributed Systems power the internet. Understanding distributed OS concepts helps you design scalable, fault-tolerant systems. DodaTech's Doda Browser syncs bookmarks and passwords across devices using a distributed backend.
Real-World Use
Google's Spanner uses TrueTime API and Paxos for globally distributed transactions. Apache ZooKeeper uses Raft/Zab for coordination. Ceph provides distributed storage across thousands of nodes. Kubernetes uses etcd (Raft) for cluster state management.
flowchart TB
subgraph "Distributed OS Architecture"
N1[Node 1] --- NET[Network]
N2[Node 2] --- NET
N3[Node 3] --- NET
end
subgraph "Services"
NET --> DFS[Distributed File System]
NET --> CONS[Consensus]
NET --> SYNC[Distributed Sync]
end
DFS --> NFS[NFS]
DFS --> CEPH[Ceph]
CONS --> PAXOS[Paxos]
CONS --> RAFT[Raft]
SYNC --> MUTEX[Mutual Exclusion]
SYNC --> ELEC[Leader Election]
Prerequisites: Python basics. Understanding of Operating Systems and Networking fundamentals helps.
Raft Consensus Algorithm
Raft is a consensus algorithm for managing a replicated log across distributed nodes. It guarantees consistency through leader election and log replication.
import time
import random
class RaftNode:
def __init__(self, node_id, peers):
self.id = node_id
self.peers = peers
self.state = "follower"
self.current_term = 0
self.voted_for = None
self.log = []
self.commit_index = 0
self.last_applied = 0
def start_election(self):
self.current_term += 1
self.state = "candidate"
self.voted_for = self.id
votes = 1
print(f"[Node {self.id}] Starting election for term {self.current_term}")
for peer in self.peers:
if peer.request_vote(self.current_term, self.id):
votes += 1
majority = len(self.peers) // 2 + 1
if votes >= majority:
self.state = "leader"
print(f"[Node {self.id}] ELECTED leader for term {self.current_term}")
return True
self.state = "follower"
return False
def request_vote(self, term, candidate_id):
if term > self.current_term:
self.current_term = term
self.state = "follower"
self.voted_for = candidate_id
print(f"[Node {self.id}] Voted for {candidate_id} in term {term}")
return True
return False
def append_entries(self, entries):
if self.state == "follower":
self.log.extend(entries)
print(f"[Node {self.id}] Appended {len(entries)} entries, log size: {len(self.log)}")
return True
return False
class RaftCluster:
def __init__(self, num_nodes):
self.nodes = {}
for i in range(num_nodes):
self.nodes[i] = RaftNode(i, [])
def simulate_election(self):
nodes = list(self.nodes.values())
# Connect peers
for n in nodes:
n.peers = [p for p in nodes if p.id != n.id]
# Simulate leader election
random.seed(42)
candidate = random.choice(nodes)
return candidate.start_election()
def replicate(self, leader, data):
print(f"\n[Leader {leader.id}] Proposing: {data}")
entry = {"term": leader.current_term, "data": data}
leader.log.append(entry)
for peer in leader.peers:
peer.append_entries([entry])
return data
cluster = RaftCluster(5)
success = cluster.simulate_election()
if success:
leader = [n for n in cluster.nodes.values() if n.state == "leader"][0]
cluster.replicate(leader, "hello")
cluster.replicate(leader, "world")
Expected output:
[Node 2] Starting election for term 1
[Node 0] Voted for 2 in term 1
[Node 3] Voted for 2 in term 1
[Node 2] ELECTED leader for term 1
[Leader 2] Proposing: hello
[Node 0] Appended 1 entries, log size: 1
[Node 1] Appended 1 entries, log size: 1
[Node 3] Appended 1 entries, log size: 1
[Node 4] Appended 1 entries, log size: 1
[Leader 2] Proposing: world
[Node 0] Appended 1 entries, log size: 2
Distributed Mutual Exclusion
Two major algorithms: Centralized (simple) and Ricart-Agrawala (decentralized).
import time
import threading
class CentralizedMutex:
def __init__(self):
self.queue = []
self.lock = threading.Lock()
self.holder = None
def request(self, node_id):
with self.lock:
if self.holder is None:
self.holder = node_id
print(f"[Coordinator] Granting lock to Node {node_id}")
return True
self.queue.append(node_id)
print(f"[Coordinator] Node {node_id} queued (holder: {self.holder})")
return False
def release(self, node_id):
with self.lock:
if self.holder == node_id:
print(f"[Coordinator] Lock released by Node {node_id}")
self.holder = None
if self.queue:
next_node = self.queue.pop(0)
self.holder = next_node
print(f"[Coordinator] Granting lock to Node {next_node}")
return self.holder
class RicartAgrawalaMutex:
def __init__(self, node_id, peers):
self.id = node_id
self.peers = peers
self.timestamp = 0
self.replies = set()
self.deferred = set()
def request_cs(self):
self.timestamp += 1
self.replies = set()
print(f"[Node {self.id}] Requesting CS at ts={self.timestamp}")
for peer in self.peers:
peer.receive_request(self.id, self.timestamp)
def receive_request(self, requester, ts):
if ts < self.timestamp or (ts == self.timestamp and requester < self.id):
print(f"[Node {self.id}] Deferring reply to {requester}")
self.deferred.add(requester)
else:
print(f"[Node {self.id}] Replying to {requester}")
return "reply"
nodes_ra = [RicartAgrawalaMutex(i, []) for i in range(3)]
for n in nodes_ra:
n.peers = [p for p in nodes_ra if p.id != n.id]
nodes_ra[0].request_cs()
Distributed Clock Synchronization
NTP (Network Time Protocol) synchronizes clocks across distributed nodes.
import random
class NTPClient:
def __init__(self, node_id, offset=0):
self.id = node_id
self.clock_offset = offset
def sync_with_server(self, server):
t1 = time.time() + self.clock_offset
t2 = server.handle_request(t1)
t3 = server.handle_response(t1)
t4 = time.time() + self.clock_offset
rtt = (t4 - t1) - (t3 - t2)
offset = ((t2 - t1) + (t3 - t4)) / 2
self.clock_offset += offset
print(f"[NTP Node {self.id}] RTT={rtt*1000:.1f}ms, offset={offset*1000:.1f}ms, new_offset={self.clock_offset*1000:.1f}ms")
return offset
class NTPServer:
def handle_request(self, t1):
self.t1 = t1
self.t2 = time.time()
return self.t2
def handle_response(self, t1):
self.t3 = time.time()
return self.t3
server = NTPServer()
client = NTPClient(1, offset=0.5)
client.sync_with_server(server)
Fault Tolerance
Distributed Systems must handle node failures gracefully:
class FaultTolerantNode:
def __init__(self, node_id, failure_prob=0.1):
self.id = node_id
self.failure_prob = failure_prob
self.alive = True
self.data = {}
def process_request(self, request):
if not self.alive:
return None
if random.random() < self.failure_prob:
self.alive = False
print(f"[Node {self.id}] CRASHED while processing {request}")
return None
self.data[request] = f"result_{request}"
print(f"[Node {self.id}] Processed {request}")
return self.data[request]
def restart(self):
self.alive = True
print(f"[Node {self.id}] Restarted")
class FaultTolerantSystem:
def __init__(self, nodes):
self.nodes = nodes
def submit_request(self, request):
for node in self.nodes:
result = node.process_request(request)
if result:
return result
for node in self.nodes:
node.restart()
return self.nodes[0].process_request(request)
random.seed(42)
nodes = [FaultTolerantNode(i, failure_prob=0.3) for i in range(3)]
system = FaultTolerantSystem(nodes)
for req in ["A", "B", "C"]:
system.submit_request(req)
print()
Distributed File System (NFS Simulation)
class NFSServer:
def __init__(self):
self.files = {}
self.file_handles = {}
def lookup(self, filename):
if filename not in self.file_handles:
self.file_handles[filename] = hash(filename) & 0xFFFFFFFF
self.files[filename] = b''
print(f"[NFS Server] lookup({filename}) â handle 0x{self.file_handles[filename]:08x}")
return self.file_handles[filename]
def read(self, file_handle, offset, count):
for name, fh in self.file_handles.items():
if fh == file_handle:
data = self.files.get(name, b'')[offset:offset + count]
print(f"[NFS Server] read({name}, {offset}, {count}) â {len(data)} bytes")
return data
return b''
def write(self, file_handle, offset, data):
for name, fh in self.file_handles.items():
if fh == file_handle:
buf = bytearray(self.files.get(name, b''))
while len(buf) < offset + len(data):
buf.append(0)
buf[offset:offset + len(data)] = data
self.files[name] = bytes(buf)
print(f"[NFS Server] write({name}, {offset}, {len(data)} bytes)")
return len(data)
return -1
class NFSClient:
def __init__(self, server):
self.server = server
self.cache = {}
def read_file(self, filename):
fh = self.server.lookup(filename)
if fh in self.cache:
print(f"[NFS Client] Cache hit for {filename}")
return self.cache[fh]
data = self.server.read(fh, 0, 4096)
self.cache[fh] = data
print(f"[NFS Client] Read {filename}: {data}")
return data
nfs_server = NFSServer()
nfs_client = NFSClient(nfs_server)
nfs_server.write(nfs_server.lookup("test.txt"), 0, b"Hello NFS!")
nfs_client.read_file("test.txt")
nfs_client.read_file("test.txt")
Expected output:
[NFS Server] lookup(test.txt) â handle 0x9e4da3fe
[NFS Server] write(test.txt, 0, 10 bytes)
[NFS Server] lookup(test.txt) â handle 0x9e4da3fe
[NFS Server] read(test.txt, 0, 4096) â 10 bytes
[NFS Client] Read test.txt: b'Hello NFS!'
[NFS Client] Cache hit for test.txt
Common Mistakes
1. Assuming Network Is Reliable
Networks drop packets, partition nodes, and have variable latency. Design for failures using retries, timeouts, and idempotent operations.
2. Ignoring Clock Skew
Distributed Systems cannot assume synchronized clocks. Use logical clocks (Lamport, vector) or hybrid logical clocks instead of wall clock time.
3. Not Handling Split-Brain
When network partitions occur, both sides may believe they're the leader. Use majority quorum (Paxos/Raft) to prevent split-brain.
4. Missing Idempotency
If a request times out, the client retries. The server must handle duplicate requests safely (dedup via request IDs).
5. Overlooking CAP Theorem
You can't have all three: Consistency, Availability, Partition tolerance. Choose two based on requirements (CP for banking, AP for social media).
Practice Questions
1. What problem does Raft solve? Raft provides consensus in the presence of node failures, ensuring all nodes agree on the same log. It handles leader election, log replication, and safety guarantees.
2. How does distributed mutual exclusion differ from centralized? Centralized mutex uses a single coordinator (simple but SPOF). Ricart-Agrawala uses distributed message passing (no SPOF but more messages). Both ensure at most one process is in the Critical Section.
3. Why can't wall clock time be used for ordering in Distributed Systems? Clock skew and drift mean different nodes have different times. Logical clocks provide a consistent ordering without relying on physical time.
4. What is split-brain and how does Raft prevent it? Split-brain occurs when both sides of a partition believe they're the leader. Raft prevents it by requiring a majority (>N/2) of nodes to elect a leader â only one partition can have a majority.
5. Challenge: Implement a key-value store using Raft consensus with 5 nodes. Handle leader election, log replication, and recovery after a node crash.
Mini Project: Distributed Key-Value Store
class DistributedKVStore:
def __init__(self, num_nodes=3):
self.nodes = {i: {} for i in range(num_nodes)}
self.node_count = num_nodes
def put(self, key, value):
# Simple quorum write: write to majority
for node_id in list(self.nodes.keys())[:2]:
self.nodes[node_id][key] = value
print(f"[KV] PUT {key}={value} (written to 2/3 nodes)")
def get(self, key):
# Read from a random node
nodes_read = [n for n in self.nodes.values() if key in n]
if nodes_read:
val = nodes_read[0][key]
print(f"[KV] GET {key} = {val}")
return val
print(f"[KV] GET {key} = NOT FOUND")
return None
def simulate_node_failure(self, node_id):
print(f"[FAILURE] Node {node_id} is down!")
del self.nodes[node_id]
kv = DistributedKVStore(5)
kv.put("name", "DodaTech")
kv.get("name")
kv.simulate_node_failure(0)
kv.get("name")
FAQ
Related Concepts
What's Next
You now understand distributed operating systems! Next, explore Virtualization & Containers for how Distributed Systems are deployed, and review Interprocess Communication for local and distributed communication patterns.
- Practice daily â Study how etcd, ZooKeeper, and Consul implement consensus
- Build a project â Implement a Raft-based distributed counter in Python
- Explore related topics â Check out Google Spanner and Amazon DynamoDB designs
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro