← Back to all projects

CONSENSUS PROTOCOLS LEARNING PROJECTS

Understanding Consensus Protocols from Scratch

Great choice! Consensus protocols are one of the most fascinating topics in distributed systems. Let me break this down and give you a learning path that builds understanding through hands-on projects.

Core Concept Analysis

What is consensus? At its heart, consensus is about getting multiple computers (nodes) to agree on something when they can’t fully trust each other or the network connecting them.

The Fundamental Building Blocks

  1. The Agreement Problem - How do nodes agree on a single value when messages can be lost or delayed?
  2. Failure Models - What can go wrong? (crashes, network partitions, malicious actors)
  3. Leader Election - How do nodes pick one “leader” to coordinate?
  4. Log Replication - How do nodes keep their data in sync?
  5. Safety vs Liveness - Can’t have perfect agreement AND guaranteed progress
  6. Byzantine Fault Tolerance - What if some nodes actively lie?

Project Recommendations

Project 1: Two Generals Problem Simulator

  • File: CONSENSUS_PROTOCOLS_LEARNING_PROJECTS.md
  • Programming Language: C
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Distributed Systems Theory
  • Software or Tool: Simulation
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A terminal-based simulation that demonstrates why achieving consensus over an unreliable network is fundamentally hard. You’ll create two “generals” (processes) trying to coordinate an attack time while messages randomly get lost.

Why it teaches consensus: This is THE foundational problem. Before you learn any algorithm, you need to viscerally understand why consensus is hard. The Two Generals Problem proves that perfect consensus is impossible over an unreliable channel. Every consensus algorithm is a practical workaround to this impossibility.

Core challenges you’ll face:

  • Simulating message loss and delays (maps to → network unreliability)
  • Implementing acknowledgment chains and seeing them fail (maps to → why acknowledgments alone don’t solve it)
  • Understanding why no finite number of messages guarantees agreement (maps to → impossibility proofs)
  • Visualizing the state of each general’s knowledge over time (maps to → distributed state)

Key Concepts:

  • Network Unreliability: “Designing Data-Intensive Applications” Ch. 8 - Martin Kleppmann
  • Impossibility Results: Original Lamport paper “The Byzantine Generals Problem” (1982)
  • Message Passing: “Computer Networks” Ch. 1 - Tanenbaum & Wetherall

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic programming (any language), understanding of processes/threads

Real world outcome: You’ll have a terminal animation showing two generals sending messengers back and forth, with some messengers getting “captured” (lost). The simulation will demonstrate that no matter how many acknowledgments you add, you can never achieve 100% certainty. You’ll see output like:

General A: "Attack at dawn!" → [MESSENGER CAPTURED]
General B: (waiting...)
General A: (retrying) "Attack at dawn!" → [DELIVERED]
General B: "Acknowledged!" → [DELIVERED]
General A: "Confirm ack!" → [MESSENGER CAPTURED]
... (uncertainty never resolves)

Learning milestones:

  1. After simulating message loss → You understand why “just send a message” doesn’t work
  2. After implementing ACK chains → You realize acknowledgments create infinite regress
  3. After running many simulations → You internalize that consensus requires different approaches (timeouts, majority voting, etc.)

Project 2: Leader Election with Bully Algorithm

  • File: leader_election_bully_algorithm.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust, Python
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: Level 1: The “Resume Gold”
  • Difficulty: Level 2: Intermediate (The Developer)
  • Knowledge Area: Distributed Systems, Consensus
  • Software or Tool: Sockets, Multi-threading
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A cluster of 5 processes that automatically elect a leader when the current leader fails, using the simple “Bully” algorithm. Each process has an ID, and the highest-ID alive process becomes leader.

Why it teaches consensus: Leader election is a sub-problem of consensus. Most consensus protocols (including Raft) use a leader to coordinate agreement. Understanding leader election in isolation makes the full protocols much clearer.

Core challenges you’ll face:

  • Detecting when the leader has failed (maps to → failure detection, heartbeats)
  • Coordinating election when multiple nodes notice failure simultaneously (maps to → race conditions)
  • Handling network partitions where nodes can’t see each other (maps to → split-brain problem)
  • Implementing timeouts correctly (maps to → liveness vs safety trade-offs)

Resources for key challenges:

  • Raft.github.io - Section on leader election explains the problem clearly

Key Concepts:

  • Failure Detection: “Designing Data-Intensive Applications” Ch. 8 (Unreliable Clocks section) - Kleppmann
  • Election Algorithms: “Distributed Systems” (Tanenbaum) Ch. 6
  • Heartbeats & Timeouts: “Operating Systems: Three Easy Pieces” (Concurrency section) - Arpaci-Dusseau

Difficulty: Beginner-Intermediate Time estimate: 1 week Prerequisites: Sockets/networking basics, multi-threading

Real world outcome: You’ll have 5 terminal windows open (one per node), and you can watch them elect a leader. When you kill the leader process (Ctrl+C), you’ll see the remaining nodes detect the failure and hold a new election. Output like:

[Node 3]: Leader Node 5 missed 3 heartbeats. Starting election.
[Node 3]: Sending ELECTION to nodes [4, 5]
[Node 4]: Received ELECTION from Node 3. I'm higher, sending OK.
[Node 4]: No higher nodes responded. I am the new leader!
[Node 4]: Broadcasting COORDINATOR message.
[Node 3]: Acknowledging Node 4 as leader.

Learning milestones:

  1. After implementing heartbeats → You understand failure detection is probabilistic, not certain
  2. After handling simultaneous elections → You see why timing and IDs matter
  3. After causing a network partition → You discover the “split-brain” problem (two leaders!)

Project 3: Raft Consensus Implementation

  • File: raft_consensus_implementation.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust, Python
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: Level 4: The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced (The Engineer)
  • Knowledge Area: Distributed Systems, Consensus Protocols
  • Software or Tool: Raft Protocol, RPC
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A complete implementation of the Raft consensus algorithm with leader election, log replication, and a simple key-value store on top.

Why it teaches consensus: Raft was explicitly designed to be understandable. It’s the same power as Paxos but decomposed into clear sub-problems. Building Raft will teach you everything about how real distributed databases maintain consistency.

Core challenges you’ll face:

  • Implementing the three states (Follower, Candidate, Leader) and transitions (maps to → state machines)
  • Log replication and ensuring logs are identical across nodes (maps to → replicated state machines)
  • Handling term numbers and preventing stale leaders (maps to → logical clocks, fencing)
  • Implementing commit rules (when is a write “safe”?) (maps to → safety guarantees)
  • Handling node crashes and recovery (maps to → persistence, durability)

Resources for key challenges:

Key Concepts:

  • Replicated State Machines: “Designing Data-Intensive Applications” Ch. 9 - Kleppmann
  • Log Replication: The Raft Paper Section 5 - Ongaro & Ousterhout
  • Term Numbers & Fencing: “Designing Data-Intensive Applications” Ch. 8 (Fencing Tokens) - Kleppmann
  • Persistence: “Operating Systems: Three Easy Pieces” (Persistence section) - Arpaci-Dusseau

Difficulty: Intermediate-Advanced Time estimate: 2-4 weeks Prerequisites: Projects 1 & 2, solid understanding of networking, concurrency

Real world outcome: You’ll have a distributed key-value store that survives node failures. You can:

# Start 5 nodes
$ ./raft-node --id=1 --peers=2,3,4,5 &
$ ./raft-node --id=2 --peers=1,3,4,5 &
# ... etc

# Write a value (goes to leader, replicates to majority)
$ curl -X PUT localhost:8001/key/name -d "Alice"
{"status": "committed", "term": 3, "index": 42}

# Kill 2 nodes - cluster still works!
$ kill %2 %3

# Read still works (majority alive)
$ curl localhost:8001/key/name
{"value": "Alice"}

Learning milestones:

  1. After leader election works → You understand how distributed systems pick a coordinator
  2. After log replication works → You understand how writes propagate reliably
  3. After surviving node failures → You truly understand fault-tolerant consensus

Project 4: Byzantine Fault Tolerant Chat Room

  • File: byzantine_fault_tolerant_chat.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust, Python
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: Level 5: The “Industry Disruptor”
  • Difficulty: Level 4: Expert (The Systems Architect)
  • Knowledge Area: Distributed Systems, Byzantine Fault Tolerance
  • Software or Tool: PBFT Protocol, Cryptographic Signatures
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A chat application where messages are ordered by consensus, but some nodes might be malicious (sending different messages to different peers, lying about votes).

Why it teaches consensus: Regular consensus (Raft, Paxos) assumes nodes are honest but might crash. Byzantine Fault Tolerance (BFT) handles actively malicious nodes. This is the foundation of blockchain technology and critical systems.

Core challenges you’ll face:

  • Implementing PBFT (Practical Byzantine Fault Tolerance) phases (maps to → multi-round voting)
  • Detecting and ignoring Byzantine nodes (maps to → cryptographic signatures, voting thresholds)
  • Understanding the 3f+1 requirement (need 3f+1 nodes to tolerate f Byzantine faults) (maps to → quorum math)
  • Handling view changes when the leader is Byzantine (maps to → Byzantine leader election)

Key Concepts:

  • Byzantine Failures: “The Byzantine Generals Problem” - Lamport, Shostak, Pease (1982)
  • PBFT Algorithm: “Practical Byzantine Fault Tolerance” - Castro & Liskov (1999)
  • Cryptographic Signatures: “Serious Cryptography” Ch. 13 - Aumasson
  • Quorum Systems: “Designing Data-Intensive Applications” Ch. 9 - Kleppmann

Difficulty: Advanced Time estimate: 3-4 weeks Prerequisites: Project 3 (Raft), basic cryptography (hashing, signatures)

Real world outcome: A chat room where users see messages in the same order, even if some nodes are lying:

[Node 1 - Honest]: "Hello everyone!"
[Node 2 - BYZANTINE]: (sends "Attack!" to Node 3, "Peace!" to Node 4)
[Node 3]: PRE-PREPARE received, checking signature... VALID
[Node 3]: PREPARE phase - collecting 2f votes...
[Node 3]: Byzantine node 2 detected! Different messages to different peers.
[Node 3]: Ignoring Node 2's message. Consensus on: "Hello everyone!" from Node 1

Learning milestones:

  1. After implementing basic PBFT → You understand why it needs more rounds than Raft
  2. After detecting a Byzantine node → You see how signatures and voting thresholds catch liars
  3. After running with f Byzantine nodes → You internalize the 3f+1 requirement

Project 5: Distributed Lock Service (like Chubby/ZooKeeper)

  • File: distributed_lock_service.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust, Java
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: Level 4: The “Open Core” Infrastructure
  • Difficulty: Level 4: Expert (The Systems Architect)
  • Knowledge Area: Distributed Systems, Coordination Services
  • Software or Tool: ZooKeeper, etcd, Consensus Protocol
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A coordination service that provides distributed locks, leader election, and configuration management - similar to Google’s Chubby or Apache ZooKeeper.

Why it teaches consensus: This is consensus applied. Real systems don’t use consensus directly; they use it to build primitives (locks, barriers, queues) that applications need. Building this shows you how consensus enables higher-level coordination.

Core challenges you’ll face:

  • Implementing locks that survive leader failures (maps to → session management, lease expiration)
  • Sequence numbers for ordered operations (maps to → linearizability)
  • Watch notifications when data changes (maps to → pub/sub over consensus)
  • Ephemeral nodes that disappear when clients disconnect (maps to → failure detection integration)

Key Concepts:

  • Distributed Locks: “Designing Data-Intensive Applications” Ch. 8 (Distributed Locks section) - Kleppmann
  • Linearizability: “Designing Data-Intensive Applications” Ch. 9 - Kleppmann
  • ZooKeeper Semantics: “ZooKeeper: Wait-free coordination for Internet-scale systems” - Hunt et al.
  • Session Management: “Building Microservices” Ch. 11 - Newman

Difficulty: Advanced Time estimate: 1 month Prerequisites: Project 3 (Raft), familiarity with ZooKeeper or etcd concepts

Real world outcome: A service that applications can use for coordination:

# Client 1
lock = client.create_lock("/locks/resource-1")
lock.acquire()  # Blocks until lock is held
# ... do critical work ...
lock.release()

# Client 2 (running simultaneously)
lock = client.create_lock("/locks/resource-1")
lock.acquire()  # Waits for Client 1 to release
# ... do critical work ...

And if Client 1 crashes while holding the lock, the service detects it and releases automatically.

Learning milestones:

  1. After distributed locks work → You understand how consensus enables mutual exclusion across machines
  2. After implementing watches → You see how consensus enables reliable notifications
  3. After handling client failures → You understand session semantics and lease expiration

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor
Two Generals Simulator Beginner Weekend Foundation - “why is this hard?” ⭐⭐⭐ (visual, intuitive)
Leader Election (Bully) Beginner-Intermediate 1 week Core sub-problem ⭐⭐⭐ (satisfying when it works)
Raft Implementation Intermediate-Advanced 2-4 weeks Deep - the real thing ⭐⭐⭐⭐⭐ (this is it!)
Byzantine Chat Advanced 3-4 weeks Specialized - malicious actors ⭐⭐⭐⭐ (feels like security)
Lock Service Advanced 1 month Applied - real-world usage ⭐⭐⭐⭐ (immediately useful)

Recommendation

Since you’re starting from zero knowledge, follow this path:

  1. Start with Project 1 (Two Generals) this weekend. It’s quick and will give you the “aha!” moment about why consensus is hard.

  2. Then do Project 2 (Leader Election) next week. This isolates one piece of consensus and makes Project 3 much easier.

  3. Before Project 3, spend an afternoon with The Secret Lives of Data visualization. It’s the single best resource for understanding Raft intuitively.

  4. Then tackle Project 3 (Raft). This is the main event. Take 2-4 weeks, don’t rush. Use Go or Rust (both have great concurrency support).

Projects 4 and 5 are “electives” based on your interests - Byzantine consensus if you’re interested in blockchain/security, Lock Service if you’re interested in practical distributed systems.

Final Overall Project: Distributed Database with Multi-Region Consensus

What you’ll build: A geo-distributed SQL database (like a mini CockroachDB or Spanner) that uses consensus to replicate data across multiple “regions” while maintaining strong consistency.

Why this is the ultimate test: This combines everything:

  • Raft for intra-region consensus
  • Cross-region consensus for global transactions
  • Distributed transactions (2PC + consensus)
  • Conflict resolution for concurrent writes
  • Real SQL queries on replicated data

Core challenges you’ll face:

  • Implementing range-based sharding with consensus per range (maps to → scaling consensus)
  • Distributed transactions across shards (maps to → two-phase commit + consensus)
  • Handling cross-region latency (maps to → async replication, consistency levels)
  • SQL query planning over distributed data (maps to → distributed query execution)
  • Clock synchronization for consistent snapshots (maps to → hybrid logical clocks)

Key Concepts:

  • Distributed Transactions: “Designing Data-Intensive Applications” Ch. 9 - Kleppmann
  • Spanner Architecture: “Spanner: Google’s Globally-Distributed Database” - Corbett et al. (Google, 2012)
  • Hybrid Logical Clocks: “Logical Physical Clocks and Consistent Snapshots” - Kulkarni et al.
  • Range-based Sharding: CockroachDB Architecture Documentation

Difficulty: Expert Time estimate: 2-3 months Prerequisites: All previous projects, understanding of databases and SQL

Real world outcome: A database you can deploy across 3 “regions” (could be different machines or even Docker containers simulating regions):

-- Connect to any region
$ psql -h region-us.mydb

-- Write in US region
INSERT INTO users (id, name) VALUES (1, 'Alice');

-- Connect to Europe region
$ psql -h region-eu.mydb

-- Read is consistent (sees the write!)
SELECT * FROM users WHERE id = 1;
-- Returns: (1, 'Alice')

-- Distributed transaction across shards
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE user_id = 1;  -- shard 1
UPDATE accounts SET balance = balance + 100 WHERE user_id = 2;  -- shard 2
COMMIT;  -- Uses 2PC + consensus, guaranteed atomic

Learning milestones:

  1. After single-region consensus works → You’ve built a fault-tolerant database
  2. After cross-region replication → You understand latency vs consistency trade-offs
  3. After distributed transactions work → You understand how NewSQL databases achieve global consistency
  4. After running realistic workloads → You truly understand production distributed systems

Sources