← Back to all projects

LEARN NOSQL DATABASES DEEP DIVE

Learn NoSQL Databases: From Zero to Deep Understanding

Goal: Deeply understand how NoSQL databases work—from basic key-value storage to building distributed systems with replication, consensus, and eventual consistency. You’ll implement the core data structures and algorithms that power databases like Redis, MongoDB, Cassandra, and DynamoDB.

Why NoSQL Databases Matter

Every modern application relies on NoSQL databases. When you use:

  • Redis to cache your sessions
  • MongoDB to store user documents
  • Cassandra for time-series data
  • DynamoDB for serverless applications
  • Neo4j for social graphs

…you’re using sophisticated distributed systems built on decades of computer science research.

After completing these projects, you will:

  • Understand every component of a storage engine
  • Know how data is persisted, indexed, and retrieved efficiently
  • Be able to design and implement your own database systems
  • Understand the CAP theorem through building, not just reading
  • Troubleshoot any database performance issue at the deepest level

Core Concept Analysis

The NoSQL Landscape

                        NoSQL Database Types
                              |
    +------------+------------+------------+------------+
    |            |            |            |            |
Key-Value    Document    Column-Family    Graph      Time-Series
    |            |            |            |            |
  Redis      MongoDB     Cassandra     Neo4j      InfluxDB
  Riak       CouchDB     HBase         TigerGraph  TimescaleDB
  etcd       Firestore   ScyllaDB      ArangoDB    QuestDB

Fundamental Concepts You Must Understand

  1. Storage Engines: How data is organized on disk
    • B-Trees: Balanced trees for read-heavy workloads (used in traditional DBs)
    • LSM Trees: Log-Structured Merge Trees for write-heavy workloads (Cassandra, RocksDB, LevelDB)
    • Hash Indexes: O(1) lookups (Bitcask, Redis)
  2. Data Structures:
    • Bloom Filters: Probabilistic “definitely not” or “maybe” checks
    • Skip Lists: Ordered data with O(log n) operations
    • Merkle Trees: Efficient data integrity verification
    • SSTables: Sorted String Tables for persistent storage
  3. Durability & Recovery:
    • Write-Ahead Log (WAL): Log before commit
    • Checkpointing: Periodic snapshots
    • Compaction: Merging and garbage collection
  4. Distribution:
    • Consistent Hashing: Distribute data with minimal reshuffling
    • Replication: Master-slave, multi-master, quorum-based
    • Consensus: Raft, Paxos for agreement in distributed systems
    • Vector Clocks: Track causality in distributed systems
    • CRDTs: Conflict-free replicated data types
  5. Consistency Models:
    • Strong Consistency: All reads see the latest write
    • Eventual Consistency: Given enough time, all nodes converge
    • Causal Consistency: Respects cause-and-effect ordering
  6. The CAP Theorem: 
    You can only have 2 of 3:

Consistency --------- Availability
     \                   /
      \                 /
       \               /
        \             /
         \           /
      Partition Tolerance

CP: MongoDB, etcd, HBase
AP: Cassandra, CouchDB, DynamoDB
CA: Traditional RDBMS (no network partitions)

Project 1: In-Memory Key-Value Store (Redis Clone Basics)

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, C, Python
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: In-Memory Storage / Data Structures
  • Software or Tool: Redis
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A networked key-value store that supports GET, SET, DEL, EXPIRE commands over TCP, storing all data in memory with a hash map.

Why it teaches NoSQL: This is the simplest possible database—a hash map with a network interface. You’ll understand the core abstraction all databases share: take a key, return a value. But you’ll also face real challenges: handling concurrent connections, implementing TTL expiration, and building a wire protocol.

Core challenges you’ll face:

  • Designing the wire protocol → maps to understanding RESP (Redis Protocol)
  • Handling concurrent access → maps to thread safety and locking strategies
  • Implementing key expiration (TTL) → maps to passive vs active expiration
  • Memory management → maps to understanding when to evict data

Key Concepts:

  • Hash Tables: “Introduction to Algorithms” Chapter 11 - Cormen et al.
  • Network Programming: “The Linux Programming Interface” Chapter 56-61 - Michael Kerrisk
  • RESP Protocol: Redis Protocol Specification
  • Concurrent Data Structures: “The Art of Multiprocessor Programming” - Herlihy & Shavit

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic understanding of TCP sockets, hash tables, and concurrent programming concepts. Familiarity with Go or your chosen language.

Real world outcome:

# Terminal 1: Start your server
$ ./minidb serve --port 6379
MiniDB listening on :6379

# Terminal 2: Use redis-cli or telnet to connect
$ redis-cli -p 6379
127.0.0.1:6379> SET user:1 "Douglas"
OK
127.0.0.1:6379> GET user:1
"Douglas"
127.0.0.1:6379> SET session:abc "token123" EX 60
OK
127.0.0.1:6379> TTL session:abc
(integer) 58
127.0.0.1:6379> DEL user:1
(integer) 1
127.0.0.1:6379> GET user:1
(nil)

Implementation Hints:

The architecture is straightforward:

Client ──TCP──> Server ──Parse──> Command ──Execute──> HashMap
                   │                                      │
                   └──────────── Response ◄───────────────┘

RESP (Redis Serialization Protocol) is simple:

  • Simple Strings: +OK\r\n
  • Errors: -ERR unknown command\r\n
  • Integers: :1000\r\n
  • Bulk Strings: $5\r\nhello\r\n
  • Arrays: *2\r\n$3\r\nGET\r\n$4\r\nuser\r\n

For TTL expiration, you have two strategies:

  1. Lazy expiration: Check if key is expired when accessed
  2. Active expiration: Background goroutine periodically scans for expired keys

Questions to guide your implementation:

  • How do you handle multiple clients simultaneously? (goroutines/threads per connection vs event loop)
  • What happens if two clients SET the same key at once?
  • How do you store the expiration time? (absolute timestamp vs relative)
  • Should you use a mutex per key or a global mutex?

Learning milestones:

  1. Server accepts connections and echoes commands → You understand network I/O
  2. GET/SET work correctly with concurrent clients → You understand thread safety
  3. TTL expiration works → You understand time-based invalidation
  4. Your server passes basic redis-cli tests → You’ve built a working database!

Project 2: Persistent Key-Value Store with Write-Ahead Log

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, C
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Durability / Crash Recovery
  • Software or Tool: WAL-based databases (PostgreSQL, etcd)
  • Main Book: “Database Internals” by Alex Petrov

What you’ll build: Extend your key-value store with persistence. Every write is first appended to a log file, then applied to memory. On restart, replay the log to recover state.

Why it teaches NoSQL: Durability is fundamental. Every serious database guarantees your data survives crashes. The WAL pattern is used everywhere—from SQLite to Kafka. You’ll understand why “write to log first” is the universal solution to crash recovery.

Core challenges you’ll face:

  • Designing the log format → maps to binary serialization and framing
  • Ensuring durability with fsync → maps to understanding storage guarantees
  • Recovering from partial writes → maps to handling corruption gracefully
  • Log compaction → maps to snapshot + truncation strategies

Key Concepts:

  • Write-Ahead Logging: “Database Internals” Chapter 3 - Alex Petrov
  • fsync semantics: “The Linux Programming Interface” Section 13.3 - Michael Kerrisk
  • Binary file formats: Protocol Buffers encoding
  • Crash recovery: “Designing Data-Intensive Applications” Chapter 3 - Martin Kleppmann

Difficulty: Intermediate Time estimate: 1 week Prerequisites: Project 1 completed. Understanding of file I/O, binary encoding.

Real world outcome:

# Start server, add data
$ ./minidb serve
MiniDB listening on :6379
$ redis-cli SET important "critical data"
OK

# Kill the server ungracefully (Ctrl+C or kill -9)
^C

# Check the WAL file
$ xxd data/wal.log | head
00000000: 0100 0000 0900 0000 696d 706f 7274 616e  ........importan
00000010: 740d 0000 0063 7269 7469 6361 6c20 6461  t....critical da
00000020: 7461 a3f2 1c42                           ta...B

# Restart - data is recovered!
$ ./minidb serve
Recovering from WAL... 1 entries replayed
MiniDB listening on :6379
$ redis-cli GET important
"critical data"

Implementation Hints:

Your log entry format needs:

┌──────────┬───────────┬───────────┬─────────────┬──────────┐
│ Op Type  │ Key Len   │   Key     │  Value Len  │  Value   │
│ (1 byte) │ (4 bytes) │ (var)     │  (4 bytes)  │  (var)   │
├──────────┴───────────┴───────────┴─────────────┴──────────┤
│                        CRC32 Checksum (4 bytes)           │
└───────────────────────────────────────────────────────────┘

The CRC32 checksum detects corruption. On recovery:

  1. Open the log file
  2. Read entries one by one
  3. Verify each checksum—if invalid, stop (partial write during crash)
  4. Apply valid entries to the in-memory hash map

Log compaction is necessary because the log grows forever:

  • Periodically snapshot the entire hash map to a file
  • Truncate the WAL after successful snapshot
  • On recovery: load snapshot, then replay WAL entries after snapshot

Questions to guide your implementation:

  • What happens if the machine crashes AFTER writing to the log but BEFORE fsync completes?
  • How do you handle concurrent writes to the log file?
  • When should you take a snapshot? (size-based, time-based, or both?)

Learning milestones:

  1. Data survives server restart → You understand basic persistence
  2. Data survives kill -9 → You understand fsync and durability
  3. Corrupted partial writes are detected → You understand checksums
  4. Compaction keeps the log small → You understand log management

Project 3: Bloom Filter Implementation

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Python
  • Alternative Programming Languages: Go, Rust, C
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Probabilistic Data Structures
  • Software or Tool: Cassandra, HBase, RocksDB
  • Main Book: “Database Internals” by Alex Petrov

What you’ll build: A space-efficient probabilistic data structure that tells you “definitely not in set” or “possibly in set”. Integrate it with your key-value store to avoid disk reads for non-existent keys.

Why it teaches NoSQL: Bloom filters are everywhere in NoSQL databases. Cassandra uses them to skip SSTables that definitely don’t contain a key. LevelDB and RocksDB use them for the same purpose. Understanding bloom filters means understanding how databases achieve O(1) “not found” responses without reading from disk.

Core challenges you’ll face:

  • Choosing hash functions → maps to independence and uniformity requirements
  • Sizing the bit array → maps to false positive rate tradeoffs
  • Handling deletions → maps to understanding counting bloom filters

Key Concepts:

  • Bloom Filters: Bloom Filters by Example - Interactive tutorial
  • Hash Functions: MurmurHash3 documentation
  • Probabilistic Analysis: “Database Internals” Chapter 7 - Alex Petrov
  • Applications: “Designing Data-Intensive Applications” Chapter 3 - Kleppmann

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic probability, hash function concepts.

Real world outcome:

# Create a bloom filter
$ python
>>> from bloomfilter import BloomFilter
>>> bf = BloomFilter(capacity=1000000, error_rate=0.01)

# Add some keys
>>> for i in range(100000):
...     bf.add(f"user:{i}")

# Check membership
>>> "user:42" in bf
True
>>> "user:999999" in bf  # Not added
False  # Definitely not in set!
>>> "user:12345678" in bf  # Also not added
True  # False positive! (happens ~1% of the time)

# Check memory usage
>>> bf.size_in_bytes()
119816  # ~120KB for 100K items with 1% error rate

# Compare to a set
>>> import sys
>>> s = set(f"user:{i}" for i in range(100000))
>>> sys.getsizeof(s)
4194528  # ~4MB - 35x more memory!

Implementation Hints:

A bloom filter is just a bit array + k hash functions:

Insert "hello":
  h1("hello") = 3  → set bit 3
  h2("hello") = 7  → set bit 7
  h3("hello") = 1  → set bit 1

Bit array: [0,1,0,1,0,0,0,1,0,0]
            ↑   ↑         ↑
           h3  h1        h2

Query "hello":
  Check bits 3, 7, 1 → all set → "possibly in set"

Query "world":
  h1("world") = 2  → bit 2 is 0 → "definitely NOT in set"

Optimal parameters:

  • Given n items and desired false positive rate p:
  • m (bits) = -n × ln(p) / (ln(2))²
  • k (hash functions) = (m/n) × ln(2)

For multiple hash functions, use double hashing:

h_i(x) = h1(x) + i × h2(x)

This gives you k hash functions from just two.

Questions to guide your implementation:

  • Why can’t you delete from a standard bloom filter?
  • How would you implement a “counting bloom filter” that supports deletion?
  • What happens to the false positive rate as you add more items?

Learning milestones:

  1. Basic bloom filter works → You understand the core algorithm
  2. False positive rate matches theory → You understand the probability
  3. Memory usage is ~2% of a hash set → You understand space efficiency
  4. Integrated with your KV store to skip disk reads → You understand practical application

Project 4: LSM Tree Storage Engine

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Rust
  • Alternative Programming Languages: Go, C++, Java
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Storage Engines / Write Optimization
  • Software or Tool: LevelDB, RocksDB, Cassandra
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A complete LSM (Log-Structured Merge) tree storage engine with MemTable (in-memory), SSTables (on-disk), Write-Ahead Log, and background compaction.

Why it teaches NoSQL: The LSM tree is THE fundamental data structure behind write-optimized databases. LevelDB, RocksDB, Cassandra, HBase, InfluxDB, and dozens of others use LSM trees. Understanding this means understanding how modern databases achieve millions of writes per second.

Core challenges you’ll face:

  • Implementing the MemTable → maps to skip lists or red-black trees
  • SSTable format design → maps to sorted, immutable files with indexes
  • Bloom filter integration → maps to avoiding unnecessary disk reads
  • Compaction strategies → maps to leveled vs size-tiered compaction
  • Read amplification → maps to searching multiple levels

Resources for key challenges:

Key Concepts:

  • LSM Tree Architecture: “Designing Data-Intensive Applications” Chapter 3 - Kleppmann
  • Skip Lists: “A Skip List Cookbook” by William Pugh
  • SSTables: “Database Internals” Chapter 7 - Alex Petrov
  • Compaction: “Designing Data-Intensive Applications” Section 3.1.2 - Kleppmann

Difficulty: Advanced Time estimate: 2-4 weeks Prerequisites: Projects 1-3 completed. Strong understanding of file I/O, binary formats, concurrent programming.

Real world outcome:

$ ./lsm-db bench --writes 1000000 --reads 100000
LSM-Tree Storage Engine Benchmark
─────────────────────────────────
Write Performance:
  Total writes: 1,000,000
  Time: 4.2s
  Throughput: 238,095 writes/sec

Read Performance:
  Total reads: 100,000
  Cache hits: 67,234
  Disk reads: 32,766
  Avg latency: 0.12ms

Storage Statistics:
  MemTable size: 64MB
  Level 0 SSTables: 4 (256MB)
  Level 1 SSTables: 10 (2.5GB)
  Level 2 SSTables: 42 (10.5GB)
  Total disk usage: 13.25GB
  Bloom filter memory: 12MB

Compaction Stats:
  Background compactions: 23
  Bytes compacted: 8.7GB
  Write amplification: 6.5x

Implementation Hints:

The LSM tree architecture:

                      WRITES
                         │
                         ▼
                 ┌───────────────┐
                 │  Write-Ahead  │ ◄── Durability
                 │     Log       │
                 └───────┬───────┘
                         │
                         ▼
                 ┌───────────────┐
                 │   MemTable    │ ◄── Sorted in-memory (Skip List)
                 │   (64 MB)     │
                 └───────┬───────┘
                         │ (when full, flush to disk)
                         ▼
    ┌───────────────────────────────────────────────────┐
    │                    Level 0                         │
    │  [SST] [SST] [SST] [SST]  (may overlap)           │
    └───────────────────────────────────────────────────┘
                         │ (compaction)
                         ▼
    ┌───────────────────────────────────────────────────┐
    │                    Level 1                         │
    │  [SST] [SST] [SST] [SST] [SST] [SST] (no overlap) │
    └───────────────────────────────────────────────────┘
                         │
                         ▼
                       (more levels...)

SSTable format:

┌─────────────────────────────────────────────────────────┐
│                     Data Blocks                          │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐                    │
│  │ Block 0 │ │ Block 1 │ │ Block 2 │ ...                │
│  │ k1:v1   │ │ k5:v5   │ │ k9:v9   │                    │
│  │ k2:v2   │ │ k6:v6   │ │ k10:v10 │                    │
│  │ ...     │ │ ...     │ │ ...     │                    │
│  └─────────┘ └─────────┘ └─────────┘                    │
├─────────────────────────────────────────────────────────┤
│                     Index Block                          │
│  Block 0 → offset 0, first_key="k1"                     │
│  Block 1 → offset 4096, first_key="k5"                  │
│  Block 2 → offset 8192, first_key="k9"                  │
├─────────────────────────────────────────────────────────┤
│                    Bloom Filter                          │
├─────────────────────────────────────────────────────────┤
│                      Footer                              │
│  index_offset, bloom_offset, magic_number               │
└─────────────────────────────────────────────────────────┘

Questions to guide your implementation:

  • How do you handle reads when data might be in MemTable OR any SSTable level?
  • What triggers a compaction? (size threshold, file count, or both?)
  • How do you handle deletes? (tombstones)
  • What’s the write amplification of your compaction strategy?

Learning milestones:

  1. MemTable fills and flushes to SSTable → You understand the basic flow
  2. Reads check MemTable then SSTables → You understand read path
  3. Compaction merges SSTables → You understand write amplification
  4. Bloom filters skip unnecessary reads → You understand read optimization
  5. Benchmark shows 200K+ writes/sec → You’ve built a real storage engine!

Project 5: B-Tree Index Implementation

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Rust, Go, C++
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Indexing / Read Optimization
  • Software or Tool: SQLite, InnoDB, BoltDB
  • Main Book: “Database Internals” by Alex Petrov

What you’ll build: A disk-based B+ tree that stores key-value pairs with O(log n) reads and writes, supporting range queries efficiently.

Why it teaches NoSQL: While LSM trees optimize for writes, B-trees optimize for reads. Understanding both lets you make informed decisions about which storage engine to use. MongoDB uses B-trees for its WiredTiger storage engine. BoltDB and LMDB are pure B-tree stores. B-trees are also the foundation of most SQL database indexes.

Core challenges you’ll face:

  • Page-based storage → maps to aligning with disk block sizes
  • Node splitting and merging → maps to maintaining balance during writes
  • Handling variable-length keys → maps to overflow pages and prefixes
  • Concurrency control → maps to latch crabbing and lock-free reads

Key Concepts:

  • B+ Tree Fundamentals: “Database Internals” Chapters 2-4 - Alex Petrov
  • Page Layout: “Database Internals” Chapter 5 - Alex Petrov
  • B-Tree Visualization: B-Tree Visualization
  • Disk I/O Optimization: “Computer Systems: A Programmer’s Perspective” Chapter 6 - Bryant & O’Hallaron

Difficulty: Advanced Time estimate: 2-3 weeks Prerequisites: Strong C programming, understanding of pointers, memory management. Understanding of tree data structures.

Real world outcome:

$ ./btree-db demo
B+Tree Database Demo
───────────────────

Creating B+Tree with page size 4096, order 128...

Inserting 1,000,000 keys...
  Insert time: 12.3s
  Tree height: 4
  Total pages: 8,234
  Disk usage: 33.7 MB

Point queries (1000 random keys):
  Avg I/O per query: 4 pages
  Avg latency: 0.08ms

Range query: keys 500000 to 500100
  Keys found: 101
  Pages read: 6
  Time: 0.2ms

Tree structure:
  Level 0 (root): 1 page
  Level 1: 8 pages
  Level 2: 512 pages
  Level 3 (leaves): 7,713 pages

Implementation Hints:

B+ Tree structure:

                    ┌─────────────────────┐
                    │   [30 | 60 | 90]    │  ◄── Internal node
                    └─────────────────────┘
                     /    |      |     \
                    ▼     ▼      ▼      ▼
    ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
    │10|15|20| │ │35|40|50| │ │65|70|80| │ │92|95|99| │ ◄── Leaf nodes
    └──────────┘ └──────────┘ └──────────┘ └──────────┘
         │             │            │            │
         └─────────────┴────────────┴────────────┘
                   Linked for range scans

Page layout (4KB page):

┌──────────────────────────────────────────────────────────┐
│ Header (16 bytes)                                         │
│   page_type (1), num_keys (2), right_ptr (8), free (5)   │
├──────────────────────────────────────────────────────────┤
│ Cell pointers (2 bytes each, grows →)                    │
│   [offset1] [offset2] [offset3] ...                      │
├──────────────────────────────────────────────────────────┤
│                    Free Space                             │
├──────────────────────────────────────────────────────────┤
│ Cells (variable size, grows ←)                           │
│   ... [cell3] [cell2] [cell1]                            │
└──────────────────────────────────────────────────────────┘

For node splitting:

  1. When a node overflows (too many keys), split it in half
  2. Push the middle key up to the parent
  3. If the parent overflows, recursively split

For range queries:

  1. Find the starting key using tree traversal
  2. Follow the leaf-level linked list until end key

Questions to guide your implementation:

  • What’s the optimal page size? (usually matches OS page size: 4KB)
  • How do you handle keys larger than ~1/4 of a page?
  • How do you ensure atomicity during splits?
  • Why do leaf nodes need to be linked?

Learning milestones:

  1. Insert/lookup works for small datasets → You understand tree traversal
  2. Node splitting works correctly → You understand balance maintenance
  3. Range queries work via leaf links → You understand the B+ tree advantage
  4. Performance matches theory (log n I/O) → You’ve built a real index

Project 6: Consistent Hashing for Distributed Storage

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, Python, Java
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Distributed Systems / Partitioning
  • Software or Tool: DynamoDB, Cassandra, Riak
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A consistent hashing ring that distributes keys across multiple nodes with virtual nodes, supporting node addition/removal with minimal key movement.

Why it teaches NoSQL: Every distributed database needs to decide which node stores which data. Naive hashing (key % num_nodes) fails catastrophically when nodes are added or removed—all keys get reshuffled. Consistent hashing solves this elegantly. Amazon’s Dynamo paper made this technique famous.

Core challenges you’ll face:

  • Hash ring implementation → maps to circular space and successor finding
  • Virtual nodes → maps to even distribution despite varying node capacities
  • Replication placement → maps to storing copies on next N nodes
  • Node membership changes → maps to graceful data migration

Resources for key challenges:

Key Concepts:

  • Consistent Hashing: “Designing Data-Intensive Applications” Chapter 6 - Kleppmann
  • Virtual Nodes: Dynamo Paper (Amazon, 2007)
  • Ring Data Structure: Typically implemented with a sorted tree/map
  • Replication Strategies: “Designing Data-Intensive Applications” Chapter 5 - Kleppmann

Difficulty: Intermediate Time estimate: 1 week Prerequisites: Project 1 completed. Understanding of hash functions, basic distributed systems concepts.

Real world outcome:

$ ./hashring demo
Consistent Hash Ring Demo
─────────────────────────

Creating ring with 3 nodes, 150 virtual nodes each...

Ring visualization (simplified):
  ────────────────────────────────────────────
  0°        90°       180°      270°      360°
  │         │         │         │         │
  ▼         ▼         ▼         ▼         ▼
  ●─●─●─●───●─●─●─●───●─●─●─●───●─●─●─●───●
  A B C A   B C A B   C A B C   A B C A   B

  A = node-1 (50 virtual nodes shown)
  B = node-2 (50 virtual nodes shown)
  C = node-3 (50 virtual nodes shown)

Key distribution (10000 keys):
  node-1: 3,312 keys (33.1%)
  node-2: 3,401 keys (34.0%)
  node-3: 3,287 keys (32.9%)
  Std deviation: 0.6% ✓

Adding node-4...
  Keys moved: 2,498 (24.98%)  ← Only ~25% moved!
  node-1: 2,489 keys (24.9%)
  node-2: 2,534 keys (25.3%)
  node-3: 2,479 keys (24.8%)
  node-4: 2,498 keys (25.0%)

Removing node-2...
  Keys moved: 2,534 (redistributed to remaining nodes)
  node-1: 3,345 keys (33.5%)
  node-3: 3,321 keys (33.2%)
  node-4: 3,334 keys (33.3%)

Implementation Hints:

The hash ring concept:

                         0
                         │
                    ┌────┴────┐
               270° │         │ 90°
                    │  Ring   │
                    │         │
                    └────┬────┘
                         │
                        180°

Keys and nodes are hashed to positions on the ring.
A key is stored on the first node clockwise from its position.

Virtual nodes solve the load balancing problem:

Without virtual nodes (uneven):      With virtual nodes (even):
Node A: ●                            Node A: ● ● ● ● ● (5 positions)
Node B: ●                            Node B: ● ● ● ● ● (5 positions)
(A gets 80% of keys!)                (Both get ~50% of keys)

Data structure (typically a balanced BST or sorted map):

type Ring struct {
    // Sorted by position
    nodes []VirtualNode  // {hash, realNode}

    // For fast lookups
    tree  *rbtree        // Red-black tree keyed by hash
}

func (r *Ring) GetNode(key string) string {
    hash := xxhash(key)
    // Find first node with hash >= key hash (with wraparound)
    return r.tree.Ceiling(hash).realNode
}

Questions to guide your implementation:

  • Why use a cryptographic hash vs a fast hash like xxHash?
  • How many virtual nodes per physical node? (typically 100-200)
  • How do you handle replication? (N successive distinct physical nodes)
  • What happens to data when a node joins/leaves?

Learning milestones:

  1. Basic ring routes keys to correct nodes → You understand the algorithm
  2. Virtual nodes give even distribution → You understand load balancing
  3. Adding/removing nodes moves ~1/N keys → You understand the efficiency
  4. Replication places copies on N nodes → You understand fault tolerance

Project 7: Simple Document Store (Mini-MongoDB)

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Python, Rust, JavaScript/Node.js
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Document Databases / Schema Flexibility
  • Software or Tool: MongoDB, CouchDB, Firestore
  • Main Book: “NoSQL Distilled” by Pramod Sadalage & Martin Fowler

What you’ll build: A document database that stores JSON documents, supports CRUD operations, allows queries on nested fields, and builds indexes for fast lookups.

Why it teaches NoSQL: Document databases are the most popular NoSQL type. Unlike key-value stores, they understand the structure of your data. You can query on any field, not just the key. Building one teaches you about schema flexibility, indexing strategies, and the query planning problem.

Core challenges you’ll face:

  • JSON/BSON storage → maps to efficient binary encoding
  • Query parsing and execution → maps to building a query engine
  • Index management → maps to maintaining secondary indexes
  • Update operators → maps to partial document modifications

Key Concepts:

  • Document Model: “NoSQL Distilled” Chapter 2 - Sadalage & Fowler
  • BSON Encoding: BSON Specification
  • Query Optimization: “Database Internals” Chapter 13 - Alex Petrov
  • Indexing Strategies: “MongoDB: The Definitive Guide” - O’Reilly

Difficulty: Advanced Time estimate: 2-3 weeks Prerequisites: Projects 1-4 completed. JSON parsing experience.

Real world outcome:

$ ./docdb shell
DocDB Shell v1.0
Connected to localhost:27017

> db.users.insertOne({
    name: "Douglas",
    email: "douglas@example.com",
    age: 32,
    address: {
        city: "San Francisco",
        country: "USA"
    },
    tags: ["developer", "musician"]
  })
{ acknowledged: true, insertedId: "65a1b2c3d4e5f6" }

> db.users.find({ "address.city": "San Francisco" })
[
  {
    _id: "65a1b2c3d4e5f6",
    name: "Douglas",
    email: "douglas@example.com",
    ...
  }
]

> db.users.createIndex({ email: 1 })
{ created: "email_1" }

> db.users.find({ email: "douglas@example.com" }).explain()
{
  queryPlanner: {
    indexUsed: "email_1",
    keysExamined: 1,
    docsExamined: 1
  }
}

> db.users.updateOne(
    { email: "douglas@example.com" },
    { $set: { age: 33 }, $push: { tags: "writer" } }
  )
{ matched: 1, modified: 1 }

Implementation Hints:

Document storage architecture:

┌──────────────────────────────────────────────────────────┐
│                     Collection: users                     │
├──────────────────────────────────────────────────────────┤
│ Primary Storage (LSM or B-Tree):                         │
│   _id → full document (BSON encoded)                     │
├──────────────────────────────────────────────────────────┤
│ Secondary Indexes:                                        │
│   email_1:  email_value → [_id1, _id2, ...]              │
│   age_1:    age_value → [_id1, _id2, ...]                │
└──────────────────────────────────────────────────────────┘

Query processing pipeline:

Query: { "address.city": "SF", age: { $gt: 25 } }
                    │
                    ▼
            ┌───────────────┐
            │  Parse Query  │
            └───────┬───────┘
                    │
                    ▼
            ┌───────────────┐
            │ Query Planner │ → Choose best index (if any)
            └───────┬───────┘
                    │
                    ▼
            ┌───────────────┐
            │  Index Scan   │ or Collection Scan
            └───────┬───────┘
                    │
                    ▼
            ┌───────────────┐
            │ Filter & Match│ → Apply remaining predicates
            └───────┬───────┘
                    │
                    ▼
                [Results]

Nested field access (“address.city”):

  1. Parse the dot notation into path segments: [“address”, “city”]
  2. Traverse the document tree following the path
  3. Handle arrays (MongoDB allows matching inside arrays)

Questions to guide your implementation:

  • How do you efficiently find documents when there’s no index? (full scan)
  • How do you maintain indexes when documents are updated?
  • What happens when you query on a field that’s sometimes missing?
  • How do you handle array fields in indexes? (multi-key indexes)

Learning milestones:

  1. CRUD operations work → You understand document storage
  2. Queries on nested fields work → You understand path traversal
  3. Indexes speed up queries → You understand secondary indexing
  4. Update operators ($set, $push) work → You understand partial updates

Project 8: Replication with Leader Election (Raft Consensus)

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, C++, Java
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 4: Expert
  • Knowledge Area: Distributed Consensus / Fault Tolerance
  • Software or Tool: etcd, CockroachDB, TiKV
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A replicated key-value store using the Raft consensus algorithm. Multiple nodes elect a leader, replicate writes, and survive node failures while maintaining consistency.

Why it teaches NoSQL: Distributed consensus is the holy grail of distributed systems. How do multiple machines agree on the same order of operations? How do you survive failures without losing data? Raft (used by etcd, TiKV, CockroachDB) is the most understandable solution. After building this, you’ll truly understand how distributed databases achieve fault tolerance.

Core challenges you’ll face:

  • Leader election → maps to terms, votes, and timeouts
  • Log replication → maps to AppendEntries RPC and commit index
  • Safety guarantees → maps to election restriction and log matching
  • Handling network partitions → maps to split-brain prevention

Resources for key challenges:

Key Concepts:

  • Raft Algorithm: “Designing Data-Intensive Applications” Chapter 9 - Kleppmann
  • Consensus: “In Search of an Understandable Consensus Algorithm” - Ongaro & Ousterhout
  • Distributed Logs: “I Heart Logs” by Jay Kreps
  • Failure Modes: “Distributed Systems” by Maarten van Steen

Difficulty: Expert Time estimate: 3-4 weeks Prerequisites: Projects 1-2 completed. Strong understanding of concurrent programming, networking. Experience with distributed systems concepts.

Real world outcome:

# Start a 3-node Raft cluster
$ ./raft-kv start --nodes=3
Node 1 starting on :8001...
Node 2 starting on :8002...
Node 3 starting on :8003...

[Node 2] Timeout, starting election for term 1
[Node 2] Received votes from [1, 3], became leader for term 1
[Node 2] Leader established, ready to accept writes

# Write through any node (forwarded to leader)
$ curl -X PUT localhost:8001/set -d 'key=name&value=Douglas'
{"status": "ok", "leader": "node2", "term": 1, "index": 1}

# Read from any node
$ curl localhost:8003/get?key=name
{"value": "Douglas", "committed": true}

# Kill the leader
$ kill -9 $(pgrep -f "raft-kv.*8002")

# Watch election happen
[Node 1] Lost heartbeat from leader
[Node 3] Lost heartbeat from leader
[Node 1] Timeout, starting election for term 2
[Node 1] Received vote from node 3, became leader for term 2
[Node 1] Leader established

# Data still available!
$ curl localhost:8003/get?key=name
{"value": "Douglas", "committed": true}

Implementation Hints:

Raft’s core state machine:

                    ┌─────────────────┐
                    │    Follower     │
                    └────────┬────────┘
                             │ timeout
                             ▼
                    ┌─────────────────┐
             ┌──────│   Candidate     │◄──────┐
             │      └────────┬────────┘       │
             │               │                 │
      loses  │               │ wins majority   │ timeout
      or     │               ▼                 │ (split vote)
      higher │      ┌─────────────────┐       │
      term   └─────►│     Leader      │───────┘
                    └─────────────────┘
                             │
                             │ discovers higher term
                             ▼
                    ┌─────────────────┐
                    │    Follower     │
                    └─────────────────┘

Key RPCs:

  1. RequestVote: Candidate asks for votes 
    Arguments: term, candidateId, lastLogIndex, lastLogTerm
Response: term, voteGranted
  2. AppendEntries: Leader replicates log entries (also heartbeat) 
    Arguments: term, leaderId, prevLogIndex, prevLogTerm, entries[], leaderCommit
Response: term, success

Safety invariants to maintain:

  • Only one leader per term
  • Leaders never overwrite their logs
  • If two logs contain an entry with same index and term, all prior entries are identical
  • If an entry is committed, it will be in the log of all future leaders

Questions to guide your implementation:

  • How do you prevent split-brain when the network partitions?
  • What happens if a follower misses some log entries?
  • How do you know when an entry is safe to apply to the state machine?
  • What’s the difference between committed and applied?

Learning milestones:

  1. Leader election works → You understand timeouts and voting
  2. Log replication works → You understand the consensus mechanism
  3. Cluster survives leader failure → You understand fault tolerance
  4. Partitioned minority can’t elect leader → You understand safety
  5. Partitioned minority rejoins cleanly → You understand log reconciliation

Project 9: Eventually Consistent Replication (Dynamo-Style)

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, Java, Python
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 4: Expert
  • Knowledge Area: Eventual Consistency / AP Systems
  • Software or Tool: DynamoDB, Cassandra, Riak
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A multi-node key-value store with tunable consistency (W and R values), vector clocks for conflict detection, and read repair for anti-entropy.

Why it teaches NoSQL: Not all applications need strong consistency. Amazon’s Dynamo (the inspiration for DynamoDB, Cassandra, and Riak) showed that sacrificing consistency for availability can enable massive scale. Understanding eventual consistency means understanding the CAP theorem through practice.

Core challenges you’ll face:

  • Vector clocks → maps to tracking causality without a global clock
  • Quorum reads/writes → maps to tunable consistency levels
  • Conflict resolution → maps to last-write-wins vs application merge
  • Anti-entropy → maps to Merkle trees and read repair

Key Concepts:

  • Dynamo Paper: “Dynamo: Amazon’s Highly Available Key-value Store” (2007)
  • Vector Clocks: “Designing Data-Intensive Applications” Chapter 5 - Kleppmann
  • Quorums: “Designing Data-Intensive Applications” Chapter 5 - Kleppmann
  • Merkle Trees: “Database Internals” Chapter 12 - Alex Petrov

Difficulty: Expert Time estimate: 3-4 weeks Prerequisites: Projects 1-2, 6 completed. Understanding of consistent hashing, distributed systems concepts.

Real world outcome:

# Start a 5-node Dynamo-style cluster
$ ./dynamo-kv start --nodes=5 --replicas=3
Starting Dynamo-style cluster...
  Node 1 (tokens: 0-204): localhost:9001
  Node 2 (tokens: 205-409): localhost:9002
  Node 3 (tokens: 410-614): localhost:9003
  Node 4 (tokens: 615-819): localhost:9004
  Node 5 (tokens: 820-1023): localhost:9005

# Write with quorum (W=2 of 3 replicas)
$ curl -X PUT localhost:9001/put -d 'key=cart:user123&value=["item1"]&w=2'
{
  "status": "ok",
  "version": {"node1": 1},
  "replicas_written": ["node2", "node3", "node4"]
}

# Concurrent write from different client (creates conflict)
$ curl -X PUT localhost:9005/put -d 'key=cart:user123&value=["item2"]&w=2'
{
  "status": "ok",
  "version": {"node5": 1},
  "replicas_written": ["node2", "node3", "node4"]
}

# Read with quorum (R=2) - detects conflict!
$ curl localhost:9001/get?key=cart:user123&r=2
{
  "conflict": true,
  "versions": [
    {"value": ["item1"], "version": {"node1": 1}},
    {"value": ["item2"], "version": {"node5": 1}}
  ],
  "message": "Client must resolve conflict"
}

# Client resolves by merging
$ curl -X PUT localhost:9001/put -d \
  'key=cart:user123&value=["item1","item2"]&context={"node1":1,"node5":1}'
{
  "status": "ok",
  "version": {"node1": 1, "node5": 1, "node1": 2}
}

Implementation Hints:

Dynamo-style architecture:

          Client
             │
             ▼
    ┌─────────────────┐
    │  Coordinator    │ (any node can coordinate)
    └────────┬────────┘
             │
   ┌─────────┼─────────┐
   │         │         │
   ▼         ▼         ▼
┌─────┐   ┌─────┐   ┌─────┐
│Node1│   │Node2│   │Node3│  (N=3 replicas)
└─────┘   └─────┘   └─────┘

Write: wait for W responses
Read: wait for R responses
If W + R > N, you get consistency

Vector clocks track causality:

Initial: {A:0, B:0, C:0}

Node A writes: {A:1, B:0, C:0}
Node B writes: {A:0, B:1, C:0}

These are CONCURRENT (neither happened-before the other)

Node C reads both, merges: {A:1, B:1, C:1}
Now this version supersedes both

Conflict detection:

def is_ancestor(v1, v2):
    """v1 is ancestor of v2 if all v1's components <= v2's"""
    return all(v1[node] <= v2.get(node, 0) for node in v1)

def compare_versions(v1, v2):
    if v1 == v2:
        return "EQUAL"
    elif is_ancestor(v1, v2):
        return "V1_BEFORE_V2"
    elif is_ancestor(v2, v1):
        return "V2_BEFORE_V1"
    else:
        return "CONCURRENT"  # Conflict!

Questions to guide your implementation:

  • What happens if W=1? (Fast but risky)
  • What if R=1 and W=N? (Fast reads, slow writes)
  • How do vector clocks grow over time? (Need pruning)
  • How does read-repair work? (Return latest, async fix stale replicas)

Learning milestones:

  1. Basic replication works → You understand distributed writes
  2. Vector clocks detect conflicts → You understand causality
  3. Quorum reads return latest → You understand tunable consistency
  4. Read repair fixes stale replicas → You understand anti-entropy
  5. System continues during partitions → You understand AP tradeoffs

Project 10: Time-Series Database

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, C++
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Time-Series Storage / Compression
  • Software or Tool: InfluxDB, TimescaleDB, Prometheus
  • Main Book: “Database Internals” by Alex Petrov

What you’ll build: A specialized database for time-series data with efficient timestamp-based storage, columnar compression, downsampling, and time-range queries.

Why it teaches NoSQL: Time-series databases are a specialized NoSQL category that’s exploding in popularity due to IoT, monitoring, and financial data. They use unique techniques: gorilla compression, columnar storage, and automatic data lifecycle management. Building one teaches you about domain-specific database optimization.

Core challenges you’ll face:

  • Time-based partitioning → maps to chunking data by time windows
  • Compression → maps to delta-of-delta and XOR for floats
  • Downsampling → maps to aggregating old data to save space
  • Efficient range queries → maps to time-indexed storage

Key Concepts:

  • Gorilla Compression: “Gorilla: A Fast, Scalable, In-Memory Time Series Database” - Facebook Paper
  • Time-Series Fundamentals: InfluxDB Documentation
  • Columnar Storage: “Designing Data-Intensive Applications” Chapter 3 - Kleppmann
  • Data Lifecycle: Retention policies and continuous queries

Difficulty: Advanced Time estimate: 2-3 weeks Prerequisites: Projects 1-4 completed. Understanding of compression concepts.

Real world outcome:

$ ./tsdb start
TimeSeries DB v1.0 listening on :8086

# Insert data points
$ curl -X POST localhost:8086/write -d '
measurement=cpu,host=server1,region=us-east value=72.5 1703116800000000000
measurement=cpu,host=server1,region=us-east value=73.2 1703116810000000000
measurement=cpu,host=server1,region=us-east value=71.8 1703116820000000000
measurement=cpu,host=server2,region=us-west value=45.3 1703116800000000000
'
{"status": "ok", "points_written": 4}

# Query with time range and aggregation
$ curl 'localhost:8086/query' -d '
SELECT mean(value), max(value), min(value)
FROM cpu
WHERE time >= 2023-12-20 AND time < 2023-12-21
GROUP BY host, time(1h)
'
{
  "results": [
    {"host": "server1", "time": "2023-12-20T00:00:00Z",
     "mean": 72.5, "max": 73.2, "min": 71.8},
    {"host": "server2", "time": "2023-12-20T00:00:00Z",
     "mean": 45.3, "max": 45.3, "min": 45.3}
  ]
}

# Check storage efficiency
$ ./tsdb stats
Storage Statistics:
  Total points: 1,000,000
  Raw size: 24 MB
  Compressed size: 1.2 MB (20x compression!)
  Chunks: 168 (1 hour each)
  Retention: 7 days

Implementation Hints:

Time-series storage architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Time-Based Shards                         │
├─────────────────────────────────────────────────────────────┤
│  2023-12-20   │  2023-12-21   │  2023-12-22   │   ...       │
│  ┌─────────┐  │  ┌─────────┐  │  ┌─────────┐  │             │
│  │ Chunk 0 │  │  │ Chunk 0 │  │  │ Chunk 0 │  │             │
│  │ (1 hour)│  │  │ (1 hour)│  │  │ (1 hour)│  │             │
│  ├─────────┤  │  ├─────────┤  │  ├─────────┤  │             │
│  │ Chunk 1 │  │  │ Chunk 1 │  │  │ Chunk 1 │  │             │
│  ├─────────┤  │  ├─────────┤  │  ├─────────┤  │             │
│  │   ...   │  │  │   ...   │  │  │   ...   │  │             │
│  └─────────┘  │  └─────────┘  │  └─────────┘  │             │
└─────────────────────────────────────────────────────────────┘

Gorilla compression for timestamps:

Instead of storing: 1609459200, 1609459210, 1609459220, 1609459230
Store:
  - First timestamp: 1609459200 (full, 64 bits)
  - Delta: 10 (needs ~4 bits)
  - Delta-of-delta: 0, 0, 0 (needs 1 bit each!)

Typical time-series have regular intervals → delta-of-delta is usually 0

Gorilla compression for floats:

Consecutive values are often similar.
XOR them together → many leading/trailing zeros → compress!

72.5 XOR 72.7 = mostly zeros with a few bits different
Store: number of leading zeros, number of meaningful bits, meaningful bits

Questions to guide your implementation:

  • How do you handle out-of-order data? (common in distributed systems)
  • What’s the optimal chunk size for your workload?
  • How do you implement downsampling (e.g., keep 1-minute averages after 7 days)?
  • How do you handle high-cardinality tags? (the #1 TSDB performance killer)

Learning milestones:

  1. Data ingestion and retrieval works → You understand the data model
  2. Time-range queries are fast → You understand time-indexed storage
  3. Compression achieves 10x+ → You understand domain-specific compression
  4. Aggregation queries work → You understand analytical processing

Project 11: Graph Database

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Python
  • Alternative Programming Languages: Go, Rust, Java
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Graph Storage / Traversal Algorithms
  • Software or Tool: Neo4j, ArangoDB, TigerGraph
  • Main Book: “Graph Databases” by Ian Robinson et al. (O’Reilly)

What you’ll build: A property graph database that stores nodes and edges with properties, supports Cypher-like queries, and efficiently traverses relationships.

Why it teaches NoSQL: Graph databases represent relationships as first-class citizens, unlike relational databases where joins are expensive. They’re essential for social networks, recommendation engines, and fraud detection. Building one teaches you about adjacency lists, graph traversal algorithms, and query optimization for connected data.

Core challenges you’ll face:

  • Graph storage format → maps to adjacency lists vs matrices
  • Traversal algorithms → maps to BFS, DFS, shortest path
  • Query language → maps to parsing and executing graph patterns
  • Index-free adjacency → maps to O(1) edge traversal

Key Concepts:

  • Property Graph Model: “Graph Databases” Chapter 2 - Robinson et al.
  • Cypher Query Language: Neo4j Cypher Manual
  • Graph Algorithms: “Graph Algorithms” by Mark Needham & Amy Hodler
  • Storage: “Seven Databases in Seven Weeks” - Neo4j chapter

Difficulty: Advanced Time estimate: 2-3 weeks Prerequisites: Understanding of graph theory, tree/graph traversal algorithms.

Real world outcome:

$ ./graphdb shell
GraphDB Shell v1.0

# Create nodes
> CREATE (alice:Person {name: "Alice", age: 30})
Node created: id=1

> CREATE (bob:Person {name: "Bob", age: 25})
Node created: id=2

> CREATE (neo4j:Database {name: "Neo4j", type: "graph"})
Node created: id=3

# Create relationships
> CREATE (alice)-[:KNOWS {since: 2020}]->(bob)
Relationship created: (1)-[KNOWS]->(2)

> CREATE (alice)-[:USES]->(neo4j)
Relationship created: (1)-[USES]->(3)

> CREATE (bob)-[:USES]->(neo4j)
Relationship created: (2)-[USES]->(3)

# Query: Find Alice's friends who use the same database
> MATCH (alice:Person {name: "Alice"})-[:KNOWS]->(friend)-[:USES]->(db)
  WHERE (alice)-[:USES]->(db)
  RETURN friend.name, db.name

+-------------+---------+
| friend.name | db.name |
+-------------+---------+
| Bob         | Neo4j   |
+-------------+---------+
1 row, 4 nodes traversed, 0.3ms

# Shortest path
> MATCH path = shortestPath((alice:Person {name:"Alice"})-[*]-(neo4j:Database))
  RETURN path

Path: (Alice)-[:USES]->(Neo4j)
Length: 1

Implementation Hints:

Property graph storage:

Nodes Table:
┌────┬────────┬────────────────────────────┐
│ ID │ Labels │ Properties (JSON/Map)       │
├────┼────────┼────────────────────────────┤
│ 1  │ Person │ {name:"Alice", age:30}     │
│ 2  │ Person │ {name:"Bob", age:25}       │
│ 3  │ Database│ {name:"Neo4j", type:"graph"}│
└────┴────────┴────────────────────────────┘

Edges Table:
┌────┬──────┬────┬───────┬───────────────────┐
│ ID │ From │ To │ Type  │ Properties        │
├────┼──────┼────┼───────┼───────────────────┤
│ 1  │ 1    │ 2  │ KNOWS │ {since: 2020}     │
│ 2  │ 1    │ 3  │ USES  │ {}                │
│ 3  │ 2    │ 3  │ USES  │ {}                │
└────┴──────┴────┴───────┴───────────────────┘

Adjacency Lists (for fast traversal):
┌────────┬────────────────────────────────┐
│ NodeID │ Outgoing Edges                 │
├────────┼────────────────────────────────┤
│ 1      │ [(KNOWS, 2), (USES, 3)]        │
│ 2      │ [(USES, 3)]                    │
│ 3      │ []                             │
└────────┴────────────────────────────────┘

Query execution (for pattern matching):

Query: MATCH (a:Person)-[:KNOWS]->(b)-[:USES]->(c)

Execution plan:
1. Scan nodes with label "Person" → candidates for 'a'
2. For each 'a', traverse KNOWS edges → candidates for 'b'
3. For each 'b', traverse USES edges → candidates for 'c'
4. Return matches

This is basically nested loop joins on the graph structure.

Questions to guide your implementation:

  • How do you store labels and types efficiently?
  • How do you make traversal O(1) per edge? (index-free adjacency)
  • How do you handle bidirectional edges?
  • What indexes speed up common queries? (node properties, edge types)

Learning milestones:

  1. CRUD for nodes and edges works → You understand the graph model
  2. Simple traversals work → You understand adjacency lists
  3. Pattern matching queries work → You understand query execution
  4. Shortest path works → You understand graph algorithms

Project 12: Full-Text Search Engine

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, Python, Java
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Information Retrieval / Indexing
  • Software or Tool: Elasticsearch, Apache Lucene, Meilisearch
  • Main Book: “Introduction to Information Retrieval” by Manning et al.

What you’ll build: A search engine with inverted indexes, tokenization, TF-IDF ranking, and boolean queries that can search through documents efficiently.

Why it teaches NoSQL: Full-text search engines like Elasticsearch are a specialized NoSQL category. They invert the relationship: instead of “document → words”, they store “word → documents”. Understanding inverted indexes is crucial for search, autocomplete, and log analytics.

Core challenges you’ll face:

  • Tokenization → maps to breaking text into searchable terms
  • Inverted index → maps to word → document ID mapping
  • Relevance ranking → maps to TF-IDF and BM25
  • Boolean queries → maps to AND, OR, NOT operations

Key Concepts:

  • Inverted Indexes: “Introduction to Information Retrieval” Chapter 1-2 - Manning et al.
  • TF-IDF: “Introduction to Information Retrieval” Chapter 6 - Manning et al.
  • Tokenization: Elasticsearch Analysis
  • Query DSL: Elasticsearch Query DSL documentation

Difficulty: Advanced Time estimate: 2-3 weeks Prerequisites: Projects 1-4 completed. Understanding of text processing.

Real world outcome:

$ ./searchdb start
SearchDB v1.0 listening on :9200

# Index some documents
$ curl -X POST localhost:9200/articles/_doc -d '{
  "title": "Introduction to NoSQL Databases",
  "content": "NoSQL databases provide flexibility and scalability...",
  "author": "Douglas"
}'
{"_id": "doc1", "result": "created"}

$ curl -X POST localhost:9200/articles/_doc -d '{
  "title": "SQL vs NoSQL: A Comparison",
  "content": "When choosing between SQL and NoSQL databases...",
  "author": "Alice"
}'
{"_id": "doc2", "result": "created"}

# Search with ranking
$ curl 'localhost:9200/articles/_search' -d '{
  "query": "NoSQL scalability"
}'
{
  "hits": {
    "total": 2,
    "hits": [
      {"_id": "doc1", "_score": 2.45, "title": "Introduction to NoSQL..."},
      {"_id": "doc2", "_score": 1.23, "title": "SQL vs NoSQL..."}
    ]
  }
}

# Boolean query
$ curl 'localhost:9200/articles/_search' -d '{
  "query": {
    "bool": {
      "must": ["NoSQL"],
      "must_not": ["SQL"]
    }
  }
}'
{
  "hits": {"total": 1, "hits": [{"_id": "doc1", ...}]}
}

# View the inverted index
$ ./searchdb explain-index articles
Inverted Index for 'articles':
  "nosql"      → [doc1: pos[0,5], doc2: pos[3]]
  "databases"  → [doc1: pos[1], doc2: pos[7]]
  "scalability"→ [doc1: pos[4]]
  "sql"        → [doc2: pos[0,4]]
  ...

Implementation Hints:

The inverted index structure:

Documents:
  doc1: "The quick brown fox"
  doc2: "The lazy brown dog"
  doc3: "Quick foxes are quick"

Inverted Index:
┌───────────┬────────────────────────────────────────────┐
│   Term    │   Posting List                             │
├───────────┼────────────────────────────────────────────┤
│ "the"     │ [(doc1, freq:1, pos:[0]), (doc2, freq:1)]  │
│ "quick"   │ [(doc1, freq:1, pos:[1]), (doc3, freq:2)]  │
│ "brown"   │ [(doc1, freq:1, pos:[2]), (doc2, freq:1)]  │
│ "fox"     │ [(doc1, freq:1, pos:[3])]                  │
│ "lazy"    │ [(doc2, freq:1, pos:[1])]                  │
│ "dog"     │ [(doc2, freq:1, pos:[3])]                  │
│ "foxes"   │ [(doc3, freq:1, pos:[1])]                  │
│ "are"     │ [(doc3, freq:1, pos:[2])]                  │
└───────────┴────────────────────────────────────────────┘

TF-IDF scoring:

TF (Term Frequency) = count of term in document / total terms in document
IDF (Inverse Document Frequency) = log(total docs / docs containing term)
Score = TF × IDF

"nosql" in doc1:
  TF = 2/100 = 0.02
  IDF = log(1000/50) = 1.3
  Score = 0.026

Boolean query execution:

Query: "NoSQL AND scalability"

1. Get posting list for "nosql": [doc1, doc2, doc5, doc7]
2. Get posting list for "scalability": [doc1, doc3, doc7, doc9]
3. Intersect lists: [doc1, doc7]
4. Return documents

Questions to guide your implementation:

  • How do you handle stemming (running → run)?
  • How do you handle stop words (the, a, an)?
  • How do you merge posting lists efficiently? (merge sort)
  • How do you persist the index to disk?

Learning milestones:

  1. Basic indexing and search works → You understand inverted indexes
  2. TF-IDF ranking returns relevant results → You understand scoring
  3. Boolean queries work → You understand query execution
  4. Performance scales to 1M+ documents → You understand real-world search

Project 13: Write-Optimized Column Store

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: C++
  • Alternative Programming Languages: Rust, Go
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 4: Expert
  • Knowledge Area: Analytical Processing / Columnar Storage
  • Software or Tool: Apache Cassandra, HBase, ClickHouse
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A column-family store like Cassandra, optimized for wide rows with many columns, efficient range scans, and write-heavy workloads.

Why it teaches NoSQL: Column-family stores are designed for massive scale. Cassandra handles petabytes at companies like Netflix and Apple. They combine LSM trees with columnar organization for both write performance and analytical queries. This teaches you about partition keys, clustering columns, and wide-row design.

Core challenges you’ll face:

  • Column-family data model → maps to partition keys and clustering
  • Wide rows → maps to storing millions of columns efficiently
  • Tombstones → maps to handling deletes in LSM trees
  • Compaction → maps to size-tiered vs leveled strategies

Key Concepts:

  • Column-Family Model: “Cassandra: The Definitive Guide” Chapter 4
  • Partition and Clustering: “Designing Data-Intensive Applications” Chapter 6 - Kleppmann
  • Cassandra Architecture: DataStax documentation
  • Compaction Strategies: “Database Internals” Chapter 7 - Alex Petrov

Difficulty: Expert Time estimate: 3-4 weeks Prerequisites: Projects 1-4 completed. LSM tree understanding essential.

Real world outcome:

$ ./columndb shell
ColumnDB Shell v1.0

# Create a table with partition key and clustering columns
> CREATE TABLE events (
    user_id text,
    event_time timestamp,
    event_type text,
    data text,
    PRIMARY KEY ((user_id), event_time, event_type)
  ) WITH CLUSTERING ORDER BY (event_time DESC);
Table created

# Insert events (wide row per user)
> INSERT INTO events (user_id, event_time, event_type, data)
  VALUES ('user1', '2023-12-20 10:00:00', 'click', '{"page": "/home"}');
> INSERT INTO events (user_id, event_time, event_type, data)
  VALUES ('user1', '2023-12-20 10:05:00', 'purchase', '{"item": "book"}');
> INSERT INTO events (user_id, event_time, event_type, data)
  VALUES ('user1', '2023-12-20 10:10:00', 'click', '{"page": "/thanks"}');

# Efficient range query on clustering columns
> SELECT * FROM events
  WHERE user_id = 'user1'
  AND event_time >= '2023-12-20 10:00:00'
  AND event_time < '2023-12-20 10:10:00';

 user_id | event_time          | event_type | data
---------+---------------------+------------+------------------
 user1   | 2023-12-20 10:05:00 | purchase   | {"item": "book"}
 user1   | 2023-12-20 10:00:00 | click      | {"page": "/home"}

# Show storage statistics
> DESCRIBE TABLE events;
Partition key: user_id
Clustering columns: event_time DESC, event_type
Partitions: 1,000
Average columns per partition: 50,000
Total size: 2.3 GB

Implementation Hints:

Column-family storage layout:

Logical View (CQL):
┌─────────┬─────────────────────┬────────────┬──────────────────┐
│ user_id │ event_time          │ event_type │ data             │
├─────────┼─────────────────────┼────────────┼──────────────────┤
│ user1   │ 2023-12-20 10:00:00 │ click      │ {"page":"/home"} │
│ user1   │ 2023-12-20 10:05:00 │ purchase   │ {"item":"book"}  │
│ user2   │ 2023-12-20 09:00:00 │ login      │ {}               │
└─────────┴─────────────────────┴────────────┴──────────────────┘

Physical Storage (sorted by partition key, then clustering columns):
┌────────────────────────────────────────────────────────────────┐
│ Partition: user1                                                │
│   Row Key: (2023-12-20 10:05:00, purchase) → {"item":"book"}   │
│   Row Key: (2023-12-20 10:00:00, click) → {"page":"/home"}     │
├────────────────────────────────────────────────────────────────┤
│ Partition: user2                                                │
│   Row Key: (2023-12-20 09:00:00, login) → {}                   │
└────────────────────────────────────────────────────────────────┘

SSTable format for column-families:

┌──────────────────────────────────────────────────────────────┐
│ Partition Index:                                              │
│   user1 → offset 0                                           │
│   user2 → offset 4096                                        │
├──────────────────────────────────────────────────────────────┤
│ Partition: user1                                              │
│   Partition Header                                            │
│   ┌──────────────────────────────────────────────────────┐   │
│   │ Clustering: (10:05:00, purchase) │ data: {...}       │   │
│   ├──────────────────────────────────────────────────────┤   │
│   │ Clustering: (10:00:00, click)    │ data: {...}       │   │
│   └──────────────────────────────────────────────────────┘   │
├──────────────────────────────────────────────────────────────┤
│ Partition: user2                                              │
│   ...                                                         │
└──────────────────────────────────────────────────────────────┘

Questions to guide your implementation:

  • How do you hash partition keys to distribute data?
  • Why must all queries include the partition key?
  • How do tombstones (deletion markers) work?
  • What’s the “tombstone problem” and how do you mitigate it?

Learning milestones:

  1. Partition key routing works → You understand data distribution
  2. Clustering column ordering works → You understand wide rows
  3. Range queries are efficient → You understand the query model
  4. Compaction handles tombstones → You understand deletion semantics

Project 14: CRDT-Based Collaborative Data Store

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Rust
  • Alternative Programming Languages: Go, TypeScript
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 4: Expert
  • Knowledge Area: Eventual Consistency / Conflict Resolution
  • Software or Tool: Riak, Automerge, Yjs
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A data store using CRDTs (Conflict-free Replicated Data Types) that allows concurrent edits from multiple clients without coordination, automatically merging changes.

Why it teaches NoSQL: CRDTs are the mathematical solution to eventual consistency. Instead of detecting and resolving conflicts, CRDTs are designed so conflicts are impossible—all merge operations are commutative, associative, and idempotent. This is how Google Docs, Figma, and modern collaborative apps work.

Core challenges you’ll face:

  • G-Counter, PN-Counter → maps to grow-only and positive-negative counters
  • LWW-Register → maps to last-writer-wins with timestamps
  • OR-Set → maps to observed-remove sets
  • Merging → maps to lattice theory and join operations

Key Concepts:

  • CRDT Theory: “A comprehensive study of CRDTs” - Shapiro et al.
  • Practical CRDTs: “Designing Data-Intensive Applications” Chapter 5 - Kleppmann
  • Automerge: Automerge CRDT Library
  • Mathematical Foundations: “CRDTs: The Hard Parts” by Martin Kleppmann (YouTube)

Difficulty: Expert Time estimate: 2-3 weeks Prerequisites: Projects 1-2, 9 completed. Understanding of set theory helpful.

Real world outcome:

$ ./crdt-store demo
CRDT Store Demo - Simulating 3 disconnected clients

# Initialize a shared shopping cart (OR-Set CRDT)
Client 1: cart = {}
Client 2: cart = {}
Client 3: cart = {}

# Concurrent modifications (clients are offline)
Client 1: ADD("milk")    → cart = {milk}
Client 2: ADD("bread")   → cart = {bread}
Client 3: ADD("milk"), REMOVE("milk"), ADD("eggs") → cart = {eggs}

# Clients sync - CRDTs merge automatically!
After merge:
  Client 1: cart = {milk, bread, eggs}
  Client 2: cart = {milk, bread, eggs}
  Client 3: cart = {milk, bread, eggs}

Note: "milk" is present because Client 1's ADD happened
concurrently with Client 3's REMOVE (causally independent).
The OR-Set keeps it because the ADD wasn't "observed" by Client 3.

# G-Counter example (increment-only counter)
Client 1: likes = {c1: 5}
Client 2: likes = {c2: 3}
Client 3: likes = {c3: 7}

Merge: likes = {c1: 5, c2: 3, c3: 7} → total = 15

# LWW-Register example
Client 1 at t=100: status = "online"
Client 2 at t=105: status = "away"
Client 3 at t=102: status = "busy"

Merge: status = "away" (highest timestamp wins)

Implementation Hints:

G-Counter (grow-only counter):

State: Map<NodeID, int>  // Each node tracks its own increments

Increment on node A:
  state[A] += 1

Merge(state1, state2):
  for each node:
    result[node] = max(state1[node], state2[node])

Value():
  return sum(state.values())

PN-Counter (allows decrement):

State: {P: G-Counter, N: G-Counter}  // Positive and Negative

Increment: P.increment()
Decrement: N.increment()
Value: P.value() - N.value()

OR-Set (Observed-Remove Set):

State: Map<Element, Set<UniqueTag>>

Add(element):
  tag = generateUniqueTag()  // e.g., (nodeID, timestamp)
  state[element].add(tag)

Remove(element):
  state[element] = {}  // Remove all observed tags

Merge(state1, state2):
  for each element:
    result[element] = state1[element] ∪ state2[element]

Contains(element):
  return state[element] is not empty

The key insight: every operation is tagged with a unique ID. Remove only removes tags you’ve seen. Concurrent adds create new tags that survive the remove.

Questions to guide your implementation:

  • Why can’t you have a simple decrement-only counter?
  • How do you garbage collect old tombstones in OR-Set?
  • What’s the memory overhead of CRDTs vs simple values?
  • How do you implement a CRDT text editor? (RGA, WOOT algorithms)

Learning milestones:

  1. G-Counter merges correctly → You understand state-based CRDTs
  2. OR-Set handles concurrent add/remove → You understand causality
  3. Multi-node demo shows convergence → You understand eventual consistency
  4. No conflicts ever! → You understand the CRDT magic

Project 15: Complete Distributed Database (Capstone Project)

  • File: LEARN_NOSQL_DATABASES_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, C++
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 5. The “Industry Disruptor”
  • Difficulty: Level 5: Master
  • Knowledge Area: Complete Distributed Systems
  • Software or Tool: TiKV, CockroachDB, FoundationDB
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A production-quality distributed key-value database combining everything: LSM storage engine, Raft consensus, consistent hashing for sharding, and a SQL-like query layer on top.

Why it teaches NoSQL: This is the culmination of everything. You’ll integrate storage engines, consensus, distribution, and query processing into a coherent system. After building this, you’ll truly understand how TiKV, CockroachDB, and YugabyteDB work under the hood.

Core challenges you’ll face:

  • Multi-Raft → maps to one Raft group per shard
  • Transaction coordination → maps to 2PC or Percolator model
  • Load balancing → maps to shard splitting and migration
  • Query routing → maps to distributed query execution

Key Concepts:

  • Multi-Raft Architecture: TiKV design documentation
  • Distributed Transactions: “Designing Data-Intensive Applications” Chapter 7 - Kleppmann
  • Percolator: Google’s Percolator paper
  • Spanner: Google Spanner paper

Difficulty: Master Time estimate: 2-3 months Prerequisites: ALL previous projects completed. Deep understanding of distributed systems, consensus, storage engines.

Real world outcome:

$ ./distributed-db start --nodes=5
Starting DistributedDB cluster...
  Node 1 (shards: 1,5,9):  localhost:10001
  Node 2 (shards: 2,6,10): localhost:10002
  Node 3 (shards: 3,7,11): localhost:10003
  Node 4 (shards: 4,8,12): localhost:10004
  Node 5 (replica):        localhost:10005

Cluster Status:
  Total shards: 12
  Replication factor: 3
  Raft groups: 12 (one per shard)
  All leaders elected ✓

$ ./distributed-db sql
DistributedDB SQL Shell

> CREATE TABLE users (
    id INT PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(100)
  );
Table created (sharded by: id)

> INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
1 row inserted (shard 7, leader: node3, committed at term 5)

> SELECT * FROM users WHERE id = 1;
+----+-------+-------------------+
| id | name  | email             |
+----+-------+-------------------+
|  1 | Alice | alice@example.com |
+----+-------+-------------------+
Execution: shard 7 → node3 (leader) → 0.8ms

# Simulate node failure
$ kill -9 $(pgrep -f "node3")

# Query still works! (Raft elects new leaders)
> SELECT * FROM users WHERE id = 1;
+----+-------+-------------------+
| id | name  | email             |
+----+-------+-------------------+
|  1 | Alice | alice@example.com |
+----+-------+-------------------+
Execution: shard 7 → node1 (new leader) → 12ms (election delay)

# Cross-shard transaction
> BEGIN;
> UPDATE accounts SET balance = balance - 100 WHERE id = 1;  -- shard 3
> UPDATE accounts SET balance = balance + 100 WHERE id = 5;  -- shard 7
> COMMIT;
Transaction committed (2PC across 2 shards)

Implementation Hints:

Multi-Raft architecture:

┌─────────────────────────────────────────────────────────────────┐
│                        SQL Query Layer                           │
│   Parse → Plan → Optimize → Route to Shards → Aggregate         │
└───────────────────────────────────────────────────────────────────┘
                                │
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
         ┌───────────┐   ┌───────────┐   ┌───────────┐
         │  Shard 1  │   │  Shard 2  │   │  Shard 3  │
         │ Raft Group│   │ Raft Group│   │ Raft Group│
         │  ┌─────┐  │   │  ┌─────┐  │   │  ┌─────┐  │
         │  │ LSM │  │   │  │ LSM │  │   │  │ LSM │  │
         │  └─────┘  │   │  └─────┘  │   │  └─────┘  │
         └───────────┘   └───────────┘   └───────────┘
              │               │               │
    ┌─────────┼───────────────┼───────────────┼─────────┐
    ▼         ▼               ▼               ▼         ▼
 ┌──────┐ ┌──────┐       ┌──────┐       ┌──────┐ ┌──────┐
 │Node 1│ │Node 2│       │Node 3│       │Node 4│ │Node 5│
 │ L    │ │ F F  │       │ L F  │       │ F L  │ │ F F  │
 └──────┘ └──────┘       └──────┘       └──────┘ └──────┘

 L = Leader for that shard's Raft group
 F = Follower

Two-Phase Commit for distributed transactions:

Phase 1 (Prepare):
  Coordinator → all participants: "Prepare to commit TX123"
  Each participant:
    - Acquire locks
    - Write to WAL
    - Reply "YES" or "NO"

Phase 2 (Commit/Abort):
  If all YES:
    Coordinator → all participants: "Commit TX123"
  Else:
    Coordinator → all participants: "Abort TX123"

  Participants release locks and apply/discard changes

Shard routing:

Key → hash(key) → shard_id → Raft group → leader node → execute

Questions to guide your implementation:

  • How do you handle shard splitting when one shard gets too hot?
  • What happens if the coordinator crashes during 2PC?
  • How do you implement snapshot isolation across shards?
  • How do you balance leaders across nodes?

Learning milestones:

  1. Single-shard operations work → You understand the basic flow
  2. Multi-Raft groups work independently → You understand sharding
  3. Cross-shard transactions work → You understand distributed transactions
  4. System survives node failures → You understand fault tolerance
  5. Performance benchmarks are competitive → You’ve built a real database!

Project Comparison Table

# Project Difficulty Time Core Concepts Fun Factor
1 In-Memory KV Store ⭐ Weekend Hash maps, networking ⭐⭐⭐
2 Persistent KV with WAL ⭐⭐ 1 week Durability, crash recovery ⭐⭐⭐
3 Bloom Filter ⭐ Weekend Probabilistic structures ⭐⭐⭐⭐
4 LSM Tree Engine ⭐⭐⭐ 2-4 weeks Write optimization ⭐⭐⭐⭐⭐
5 B-Tree Index ⭐⭐⭐ 2-3 weeks Read optimization ⭐⭐⭐⭐
6 Consistent Hashing ⭐⭐ 1 week Data distribution ⭐⭐⭐⭐
7 Document Store ⭐⭐⭐ 2-3 weeks Schema flexibility, indexing ⭐⭐⭐⭐
8 Raft Consensus ⭐⭐⭐⭐ 3-4 weeks Distributed consensus ⭐⭐⭐⭐⭐
9 Dynamo-Style Replication ⭐⭐⭐⭐ 3-4 weeks Eventual consistency ⭐⭐⭐⭐⭐
10 Time-Series DB ⭐⭐⭐ 2-3 weeks Domain-specific optimization ⭐⭐⭐⭐
11 Graph Database ⭐⭐⭐ 2-3 weeks Relationship traversal ⭐⭐⭐⭐⭐
12 Full-Text Search ⭐⭐⭐ 2-3 weeks Information retrieval ⭐⭐⭐⭐
13 Column-Family Store ⭐⭐⭐⭐ 3-4 weeks Wide-row storage ⭐⭐⭐⭐
14 CRDT Store ⭐⭐⭐⭐ 2-3 weeks Conflict-free merging ⭐⭐⭐⭐⭐
15 Distributed Database ⭐⭐⭐⭐⭐ 2-3 months Everything combined! ⭐⭐⭐⭐⭐

Phase 1: Foundations (Weeks 1-4)

Start here to understand core storage concepts

  1. Project 1: In-Memory KV Store - Get something working fast
  2. Project 2: Persistent KV with WAL - Understand durability
  3. Project 3: Bloom Filter - Learn probabilistic structures

Phase 2: Storage Engines (Weeks 5-10)

Deep dive into how data is stored and retrieved

  1. Project 4: LSM Tree Engine - Master write optimization
  2. Project 5: B-Tree Index - Master read optimization

Phase 3: Distribution Basics (Weeks 11-14)

Learn to spread data across machines

  1. Project 6: Consistent Hashing - Data partitioning
  2. Project 7: Document Store - Apply your storage engine knowledge

Phase 4: Distributed Consensus (Weeks 15-22)

The hardest but most rewarding part

  1. Project 8: Raft Consensus - Strong consistency
  2. Project 9: Dynamo-Style Replication - Eventual consistency

Phase 5: Specialized Databases (Weeks 23-30)

Pick based on your interests

  1. Project 10: Time-Series DB - If you work with metrics/IoT
  2. Project 11: Graph Database - If you work with relationships
  3. Project 12: Full-Text Search - If you work with text

Phase 6: Advanced Topics (Weeks 31-40)

For those who want to go deep

  1. Project 13: Column-Family Store - Cassandra-style
  2. Project 14: CRDT Store - Conflict-free collaboration

Phase 7: Capstone (Weeks 41-52)

Combine everything

  1. Project 15: Distributed Database - Build a complete system

Recommendation Based on Your Context

If you’re a beginner with limited time:

Start with Projects 1-3. You’ll understand the fundamentals that underpin all databases in just 3-4 weeks.

If you’re an intermediate developer:

Do Projects 1-6 in order. This gives you complete understanding of storage engines and distribution basics in about 3 months.

If you’re preparing for system design interviews:

Focus on Projects 4, 6, 8, and 9. These cover the most common interview topics: LSM trees, consistent hashing, Raft, and eventual consistency.

If you want the most impressive resume project:

Build Project 8 (Raft) or Project 15 (Distributed Database). Few developers have actually implemented consensus algorithms or complete distributed systems.

If you want to contribute to open source databases:

Complete Projects 4, 8, and 13. You’ll understand enough to contribute to RocksDB, etcd, or Cassandra.

Final Capstone: Build TiKV-Lite

After completing all projects, you’ll have all the pieces to build a TiKV-like distributed transactional key-value database:

  • Storage: LSM tree engine (Project 4)
  • Consensus: Multi-Raft for replication (Project 8)
  • Distribution: Consistent hashing for sharding (Project 6)
  • Transactions: Two-phase commit (Project 15)
  • Query Layer: SQL parsing and optimization (Project 7 principles)

This is a production-quality database that could be the foundation of a startup or a major open-source project.

Essential Resources

Books (In Order of Priority)

  1. “Designing Data-Intensive Applications” by Martin Kleppmann
    • THE bible for distributed databases
    • Covers everything from storage to consensus
  2. “Database Internals” by Alex Petrov
    • Deep dive into storage engine implementation
    • Essential for Projects 4-5
  3. “NoSQL Distilled” by Pramod Sadalage & Martin Fowler
    • Quick overview of NoSQL categories
    • Great starting point

Online Courses

  1. Mini-LSM: LSM in a Week
    • Hands-on LSM tree implementation
    • Perfect companion for Project 4
  2. CMU 15-445: Database Systems
    • Free lectures on database internals
    • Covers B-trees, query processing

Papers

  1. Dynamo Paper (Amazon, 2007) - Foundation for eventual consistency
  2. Raft Paper (Ongaro & Ousterhout) - Understandable consensus
  3. Bigtable Paper (Google) - Column-family inspiration
  4. Spanner Paper (Google) - Globally distributed transactions

Summary

# Project Main Language
1 In-Memory Key-Value Store Go
2 Persistent Key-Value Store with WAL Go
3 Bloom Filter Implementation Python
4 LSM Tree Storage Engine Rust
5 B-Tree Index Implementation C
6 Consistent Hashing for Distributed Storage Go
7 Simple Document Store (Mini-MongoDB) Go
8 Replication with Leader Election (Raft) Go
9 Eventually Consistent Replication (Dynamo-Style) Go
10 Time-Series Database Go
11 Graph Database Python
12 Full-Text Search Engine Go
13 Write-Optimized Column Store C++
14 CRDT-Based Collaborative Data Store Rust
15 Complete Distributed Database (Capstone) Go

Sources: