← Back to all projects

EVENT SOURCING CQRS DEEP DIVE

In traditional CRUD (Create, Read, Update, Delete) systems, we store the *current state*. When a user changes their address, we overwrite the old one. The history is lost forever.

Learn Event Sourcing & CQRS: From Zero to Event-Driven Master

Goal: Deeply understand Event Sourcing and CQRS (Command Query Responsibility Segregation) by building a custom event store from first principles. You will move from simple append-only logs to complex distributed systems that handle event versioning, snapshotting, and high-performance projections, internalizing why “state” is just the current fold of a history of immutable facts.

Why Event Sourcing & CQRS Matters

In traditional CRUD (Create, Read, Update, Delete) systems, we store the current state. When a user changes their address, we overwrite the old one. The history is lost forever.

Event Sourcing flips this. We store every change as an immutable fact (an Event). The state is simply the result of replaying these facts. CQRS takes this further by separating the logic used to change data (Commands) from the logic used to read data (Queries).

The Power of “Why”

  • Perfect Audit Log: You don’t just know what the balance is; you know every cent that ever moved and why.
  • Time Travel: You can reconstruct the state of the system at any point in history.
  • Future-Proofing: You can create new “Read Models” (projections) today using data you captured years ago.
  • Scalability: Write-heavy and read-heavy workloads can be scaled independently.

Core Concept Analysis

1. The Anatomy of an Event

An Event is a statement of fact about the past. It is immutable. It should be named in the past tense (e.g., OrderPlaced, UserEmailChanged).

Event: {
  "id": "uuid",
  "type": "FundsDeposited",
  "aggregate_id": "account_123",
  "version": 4,
  "timestamp": "2025-12-28T10:00:00Z",
  "data": { "amount": 100.00, "currency": "USD" }
}

2. Event Sourcing Flow

Instead of UPDATE accounts SET balance = balance + 100, we append an event.

[Store]
  |
  |-- Event 1: AccountOpened {id: 1, initial: 0}
  |-- Event 2: FundsDeposited {id: 1, amount: 100}
  |-- Event 3: FundsWithdrawn {id: 1, amount: 20}
  |
[Replay/Fold]
  |
  V
Current State: {balance: 80}

3. CQRS: The Great Divide

CQRS separates the system into two distinct sides.

      [ COMMAND SIDE ]                  [ QUERY SIDE ]
    (Write Model / Logic)             (Read Model / View)
             |                                 |
             V                                 V
      +--------------+                 +----------------+
      |  Validation  |                 | Fast Retrieval |
      |  Invariants  |                 | (SQL/NoSQL)    |
      +--------------+                 +----------------+
             |                                 ^
             |         [ EVENT BUS ]           |
             +----( Publishing Events )--------+

Concept Summary Table

Concept Cluster What You Need to Internalize
Immutability Facts cannot be changed or deleted. To correct an error, you must append a new “compensating” event.
Event Store An append-only log that is the “Single Source of Truth.” It must support optimistic concurrency.
Aggregate The unit of consistency. It loads its history to validate commands and emits new events.
Projection A derived view of data. It is eventually consistent and can be thrown away and rebuilt from the log.
Snapshotting An optimization. Instead of replaying 1 million events, load a state checkpoint and replay only recent events.
Conflict Resolution Using versions to detect if two users tried to change the same aggregate simultaneously.

Deep Dive Reading by Concept

Foundational Principles

Concept Book & Chapter
The Log-Structured Approach “Designing Data-Intensive Applications” by Martin Kleppmann — Ch. 3: “Storage and Retrieval”
Event Sourcing Intro “Implementing Domain-Driven Design” by Vaughn Vernon — Ch. 14: “Application”
Command/Query Separation “Patterns, Principles, and Practices of Domain-Driven Design” by Scott Millett — Ch. 23: “CQRS”

Project 4: The Time Machine (Snapshotting)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Main Programming Language: Go
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Performance Optimization
  • Main Book: “Event Streams in Action”

What you’ll build: A system that saves the state every 100 events to speed up loading.

Why it teaches Event Sourcing: It solves the performance issue of replaying long histories.

Real World Outcome

A load time that stays constant even as the history grows.

The Core Question You’re Answering

“Is it okay to store state if the log is the truth?”

Concepts You Must Understand First

  1. Memento Pattern
    • Capturing state.
  2. Snapshot Versioning
    • Linking state to an event offset.

Questions to Guide Your Design

  1. Frequency
    • When should you take a snapshot?
  2. Storage
    • Should snapshots be in the same log?

Thinking Exercise

The Fast Forward

Trace how you load an aggregate with a snapshot at v100 and events up to v150.

The Interview Questions They’ll Ask

  1. “When are snapshots necessary?”
  2. “How do you handle snapshot schema changes?”
  3. “Can you delete the log once you have a snapshot?”
  4. “How do you store snapshots efficiently?”
  5. “What is the trade-off of snapshot frequency?”

Hints in Layers

Hint 1: The Trigger Check version % 100 == 0.

Hint 2: Serializing State Use JSON to dump the struct fields.

Hint 3: Loading Logic Find the latest snapshot first.

Hint 4: Appending Events Replay only events with version > snapshot_version.

Books That Will Help

Topic Book Chapter
Snapshots “Event Streams in Action” Ch. 4
Optimization “Designing Data-Intensive Applications” Ch. 3

Implementation Hints

Snapshots should be stored in a separate directory named by aggregate ID.

Learning Milestones

  1. Performance Win - You see loading speed up.
  2. Checkpointing - You understand state/history synchronization.
  3. State Management - You can handle long-lived streams.

Project 5: The Evolving Fact (Upcasting)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Main Programming Language: Go
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Versioning
  • Main Book: “Versioning in Event Sourced Systems”

What you’ll build: A middleware that transforms old event schemas into new ones on-the-fly.

Why it teaches Event Sourcing: It handles the reality of changing requirements in an immutable system.

Real World Outcome

Running new code against old data without errors.

The Core Question You’re Answering

“How do I change the past without breaking immutability?”

Concepts You Must Understand First

  1. Schema Evolution
    • Forward and backward compatibility.
  2. Upcasting Pattern
    • Transformation chains.

Questions to Guide Your Design

  1. Lazy vs Eager
    • When should you transform?
  2. Chain of Responsibility
    • How to go from v1 to v3?

Thinking Exercise

The Translator

Write a pseudo-code function that adds a default Currency field to an event that only has Amount.

The Interview Questions They’ll Ask

  1. “What is an Upcaster?”
  2. “How do you handle breaking changes?”
  3. “Why not just migrate the database?”
  4. “How do you test upcasters?”
  5. “What is a ‘Weak Schema’?”

Hints in Layers

Hint 1: The Map Use a map of versions to functions.

Hint 2: Intermediate Objects Parse into a map before the final struct.

Hint 3: Chaining Recursive calls to move version by version.

Hint 4: Metadata Store the schema version in the event envelope.

Books That Will Help

Topic Book Chapter
Versioning “Versioning in Event Sourced Systems” Entire Book

Implementation Hints

Upcasters should be pure functions. They take a JSON map and return a JSON map.

Learning Milestones

  1. Schema Mapping - You can handle versioned data.
  2. Legacy Support - You understand how to maintain old facts.
  3. Data Transformation - You can evolve models safely.

Project 6: The Messenger (The Event Bus)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Main Programming Language: Go
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Pub/Sub
  • Main Book: “Enterprise Integration Patterns”

What you’ll build: A system that publishes log events to NATS or Redis for side effects.

Why it teaches CQRS: It decouples consistency from side effects.

Real World Outcome

An email being sent automatically when a user registers.

The Core Question You’re Answering

“How do I notify the world without slowing down my database?”

Concepts You Must Understand First

  1. Outbox Pattern
    • Ensuring delivery.
  2. At-least-once delivery
    • Handling failures.

Questions to Guide Your Design

  1. Reliability
    • What if the bus is down?
  2. Ordering
    • Does order matter for side effects?

Thinking Exercise

The Double Write

What happens if you save the event but the server crashes before you can publish it?

The Interview Questions They’ll Ask

  1. “What is the Outbox Pattern?”
  2. “How do you handle duplicate messages?”
  3. “Orchestration vs Choreography?”
  4. “What is a Poison Pill?”
  5. “How do you ensure message ordering?”

Hints in Layers

Hint 1: The Outbox Save events to a ‘pending’ table.

Hint 2: The Poller A separate loop that reads pending events.

Hint 3: The Bus Use NATS ‘Publish’.

Hint 4: Acknowledgement Delete from ‘pending’ only after the bus confirms.

Books That Will Help

Topic Book Chapter
Messaging “Enterprise Integration Patterns” Ch. 3
Outbox “Microservices Patterns” Ch. 3

Implementation Hints

The Outbox should be in the same transaction as the event append.

Learning Milestones

  1. Decoupling - You can separate logic from side effects.
  2. Reliability - You understand the Outbox pattern.
  3. Integration - You can connect services via events.

Project 7: The Coordinator (Saga)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Difficulty: Level 4: Expert
  • Knowledge Area: Distributed Systems

What you’ll build: A multi-step transaction manager (e.g., Order -> Payment -> Inventory).

Real World Outcome

A complex process that rolls back (compensates) if any step fails.

The Interview Questions They’ll Ask

  1. “What is a Saga?”
  2. “Compensating transactions vs rollback?”
  3. “Stateful vs Stateless sagas?”
  4. “How to handle timeouts in sagas?”
  5. “Saga isolation issues?”

Hints in Layers

Hint 1: The State Machine Define the steps clearly.

Hint 2: Listening Subscribe to events from all services.

Hint 3: Compensation Define the ‘Inverse’ of every action.

Hint 4: Persistence Store the saga state as its own event stream.

Project 8: The Auditor (Time-Travel Debugger)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Difficulty: Level 2: Intermediate

What you’ll build: A UI to visualize the event timeline of an aggregate.

Real World Outcome

Seeing why an account is locked by replaying the exact sequence.

Project 9: The Performance Beast (Aggregate Cache)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Difficulty: Level 3: Advanced

What you’ll build: An in-memory cache that keeps ‘hot’ aggregates ready for commands.

Project 10: The Untouchable Ledger (Merkle Trees)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Difficulty: Level 4: Expert

What you’ll build: A cryptographic chain for your events to prevent tampering.

Project Comparison Table

Project Difficulty Time Depth Fun
1. Ledger 1 Weekend High 3
2. Aggregate 2 1 Week High 4
3. View Maker 2 1 Week Med 3
4. Snapshots 3 1 Week Med 3
5. Upcasting 3 1 Week High 2
6. Event Bus 2 Weekend Med 4
7. Sagas 4 2 Weeks High 5
8. Auditor 2 1 Week Med 5
9. Cache 3 1 Week Med 3
10. Merkle 4 1 Week High 4

Recommendation

Start with Projects 1, 2, and 3. This gives you the core ES/CQRS loop.

Final Overall Project: The Event-Driven Bank

Build a complete banking system with transfers, interest, auditing, and a real-time UI.

Summary

This path covers Event Sourcing and CQRS through 10 hands-on projects.

# Name Language Difficulty
1 Ledger Go Beginner
2 Aggregate Go Intermediate
3 View Maker Go Intermediate
4 Snapshots Go Advanced
5 Upcasting Go Advanced
6 Event Bus Go Intermediate
7 Saga Go Expert
8 Auditor Go Intermediate
9 Cache Go Advanced
10 Merkle Go Expert

Expected Outcome: You will deeply understand how to build reliable, auditable, and scalable systems using events.

Project 1: The Ledger (The Append-Only Log)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, Python, C#
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: File I/O / Serialization
  • Software or Tool: JSON/Protobuf, Filesystem
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A CLI tool that appends events to a flat file and allows reading them back. It must ensure that once an event is written, it can never be modified (immutability).

Why it teaches Event Sourcing: It forces you to think about data as a sequence of discrete facts rather than a table of current values. You’ll grapple with how to structure an event so it contains enough context to be useful later.

Core challenges you’ll face:

  • Defining the Schema → What metadata does every event need? (ID, Timestamp, Type)
  • Atomic Appends → How do you ensure two writes don’t corrupt the file?
  • Serialization → Transforming objects/structs into bytes and back.

Real World Outcome

You will have a file (events.log) where every line is a JSON-encoded event. You can’t “edit” a transaction; you can only add a new one.

Example Output:

$ ./ledger append --type "UserRegistered" --data '{"id": 1, "name": "Alice"}'
Event 001 appended.

$ ./ledger list
[001] 2025-12-28 10:00:00 | UserRegistered | {"id": 1, "name": "Alice"}

The Core Question You’re Answering

“If I can never delete or update a row, how do I keep track of what happened?”

Before you write any code, sit with this question. In a database, we usually overwrite. Here, we accumulate. How do you find the “latest” version of something if the old versions are still there?

Concepts You Must Understand First

Stop and research these before coding:

  1. Serialization
    • How do you turn a struct into bytes?
    • Book Reference: “Designing Data-Intensive Applications” Ch. 4
  2. File Append Atomicitiy
    • Is O_APPEND enough for multiple processes?
    • Book Reference: “The Linux Programming Interface” Ch. 4

Questions to Guide Your Design

  1. The Header
    • Should the metadata be separate from the event data?
  2. Indexing
    • How do you find an event by ID without reading the whole file?

Thinking Exercise

The Eraser Test

Imagine you are building a banking system. A user deposits $100. Then they withdraw $50. Draw the “Event Log” (Event Sourcing style). Questions while tracing:

  • In the Event Log, how do you calculate the balance?

The Interview Questions They’ll Ask

  1. “Why is an append-only log more performant for writes than a B-Tree?”
  2. “How do you handle schema changes in an immutable file?”
  3. “What happens if the disk fills up during an append?”
  4. “Is JSON a good choice for a high-performance event store?”
  5. “How do you verify the integrity of the log?”

Hints in Layers

Hint 1: The Structure Create a generic Event struct with ID, Type, and Payload.

Hint 2: The File Use os.OpenFile with O_APPEND.

Hint 3: JSON Lines Use one JSON object per line.

Hint 4: Verification Verify the JSON is valid on read.

Books That Will Help

Topic Book Chapter
Storage “Designing Data-Intensive Applications” Ch. 3
Encoding “Designing Data-Intensive Applications” Ch. 4

Implementation Hints

Focus on creating a single Append(event Event) error function. Ensure it handles the newline character correctly. For reading, use a Scanner to go line-by-line.

Learning Milestones

  1. First Append - You understand how to store facts.
  2. Log Listing - You can retrieve the history.
  3. Validation - You understand data integrity in logs.

Project 2: The Invariant Guardian (The Aggregate)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Java, C#, TypeScript
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Domain Modeling / State Machines
  • Software or Tool: Unit Testing Framework
  • Main Book: “Implementing Domain-Driven Design” by Vaughn Vernon

What you’ll build: A “Bank Account” aggregate that replays history to find the balance and validates new commands.

Why it teaches Event Sourcing: You’ll learn how to separate validation from state transition.

Real World Outcome

A test suite that ensures you can’t withdraw more money than you have.

Example Test:

// Given: AccountOpened(50)
// When: Withdraw(100)
// Then: Error("Insufficient Funds")

The Core Question You’re Answering

“How do I make a decision today based on everything that happened in the past?”

Concepts You Must Understand First

  1. State Folding
    • How to reduce events into state.
    • Book Reference: “Implementing Domain-Driven Design” Ch. 10
  2. Invariants
    • What are the rules that must never be broken?

Questions to Guide Your Design

  1. The Apply Method
    • Should it ever return an error?
  2. Command vs Event
    • What is the difference between “Withdraw” and “Withdrawn”?

Thinking Exercise

The State Machine

Map out the states of a Bank Account. How do events trigger transitions?

The Interview Questions They’ll Ask

  1. “What is an Aggregate Root?”
  2. “Why is the Apply method purely functional?”
  3. “How do you handle commands that depend on external state?”
  4. “What is a Bounded Context?”
  5. “How do you handle concurrent updates to the same aggregate?”

Hints in Layers

Hint 1: The State Keep a private balance field in your struct.

Hint 2: Replay Loop through events and call an apply method for each.

Hint 3: Validation Check the balance before emitting a new event.

Hint 4: Versioning Keep track of how many events you’ve applied to detect conflicts.

Books That Will Help

Topic Book Chapter
Aggregates “Implementing Domain-Driven Design” Ch. 10
Domain Events “Domain-Driven Design” Ch. 5

Implementation Hints

The Handle(command) method should be the only place where business logic lives. The Apply(event) method should only update fields and never fail.

Learning Milestones

  1. State Reconstruction - You can turn events into an object.
  2. Business Rules - You can enforce logic in an event-sourced way.
  3. Consistency - You understand the aggregate boundary.

Project 3: The View Maker (The Projection)

  • File: EVENT_SOURCING_CQRS_DEEP_DIVE.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Node.js, Python
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Read Models / Denormalization
  • Software or Tool: SQLite or an In-Memory Map
  • Main Book: “Patterns, Principles, and Practices of Domain-Driven Design” by Scott Millett

What you’ll build: A background worker that maintains a summary table in SQLite based on events.

Why it teaches CQRS: You learn that the data we show to users doesn’t have to look like the data we use to process commands.

Real World Outcome

A SQL table that is always updated when events are appended.

The Core Question You’re Answering

“If the Event Store is the truth, why do I need a separate database for reading?”

Concepts You Must Understand First

  1. Eventual Consistency
    • Why views lag behind the log.
    • Book Reference: “Designing Data-Intensive Applications” Ch. 11
  2. Denormalization
    • Why redundant data is okay in CQRS.

Questions to Guide Your Design

  1. Idempotency
    • What happens if you process the same event twice?
  2. Checkpointing
    • How do you remember your position in the log?

Thinking Exercise

The View Reconstruction

Imagine your SQL database is deleted. How do you recover it using ONLY the event log?

The Interview Questions They’ll Ask

  1. “What is Eventual Consistency?”
  2. “How do you handle projection failures?”
  3. “Can a projection use data from multiple aggregates?”
  4. “How do you scale projectors?”
  5. “When would you use a Document Store instead of SQL for a projection?”

Hints in Layers

Hint 1: Tail the Log Read the log file from the beginning.

Hint 2: The DB Open a SQLite connection.

Hint 3: Update Logic For each event, run an UPDATE or INSERT.

Hint 4: Offset Store the line number you last read in a separate file.

Books That Will Help

Topic Book Chapter
CQRS “Patterns, Principles, and Practices of DDD” Ch. 23
Read Models “Building Event-Driven Microservices” Ch. 6

Implementation Hints

Your projector should be an infinite loop. If it reaches the end of the file, it should sleep for 100ms before checking again.

Learning Milestones

  1. Data Sync - You can move data from log to table.
  2. Async Views - You understand that reads can be separate from writes.
  3. Recovery - You can rebuild views from scratch.