← Back to all projects

LEARN QUEUES MESSAGE BROKERS PROJECTS

**File**: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md

Learn Message Queues (RabbitMQ, Kafka, etc.) by Building

File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md

0. Your Position (What this plan assumes)

You want to understand “queues” the way a broker implements them, not just how to call a client SDK. That means you will build small, working slices that force you to confront:

• Delivery guarantees vs. throughput
• Ordering vs. parallelism
• Persistence vs. latency
• Backpressure, retries, poison messages, and dead-lettering
• Replication, leader election, and failure recovery (Kafka-style)

1. Core Concept Analysis (what you must internalize)

A queue/broker is not “a list of messages.” It is a set of coordinated mechanisms:

1) The contract (what does “delivered” mean?)

• At-most-once, at-least-once, effectively-once; ack/nack; redelivery rules; deduplication.

2) Flow control

• Prefetch/windowing, batching, push vs pull, consumer lag, backpressure.

3) Routing / topology

• Simple queue vs exchange-based routing (RabbitMQ-style), partitions (Kafka-style).

4) State & durability

• In-memory vs durable; write-ahead log; fsync policy; retention; compaction.

5) Coordination

• Consumer groups, rebalancing, offset tracking; membership and heartbeats.

6) Replication & failure

• Leader/follower, ISR/quorums, elections, fencing, replay and recovery.

7) Operational reality

• Metrics (lag, throughput, redelivery), tuning, overload behavior, upgrade safety.

Key mental model:

• RabbitMQ (classic AMQP) is fundamentally a routing + queueing system with explicit topology (exchanges/bindings) and consumer acknowledgements/prefetch.
• Kafka is fundamentally a partitioned replicated log where “consuming” is reading from a log position (offset), coordinated by consumer groups.

(References in chat response include RabbitMQ docs on exchanges, acknowledgements/prefetch, and Kafka docs on offsets, protocol, replication/idempotence.)

---

2. Project Ideas (14 projects)

Project 1: In-Memory Work Queue + Backpressure Simulator

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, Python, Elixir
• Coolness Level: Level 2: Practical but Forgettable
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 2: Intermediate
• Knowledge Area: Concurrency / Backpressure
• Software or Tool: Queue runtime (build)
• Main Book: Computer Systems: A Programmer’s Perspective (Bryant & O’Hallaron)

What you’ll build: A local queue service that accepts “jobs,” dispatches to workers, and visibly throttles producers when workers fall behind.

Why it teaches queues: You will implement the first non-negotiable reality: producers and consumers rarely match throughput, so you must design backpressure instead of hoping.

Core challenges you’ll face:

• Designing a bounded buffer and deciding what happens on overflow → backpressure semantics
• Worker concurrency vs ordering (FIFO vs parallel) → ordering contracts
• Visibility: show queue depth, worker utilization, and drop/throttle events → operational thinking

Key Concepts

• Backpressure & load shedding: Designing Data-Intensive Applications (Kleppmann) — the sections on throughput, buffering, and backpressure
• Concurrency primitives: CS:APP — synchronization concepts and reasoning about concurrency

Difficulty: Intermediate
Time estimate: 1–2 weeks
Prerequisites: Basic concurrency in your chosen language; comfort with networking basics (local TCP/HTTP is enough)

Real world outcome

• A terminal dashboard (or simple web page) showing queue depth, rate in/out, and when throttling activates.

Implementation Hints

• Force overload (producer faster than consumer). Your design is “correct” only if it degrades predictably and doesn’t crash.
• Decide explicitly: block producers, drop, or spill to disk (spill comes in Project 2).

Learning milestones

1. Queue works under normal load → you understand producer/consumer coordination
2. Under overload it stays stable (no runaway memory) → you understand backpressure
3. You can explain the tradeoff of “block vs drop vs buffer” → you understand queue contracts

Project 2: Durable File-Backed Queue (Write-Ahead Log)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, C, Java
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 3. The “Service & Support” Model
• Difficulty: Level 3: Advanced
• Knowledge Area: Persistence / WAL
• Software or Tool: Write-Ahead Log (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A queue that survives restarts using an append-only log + an index of “in-flight” vs “acked.”

Why it teaches queues: Durability is not “save it somewhere.” It is defining when a message is considered safely stored and how you recover after crashes.

Core challenges you’ll face:

• WAL format: framing, checksums, partial writes → crash safety
• Ack tracking and replay on restart → at-least-once delivery mechanics
• Compaction: reclaim space without losing state → log maintenance

Key Concepts

• Append-only logs & recovery: DDIA — log-structured storage concepts
• Crash consistency: CS:APP — storage and system-level I/O fundamentals

Difficulty: Advanced
Time estimate: 2–4 weeks
Prerequisites: Project 1; familiarity with files and durability tradeoffs (what fsync implies conceptually)

Real world outcome

• Kill the process mid-stream; restart; prove which messages re-deliver and why (with a replay report).

Implementation Hints

• Treat “power loss” as a first-class scenario: truncated last record, corrupt tail, etc.
• Make recovery a visible step: print recovery actions and the reconstructed queue state.

Learning milestones

1. Survives clean restart → persistence basics
2. Survives crash mid-write → you understand WAL and partial writes
3. Space doesn’t grow forever → you understand compaction

Project 3: Mini AMQP Router (Exchanges, Bindings, Routing Keys)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Elixir, Rust, Java
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 2. The “Micro-SaaS / Pro Tool”
• Difficulty: Level 3: Advanced
• Knowledge Area: Message Routing / Topology
• Software or Tool: Exchange/Binding router (build)
• Main Book: RabbitMQ in Depth (Alvaro Videla & Jason J. W. Williams) (recommended)

What you’ll build: A simplified “broker core” that routes published messages to queues using direct and topic-style rules.

Why it teaches queues: RabbitMQ’s power is topology. You will understand why messages are not “sent to a queue” but routed via exchange rules.

Core challenges you’ll face:

• Implementing direct routing and topic wildcards (* and #) → routing semantics
• Managing bindings efficiently as they grow → data structure design
• Observability: show why a message went to Q1 vs Q2 → debuggability

Key Concepts

• Exchange types and routing behavior: RabbitMQ docs — “Exchanges” (direct/topic/fanout)
• Topic wildcard matching rules: AMQP 0-9-1 specification — topic exchange definition

Difficulty: Advanced
Time estimate: 2–3 weeks
Prerequisites: Project 1; comfort implementing small parsers and maps/tries

Real world outcome

• A demo where publishing with different routing keys visibly lands in different queues; unmatched routes are reported.

Implementation Hints

• Start with only: publish, declare queue, bind, consume.
• For topic routing, choose one strategy: naive scan first, then optimize with a trie-like matcher.

Learning milestones

1. Direct routing works → you understand binding keys
2. Topic wildcards work → you understand pattern routing
3. You can explain topology design choices → you understand why RabbitMQ has exchanges

Project 4: Ack/Nack, Redelivery, and Poison-Message Lab

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Python, Elixir, Java
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 3. The “Service & Support” Model
• Difficulty: Level 2: Intermediate
• Knowledge Area: Delivery Guarantees
• Software or Tool: Redelivery / DLQ (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A consumer runtime that supports manual ack, negative ack, requeue, and a dead-letter queue after N failures.

Why it teaches queues: “At-least-once” only becomes real when you see duplicates, redeliveries, and the need to isolate poison messages.

Core challenges you’ll face:

• Defining when a message is “done” (ack) vs “failed” (nack/requeue) → delivery semantics
• Implementing retry policy and DLQ thresholds → failure isolation
• Ensuring requeued messages don’t starve others → fairness and scheduling

Key Concepts

• Consumer acknowledgements vs auto-ack: RabbitMQ docs — “Consumer Acknowledgements”
• Negative acknowledgements: RabbitMQ docs — basic.nack / reject semantics

Difficulty: Intermediate
Time estimate: 1–2 weeks
Prerequisites: Project 1

Real world outcome

• A run report showing: processed count, redelivery count, DLQ count, and sample poison message trace.

Implementation Hints

• Force failures deterministically (e.g., “fail every 5th message”) so you can prove behavior.
• Track message attempts and first-seen time; this becomes your “message lifecycle.”

Learning milestones

1. Manual ack works → you understand “done means acked”
2. Requeue causes duplicates → you understand at-least-once
3. DLQ prevents system poisoning → you understand production queue hygiene

Project 5: Prefetch Tuning Playground (Flow Control You Can See)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Python, Java, Elixir
• Coolness Level: Level 2: Practical but Forgettable
• Business Potential: 2. The “Micro-SaaS / Pro Tool”
• Difficulty: Level 2: Intermediate
• Knowledge Area: Flow Control / QoS
• Software or Tool: Prefetch / windowing (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A consumer that enforces a configurable “unacked window” and produces graphs of throughput vs latency vs redeliveries as you change the window.

Why it teaches queues: Prefetch is the broker’s lever for balancing throughput and fairness; you’ll learn why “bigger buffer” can worsen tail latency and starvation.

Core challenges you’ll face:

• Enforcing “max unacked per consumer” → QoS enforcement
• Measuring tail latency and head-of-line blocking → queueing effects
• Designing experiments to isolate variables → systems thinking

Key Concepts

• Prefetch / QoS limits: RabbitMQ docs — “Consumer Prefetch”
• Unacked messages as the true backlog: RabbitMQ docs — acknowledgement behavior

Difficulty: Intermediate
Time estimate: Weekend–1 week
Prerequisites: Project 4

Real world outcome

• A single chart or dashboard: throughput and p95 latency as prefetch changes.

Implementation Hints

• Keep the workload constant (same message size/processing time) while changing prefetch.
• Record both “queue depth” and “in-flight (unacked) depth.”

Learning milestones

1. Windowing works → you understand flow control
2. You can predict starvation scenarios → you understand fairness
3. You can justify a prefetch choice → you understand tuning tradeoffs

Project 6: Kafka-Lite (Partitioned Append-Only Log Broker)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, Java, C++
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 4: Expert
• Knowledge Area: Logs / Streaming
• Software or Tool: Partitioned log broker (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A minimal broker that stores messages in topic partitions as append-only segment files and supports a pull-based consumer reading by offset.

Why it teaches queues: This is the core “Kafka mental model”: messages are not removed when consumed; consumers advance a position in a replicated log.

Core challenges you’ll face:

• Segment file design (rollover, indexing) → log structure
• Consumer read API (fetch from offset, batching) → pull-based consumption
• Retention policy (time/size based) → data lifecycle management

Key Concepts

• Kafka protocol as request/response over TCP: Apache Kafka “Protocol” guide
• Offsets & consumer progress: Apache Kafka “Distribution / consumer offset tracking” docs

Difficulty: Expert
Time estimate: 1 month+
Prerequisites: Project 2; comfort with file formats and incremental reading

Real world outcome

• A demonstrator where two consumers read the same partition independently at different offsets, proving “log not queue.”

Implementation Hints

• Do not start with replication; start with a single broker, one partition, one segment file.
• Make offsets visible: every fetch prints the offset range returned and next offset.

Learning milestones

1. Single partition works → you understand append-only logs
2. Segment rollover works → you understand retention primitives
3. Multiple consumers at different offsets work → you understand streaming semantics

Project 7: Consumer Group Coordinator (Rebalance + Offset Commit)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Java, Rust, Elixir
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 4: Expert
• Knowledge Area: Coordination / Membership
• Software or Tool: Group coordinator (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A coordinator service that assigns partitions to group members, detects failures via heartbeats, and stores committed offsets per group.

Why it teaches queues: “Consumer groups” are the hidden complexity of Kafka. Rebalancing is where ordering, latency spikes, and duplicates appear.

Core challenges you’ll face:

• Membership protocol (join/leave/heartbeat) → distributed coordination
• Partition assignment strategies (range/round-robin) → fairness vs locality
• Offset commit semantics (commit timing, replay after restart) → processing guarantees

Key Concepts

• Group coordinator and offset storage concept: Apache Kafka docs — consumer offset tracking via coordinator
• Wire protocol framing: Apache Kafka “Protocol” guide (size-delimited messages)

Difficulty: Expert
Time estimate: 1 month+
Prerequisites: Project 6; basic distributed systems instincts

Real world outcome

• Start 3 consumers in a group; stop one; watch partitions rebalance and offsets resume correctly.

Implementation Hints

• Make rebalances explicit events and log a “before/after” assignment map.
• Commit offsets only after “processing done” to see why commit timing matters.

Learning milestones

1. Stable assignment works → you understand consumer groups
2. Failure triggers rebalance → you understand coordination under churn
3. Restart resumes from committed offsets → you understand stateful consumption

Project 8: Replication + Leader Election (3-Node Log with ISR)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, Java, C++
• Coolness Level: Level 5: Pure Magic (Super Cool)
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 4: Expert
• Knowledge Area: Replication / Consensus-lite
• Software or Tool: Replicated log (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A 3-node replicated partition with leader/follower replication, ISR tracking, and controlled leader failover.

Why it teaches queues: This is the durability core behind “acks=all” and “min ISR.” You will learn what data is safe when nodes fail mid-write.

Core challenges you’ll face:

• Replication protocol: append, replicate, advance high-watermark → durability model
• ISR membership changes under lag → safety vs availability
• Leader election and fencing old leaders → split-brain prevention

Key Concepts

• Durability vs ack settings (leader waits for ISR): Kafka producer configs (acks, idempotence prerequisites)
• Leader election and ISR/ELR mechanics (modern Kafka): Kafka operations docs on replication and leader eligibility

Difficulty: Expert
Time estimate: 1–2 months
Prerequisites: Project 6; comfort with failure injection

Real world outcome

• A scripted failure demo: kill leader during writes; show which offsets are committed vs rolled back, with a clear report.

Implementation Hints

• Start with a deterministic “test harness” that can pause a follower, delay replication, or crash a node.
• Track and display “high watermark” and “last replicated” per follower.

Learning milestones

1. Replication works normally → you understand leader/follower
2. Failure doesn’t corrupt log → you understand safety invariants
3. You can explain min-ISR tradeoffs → you understand durability in practice

Project 9: Delivery Semantics Lab (Duplicates, Idempotence, “Effectively Once”)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Java, Python, Rust
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 3. The “Service & Support” Model
• Difficulty: Level 3: Advanced
• Knowledge Area: Semantics / Idempotency
• Software or Tool: Deduplication + idempotent producer (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A producer/consumer pair that can be switched between at-most-once / at-least-once / “idempotent” modes, with a ledger proving duplicates or their absence.

Why it teaches queues: The industry’s hardest truth: “exactly once” is rarely end-to-end unless you control the entire pipeline and state changes are transactional.

Core challenges you’ll face:

• Generating stable message IDs and deduplicating safely → idempotence patterns
• Showing duplicates under retries and failures → at-least-once reality
• Demonstrating the “side effects” problem (DB writes, external APIs) → end-to-end correctness

Key Concepts

• Idempotent producer constraints: Apache Kafka producer configs (enable.idempotence, acks=all, in-flight limits)
• Publisher confirms vs consumer acks: RabbitMQ docs — confirms and acknowledgements

Difficulty: Advanced
Time estimate: 2–3 weeks
Prerequisites: Project 4 (acks), plus basic persistence (a simple local “ledger” file is enough)

Real world outcome

• A reproducible report: “we injected 5 failures; duplicates occurred in mode X, not in mode Y, for reason Z.”

Implementation Hints

• Separate “message delivered” from “business effect applied.” Your ledger should track both.
• Make retries explicit and controlled (fixed schedule, injected timeouts).

Learning milestones

1. You can force duplicates → you understand why at-least-once duplicates happen
2. Deduplication works → you understand idempotency patterns
3. You can explain why “exactly once” needs transactional boundaries → you understand the real problem

Project 10: Retention + Log Compaction (Keep What Matters, Drop the Rest)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, Java, C++
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 2. The “Micro-SaaS / Pro Tool”
• Difficulty: Level 4: Expert
• Knowledge Area: Storage / Data Lifecycle
• Software or Tool: Compaction engine (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A segment maintenance job that enforces retention and optionally compacts by key (keeping only the latest value per key).

Why it teaches queues: This is the hidden feature that turns a log into a durable “source of truth” rather than an infinite tape.

Core challenges you’ll face:

• Segment scanning and rewrite without breaking offsets → log invariants
• Key index maintenance and tombstones → state representation
• Compaction scheduling to avoid impacting reads/writes → operational constraints

Key Concepts

• Log-structured storage and compaction concepts: DDIA — LSM/compaction mental model
• Kafka’s “record batches” and binary framing mindset: Kafka protocol guide (to think in batches/segments)

Difficulty: Expert
Time estimate: 1 month+
Prerequisites: Project 6

Real world outcome

• A before/after visualization: storage size shrinks while “latest state per key” remains correct.

Implementation Hints

• Start with retention only; add compaction later.
• Maintain a “mapping report” so you can explain what got removed and why.

Learning milestones

1. Retention works → you understand data lifecycle
2. Compaction preserves latest state → you understand key-based semantics
3. You can explain tombstones and deletes → you understand real stream storage

Project 11: Delay Queues and Scheduled Delivery (Timers Are Hard)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Elixir, Java, Python
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 2. The “Micro-SaaS / Pro Tool”
• Difficulty: Level 3: Advanced
• Knowledge Area: Scheduling / Time
• Software or Tool: Delay queue (build)
• Main Book: Operating Systems: Three Easy Pieces (Arpaci-Dusseau)

What you’ll build: A queue that supports “deliver not before time T” and a scheduler that releases messages when due, under load.

Why it teaches queues: Delays interact with ordering, persistence, and crash recovery; it’s a great trapdoor into “this is harder than it looks.”

Core challenges you’ll face:

• Efficient time-ordered storage (timing wheel, heap) → data structures for time
• Correctness across restarts (no lost timers) → durable scheduling
• Burst handling when many timers fire simultaneously → thundering herd control

Key Concepts

• Time and scheduling in systems: OSTEP — time and scheduling mindset (timers, waiting, wakeups)
• Queue fairness under burst: RabbitMQ prefetch concepts (think in windows of in-flight work)

Difficulty: Advanced
Time estimate: 2–4 weeks
Prerequisites: Project 2 (durability) recommended

Real world outcome

• A demo that schedules 10,000 delayed tasks and fires them with bounded jitter, even after a restart.

Implementation Hints

• Decide whether “due time” is strict or best-effort; measure jitter and report it.
• Persist the schedule state; recovery must reconcile “now” with overdue items.

Learning milestones

1. Delayed delivery works → you understand time-ordered queues
2. Restart correctness holds → you understand durable scheduling
3. Under burst it stays stable → you understand overload control

Project 12: Observability for Queues (Lag, Throughput, Redelivery)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Python, Elixir, Rust
• Coolness Level: Level 2: Practical but Forgettable
• Business Potential: 3. The “Service & Support” Model
• Difficulty: Level 2: Intermediate
• Knowledge Area: Observability / Metrics
• Software or Tool: Metrics + dashboard (build)
• Main Book: Release It! (Michael T. Nygard) (recommended)

What you’ll build: A metrics package that reports queue depth, in-flight, consumer lag, retry rates, and “time in queue.”

Why it teaches queues: You’ll learn what production operators watch and why “queue depth” alone is often misleading.

Core challenges you’ll face:

• Defining the right metrics and invariants → operational correctness
• Aggregation under high volume → systems efficiency
• Alert conditions (lag increasing, retries spiking) → SRE thinking

Key Concepts

• In-flight/unacked vs queued: RabbitMQ docs — ack + prefetch implications
• Consumer lag as a first-class metric: Kafka offset tracking model (consumer progress)

Difficulty: Intermediate
Time estimate: 1–2 weeks
Prerequisites: Any earlier project

Real world outcome

• A dashboard showing: lag per consumer, retries per minute, p95 “time in queue,” and DLQ growth.

Implementation Hints

• Make metrics “pull” (scrape) or “push” intentionally; document tradeoffs.
• Show one anomaly (e.g., slow consumer) and prove the metrics identify it.

Learning milestones

1. Metrics reflect reality → you understand what to measure
2. Alerts catch real failure modes → you understand operational signals
3. You can debug a slowdown using only metrics → you understand production queueing

Project 13: Load Generator + Benchmark Harness for Brokers

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, Python, Java
• Coolness Level: Level 2: Practical but Forgettable
• Business Potential: 2. The “Micro-SaaS / Pro Tool”
• Difficulty: Level 3: Advanced
• Knowledge Area: Performance / Benchmarking
• Software or Tool: Benchmark harness (build)
• Main Book: CS:APP (Bryant & O’Hallaron)

What you’ll build: A repeatable harness that produces load patterns (steady, bursty, fanout) and measures throughput, latency percentiles, and tail behavior.

Why it teaches queues: Benchmarks reveal the hidden costs of acknowledgements, persistence, batching, and flow control.

Core challenges you’ll face:

• Generating stable load without measuring yourself → benchmark validity
• Capturing latency distributions (p50/p95/p99) → tail latency thinking
• Comparing modes (acks, batching, prefetch) → tradeoff quantification

Key Concepts

• Windowing/batching tradeoffs: Kafka protocol and batching mindset
• Prefetch impacts: RabbitMQ consumer prefetch concepts

Difficulty: Advanced
Time estimate: 2–3 weeks
Prerequisites: Project 5 or 6 strongly recommended

Real world outcome

• A benchmark report that ranks configurations by throughput and p99 latency, with clear conclusions.

Implementation Hints

• Keep “clocking” and measurement overhead minimal; validate the harness itself.
• Include failure mode benchmarks: consumer restarts, network delay injection.

Learning milestones

1. Reproducible results → you understand measurement discipline
2. Clear tradeoffs emerge → you understand broker tuning knobs
3. You can design a fair comparison → you understand systems evaluation

Project 14: Workflow Engine on Top of Queues (Retries, DLQ, Idempotency)

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Python, Elixir, Java
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 4. The “Open Core” Infrastructure
• Difficulty: Level 3: Advanced
• Knowledge Area: Distributed Workflows
• Software or Tool: Worker framework (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A small worker framework that supports task definitions, retries with backoff, idempotency keys, DLQs, and a minimal admin UI.

Why it teaches queues: Real systems are not “consume and print.” They are workflows with retries, partial failures, and external side effects.

Core challenges you’ll face:

• Defining task lifecycle states (queued, running, retrying, dead) → state machines
• Exactly-what-happened audit trail → debuggability
• Preventing duplicate side effects → idempotency in practice

Key Concepts

• Ack/retry/DLQ patterns: RabbitMQ ack/nack and reliability docs
• Idempotence constraints: Kafka producer idempotence requirements

Difficulty: Advanced
Time estimate: 1 month+
Prerequisites: Projects 4 and 9

Real world outcome

• A usable local “job runner” where you can submit tasks and watch them retry, succeed, or land in DLQ with audit trails.

Implementation Hints

• Treat every task execution as producing an “execution record” (attempt number, result, timestamps).
• Build a minimal UI: list tasks, filter failures, requeue from DLQ.

Learning milestones

1. Retries + DLQ behave predictably → you understand robust queue-based workflows
2. You can diagnose failures from audit logs → you understand operational needs
3. You can prevent duplicate side effects → you understand correctness under retries

---

2.1.2 Project Comparison Table

Project	Difficulty	Time	Depth of Understanding	Fun Factor
In-Memory Work Queue + Backpressure	Intermediate	1–2 weeks	Medium	Medium
Durable File-Backed Queue (WAL)	Advanced	2–4 weeks	High	Medium
Mini AMQP Router (Exchanges/Bindings)	Advanced	2–3 weeks	High	High
Ack/Nack + DLQ Lab	Intermediate	1–2 weeks	High	Medium
Prefetch Tuning Playground	Intermediate	Weekend–1 week	Medium	Medium
Kafka-Lite Partitioned Log	Expert	1 month+	Very High	High
Consumer Group Coordinator	Expert	1 month+	Very High	Medium
Replication + Leader Election	Expert	1–2 months	Extremely High	High

2.1.3 Recommendation (what to start with)

Start with Project 4 (Ack/Nack + DLQ Lab), then Project 3 (Mini AMQP Router), then Project 6 (Kafka-Lite).

Why this order: 1) Project 4 makes “delivery semantics” real immediately (duplicates, redelivery, poison messages).
2) Project 3 makes RabbitMQ’s topology intuition concrete (routing is the point).
3) Project 6 builds the Kafka mental model (log + offsets), which is a fundamentally different paradigm.

If you only have weekends: do 4 → 5 → 9 (you’ll still learn the core semantics deeply).

---

2.1.4 Final Overall Project (capstone): QueueLab — A Unified Broker Playground

• File: LEARN_QUEUES_MESSAGE_BROKERS_PROJECTS.md
• Main Programming Language: Go
• Alternative Programming Languages: Rust, Java, Elixir
• Coolness Level: Level 5: Pure Magic (Super Cool)
• Business Potential: 4. The “Open Core” Infrastructure
• Difficulty: Level 4: Expert
• Knowledge Area: Messaging Systems / Distributed Systems
• Software or Tool: Local broker platform + UI (build)
• Main Book: Designing Data-Intensive Applications (Kleppmann)

What you’ll build: A local “queue laboratory” that can run in two modes:

• AMQP-mode: exchange/binding routing, push consumers, ack/nack, prefetch, DLQs
• Log-mode: topics/partitions, pull consumers by offset, consumer groups, retention/compaction
  Plus a single dashboard that shows: routing decisions, in-flight windows, consumer lag, replication status, and redelivery/DLQ rates.

Why it teaches queues: It forces you to build and compare the two dominant broker models side-by-side, with the same workload, so the differences become undeniable.

Core challenges you’ll face

• Designing a common “message lifecycle model” across both paradigms → conceptual mastery
• Failure injection suite (kill node, delay follower, crash consumer) → real-world resilience
• Operators’ dashboard with actionable metrics and timelines → production-grade thinking

Key Concepts

• RabbitMQ reliability and replicated queues concept: RabbitMQ reliability/quorum queue docs
• Kafka protocol, offsets, and idempotence constraints: Apache Kafka protocol + producer config docs

Real world outcome

• A demo environment where you can run the same “order processing” workload on both modes and generate a report:
  • throughput, p95/p99 latency
  • duplicate rate
  • time-to-recover after node failure
  • operational signals (lag, retries, DLQ)

Implementation Hints

• Treat this as “a product.” Define 3–5 standard scenarios: fanout logs, task queue, event sourcing stream, burst load, node failure.
• Make every scenario produce a saved report so the outcome is measurable and repeatable.

Learning milestones

1. Both modes pass the same workload → you understand conceptual differences
2. Failure injection produces explainable outcomes → you understand durability and recovery
3. Dashboard tells you what’s wrong without guessing → you understand operating message systems

---

Summary (Projects + Main Language)

1. In-Memory Work Queue + Backpressure Simulator — Go
2. Durable File-Backed Queue (Write-Ahead Log) — Go
3. Mini AMQP Router (Exchanges, Bindings, Routing Keys) — Go
4. Ack/Nack, Redelivery, and Poison-Message Lab — Go
5. Prefetch Tuning Playground — Go
6. Kafka-Lite (Partitioned Append-Only Log Broker) — Go
7. Consumer Group Coordinator (Rebalance + Offset Commit) — Go
8. Replication + Leader Election (3-Node Log with ISR) — Go
9. Delivery Semantics Lab (Duplicates, Idempotence, “Effectively Once”) — Go
10. Retention + Log Compaction — Go
11. Delay Queues and Scheduled Delivery — Go
12. Observability for Queues (Lag, Throughput, Redelivery) — Go
13. Load Generator + Benchmark Harness for Brokers — Go
14. Workflow Engine on Top of Queues — Go
    Capstone: QueueLab — Go