Project 8: Protocol Buffers Message Queue

A simple message queue server using Protocol Buffers for serialization. Clients can publish messages to topics and subscribe to receive them. Messages are persisted to disk for durability.

Quick Reference

Attribute Value
Primary Language C++
Alternative Languages Go, Rust, Java
Difficulty Level 3: Advanced
Time Estimate 2 weeks
Knowledge Area Binary Serialization, Message Queues, Pub/Sub
Tooling Redis/RabbitMQ alternative
Prerequisites Projects 3 and 7, understanding of binary formats

What You Will Build

A simple message queue server using Protocol Buffers for serialization. Clients can publish messages to topics and subscribe to receive them. Messages are persisted to disk for durability.

Why It Matters

This project builds core skills that appear repeatedly in real-world systems and tooling.

Core Challenges

  • Protocol Buffer schema design → maps to .proto files and protoc compiler
  • Efficient binary framing → maps to varint length prefixes
  • Topic routing → maps to subscriber management per topic
  • Message persistence → maps to append-only log files

Key Concepts

  • Protocol Buffers: https://protobuf.dev/programming-guides/encoding/
  • Message Queue Semantics: “Designing Data-Intensive Applications” Chapter 11 - Kleppmann
  • Pub/Sub Pattern: “Enterprise Integration Patterns” - Hohpe & Woolf
  • Append-Only Logs: “Designing Data-Intensive Applications” Chapter 3 - Kleppmann

Real-World Outcome

$ ./msgqueue_server 9000
Message queue server listening on port 9000

# Terminal 2: Subscriber
$ ./msgqueue_client subscribe notifications
Subscribed to 'notifications'
Waiting for messages...
[notifications] User "alice" logged in at 2024-01-15T10:30:00Z
[notifications] New order #1234 received

# Terminal 3: Publisher
$ ./msgqueue_client publish notifications '{"event": "login", "user": "alice"}'
Published to 'notifications'

# Server stats
$ ./msgqueue_client stats
Topics: 3
  notifications: 2 subscribers, 145 messages
  orders: 1 subscriber, 89 messages
  logs: 0 subscribers, 10,234 messages
Messages/sec: 1,245

Implementation Guide

  1. Reproduce the simplest happy-path scenario.
  2. Build the smallest working version of the core feature.
  3. Add input validation and error handling.
  4. Add instrumentation/logging to confirm behavior.
  5. Refactor into clean modules with tests.

Milestones

  • Milestone 1: Minimal working program that runs end-to-end.
  • Milestone 2: Correct outputs for typical inputs.
  • Milestone 3: Robust handling of edge cases.
  • Milestone 4: Clean structure and documented usage.

Validation Checklist

  • Output matches the real-world outcome example
  • Handles invalid inputs safely
  • Provides clear errors and exit codes
  • Repeatable results across runs

References

  • Main guide: LEARN_CPP_NETWORK_PROGRAMMING.md
  • “Designing Data-Intensive Applications” by Martin Kleppmann