Project 3: Message Bus + Shared Memory
Build a structured message bus and a shared memory ledger with validation gates.
Quick Reference
| Attribute | Value |
|---|---|
| Difficulty | Level 3 |
| Time Estimate | 12-18 hours |
| Language | Python (Alternatives: TypeScript, Go) |
| Prerequisites | Messaging basics, data modeling |
| Key Topics | Protocols, memory consistency, validation |
1. Learning Objectives
By completing this project, you will:
- Design a message schema for agent communication.
- Implement a shared memory store with versioning.
- Add review-before-commit validation gates.
- Detect and log conflicting updates.
2. Theoretical Foundation
2.1 Core Concepts
- Message Protocols: Structured messages reduce ambiguity.
- Shared State: Canonical facts must be validated.
- Consistency: Avoid contradictions through versioning and review.
2.2 Why This Matters
Without strict protocols, agents misinterpret each other and corrupt shared memory. This project makes information flow explicit and auditable.
2.3 Historical Context / Background
Message buses and append-only logs are proven patterns in distributed systems. They map directly to multi-agent communication and memory.
2.4 Common Misconceptions
- “Shared memory is just a chat log.” It must be validated and structured.
- “Schemas slow things down.” They reduce downstream errors.
3. Project Specification
3.1 What You Will Build
A message bus where agents exchange structured messages, plus a shared memory ledger with versioning and validation.
3.2 Functional Requirements
- Message Schema: Required fields (sender, task, evidence, request).
- Shared Memory: Append-only entries with version IDs.
- Validation Gate: Reviewer approves before commit.
- Conflict Detector: Flags contradictory entries.
3.3 Non-Functional Requirements
- Traceability: Every entry is linked to a message.
- Auditability: Log all updates and conflicts.
- Reliability: Reject malformed messages.
3.4 Example Usage / Output
$ run-message-bus --task "Summarize X"
[Bus] message accepted from Researcher (T-002)
[Memory] entry staged, awaiting review
[Reviewer] approved entry v3
3.5 Real World Outcome
You can inspect the shared memory ledger and see exactly which agent added each fact, when it was reviewed, and which evidence supports it.
4. Solution Architecture
4.1 High-Level Design
Agent -> Message Bus -> Validator -> Shared Memory -> Audit Log
4.2 Key Components
| Component | Responsibility | Key Decisions |
|---|---|---|
| Message Bus | Accept structured messages | Schema enforcement |
| Validator | Check evidence and format | Review-before-commit |
| Shared Memory | Store validated facts | Append-only log |
| Conflict Detector | Identify contradictions | Rule-based checks |
4.3 Data Structures
Pseudo-structures:
STRUCT Message:
sender_role
task_id
summary
evidence_links
request_type
STRUCT MemoryEntry:
entry_id
task_id
content
evidence_links
status
4.4 Algorithm Overview
Validation Pipeline
- Accept message.
- Validate schema.
- Stage entry for review.
- Approve or reject.
Complexity Analysis:
- Time: O(M) messages
- Space: O(E) entries
5. Implementation Guide
5.1 Development Environment Setup
Use a local file-based log or lightweight database to store memory entries.
5.2 Project Structure
project-root/
├── bus/
├── memory/
├── validation/
├── audit/
└── logs/
5.3 The Core Question You’re Answering
“How do multiple agents share information without corrupting the system’s memory?”
5.4 Concepts You Must Understand First
- Message schemas
- What fields are mandatory?
- Book Reference: “Patterns of Enterprise Application Architecture” - Ch. 10
- Consistency
- How do you handle conflicting facts?
- Book Reference: “Designing Data-Intensive Applications” - Ch. 5
5.5 Questions to Guide Your Design
- Validation rules
- What makes an entry valid?
- Conflict detection
- What triggers a contradiction warning?
5.6 Thinking Exercise
Simulate two agents submitting contradictory facts and decide which policy resolves it.
5.7 The Interview Questions They’ll Ask
- “How do you design message protocols for agents?”
- “Why use append-only memory?”
- “How do you detect contradictions?”
- “How do you validate evidence?”
- “What is the difference between staging and committing?”
5.8 Hints in Layers
Hint 1: Start with schema Define message fields and validate them.
Hint 2: Add staging Store entries as pending before review.
Hint 3: Add approval Require a reviewer to commit.
Hint 4: Add audit log Record every decision.
5.9 Books That Will Help
| Topic | Book | Chapter |
|---|---|---|
| Messaging patterns | “Patterns of Enterprise Application Architecture” | Ch. 10 |
| Consistency | “Designing Data-Intensive Applications” | Ch. 5 |
5.10 Implementation Phases
Phase 1: Foundation (3-4 hours)
Goals:
- Define schemas
- Implement basic message bus
Tasks:
- Create message schema
- Accept and log messages
Checkpoint: Schema validation rejects malformed input.
Phase 2: Core Functionality (4-6 hours)
Goals:
- Add shared memory store
- Add review workflow
Tasks:
- Stage entries
- Approve entries
Checkpoint: Entries only appear after review.
Phase 3: Polish & Edge Cases (3-4 hours)
Goals:
- Add conflict detection
- Add audit reports
Tasks:
- Flag contradictions
- Report conflicts
Checkpoint: Conflicts appear in logs.
5.11 Key Implementation Decisions
| Decision | Options | Recommendation | Rationale |
|---|---|---|---|
| Memory model | Mutable vs append-only | Append-only | Auditability |
| Validation | Immediate vs staged | Staged | Safer and reviewable |
6. Testing Strategy
6.1 Test Categories
| Category | Purpose | Examples |
|---|---|---|
| Unit Tests | Schema checks | Missing fields fail |
| Integration Tests | Review pipeline | Entry staged then approved |
| Edge Case Tests | Contradictions | Conflicts flagged |
6.2 Critical Test Cases
- Missing evidence links should reject entry.
- Conflicting facts should be flagged.
- Unauthorized commit should fail.
6.3 Test Data
Entry A: "Fact X is true"
Entry B: "Fact X is false"
Expected: conflict flag
7. Common Pitfalls & Debugging
7.1 Frequent Mistakes
| Pitfall | Symptom | Solution |
|---|---|---|
| No review gate | Bad facts committed | Require approval |
| Overwriting entries | Lost history | Use append-only log |
| Unstructured messages | Misrouted tasks | Enforce schema |
7.2 Debugging Strategies
- Trace entry provenance by task ID.
- Compare conflicting entries side-by-side.
7.3 Performance Traps
- Excessive validation can slow throughput.
8. Extensions & Challenges
8.1 Beginner Extensions
- Add message priorities.
- Add simple conflict tags.
8.2 Intermediate Extensions
- Add a reviewer queue.
- Add memory snapshots.
8.3 Advanced Extensions
- Build a knowledge graph view.
- Add automated contradiction repair.
9. Real-World Connections
9.1 Industry Applications
- Knowledge base curation
- Multi-agent data validation pipelines
9.2 Related Open Source Projects
- Redis Streams (message log concepts)
- LangGraph (agent messaging)
9.3 Interview Relevance
- Message schemas and consistency models are common system design topics.
10. Resources
10.1 Essential Reading
- “Designing Data-Intensive Applications” - consistency and logs
- “Patterns of Enterprise Application Architecture” - messaging
10.2 Tools & Documentation
- FIPA ACL Specification: http://www.fipa.org/specs/fipa00061/
10.3 Related Projects in This Series
- Previous Project: Planning Board (P02)
- Next Project: Negotiation & Conflict Lab (P04)
11. Self-Assessment Checklist
11.1 Understanding
- I can design a message schema and explain why it matters
11.2 Implementation
- Shared memory is append-only and validated
11.3 Growth
- I can reason about trade-offs in consistency vs speed
12. Submission / Completion Criteria
Minimum Viable Completion:
- Message schema and shared memory working
Full Completion:
- Review gate and conflict detection added
Excellence (Going Above & Beyond):
- Knowledge graph visualization added
This guide was generated from LEARN_COMPLEX_MULTI_AGENT_SYSTEMS_DEEP_DIVE.md. For the complete learning path, see the README.