Project 3: Contract-Aware Domain Event Hub

Turn the local topic bus into a documented domain protocol with stable envelopes, server-derived topics, version compatibility, and revision-gap recovery.

Quick Reference

Attribute Value
Difficulty Level 2: Intermediate
Time Estimate 10–14 hours
Language Elixir (alternative: Erlang, F#)
Prerequisites Projects 1–2, structs and typespecs, process messaging, basic domain modeling, ExUnit
Key Topics Commands versus events, event envelopes, schema evolution, correlation, causation, stream revisions, duplicate/stale/gap handling, tenant-safe routing
Primary Tools Elixir, Registry-backed bus, ExUnit, StreamData optional
Produces A node-local domain event hub and compatibility/recovery evidence suite

1. Learning Objectives

By completing this project, you will:

  1. Distinguish a command asking one owner to act from an event announcing a committed fact to many observers.
  2. Design a stable event envelope carrying identity, type, version, tenant, subject, stream revision, time, correlation, and causation.
  3. Derive bounded topics inside the application after authorization instead of accepting arbitrary caller-provided topic strings.
  4. Define a v1/v2 compatibility matrix before evolving an event payload.
  5. Specify deterministic policies for duplicate, stale, gapped, and unsupported-version events.
  6. Use event identity for deduplication and stream revision for ordering/gap reasoning without confusing their roles.
  7. Keep UI notifications small and reload authoritative state when correctness requires it.
  8. Explain why a local event hub remains ephemeral and why a transactional outbox is needed when database commit and publication must be atomic.
  9. Test tenant isolation and protocol compatibility as contract properties rather than incidental examples.

2. Theoretical Foundation

2.1 Core Concepts

Commands, events, queries, and state

A command expresses intent: reserve an order, cancel an invoice, or send an email. It should have one responsible owner that validates rules and decides whether the action occurs. An event is a past-tense fact: order reserved, invoice cancelled, or email sent. It can have many observers, but those observers must not reinterpret the event as permission to perform the original decision again. A query asks for current information. A state document or projection represents what is currently believed to be true.

This distinction prevents a common Pub/Sub failure: broadcasting command-like messages to many subscribers and hoping exactly one performs the action. Pub/Sub is naturally suited to facts that several independent observers may react to. The authoritative aggregate or service accepts the command, commits its transition, and only then emits the event.

Stable event identity

Every logical event needs a globally unique event ID. The ID lets logs, metrics, and subscribers refer to the same fact across retries and duplicate fanout. Event identity is not ordering. Two different events have different IDs but may belong to one ordered aggregate stream.

Stream revision

A revision is a monotonic sequence scoped to a defined stream, such as tenant 42’s order 913. If a projection has applied revision 18 and receives revision 19, it can apply the next fact. If it receives revision 18 again, the event is duplicate or stale. If it receives revision 20, revision 19 is missing from its observation and it should mark itself stale or reload authoritative state. Revision only makes sense when its scope is explicit; a global integer without one sequencer is misleading.

Correlation and causation

Correlation groups messages belonging to one larger workflow or request. Causation identifies the direct predecessor that caused this event. For example, a checkout correlation can span payment and fulfillment, while order.status_changed may be caused by command reserve_order. These fields turn distributed logs into an explainable chain and help detect cycles or accidental duplicate reactions.

Schema evolution

Publishers and subscribers rarely upgrade simultaneously. An envelope therefore has an explicit schema version and a compatibility policy. Backward-compatible additions may be ignored by older consumers if their decoder is designed for it. Removing a required field, changing meaning, or reusing an event type for a different fact is breaking. Version numbers alone do not create compatibility; a matrix describing which publisher and subscriber versions interoperate does.

Topics and authorization

A topic is a routing address, not an access-control system. If a caller can request topic tenant:42:orders directly, it may forge another tenant’s address. A safe domain API accepts authenticated context and domain identifiers, verifies access, then derives topics internally. Subscriber-side filtering is too late because data has already crossed the boundary.

2.2 The Event Envelope

A useful envelope separates universal routing/diagnostic fields from event-specific data:

EventEnvelope
  event_id
  event_type
  schema_version
  occurred_at_utc
  tenant_id
  subject_type
  subject_id
  stream_name
  stream_id
  stream_revision
  correlation_id
  causation_id
  actor_reference
  trace_context
  data

Field responsibilities:

  • event_id identifies one fact and supports deduplication.
  • event_type is a stable past-tense semantic name.
  • schema_version selects compatibility rules.
  • occurred_at_utc communicates domain time for humans and cross-system records; it should not be used as the sole ordering mechanism.
  • tenant_id and subject fields scope routing and authorization.
  • stream_name, stream_id, and stream_revision define the exact ordered stream.
  • correlation_id connects the broader workflow.
  • causation_id points to the message directly responsible.
  • actor_reference is a safe identifier, not a full user record or secret.
  • trace_context connects observability systems.
  • data contains the smallest facts subscribers need.

The envelope should not contain subscriber-specific presentation data. A UI may receive order ID, revision, and status, then reload authorized detail. An audit subscriber may retain the immutable event. A notification subscriber may render a message from current templates. One stable event can therefore drive distinct reactions without pretending its payload is the entire authoritative state.

2.3 Event Processing States

For a projection tracking current revision r:

incoming event
      |
      v
supported schema? ---- no ----> quarantine + metric
      |
     yes
      |
      v
event_id already seen? -- yes --> duplicate: no reapply
      |
     no
      |
      v
revision == r + 1? ----- yes --> apply; set r = incoming
      |
     no
      |
      +-- revision <= r -------> stale/out-of-order: do not regress
      |
      +-- revision > r + 1 ----> gap: mark STALE; request reload

This state machine deliberately favors correctness over pretending every local Pub/Sub notification forms a complete log. A gap is evidence that the observer cannot prove its projection is current. Reloading authoritative state is often safer than attempting to invent missing history.

Deduplication memory must be bounded in real systems. In this project, retain a limited set for evidence and pair it with revision logic. Project 8 later introduces durable broker and consumer policies.

2.4 Compatibility Matrix

Use one intentional evolution:

  • Schema v1 contains order_id, previous_status, and current_status.
  • Schema v2 adds an optional reason_code and safe actor_reference.
  • v2 publishers retain all v1 fields during the overlap window.
  • v1 subscribers ignore unknown additions.
  • v2 subscribers accept v1 and treat new fields as absent.

  • Schema v3 in the lab represents an unsupported breaking change and is quarantined.

    Subscriber v1 event v2 event v3 event v1 ACCEPT ACCEPT* QUARANTINE v2 ACCEPT** ACCEPT QUARANTINE

    * ignores optional additions
** supplies explicit absent/default semantics

The matrix belongs in tests and release documentation. Compatibility is a contract, not a hopeful decoder behavior.

2.5 Mental Model

authenticated principal
         |
         | command: reserve_order(tenant=42, order=913)
         v
+---------------------+
| Order Owner         |
| validate invariant  |
| commit revision 18  |
+----------+----------+
           |
           | committed fact
           v
+-----------------------------+
| Domain Event Hub            |
| validate envelope           |
| derive authorized topics    |
+-----------+-----------------+
            |
     +------+-------------------------+
     |                                |
     v                                v
tenant:42:orders              tenant:42:order:913
   /       \                          |
  v         v                         v
audit     UI list                detail projection
           |                         |
           +---- reload authorized --+
                 authoritative state

Event hub = notification protocol
Order owner/state store = authority

2.6 Definitions and Key Terms

  • Command: A request for one responsible owner to attempt an action.
  • Domain event: An immutable statement that a meaningful fact occurred.
  • Envelope: Shared metadata surrounding event-specific data.
  • Aggregate: A consistency boundary that owns invariants and transitions.
  • Stream: Ordered sequence of events or revisions for a defined subject.
  • Revision: Monotonic position within one stream.
  • Event ID: Stable identity for one logical event.
  • Correlation ID: Identity grouping a larger workflow.
  • Causation ID: Identity of the direct message that produced an event.
  • Schema version: Explicit representation version with documented compatibility.
  • Quarantine: Isolating an unsupported or malformed event without crashing the subscriber.
  • Projection: Read model derived from events or authoritative state.
  • Server-derived topic: Routing name constructed only after trusted authorization and validation.

2.7 Minimal Concrete Example

Envelope outline:

event_id: evt_01J...
event_type: order.status_changed
schema_version: 2
occurred_at_utc: 2026-07-15T14:32:10Z
tenant_id: 42
subject: [order, 913]
stream: [order, 913, revision=18]
correlation_id: checkout_7K
causation_id: cmd_9P
actor_reference: user_81
data:
  previous_status: reserved
  current_status: packed
  reason_code: warehouse_confirmed

Topic derivation outline:

authorize principal for tenant and order
derive:
  tenant:42:orders
  tenant:42:order:913
publish validated envelope

Subscriber outline:

decode supported schema
reject duplicate event_id
compare stream revision
apply next revision or mark projection stale
reload authoritative record on gap

2.8 Common Misconceptions

  • “An event is any message.” Commands, replies, queries, and events have different ownership and semantics.
  • “A timestamp orders events.” Clocks and concurrency make timestamps insufficient for aggregate order.
  • “Event ID and revision are interchangeable.” ID identifies; revision positions within a stream.
  • “Adding a version field solves schema evolution.” Compatibility requires field semantics, decoder behavior, and rollout tests.
  • “Subscribers can filter unauthorized messages.” Receiving the message is already a data leak.
  • “A rich event payload eliminates reads.” It can become stale, oversized, and overexposed; notification-plus-reload is often safer.
  • “Local Pub/Sub is an event store.” It has no durable history or offline replay.
  • “Unknown events should crash loudly.” One malformed or future event should be quarantined and observable without taking down unrelated handling.

2.9 Check Your Understanding

  1. Why is reserve_order a command and order_reserved an event?
  2. Why carry both event ID and stream revision?
  3. What should a projection do with revision 20 when it last applied 18?
  4. Why derive topics after authorization?
  5. What makes a v2 addition backward compatible?
  6. When should a UI reload authoritative state?
  7. Which crash window remains between state commit and local publication?

Answers

  1. The first asks one owner to decide; the second reports a completed fact.
  2. Event ID detects the same logical fact; revision detects position and gaps in one stream.
  3. Mark itself stale and reload or recover missing history; do not silently apply as consecutive.
  4. It prevents callers from forging another principal’s routing address.
  5. v1-required fields and meanings remain, additions are optional, and old decoders ignore them safely.
  6. On a revision gap, reconnect, suspected missed notification, or when the payload is intentionally minimal.
  7. The owner can commit and crash before publishing; the transactional outbox project addresses it.

2.10 Historical Context and Evolution

Early message-oriented middleware often emphasized transport-level messages and integration channels. Domain-driven design sharpened the distinction between commands and domain events, while event sourcing made stream identity and version explicit. Modern real-time applications frequently combine a durable source of truth with ephemeral Pub/Sub notifications. This project adopts that hybrid view: contracts must be strong even when the transport is intentionally volatile.

2.11 Why This Matters

The most expensive Pub/Sub bugs usually come from vague contracts rather than failed send syntax. A subscriber crashes because a field changed meaning. A projection silently skips a revision. A tenant receives metadata from a global topic. A UI treats a notification payload as canonical state. This project forces those decisions into envelopes, matrices, state machines, and tests before Phoenix makes broadcast mechanically easy.

2.12 References

  • “Enterprise Integration Patterns” by Gregor Hohpe and Bobby Woolf — Message Construction, Correlation Identifier, Publish-Subscribe Channel.
  • “Designing Data-Intensive Applications” by Martin Kleppmann — encoding evolution, streams, and distributed data.
  • “Domain-Driven Design” by Eric Evans — aggregates, domain events, and bounded contexts.
  • OWASP WebSocket Security Cheat Sheet — authorization and message validation principles.
  • Elixir Registry — local routing primitive used by the hub.

3. Project Specification

3.1 What You Will Build

Build an order-focused local event hub on top of Project 2’s Registry bus. It exposes domain functions rather than arbitrary topics. An order owner accepts commands, validates a simple status transition, commits a new in-memory revision for the lab, constructs a validated envelope, and asks the hub to publish on server-derived topics.

Three subscriber fixtures demonstrate different contracts:

  • A v2 UI projection reloads authorized order state.
  • A v1 audit subscriber decodes both supported versions during rollout.
  • A quarantine observer records unsupported or malformed events without crashing the bus.

The demo injects duplicates, stale revisions, a revision gap, an unsupported schema, and a cross-tenant topic-forging attempt.

3.2 Functional Requirements

  1. Command ownership: One order-owner process decides valid status changes.
  2. Post-commit publication: Events represent accepted, committed transitions.
  3. Stable envelope: Every field has validation and documented meaning.
  4. Past-tense event types: Event names describe facts, not requested actions.
  5. Authorized topic derivation: External callers cannot submit raw topic strings.
  6. Bounded fanout: Publish to tenant collection and aggregate detail topics.
  7. Version matrix: v1 and v2 interoperability is explicit and tested.
  8. Unknown-version policy: Unsupported versions are quarantined and counted.
  9. Duplicate policy: Repeated event IDs do not reapply effects.
  10. Stale policy: Older revisions do not regress projections.
  11. Gap policy: Non-consecutive future revisions mark projections stale and request reload.
  12. Minimal UI notification: UI can reload authoritative state with tenant and subject identity.
  13. Correlation chain: Command, event, and subscriber logs retain correlation and causation.
  14. Sender exclusion option: Internal publication may omit a PID that already applied the local transition.
  15. Ephemeral disclaimer: No claim that the hub makes state commit and publish atomic.

3.3 Non-Functional Requirements

  • Security: Tenant and subject topics are derived only from validated, authorized identifiers.
  • Compatibility: Supported combinations are covered by contract tests.
  • Observability: Unknown schemas, duplicates, stale events, and gaps have bounded metrics.
  • Privacy: Envelopes contain references and minimum facts, not user records or secrets.
  • Resilience: One bad envelope cannot crash unrelated subscribers.
  • Clarity: Event and topic naming is documented in a protocol catalog.
  • Maintainability: Subscriber behavior depends on envelope contracts, not producer implementation details.

3.4 Example Usage and Exact Output

$ mix hub.demo
command id=cmd_9P type=reserve_order tenant=42 order=913
authorization principal=user_81 tenant=42 result=allow
state commit order=913 revision=18 status=packed
event id=evt_01J type=order.status_changed schema=2 correlation=checkout_7K
topics derived=[tenant:42:orders,tenant:42:order:913]
subscriber=ui_v2 revision_before=17 revision_in=18 action=reload result=ok
subscriber=audit_v1 schema_in=2 action=compat_decode result=ok
inject event_id=evt_01J revision=18 -> duplicate action=ignore
inject event_id=evt_stale revision=17 -> stale action=ignore
inject event_id=evt_gap revision=20 after=18 -> gap missing=[19] projection=STALE action=reload
inject schema=3 -> quarantine reason=unsupported_version count=1
forge principal=tenant:99 requested_tenant=42 -> denied deliveries=0 audit=recorded

3.5 Real World Outcome

The terminal tells one complete causal story. A command reaches one owner, authorization succeeds, the order changes to revision 18, and only then does a stable event appear. The hub prints the two topics it derived. The v2 UI fixture reloads the order, while the v1 audit fixture proves the rollout contract by decoding the v2 event without losing v1-required meaning.

The second half is deliberately adversarial. Replaying the same event ID produces duplicate/ignore. An unknown older event produces stale/ignore. Jumping to revision 20 turns the projection visibly STALE and triggers a reload rather than silently applying an incomplete sequence. An unsupported schema is quarantined without a subscriber crash. A tenant-forging attempt results in zero deliveries and an audit record.

4. Solution Architecture

4.1 High-Level Design

+---------------- trusted command boundary ----------------+
| principal + domain identifiers + command                |
+--------------------------+-------------------------------+
                           |
                           v
                +----------------------+
                | Authorization policy |
                +----------+-----------+
                           |
                           v
                +----------------------+
                | Order owner          |
                | validate transition  |
                | commit revision      |
                +----------+-----------+
                           |
                     event envelope
                           |
                           v
                +----------------------+
                | Domain Event Hub     |
                | validate + derive    |
                +----------+-----------+
                           |
              +------------+-------------+
              | Project 2 local topic bus|
              +-----+-------------+------+
                    |             |
                    v             v
                UI projection   audit subscriber
                gap -> reload   version adapter
                    |
                    v
             authoritative order

4.2 Key Components

Component Responsibility Key Decisions
Command boundary Accept principal, IDs, and intent No raw topics or prebuilt events from untrusted caller
Authorization policy Verify tenant and subject access Runs before topic derivation
Order owner Enforce transitions and increment revision One owner for command decision
Envelope factory Construct and validate universal fields Stable event catalog
Topic derivation Produce collection and detail topics Bounded server-side rules
Event hub Publish supported envelopes and instrument No durable claim
Compatibility decoder Normalize supported schema versions Matrix-driven overlap
Projection fixture Apply next revisions and reload on gaps Authority remains state owner/store
Quarantine observer Isolate unsupported/malformed events Bounded evidence and metrics

4.3 Data Structures

OrderState:
  tenant_id
  order_id
  status
  revision

CommandContext:
  command_id
  principal_id
  tenant_id
  correlation_id
  requested_at

EventEnvelope:
  universal fields described in Section 2

ProjectionState:
  tenant_id
  order_id
  last_revision
  status: CURRENT | STALE | RELOADING
  recently_seen_event_ids: bounded set

QuarantineRecord:
  event_id
  event_type
  schema_version
  safe_reason
  observed_at
  payload_digest

Never place raw secrets, full principal data, or unbounded payload dumps in quarantine records.

4.4 Algorithm Overview

Command-to-event path

  1. Validate command shape and authenticated context.
  2. Authorize tenant and order access.
  3. Route the command to the one order owner.
  4. Validate status transition.
  5. Commit new state and revision.
  6. Create the event envelope with command ID as causation.
  7. Derive authorized topics.
  8. Publish the event and report routing evidence.

Subscriber processing

  1. Validate envelope and supported type.
  2. Normalize v1 or v2 into one internal representation.
  3. Check bounded recent event IDs.
  4. Compare revision with local projection revision.
  5. Apply, ignore, quarantine, or mark stale according to the state machine.
  6. Reload authorized state after a gap.
  7. Emit bounded metrics and a receipt for the lab.

Complexity

  • Envelope validation: O(number of envelope fields plus bounded payload fields).
  • Topic derivation: O(1) for the fixed topic set.
  • Fanout: O(matching local subscribers).
  • Bounded dedup lookup: approximately O(1) with an indexed recent-ID structure.
  • Gap recovery: cost of authoritative reload, intentionally outside Pub/Sub fanout.

4.5 Contract Invariants

Invariant Enforcement
Commands have one owner Command routing API
Events describe committed facts Owner emits only after transition commit
Topic cannot be forged by caller Domain API accepts IDs, not topic
Revision never regresses projection Subscriber state machine
Unknown version does not crash subscriber Quarantine path
UI can recover after missed notification Authoritative reload
Commit and local publish are not called atomic Documentation and failure test

5. Implementation Guide

5.1 Development Environment Setup

$ elixir --version
Erlang/OTP 27 [...]
Elixir 1.18.[...]

$ mix new domain_event_hub --sup
* creating mix.exs
* creating lib/...
* creating test/...

StreamData is optional for property-based envelope and revision-state tests. Complete the core suite with ExUnit first.

5.2 Suggested Project Structure

domain_event_hub/
├── lib/
│   ├── domain_event_hub/application.ex
│   └── domain_event_hub/
│       ├── orders/owner.ex
│       ├── orders/policy.ex
│       ├── events/envelope.ex
│       ├── events/catalog.ex
│       ├── events/compatibility.ex
│       ├── events/topic_derivation.ex
│       ├── events/hub.ex
│       ├── projections/order_status.ex
│       └── quarantine.ex
├── lab/
│   └── order_scenario.exs
├── test/
│   ├── envelope_contract_test.exs
│   ├── compatibility_test.exs
│   ├── revision_policy_test.exs
│   ├── tenant_isolation_test.exs
│   └── command_event_boundary_test.exs
└── README.md

5.3 The Core Question You Are Answering

“What must travel with an event so independent subscribers can remain correct across versions, restarts, gaps, and tenants?”

The answer is not “the whole database row.” It is a stable, minimal contract plus enough identity and revision context for subscribers to detect when they cannot safely proceed.

5.4 Concepts You Must Understand First

  1. Command versus event
    • Who decides whether an action occurs?
    • Why is an event named in past tense?
    • Book: “Enterprise Integration Patterns,” Ch. 4 Message Construction.
  2. Identity and ordering scope
    • What does event ID identify?
    • What exact aggregate does revision order?
    • Book: “Designing Data-Intensive Applications,” streams and encoding.
  3. Correlation and causation
    • Which workflow does this event belong to?
    • Which command or event directly produced it?
    • Book: “Enterprise Integration Patterns,” Correlation Identifier.
  4. Schema compatibility
    • Which v1-required meanings remain in v2?
    • How does each subscriber handle absent additions?
    • Book: “Designing Data-Intensive Applications,” encoding evolution.
  5. Authorization before routing
    • Why is subscriber filtering too late?
    • Which inputs can an untrusted caller control?
    • Reference: OWASP WebSocket Security Cheat Sheet.
  6. Ephemeral versus authoritative state
    • How does a projection recover missed notifications?
    • Which component remains the source of truth?

5.5 Questions to Guide Your Design

  1. Which envelope fields are required for every event?
  2. What is the exact scope of stream revision?
  3. Which fields are safe to expose to every subscriber on a topic?
  4. What is the v1/v2 compatibility matrix?
  5. How large is the schema overlap window?
  6. What is the policy for duplicate, stale, gap, malformed, and unknown-version events?
  7. How will dedup memory stay bounded?
  8. Which topics are derived for collection and aggregate views?
  9. Which subscriber should reload rather than apply the payload?
  10. What happens if the owner commits and crashes before publish?

5.6 Thinking Exercise

Classify each item:

  • send_email
  • email_sent
  • order_status
  • get_order
  • reserve_order
  • order_reserved

Use command, event, state/document, or query. For every command, name the one owner. For every event, write a past-tense sentence that remains true forever.

Then design one unsafe global topic and replace it with server-derived tenant and aggregate topics. Finally, trace revisions 10, 11, duplicate 11, stale 9, and gap 14 through the projection state machine.

5.7 The Interview Questions They Will Ask

  1. How is an event different from a command?
  2. Why carry both event ID and stream revision?
  3. How do you roll out a new event schema safely?
  4. What should a subscriber do after detecting a revision gap?
  5. Why is a topic name not an authorization boundary?
  6. When is notification-plus-reload better than a rich event payload?

5.8 Hints in Layers

Hint 1: Start with one aggregate

Use one order and one monotonic revision. Avoid a fake global order.

Hint 2: Design the envelope before handlers

Write field meanings and validation rules. Give two example events and one invalid event.

Hint 3: Write the matrix before v2

Specify old-subscriber/new-publisher and new-subscriber/old-publisher behavior before changing the payload.

Hint 4: Reload on uncertainty

When the projection cannot prove continuity, mark it stale and fetch authoritative state.

5.9 Books That Will Help

Topic Book Chapter or Section
Message construction “Enterprise Integration Patterns” Ch. 4
Publish-subscribe channels “Enterprise Integration Patterns” Ch. 3
Schema evolution “Designing Data-Intensive Applications” Encoding and Evolution
Streams and ordering “Designing Data-Intensive Applications” Stream Processing
Aggregate boundaries “Domain-Driven Design” Aggregates

5.10 Implementation Phases

Phase 1: Domain Boundary — 3 hours

  • Define command ownership and order transition rules.
  • Define state and revision scope.
  • Produce a committed transition transcript without Pub/Sub.

Checkpoint: An invalid transition produces no event; a valid transition increments exactly one revision.

Phase 2: Envelope and Catalog — 3 hours

  • Define required fields and validation.
  • Create v1 and v2 catalog entries.
  • Document correlation and causation.

Checkpoint: Contract tests accept supported examples and reject malformed envelopes.

Phase 3: Authorized Topic Derivation — 2–3 hours

  • Accept principal and domain IDs.
  • Authorize before deriving topics.
  • Route to collection and detail subscribers.

Checkpoint: A cross-tenant forging attempt produces zero deliveries.

Phase 4: Compatibility and Revision Policies — 4 hours

  • Implement the compatibility normalizer.
  • Add duplicate, stale, gap, and unknown-version state transitions.
  • Add bounded quarantine evidence.

Checkpoint: The exact adversarial transcript matches the specification.

Phase 5: Recovery and Documentation — 2–4 hours

  • Add projection reload after gap.
  • Document the commit/publish crash window.
  • Complete event catalog and rollout matrix.

Checkpoint: Restart and gap scenarios recover from authority without Pub/Sub replay.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
API input Raw topic, principal plus IDs Principal plus IDs Prevents forged routing
Event naming Imperative, past tense Past tense Describes immutable fact
Ordering Timestamp, scoped revision Scoped revision Detects gaps deterministically
UI payload Full record, identity/revision Identity/revision plus minimal fact Reduces leakage and staleness
Unknown version Crash, silently ignore, quarantine Quarantine and metric Preserves availability and evidence
Gap handling Apply anyway, mark stale/reload Mark stale/reload Avoids incomplete projection
Commit/publish atomicity Pretend atomic, document boundary Document and defer to outbox Truthful reliability model

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Contract Validate envelope fields and meanings Missing event ID, invalid revision
Compatibility Validate v1/v2 matrix Old subscriber reads v2 addition
State machine Validate revision policies next, duplicate, stale, gap
Domain Validate command/event boundary Rejected command emits no event
Security Validate topic derivation Tenant forging gets zero delivery
Recovery Validate reload behavior Gap marks stale then current
Resilience Isolate bad events Unsupported version quarantined

6.2 Critical Test Cases

  1. Accepted command commits state before event publication.
  2. Rejected command emits no event.
  3. Every event includes required identity, tenant, stream, version, and causality fields.
  4. v1 subscriber accepts supported v1 and v2 envelopes.
  5. v2 subscriber accepts v1 with explicit absent semantics.
  6. Duplicate event ID does not reapply.
  7. Unknown older revision does not regress projection.
  8. Consecutive revision applies and advances.
  9. Gap marks projection stale and triggers one reload.
  10. Unsupported schema enters quarantine without subscriber exit.
  11. Unauthorized tenant/subject input derives no topic and sends no event.
  12. Subscriber restart reloads current state rather than expecting replay.
  13. Simulated crash after commit but before publish demonstrates the documented reliability gap.

6.3 Test Data

stream:
  tenant_id: 42
  stream_name: order
  stream_id: 913
  current_revision: 18

accepted sequence:
  evt-19 revision=19 schema=1
  evt-20 revision=20 schema=2

adversarial:
  evt-20 again       -> duplicate
  evt-stale rev=17   -> stale
  evt-gap rev=23     -> gap; missing 21..22
  evt-future schema=3 -> quarantine
  tenant=99 principal requesting tenant=42 -> deny

compatibility:
  subscriber v1 + event v1 -> accept
  subscriber v1 + event v2 -> accept optional additions
  subscriber v2 + event v1 -> accept with absent additions
  any supported subscriber + event v3 -> quarantine

6.4 Property-Oriented Tests

Optional StreamData properties:

  • Projection revision never decreases.
  • Applying the same event ID twice changes state at most once.
  • Any positive gap changes status away from CURRENT until reload.
  • Topic derivation never emits a tenant different from authorized context.
  • Normalizing supported event versions preserves required v1 semantic fields.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Commands broadcast as events Multiple subscribers perform one action Route command to one owner; publish fact afterward
Timestamp used as order Concurrent updates reorder unpredictably Use scoped stream revision
Version field without matrix Old subscriber crashes after rollout Test explicit publisher/subscriber combinations
Global tenant topic Cross-tenant metadata leak Authorize then derive scoped topic
Full record in event Stale or excessive data exposure Publish minimal fact and reload
Gap silently applied Projection appears current but is incomplete Mark stale and reload
Unknown schema crashes process One future event stops subscriber Quarantine safely
Event log includes full bad payload Secrets leak into diagnostics Store digest and safe metadata
Local hub called durable Missed event after restart surprises users Document ephemeral boundary

7.2 Debugging Strategies

  • Reconstruct one event’s causal chain using command, correlation, causation, and event IDs.
  • Print stream scope and current/incoming revisions on every state-machine decision.
  • Run the compatibility matrix as a focused suite before deployment.
  • Compare derived topics with authenticated tenant context.
  • Inspect quarantine reason and payload digest, not unrestricted payload.
  • When a UI looks stale, compare projection revision with authoritative record before blaming rendering.

7.3 Performance Traps

Large envelopes increase copying and serialization cost. Unbounded recent-event ID sets leak memory. High-cardinality metrics using tenant, subject, or event ID labels overwhelm observability backends. Keep topic derivation fixed and cheap, envelope data minimal, dedup bounded, and detailed identifiers in structured logs or traces rather than metric labels.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add one more event type with a complete catalog entry.
  • Generate human-readable documentation from the event catalog representation.
  • Add an audit transcript grouped by correlation ID.

8.2 Intermediate Extensions

  • Add property-based state-machine tests for arbitrary revision sequences.
  • Model a deprecation window for schema v1 and define exit criteria.
  • Add redacted quarantine export for offline analysis.

8.3 Advanced Extensions

  • Design an anti-corruption adapter translating an external event into the internal contract.
  • Compare notification-plus-reload with event-carried state transfer for one use case.
  • Write the transactional outbox design that closes the commit/publish crash window, but defer implementation to Project 8.

9. Real-World Connections

9.1 Industry Applications

  • Order and fulfillment systems: Multiple projections react to committed status facts.
  • SaaS dashboards: Tenant-scoped notifications trigger authorized reloads.
  • Audit trails: Event identity and causality explain who changed what and why.
  • Workflow orchestration: Correlation connects independent steps without conflating commands and events.
  • Rolling deployments: Version matrices let old and new subscribers overlap safely.
  • Cache invalidation: Identity and revision tell caches what to refresh and whether they missed changes.
  • Commanded — CQRS/ES concepts in Elixir; useful for comparison, not required.
  • Broadway — later durable ingestion and processing boundary.
  • Phoenix.PubSub — framework transport adopted in Project 4.
  • CloudEvents specification — industry envelope vocabulary for external interoperability.

9.3 Interview Relevance

This project supports architecture interviews about event-driven systems, schema rollout, data ownership, tenant isolation, and missed events. You can explain concrete policies instead of using “eventual consistency” as a catch-all phrase.

10. Resources

10.1 Essential Reading

  • “Enterprise Integration Patterns” — Ch. 3–4.
  • “Designing Data-Intensive Applications” — Encoding and Evolution; Stream Processing.
  • “Domain-Driven Design” — Aggregates and domain concepts.
  • CloudEvents — external event-envelope standard and ecosystem.
  • OWASP WebSocket Security Cheat Sheet — authorization and validation guidance.

10.2 Video and Talk Topics

  • Domain modeling talks on commands versus events.
  • Conference talks on schema evolution without lockstep deployments.
  • Case studies on multi-tenant real-time authorization.
  • Event-sourcing talks should be treated as a distinct architecture, not assumed by this local notification hub.

10.3 Tools and Documentation

  • ExUnit: Compatibility and state-machine contract tests.
  • StreamData, optional: Generate revision sequences and envelope variations.
  • Typespecs and structs: Make expected shapes visible while retaining runtime validation.
  • Structured logging: Correlate event, command, tenant, and revision safely.
  • Previous: Project 2 — Registry Topic Bus provides the node-local routing primitive.
  • Next: Project 4 — Phoenix.PubSub Notification Hub replaces the custom bus while preserving this contract layer.
  • Project 9 — Durable Notification Outbox closes the database commit/publication reliability gap.

11. Self-Assessment Checklist

11.1 Understanding

  • I can distinguish command, event, query, and state.
  • I can explain every envelope field.
  • I can distinguish event identity from stream revision.
  • I can trace correlation and causation.
  • I can explain the v1/v2 compatibility matrix.
  • I can state policies for duplicate, stale, gap, and unknown version.
  • I can explain why topics are not authorization.
  • I can identify the commit/publish crash window.

11.2 Implementation

  • One owner validates and commits order commands.
  • Events are emitted only for accepted transitions.
  • Topics are server-derived after authorization.
  • Envelopes are validated against a catalog.
  • Supported schema combinations have tests.
  • Projection revision never regresses.
  • Gaps mark stale and trigger reload.
  • Unsupported schemas are quarantined safely.
  • Metrics use bounded labels.

11.3 Growth

  • I can review an event contract for privacy and compatibility.
  • I can explain notification-plus-reload tradeoffs.
  • I can propose a safe schema deprecation plan.
  • I documented one failure that Pub/Sub transport alone cannot solve.

12. Submission / Completion Criteria

Minimum Viable Completion

  • A committed order transition emits one validated event.
  • The event reaches collection and detail subscribers through derived topics.
  • Event ID, schema version, tenant, stream, and revision are present.

Full Completion

  • Command/event ownership is explicit.
  • v1/v2 compatibility matrix passes.
  • Duplicate, stale, gap, and unsupported-version policies are observable.
  • Gap recovery reloads authoritative state.
  • Tenant-forging attempts produce zero delivery and safe audit evidence.
  • Documentation states the local, volatile, non-transactional nature of publication.

Excellence

  • Property-based tests preserve revision and tenant invariants.
  • The event catalog generates a compatibility report.
  • A failure-timeline document shows the exact commit/publish window and previews the outbox solution without implementing it prematurely.

Return to the Pub/Sub in Elixir mastery guide.

See the expanded project index.