Project 12: Multi-Tenant Real-Time Event Platform

Build and chaos-certify a three-node incident and fleet operations platform with tenant-safe LiveView boards, Presence, bounded telemetry, transactional domain events, and durable broker integrations.

Quick Reference

Attribute	Value
Difficulty	Level 5 — Master
Time Estimate	45–60 hours
Language	Elixir (alternatives: Erlang services, TypeScript clients)
Prerequisites	Projects 1–11 or equivalent Phoenix, OTP, PubSub, Presence, Ecto, Oban/RabbitMQ, Broadway, distributed-systems, and SRE knowledge
Key Topics	Four-plane architecture, tenant isolation, event identity, durable/transient boundaries, overload policy, partitions, chaos engineering, SLO evidence

1. Learning Objectives

By completing this capstone, you will be able to:

Decompose a realtime platform into authoritative state, transient notification, durable work, and operational control planes.
Assign an explicit loss, ordering, duplication, authorization, capacity, and recovery contract to every event path.
Preserve event identity and aggregate revision across command, transaction, outbox, broker, projection, PubSub, LiveView, and audit evidence.
Enforce tenant isolation at HTTP, connected LiveView, socket, topic, database, job, broker, metrics, logs, cache, and operator-tool boundaries.
Combine Phoenix.PubSub and Presence with durable state without treating either as a database or consensus service.
Keep high-rate telemetry bounded through sampling, coalescing, demand, and explicit degraded modes.
Recover after node partitions, app kills, broker pauses, duplicate delivery, and role revocation without losing durable facts.
Define quality-attribute scenarios with stimulus, environment, response, and quantitative measure.
Automate chaos drills whose assertions fail when the recovery mechanism is deliberately disabled.
Produce an architecture decision record, threat model, capacity model, operations runbook, and certification report suitable for senior review.

2. Theoretical Foundation

2.1 Core Concepts

The four-plane model

Framework-first diagrams tend to label boxes “Phoenix,” “RabbitMQ,” and “PostgreSQL” without stating why an event uses one rather than another. The four-plane model begins with responsibility:

The state plane owns authoritative domain facts, invariants, revisions, and durable projections. PostgreSQL and Ecto are the reference mechanisms.
The notification plane reduces observation latency for connected processes and clients. Phoenix.PubSub, Channels, LiveView messages, and Presence diffs belong here. Loss is expected and repaired through reload.
The work plane retains asynchronous responsibility across restarts and independent service pace. Oban jobs, outbox rows, RabbitMQ queues, and Broadway acknowledgements belong here.
The control plane observes, configures, deploys, rate-limits, revokes, drains, and repairs the other planes. Telemetry, alerts, feature flags, runbooks, admin commands, and chaos controls belong here.

The same domain event may cross all four planes, but each transition must be named. “Incident severity raised” is a state-plane fact at revision 18. A PubSub refresh is a notification. A pager integration is durable work. A queue-age alert and replay command are control-plane operations.

Event identity, revisions, and causality

The platform uses an immutable event ID for one logical occurrence and a monotonically increasing revision within each aggregate stream. The event ID supports deduplication and trace correlation. The revision supports stale-message rejection and gap detection. A request ID groups one inbound interaction, while a causation ID points to the event or command that caused another event. These identities are related but not interchangeable.

Ordering is scoped. PostgreSQL can serialize one incident’s revision update, but Phoenix.PubSub does not create a global total order across topics and nodes. A broker queue may order deliveries within one queue but concurrency, retries, and multiple partitions alter processing order. The platform defines ordering only where business invariants require it and detects stale or missing revisions elsewhere.

Tenant isolation as an end-to-end invariant

Authentication identifies an actor; authorization decides whether that actor can act on a specific tenant/resource now. Phoenix LiveView has both an HTTP request and a connected stateful lifecycle, so security checks must cover connected mount and subsequent events. A user-provided topic string is never authority.

Tenant scope travels through server-derived context. Queries include tenant predicates or a stronger database isolation mechanism. Outbox/job arguments carry tenant identity but workers revalidate against durable records. Broker credential, route, envelope, and projection tenant must agree. PubSub topics are generated from validated server context. Metrics avoid sensitive tenant labels or use bounded internal identifiers. Logs and traces are redacted. Admin tools require an explicit tenant scope and audit actor.

Revocation is a temporal requirement. A user authorized at mount may lose a role while the socket remains connected. The system needs a revocation signal or short-lived policy cache, reauthorization for sensitive events, unsubscribe/disconnect behavior, and tests proving no post-revocation delivery or action beyond the defined convergence window.

Presence is awareness, not authority

Phoenix.Tracker and Presence replicate metadata with a heartbeat and CRDT design. Their views are eventually consistent. During node failure or partition, a responder may temporarily appear present or absent differently across nodes. Presence is suitable for operator awareness, collaborative cursors, and routing hints. It is not a billing, lock ownership, or security authority.

The platform displays Presence with freshness and convergence semantics. It may say “last seen 7 seconds ago” rather than asserting a globally current truth. A critical incident assignment is a state-plane record with transaction and audit, even if Presence suggests which responders are online.

Bounded realtime telemetry

Device gauges may produce thousands of readings per second while humans can absorb only a few visual updates per second. Sending every reading through PubSub to every LiveView creates mailbox growth, stale work, serialization cost, and browser churn. The platform separates raw ingestion, durable exceptions, current aggregate state, and optional visual freshness.

High-rate gauges are sampled or coalesced by logical key. A bounded latest-value table or demand pipeline retains current values; a fixed render tick publishes only the latest authorized snapshot. Critical state transitions bypass optional coalescing and become durable facts/work. Under overload, the system degrades visual frequency before losing required work.

Durability and idempotency

Domain state and outbox/job responsibility commit atomically. Durable broker publication uses confirms. Consumption acknowledges after an idempotent projection or effect. Duplicates are expected at transaction retry, publisher retransmission, broker redelivery, and operator replay boundaries. Stable event identity plus durable inbox/effect records turn duplicate attempts into one outcome.

PubSub remains best effort. If a node dies after commit and before refresh, connected clients may wait until another invalidation or reconnect. Every board reloads current state on mount, rejects stale revisions, and repairs gaps by querying authoritative state or a durable event stream.

Partitions, convergence, and degraded modes

A three-node partition divides current delivery components. Each component may continue permitted local operations, but only the database/broker topology and domain policy determine which writes remain safe. PubSub messages missed across the split do not replay. Presence views converge later. The application reloads durable facts and labels the period of ephemeral loss.

The platform has named degraded modes:

fresh: all latency and connectivity objectives met.
stale_ui: durable state available but transient refresh age exceeds budget.
integration_delayed: state commits while outbox/broker age is elevated.
telemetry_coalesced: optional gauge detail reduced to protect capacity.
partitioned: node components diverge; durable writes follow documented policy.
authorization_uncertain: sensitive actions fail closed until policy authority is available.

Quality attributes and chaos evidence

A quality-attribute scenario names a source, stimulus, artifact, environment, response, and measure. “The system is resilient” is not testable. “During a single application-node partition under reference load, durable incident updates commit without loss, connected clients in the isolated component may miss refreshes, membership converges within 20 seconds after heal, and every board reloads the latest revision within 35 seconds” is testable.

Chaos engineering validates hypotheses, not random destruction. Each drill records preconditions, injected fault, predicted degraded state, invariants, recovery mechanism, deadline, and cleanup. A good drill includes a sabotage control: disable the recovery scan or idempotency check and prove the certification fails.

2.2 Why This Matters

Realtime platforms often work in demos and fail in operation because low-latency delivery is mistaken for durable state, Presence is mistaken for current truth, or one generic event bus hides incompatible promises. Multi-tenancy magnifies every error: a wrong topic can become a data breach, and an unbounded subscriber can affect unrelated customers.

The four-plane architecture provides a review vocabulary. Architects can ask which plane owns a flow, what happens when it is removed, and which mechanism can evolve independently. Operators can see whether a user complaint is stale notification, delayed durable work, unavailable state, or a control-plane failure.

The certification report is a product artifact. It turns latency, isolation, durability, boundedness, convergence, and revocation from claims into repeatable evidence. It also creates fitness functions that protect architecture during later changes.

2.3 Historical Context / Background

Realtime web architectures evolved from polling and long-lived socket systems toward frameworks that map connections to lightweight server processes. Erlang/OTP supplied fault isolation and supervision; Phoenix combined those runtime strengths with Channels, PubSub, Presence, and LiveView. Durable job and broker ecosystems addressed asynchronous work across time and service boundaries.

The modern challenge is less “can the server push an update?” and more “can the platform preserve correctness and isolation when pushes are missed, duplicated, delayed, or overwhelming?” This capstone focuses on that operational maturity.

2.4 Common Misconceptions

“One event bus should carry every flow.” State, notification, work, and control have incompatible guarantees.
“Presence says who is online right now.” It is eventually consistent awareness with failure-detection delay.
“PubSub across three nodes makes the platform durable.” It remains transient fanout.
“Tenant ID in the topic is tenant isolation.” Authorization must be established independently and checked across every hop.
“A restarted process means the user recovered.” Business invariants and freshness SLOs may still be broken.
“If the chaos script completed, the drill passed.” It passes only when measurable hypotheses and sabotage controls behave as expected.
“Backpressure means dropping nothing.” Optional telemetry may be sampled/coalesced so critical work stays safe.
“Exactly once simplifies the architecture.” Duplicate attempts remain possible; idempotent effects and explicit local invariants are the practical tools.

3. Project Specification

3.1 What You Will Build

Build a fleet and incident operations platform for tenants 42 and 77. Each tenant has devices, incidents, responders, roles, integrations, and realtime boards. Three Phoenix application nodes serve users and run PubSub/Presence. PostgreSQL owns domain facts, revisions, tenant policy, outbox/inbox records, and durable projections. Oban or an outbox plus RabbitMQ/Broadway executes durable integrations. A bounded telemetry path updates current device gauges.

The reference journey is “incident severity raised.” An authorized command validates tenant and expected revision, commits revision 18 plus an immutable outbox event, and publishes a compact after-commit refresh. Online boards reload. A pager integration is delivered durably and idempotently. Presence helps show which responders may be available but does not assign ownership.

The platform includes a certification console that runs slow subscriber, application kill, node partition, broker pause, duplicate delivery, unsupported schema, and role-revocation drills. Each drill has predicted degraded status, quantitative SLOs, cross-tenant probes, and automated cleanup.

3.2 Functional Requirements

Tenant-scoped domain model: Devices, incidents, responders, memberships, policies, integrations, revisions, and events include enforced tenant ownership.
Authorized LiveView lifecycle: Authenticate and authorize HTTP and connected mounts; recheck sensitive events.
Server-derived topics: Subscribe and publish only through validated topic constructors.
Authoritative incident commands: Use optimistic revision and transactional policy checks.
Atomic durable handoff: Commit domain event and durable integration responsibility together.
Realtime invalidation: Publish compact tenant/resource/revision refresh after commit.
Reconnect repair: Reload state on mount and recover revision gaps.
Presence awareness: Track responder metadata with freshness and convergence labels.
Bounded telemetry: Coalesce gauges, preserve critical alerts, expose dropped/coalesced counts, and limit UI render frequency.
Durable integration: Use Oban or RabbitMQ/Broadway with stable event IDs and idempotent receivers.
Revocation: Remove sensitive access within a measured deadline and disconnect or downgrade existing sessions.
Operational dashboard: Show node topology, event age, mailbox bounds, outbox/broker lag, Presence convergence, and tenant-isolation probes.
Chaos certification: Run all reference faults with predicted and measured outcomes.
Audit export: Produce redacted event journeys, decisions, drill results, and operator actions.

3.3 Non-Functional Requirements

Isolation: Cross-tenant observed leaks remain zero across subscriptions, publications, queries, jobs, broker routes, caches, logs, metrics, traces, and admin tools.
Durability: Committed facts and required work survive application/node restart and duplicate delivery.
Freshness: Reference-load UI event-age p99 is below 500 ms in healthy local conditions.
Boundedness: Subscriber mailbox p99 remains below 1,000; every queue, buffer, batch, payload, and history has a limit.
Work recovery: Oldest outbox/job age normally remains below 30 seconds and returns below threshold after outage within the recovery budget.
Presence convergence: Reference node-loss/heal convergence p99 remains below 35 seconds, with UI labeling uncertainty.
Revocation: Sensitive action and delivery cease within the documented policy propagation target.
Availability policy: Each partitioned operation is classified as continue, degrade, or fail closed.
Evidence: Every SLO is computed from reproducible, version-stamped test runs.

3.4 Example Usage / Output

$ mix platform.certify --profile reference
EVENT PLATFORM — CHAOS CERTIFICATION
run=cert-2026-07-15-01 version=git:4ad2e8a nodes=3 tenants=2

normal_load:
  ui_event_age_p99=182ms target<500ms PASS
  mailbox_p99=44 bound<1000 PASS
  telemetry_received=500000 rendered=3000 coalesced=497000 PASS
  outbox_oldest=1.8s target<30s PASS

duplicate_delivery:
  broker_redeliveries=17 durable_effects=17 duplicate_effects=0 PASS

node_partition:
  components=[[alpha,beta],[gamma]]
  transient_revisions_missed=expected:23
  durable_facts_lost=0 PASS
  membership_convergence=6.2s target<35s PASS
  board_reload_latest_revision=PASS

role_revocation:
  sessions_targeted=6
  sensitive_delivery_after_deadline=0 PASS
  sensitive_actions_after_deadline=0 PASS

tenant_isolation:
  attempted_cross_tenant_paths=240 observed_leaks=0 PASS

sabotage_control disable_revision_reload:
  expected_certification_failure=true observed=true PASS

overall=PASS_WITH_DOCUMENTED_EPHEMERAL_LOSS

3.5 Real World Outcome

Open six browser sessions distributed across three application nodes: two tenant-42 operators, two tenant-77 operators, and two responder sessions. The tenant-42 board shows only its devices, incidents, current gauges, and responders. Tenant 77 has visually identical but independently scoped data. A topology panel identifies the serving node and current component without exposing privileged internals to ordinary users.

Raise incident 913 from severity 2 to 3. Both authorized tenant-42 boards update to revision 18. Tenant-77 boards show no event. The event-journey drawer traces command, transaction, outbox, PubSub refresh, broker integration, idempotent pager result, and audit record using the same event ID. Disconnect one tenant-42 board before the change; on reconnect it loads revision 18 from PostgreSQL and labels the repair source.

Feed high-rate gauges. The ingestion counter rises rapidly, while each board renders at a human-scale cadence. Cards show how many readings were coalesced and whether telemetry is fresh or degraded. Trigger a critical threshold event; it becomes a durable incident fact/work item rather than being lost as an optional gauge sample.

Run the chaos console. During gamma’s partition, boards in different components may miss transient revisions and Presence may diverge, but durable facts remain intact. After heal, topology and Presence converge and every board reloads the latest revision. Pause RabbitMQ; outbox/broker age turns amber while incident commands and local refresh continue according to policy. Resume it and watch backlog drain without duplicate pager effects.

Revoke a tenant-42 operator’s incident-management role while their LiveView remains connected. Within the measured deadline, sensitive subscriptions/actions stop, the UI downgrades or disconnects, and later broadcasts produce zero restricted content. The final certification report shows every SLO, expected degraded state, observed behavior, and sabotage control.

4. Solution Architecture

4.1 High-Level Design

                              CLIENT / EDGE
        ┌──────────────────────────────────────────────────────┐
        │ LiveView boards | Channels | reconnect | auth policy│
        └──────────────────────────┬───────────────────────────┘
                                   │
      ┌────────────────────────────┼─────────────────────────────┐
      │                            │                             │
      v                            v                             v
┌─────────────────┐       ┌──────────────────┐         ┌──────────────────┐
│ STATE PLANE     │       │ NOTIFICATION     │         │ WORK PLANE       │
│ PostgreSQL/Ecto │       │ PLANE            │         │ Outbox/Oban      │
│ facts/revisions │──────▶│ PubSub/Presence  │         │ RabbitMQ/Broadway│
│ policy/audit    │       │ transient refresh│         │ confirms/acks    │
└────────┬────────┘       └─────────┬────────┘         └─────────┬────────┘
         │                          │                            │
         └──────────────────────────┼────────────────────────────┘
                                    v
                         ┌────────────────────────┐
                         │ CONTROL PLANE          │
                         │ telemetry + SLOs       │
                         │ topology + revocation  │
                         │ deploy/drain/replay    │
                         │ chaos + audit evidence│
                         └────────────────────────┘

Three application nodes host notification and edge components.
State/work durability remains explicit outside transient process mailboxes.
Every arrow carries tenant context, event identity, capacity, and recovery policy.

Incident severity journey

authorized command
  -> transaction(order/incident revision + outbox event)
  -> after-commit PubSub refresh -> authorized boards reload
  -> durable publisher confirm -> broker queue
  -> Broadway projection/effect -> consumer ack
  -> metrics/audit certify latency, durability, isolation

4.2 Key Components

Component	Responsibility	Key Decisions
Identity/policy boundary	Authenticate actor and authorize tenant/resource/action	Connected-mount plus ongoing sensitive-event checks
Incident command service	Enforce domain policy and optimistic revision	No PubSub as command path
State repository	Own facts, events, policy, and audit	Tenant constraints and transactional handoff
Topic constructor	Create bounded tenant/resource topics	No arbitrary client topic strings
PubSub/Presence	Fan out refreshes and awareness	Transient, eventually convergent, reloadable
Telemetry aggregator	Coalesce high-rate gauges and detect critical transitions	Bounded latest-value state and render cadence
Outbox/Oban publisher	Retain required integration responsibility	Stable event ID and observable age
Broker/Broadway edge	Cross service boundary durably	Confirm, ack, idempotency, DLQ
Revocation coordinator	Invalidate policy caches and sessions	Measured propagation and fail-closed behavior
SLO/chaos controller	Run quality scenarios and publish evidence	Sabotage controls and idempotent cleanup

4.3 Data Structures

TenantContext
  tenant_id
  actor_id
  roles/policy_version
  authenticated_session_id
  authorization_checked_at

DomainEvent
  event_id
  tenant_id
  aggregate_type/id/revision
  event_type/schema_version
  occurred_at/request_id/causation_id/actor_ref
  minimal_payload

RealtimeRefresh
  tenant_id + resource_type + resource_id + revision + event_id
  no sensitive snapshot

PresenceMetadata
  actor_ref + role_summary + status + connected_at + serving_node
  no authorization authority or private profile

TelemetrySnapshot
  tenant_id + device_id + window_end + latest/min/max/count
  coalesced_count + freshness

CertificationResult
  run_id + version + environment
  scenario + stimulus + expected_state
  measures + thresholds + verdict
  evidence_refs + cleanup_result

Database constraints and query APIs must make tenant omission difficult. Cache keys, inbox/outbox identities, and unique constraints include tenant where global IDs are not cryptographically guaranteed unique.

4.4 Algorithm Overview

Algorithm: Incident command and fanout

AUTHENTICATE session and load current policy version
AUTHORIZE action for tenant/resource
DECIDE transition using expected aggregate revision
TRANSACTION
  update incident at expected revision
  insert immutable domain event
  insert durable integration handoff
COMMIT
BEST-EFFORT broadcast compact refresh on server-derived topic
RETURN committed revision/event_id

Algorithm: Board refresh

ON connected mount:
  authenticate + authorize tenant board
  subscribe to server-derived topics
  load current authoritative snapshot and revision cursors
ON refresh:
  reject if tenant/resource not in authorized context
  ignore stale revision
  if next revision or invalidation: reload authorized current state
  if gap detected: mark recovering and reload/replay durable facts
ON policy revocation:
  reauthorize, unsubscribe restricted topics, clear sensitive assigns,
  disconnect or downgrade before deadline

Algorithm: Bounded telemetry

ingest validated reading under bounded demand
update latest-value/window aggregate by tenant/device
if critical threshold transition:
  commit durable incident fact/work
at fixed render cadence:
  emit latest authorized snapshot
  record received/rendered/coalesced/dropped counts
under overload:
  lower optional render rate before rejecting critical transitions

Algorithm: Certification scenario

capture version/config/baseline
state hypothesis and thresholds
start tenant-isolation and sequence probes
inject one bounded fault
observe predicted degraded state
assert durable, isolation, capacity, and recovery invariants
heal fault and wait for explicit recovery conditions
run sabotage control when safe
export redacted evidence
perform and verify cleanup

Complexity analysis

Tenant/resource lookups and event/inbox/outbox operations are O(log n) with scoped indexes.
PubSub fanout is O(subscribers to the topic); topic granularity bounds unnecessary work.
Latest-value telemetry space is O(active tenant-device keys), not O(all readings).
Presence replication and cluster fanout grow with nodes and tracked changes; test realistic churn.
Certification evidence is O(scenarios × probes) with fixed retention and summary aggregation.

5. Implementation Guide

5.1 Development Environment Setup

Use an isolated local or ephemeral environment with three application nodes, PostgreSQL, RabbitMQ, and a reverse proxy/load balancer. Generate synthetic tenants and credentials only.

$ ./platform doctor
elixir=1.20.x otp=29 phoenix=resolved-from-lock
nodes=[alpha@platform,beta@platform,gamma@platform]
postgres=healthy rabbitmq=healthy
tenant_fixtures=[42,77]
cross_tenant_probe_users=ready
chaos_backend=container_network
distribution_boundary=isolated
doctor result=PASS

$ ./platform seed --dataset certification-small
tenant42 incidents=20 devices=100 responders=12
tenant77 incidents=20 devices=100 responders=12
cross_tenant_shared_ids=deliberately_included
seed result=PASS

The deliberately overlapping resource IDs between tenants expose missing tenant predicates. Record resolved versions and configuration hashes in every certification report. Never reuse production credentials or data.

5.2 Project Structure

multitenant_event_platform/
├── apps/
│   ├── domain/                 # incidents, devices, policy, event identity
│   ├── realtime/               # PubSub topics, Presence, bounded telemetry
│   ├── integrations/           # outbox/Oban, broker/Broadway, idempotency
│   ├── web/                    # LiveViews, Channels, revocation handling
│   └── certification/          # scenarios, probes, reports, cleanup
├── config/
│   ├── runtime.exs
│   └── certification.example
├── priv/repo/migrations/
├── test/
│   ├── domain/
│   ├── isolation/
│   ├── realtime/
│   ├── durable_work/
│   └── chaos/
├── docs/
│   ├── architecture.md
│   ├── threat-model.md
│   ├── quality-scenarios.md
│   ├── capacity-model.md
│   └── operations-runbook.md
└── evidence/

An umbrella is optional. The critical structure is boundary ownership, not the exact directory layout.

5.3 The Core Question You’re Answering

“Can I justify, observe, and recover every event boundary without using Pub/Sub as a universal substitute for state, work queues, authorization, or consensus?”

Answer with one complete event journey. For each arrow, name the plane, source of authority, delivery semantics, ordering scope, duplicate policy, tenant decision, capacity limit, observability signal, and recovery method. Any unnamed arrow is an unreviewed failure boundary.

5.4 Concepts You Must Understand First

All prior Pub/Sub concepts
- Can you distinguish raw process messages, Registry, Phoenix.PubSub, Presence, demand pipelines, outbox, and brokers?
- Which earlier project provides evidence for each mechanism?
Quality-attribute scenarios
- What source, stimulus, artifact, environment, response, and measure define the scenario?
- How will a threshold fail deterministically?
- Book Reference: “Software Architecture in Practice, 4th Edition” — Quality Attributes.
Multi-tenant authorization
- Where does identity become trusted context?
- How do connected sessions react to policy change?
- Reference: Phoenix LiveView security model and OWASP authorization guidance.
Durable event processing
- Where are transactional handoff, publisher confirm, consumer ack, and idempotent effect?
- Book Reference: “Enterprise Integration Patterns” — Guaranteed Delivery and Idempotent Receiver.
Distributed convergence
- Which views are transient/eventually consistent?
- Which durable source repairs missed revisions?
- Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — distributed systems and streams.
Capacity and stability
- Which queues/buffers are bounded and which work degrades first?
- Book Reference: “Release It!, 2nd Edition” — Stability Patterns.

5.5 Questions to Guide Your Design

Plane ownership
- If PubSub is disabled for one minute, which facts/work remain correct?
- If RabbitMQ is paused, which commands and UI paths continue?
Tenant isolation
- Which helper makes an unscoped query/topic/job impossible or obvious?
- How are tenant IDs represented in telemetry without leakage or cardinality explosion?
Event semantics
- Where can duplicates occur, and which durable key neutralizes each one?
- What revision gaps are repairable from state versus event replay?
Overload
- Which optional updates are coalesced or sampled?
- Which critical events must be rejected explicitly rather than silently dropped?
Failure and recovery
- Which operations continue during a node partition?
- What evidence proves the user-facing board, not merely a process, recovered?
Evolution
- Can the work plane move from Oban to RabbitMQ without changing domain event identity?
- Which automated fitness functions protect boundaries during that change?

5.6 Thinking Exercise

Create a full journey for “incident severity raised”:

browser command -> connected authorization -> domain decision -> DB transaction
-> domain event/outbox -> PubSub refresh -> LiveView reload
-> broker publication/confirm -> Broadway delivery -> integration effect/ack
-> Presence hint -> metrics/audit -> reconnect/replay

For every arrow, fill a table with:

source and destination plane;
tenant authority;
event identity and revision;
ordering guarantee;
loss and duplicate behavior;
capacity/buffer limit;
observability signal;
recovery source;
security/privacy classification.

Then remove PubSub, PostgreSQL, RabbitMQ, one application node, policy service, and one subscriber in turn. Predict continue, degrade, or fail closed before running any drill.

5.7 The Interview Questions They’ll Ask

“How do you choose between Phoenix.PubSub, Oban, GenStage/Broadway, and RabbitMQ?”
“How does the platform recover after missing PubSub messages during a partition?”
“Where can duplicates occur, and how do you prevent duplicate effects?”
“How do you prevent cross-tenant leakage through topics, jobs, metrics, and logs?”
“What happens when one subscriber or integration cannot keep up?”
“Which evidence proves the platform meets its SLOs under chaos?”

5.8 Hints in Layers

Hint 1: Draw planes before components

Assign every flow to state, notification, work, or control. Only then select PostgreSQL, PubSub, Oban, RabbitMQ, or Telemetry.

Hint 2: Build one tenant end to end

Complete identity, revision, durability, reconnect, and SLO evidence for tenant 42. Add tenant 77 specifically to attack isolation assumptions.

Hint 3: Add failure before scale

Prove one post-effect duplicate, one missed refresh, and one revocation before increasing rates or node count.

Hint 4: Certification is the deliverable

Every drill needs a hypothesis, threshold, sabotage control, and cleanup assertion. A dashboard screenshot alone is not evidence.

5.9 Books That Will Help

Topic	Book	Chapter / Section
Quality attributes	“Software Architecture in Practice, 4th Edition”	Quality Attribute Scenarios
Evolution fitness	“Building Evolutionary Architectures, 2nd Edition”	Fitness Functions
Messaging boundaries	“Enterprise Integration Patterns”	Ch. 2–4; Guaranteed Delivery; Idempotent Receiver
Data and distribution	“Designing Data-Intensive Applications, 2nd Edition”	Transactions, Distributed Systems, Streams
Stability	“Release It!, 2nd Edition”	Stability Patterns and Production
Service architecture	“Building Microservices, 2nd Edition”	Coupling and Asynchronous Communication

5.10 Implementation Phases

Phase 1: One-Tenant Vertical Slice (16–24 hours)

Goals

Establish state, identity, revision, and realtime reload boundaries.
Produce one complete incident journey.

Tasks

Model tenant, incident, device, responder, membership, and policy constraints.
Implement authenticated/authorized connected board lifecycle.
Add command transaction, domain event, and atomic durable handoff.
Add compact PubSub refresh and reconnect/gap reload.
Trace one event ID through the journey.

Checkpoint: Tenant 42 raises severity, two boards reach revision 18, a disconnected board catches up, and durable integration responsibility remains queryable.

Phase 2: Isolation, Presence, and Bounded Telemetry (16–24 hours)

Goals

Add a hostile second tenant and distributed awareness.
Protect capacity under high-rate input.

Tasks

Seed tenant 77 with deliberately overlapping resource IDs.
Add cross-tenant query/topic/job/cache/log/metric tests.
Add Presence with freshness and convergence labeling.
Build bounded telemetry coalescing and critical-transition promotion.
Add role revocation and connected-session enforcement.

Checkpoint: 240 cross-tenant probes observe zero leaks; high-rate telemetry remains bounded; critical transitions persist; revocation meets deadline.

Phase 3: Durable Broker Integration and Three Nodes (16–28 hours)

Goals

Extend work across a durable broker boundary.
Exercise distributed PubSub and Presence behavior.

Tasks

Start three application nodes and prove topology.
Add confirmed broker publication and Broadway consumption.
Add durable inbox/idempotent integration effect and DLQ.
Inject duplicate delivery and processor/app restarts.
Partition one node and verify transient loss plus durable recovery.

Checkpoint: Broker redeliveries create zero duplicate effects; partitioned boards reload latest facts; Presence converges within budget.

Phase 4: SLOs, Chaos, and Certification (12–24 hours)

Goals

Turn architectural claims into automated evidence.
Produce operations and review artifacts.

Tasks

Define reference load and quality-attribute scenarios.
Instrument latency, mailbox, outbox/broker, Presence, isolation, and revocation measures.
Automate slow subscriber, node kill/partition, broker pause, duplicate, poison, and revocation drills.
Add sabotage controls and idempotent cleanup.
Publish architecture, threat, capacity, runbook, and certification documents.

Checkpoint: The complete certification passes with documented ephemeral loss, and disabling each selected recovery mechanism causes the related scenario to fail.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Architecture vocabulary	Framework boxes; four planes	Four planes, then mechanisms	Keeps promises and authority reviewable
Realtime payload	Full snapshot; ID/revision invalidation	Compact invalidation and authorized reload	Limits stale data and tenant exposure
Presence use	Lock/authority; awareness	Awareness only	Eventually consistent by design
Telemetry flow	Every sample to every UI; coalesced latest/window	Bounded coalescing plus durable critical transitions	Protects mailbox and human-scale rendering
Durable work	PubSub; transactional outbox/job + broker as needed	Explicit durable handoff	Survives restarts and independent pace
Tenant isolation	Convention only; layered constraints/helpers/tests	Layered enforcement and hostile probes	Prevents one missed check becoming a breach
Partition policy	Implicit best effort; operation-by-operation policy	Continue/degrade/fail-closed matrix	Makes availability and safety explicit
Certification	Manual demo; automated quality scenarios	Automated with sabotage controls	Creates repeatable architecture evidence

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Domain	Preserve business invariants	Revision conflict, policy denial, atomic event handoff
Authorization/isolation	Prevent tenant leakage	Connected mount, topic, query, job, broker, logs, metrics
Realtime	Verify transient semantics	Two clients, stale message, revision gap, reconnect
Presence	Verify awareness convergence	Join/leave, node kill, partition/heal, stale metadata
Capacity	Verify bounded behavior	High-rate gauges, slow subscriber, DB/broker slowdown
Durable work	Verify restart/idempotency	Outbox restart, broker redelivery, DLQ, replay
Distributed	Verify component behavior	Three-node topology, node kill, network partition
Revocation	Verify temporal authorization	Role removal while sockets/jobs remain active
Chaos/SLO	Verify measurable resilience	Fault hypothesis, threshold, recovery, sabotage control
Privacy	Verify evidence safety	Payload/log/trace/DLQ export redaction

6.2 Critical Test Cases

Atomic incident change: Revision and durable handoff commit or roll back together.
Two-client refresh: Authorized boards reach the committed revision under healthy conditions.
Disconnected repair: A board missing PubSub loads the latest revision on reconnect.
Stale/gap handling: Stale revisions are ignored; gaps cause explicit recovery.
Cross-tenant subscription: Tenant-42 session cannot subscribe to tenant-77 topics.
Cross-tenant publication: Forged event cannot create tenant-77 projection or refresh.
Shared resource IDs: Every query still returns only current tenant rows.
Presence partition: Divergent awareness is labeled and converges after heal.
High-rate telemetry: Received volume grows while buffer/mailbox/render bounds hold.
Critical promotion: Threshold transition persists even when optional gauges are coalesced.
Duplicate durable delivery: Multiple attempts create one effect.
Broker pause: Outbox/broker age rises visibly and drains after recovery.
Node partition: Transient loss is documented; durable facts lost remain zero.
Role revocation: Existing session loses restricted delivery/action within deadline.
Sabotage control: Disabling revision reload, idempotency, or revocation propagation fails the expected certification scenario.

6.3 Test Data

tenants:
  42: operators=[alice,bob] responders=[r1..r6]
  77: operators=[carol,dan] responders=[r7..r12]

shared logical IDs across tenants:
  incident=913
  device=abc
  responder=1001

reference event:
  event_id=evt-77
  tenant=42
  incident=913
  revision=18
  type=incident.severity_raised
  severity_before=2
  severity_after=3

reference load:
  devices_per_tenant=100
  gauge_rate_total=5000/second
  board_sessions=60
  critical_transition_rate=5/second
  integration_rate=50/second
  duration=10 minutes

SLO targets:
  ui_event_age_p99<500ms
  mailbox_p99<1000
  outbox_oldest_normal<30s
  presence_convergence_p99<35s
  cross_tenant_leaks=0
  duplicate_effects=0

Use deterministic synthetic generators and print seed, version, config hash, and environment with every result.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Generic event bus owns everything	No one can state durability or authority	Assign every flow to one plane
Topic contains tenant but auth is absent	Cross-tenant messages leak	Derive topics after authorization and test hostile paths
Presence used as lock	Two components make conflicting decisions	Store authoritative assignment in state plane
Full telemetry fanout	Mailboxes and browser updates grow	Coalesce/sample optional gauges; persist critical transitions
PubSub assumed to replay	Boards remain stale after partition	Detect gaps/reconnect and reload durable state
Ack before effect commit	Broker deletes unfinished work	Transfer responsibility only after durable effect
“Process restarted” success criterion	User board still stale or duplicate effect exists	Assert business/user invariants and SLOs
Metrics use raw tenant/event IDs	Privacy/cardinality incident	Use bounded labels; correlate IDs in protected traces
Chaos without cleanup	Later runs inherit network faults	Make cleanup idempotent and certify residual state
Green chaos without sabotage	Assertions never tested recovery mechanism	Disable one mechanism and expect failure

7.2 Debugging Strategies

Locate the plane first: Is the symptom wrong state, stale notification, delayed work, or control-plane blindness?
Trace one event ID and revision across all durable and transient boundaries.
Compare authoritative revision with each board cursor to distinguish loss from stale rendering.
Inspect component topology and Presence views per node, not one “global” snapshot.
Compare received/rendered/coalesced/dropped counts for telemetry capacity issues.
Run cross-tenant canaries continuously during load and chaos, not only in unit tests.
Freeze one dependency at a time so backlog location and degraded state remain attributable.
Verify recovery from the user’s perspective: correct board, role, integration effect, and deadline.

7.3 Performance Traps

One PubSub topic per overly granular object can create excessive subscription churn; one global topic creates excessive filtering and leakage risk. Measure the middle ground.
Large LiveView assigns and frequent diffs increase memory and serialization cost.
Presence metadata that changes on every telemetry reading creates replication storms.
Database pool, Broadway concurrency, Oban queue concurrency, and broker prefetch can multiply beyond downstream capacity.
Synchronous external calls in command or notification paths increase lock/ack latency.
Unbounded evidence collection can become the largest workload during chaos.
Tenant-fairness failure allows one noisy tenant to consume all queue, process, or render capacity.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a per-event journey drawer with plane labels.
Add explicit fresh/stale/recovering UI badges.
Add one role-revocation workflow with a visible deadline counter.

8.2 Intermediate Extensions

Add per-tenant quotas and weighted-fair durable queues.
Add schema-version compatibility and one upcaster.
Add a read-model rebuild from durable events.
Add blue/green deployment under continuous PubSub and durable-work probes.

8.3 Advanced Extensions

Add a second region with explicit federation rather than transparent full-mesh distribution.
Implement cell-based tenant isolation and compare blast radius.
Add OpenTelemetry baggage controls and cross-service trace sampling.
Build automated architecture fitness functions in CI for topic construction, tenant-scoped queries, payload size, and SLO regression.
Run disaster recovery from database backup plus broker/outbox evidence and measure RPO/RTO.

9. Real-World Connections

9.1 Industry Applications

Fleet operations: Device telemetry, critical incidents, responder awareness, and third-party dispatch have different event contracts.
Healthcare operations: Tenant and patient isolation, durable escalation, and audit are mandatory.
Financial platforms: Durable facts and idempotent integrations coexist with low-latency dashboards.
Collaboration SaaS: Presence and transient document notifications require durable document authority.
Security operations: High-volume signals must be bounded while critical incidents and response actions remain durable.

Phoenix: Web, Channels, and realtime application framework.
Phoenix.PubSub: Local and distributed transient fanout.
Phoenix LiveView: Stateful server-rendered realtime UI.
Oban: PostgreSQL-backed durable jobs.
RabbitMQ: Durable cross-service messaging.
Broadway: Demand-driven external event processing.
OpenTelemetry: Cross-boundary traces and metrics.

9.3 Interview Relevance

This capstone prepares you for principal-level architecture discussions. You can explain why one domain event crosses multiple planes, where tenant authority is established, how duplicate attempts become one effect, how overload degrades optional freshness, and which chaos evidence supports latency, durability, convergence, and isolation claims.

10. Resources

10.1 Essential Reading

“Software Architecture in Practice, 4th Edition” by Len Bass et al. — quality attributes and measurable scenarios.
“Building Evolutionary Architectures, 2nd Edition” by Neal Ford et al. — architecture fitness functions.
“Enterprise Integration Patterns” by Gregor Hohpe and Bobby Woolf — messaging channels, routing, guaranteed delivery, and idempotency.
“Designing Data-Intensive Applications, 2nd Edition” by Martin Kleppmann et al. — transactions, distributed systems, and streams.
“Release It!, 2nd Edition” by Michael Nygard — stability and production operations.
“Building Microservices, 2nd Edition” by Sam Newman — service and asynchronous boundaries.

10.2 Video Resources

Official ElixirConf/Phoenix talks on LiveView operations, PubSub, Presence, and telemetry.
RabbitMQ official sessions on reliability, quorum queues, and capacity.
SRE and chaos-engineering conference talks focused on hypothesis-driven experiments and user-facing invariants.

10.3 Tools & Documentation

Phoenix.PubSub — transient fanout and pool migration.
Phoenix.Tracker — CRDT/heartbeat Presence foundation and convergence behavior.
Phoenix LiveView security model — HTTP and connected authorization lifecycle.
Ecto.Multi — transactional domain and handoff operations.
Oban — durable database job insertion and execution.
RabbitMQ reliability — confirms, acknowledgements, and recovery.
Broadway.Acknowledger — source responsibility after processing.
OpenTelemetry Erlang/Elixir — observability across services.

Projects 1–4 establish process messages, local routing, domain contracts, and Phoenix.PubSub semantics.
Project 5 adds the reconnect-safe LiveView interface, and Project 6 adds Presence and convergence semantics.
Project 7 adds mailbox-pressure controls, while Project 8 adds demand-aware processing boundaries.
Project 9 adds transactional durable handoff.
Project 10 proves three-node partition and migration behavior.
Project 11 adds durable broker responsibility and tenant-safe edge validation.

11. Self-Assessment Checklist

11.1 Understanding

I can assign every event flow to state, notification, work, or control.
I can explain event ID, aggregate revision, request ID, and causation ID separately.
I can explain why Presence is not authorization or lock ownership.
I can name every boundary where duplicate delivery may occur.
I can explain how optional telemetry degrades before critical work.
I can state the continue/degrade/fail-closed policy for each reference fault.
I can define a complete quality-attribute scenario and sabotage control.

11.2 Implementation

Three application nodes prove topology and tenant-safe PubSub/Presence.
Domain state and durable handoff commit atomically.
Boards reload on mount and repair revision gaps.
High-rate telemetry remains within queue/mailbox/render bounds.
Broker redelivery produces zero duplicate effects.
Cross-tenant probes cover every listed boundary and observe zero leaks.
Role revocation meets the measured deadline for existing sessions.
Chaos drills assert user/business invariants and clean up reliably.
Evidence records versions, config, seeds, thresholds, and redaction.

11.3 Growth

I documented at least three architecture assumptions disproved by drills.
I can propose a safe evolution from database jobs to broker-based work without changing domain identity.
I can defend capacity and SLO targets with measurements.
I can present the certification report to architecture, security, and operations reviewers.

12. Submission / Completion Criteria

Minimum Viable Completion

Two tenants have isolated LiveView incident boards and server-derived topics.
One incident transition atomically commits state, event identity, and durable handoff.
Connected clients refresh through PubSub; disconnected clients recover from durable state.
One bounded telemetry path exposes received/rendered/coalesced counts.
One durable integration tolerates duplicate delivery with one effect.

Full Completion

All minimum criteria deployed across three application nodes with Presence and proven topology.
Cross-tenant probes cover subscription, publication, query, job, broker, cache, metrics, logs, traces, and admin tools with zero observed leaks.
Node partition documents expected transient loss and recovers all durable facts/revisions.
Broker pause, processor kill, slow subscriber, poison event, duplicate delivery, and role revocation meet defined degraded-state and recovery targets.
Certification reports UI event age, mailbox bounds, telemetry coalescing, outbox/broker age, duplicate effects, Presence convergence, revocation, and isolation.
Sabotage controls prove selected certification assertions can fail.

Excellence (Going Above & Beyond)

CI enforces architecture fitness functions for tenant scoping, topic construction, event contracts, and performance budgets.
A second region or cell architecture contains blast radius and documents federation semantics.
Disaster recovery proves backup restore, outbox/broker reconciliation, and measured RPO/RTO.
The complete portfolio includes ADRs, threat model, quality scenarios, capacity model, load results, runbooks, chaos evidence, and an executive architecture summary.

This guide was expanded from LEARN_ELIXIR_PUBSUB_DEEP_DIVE.md. For the complete expanded project index, see README.md.

Project 12: Multi-Tenant Real-Time Event Platform

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Historical Context / Background

2.4 Common Misconceptions

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Structures

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: One-Tenant Vertical Slice (16–24 hours)

Phase 2: Isolation, Presence, and Bounded Telemetry (16–24 hours)

Phase 3: Durable Broker Integration and Three Nodes (16–28 hours)

Phase 4: SLOs, Chaos, and Certification (12–24 hours)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

7.3 Performance Traps

8. Extensions & Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools & Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria