Sprint: Elixir Pub/Sub Mastery - Real World Projects

Goal: Master publish/subscribe from the BEAM’s smallest useful primitive—a message sent to a process—through local topic registries, Phoenix.PubSub, presence, backpressure-aware pipelines, durable handoff, and multi-node operation. You will learn what Pub/Sub decouples, what it does not guarantee, how mailboxes fail under overload, and where ordering, acknowledgement, replay, authorization, and persistence must be designed explicitly. By the end, you will be able to build and operate a multi-tenant real-time event platform in Elixir and explain why each boundary is ephemeral, durable, local, or distributed.

Introduction

Publish/subscribe is a communication pattern in which publishers emit messages to logical topics without naming every receiver, while subscribers express interest in those topics without being coupled to each publisher. In Elixir, this pattern begins with BEAM process mailboxes and can grow through Registry, Phoenix.PubSub, Phoenix.Tracker, distributed Erlang, demand-driven pipelines, database outboxes, and external brokers.

Pub/Sub solves routing decoupling: the producer need not know which processes currently care. It does not automatically solve durable storage, replay, end-to-end acknowledgement, slow-consumer protection, global ordering, authorization, or exactly-once effects. This guide teaches those missing contracts rather than treating a successful broadcast/3 call as proof of reliable delivery.

Across twelve projects you will build:

• a raw mailbox broadcast laboratory;
• a local topic bus with duplicate-key Registry dispatch;
• a contract-aware domain event hub;
• a Phoenix.PubSub notification service and a LiveView operations board;
• a distributed presence workspace;
• a mailbox-pressure and overload laboratory;
• a demand-aware Broadway bridge;
• a transactional outbox with durable workers;
• a three-node Pub/Sub partition laboratory;
• a RabbitMQ-to-Phoenix edge bridge; and
• a secure multi-tenant real-time event platform.

The scope includes in-VM and distributed Pub/Sub, delivery semantics, topic design, event envelopes, load control, durability boundaries, cluster behavior, observability, testing, and authorization. It does not attempt to reimplement Kafka, RabbitMQ, a consensus protocol, or the Erlang distribution protocol. External brokers appear only where their guarantees clarify the boundary between transient notification and durable work.

       command / state change                  ephemeral notification plane
┌───────────────────────────┐              ┌───────────────────────────────────┐
│ HTTP, job, device, domain │              │ local and distributed subscribers │
│ input                     │              │ LiveView • Channel • worker • log │
└─────────────┬─────────────┘              └───────────────▲───────────────────┘
              │                                            │
              v                                            │ broadcast
┌───────────────────────────┐      commit      ┌────────────┴───────────────────┐
│ authoritative state owner │────────────────▶│ event envelope + topic policy │
│ GenServer or database     │                 └────────────┬───────────────────┘
└─────────────┬─────────────┘                              │
              │ durable requirement                        │ local / cluster fanout
              v                                            v
┌───────────────────────────┐                 ┌────────────────────────────────┐
│ outbox / Oban / broker    │                 │ Registry / Phoenix.PubSub / :pg│
│ retry • ack • dedupe      │                 │ asynchronous BEAM messages     │
└─────────────┬─────────────┘                 └───────────────┬────────────────┘
              │                                              │
              v                                              v
┌───────────────────────────┐                 ┌────────────────────────────────┐
│ external side effects     │                 │ one mailbox per subscriber     │
│ email • webhook • billing │                 │ fast, isolated, but unbounded  │
└───────────────────────────┘                 └────────────────────────────────┘

How to Use This Guide

1. Read the nine theory chapters before choosing a production-looking project. Each chapter names the guarantees you must write down before implementation.
2. Build Projects 1–3 in order. They expose the raw mechanics that framework APIs deliberately hide.
3. For every project, write the event contract, failure matrix, and overload policy before implementing the happy path.
4. Treat every CLI transcript as a target observation, not code to copy. Instrument the system until your output proves the stated invariant.
5. Use the Definition of Done as an acceptance test. A UI that “looks live” is not complete if reconnects, mailbox growth, tenant isolation, or stale events are unmeasured.
6. After each project, explain which facts are durable, which messages may be missed, and which component owns recovery.

Prerequisites & Background Knowledge

Essential Prerequisites (Must Have)

• Comfortable Elixir syntax: pattern matching, tagged tuples, maps, structs, modules, functions, and case/receive.
• Basic BEAM process operations and OTP supervision vocabulary.
• Ability to run Mix tests and inspect a process from IEx.
• Basic HTTP and database transaction knowledge for later projects.
• Recommended reading: “Programming Elixir” concurrency and OTP chapters; “Designing for Scalability with Erlang/OTP” process and distribution chapters; “Designing Data-Intensive Applications, 2nd Edition” chapters on encoding, replication, streams, and distributed-system failure.

Helpful But Not Required

• Phoenix LiveView and Channels, introduced in Projects 4–6.
• Ecto transactions and Oban, introduced in Project 9.
• Distributed Erlang node naming, cookies, and clustering, introduced in Project 10.
• RabbitMQ acknowledgement and dead-letter vocabulary, introduced in Project 11.

Self-Assessment Questions

1. Can you explain why send/2 succeeding does not mean the receiver processed the message?
2. Can you distinguish a process PID, a registered name, a topic, and a durable event ID?
3. What happens to a GenServer’s in-memory state and mailbox when it crashes?
4. Why can two senders produce different interleavings at one receiver even when each sender preserves its own order?
5. Can you name one explicit policy for a subscriber whose mailbox grows faster than it drains?
6. Can you explain why publishing after a database commit can lose a notification and publishing before commit can announce state that later rolls back?

Development Environment Setup

Required tools:

• Elixir 1.20.x and Erlang/OTP 29, or a currently supported compatible pair.
• Mix and IEx.
• PostgreSQL for Project 9 and RabbitMQ for Project 11.
• Two or three local terminals for distributed-node experiments.

Recommended tools:

• Observer or :observer_cli, recon, Telemetry/Metrics, and a Prometheus-compatible backend.
• Docker or another reproducible service runner for PostgreSQL and RabbitMQ.
• tc/firewall tooling or a container network for controlled partition tests.

Testing your setup:

$ elixir --version
Erlang/OTP 29 [...]
Elixir 1.20.2 (compiled with Erlang/OTP 29)

$ iex --sname pubsub_check -e 'IO.inspect({node(), Node.alive?()})'
{:pubsub_check@your-host, true}

The versions above are the researched July 2026 baseline, not a rule that future patch releases must be avoided.

Time Investment

• Foundation projects: 6–10 hours each.
• Application projects: 12–20 hours each.
• Distributed and durability projects: 20–35 hours each.
• Capstone: 40–60 hours.
• Full sprint: roughly 5–8 months part-time.

Important Reality Check

The easy part of Pub/Sub is sending a term to many PIDs. The hard part is deciding what a message means when subscribers crash, lag, reconnect, duplicate effects, change versions, cross nodes, or belong to different tenants. Expect the most useful project evidence to come from failure injection and measurement rather than the first successful demo.

Big Picture / Mental Model

Think in four independent planes. The state plane owns facts. The notification plane makes current processes aware of changes. The work plane provides durable, acknowledged execution. The control plane defines membership, authorization, topology, and observability. A production system often uses all four; calling all of them “events” hides important differences.

                           CONTROL PLANE
             topic policy • auth • membership • metrics
                    │             │             │
                    v             v             v
┌────────────────────────┐  ┌──────────────────────────┐
│ STATE PLANE            │  │ NOTIFICATION PLANE       │
│ database / state owner │  │ Registry/Phoenix.PubSub │
│ authoritative facts    │─▶│ fast, transient fanout   │
└────────────┬───────────┘  └────────────┬─────────────┘
             │                            │
             │ committed outbox           │ mailbox message
             v                            v
┌────────────────────────┐  ┌──────────────────────────┐
│ WORK PLANE             │  │ SUBSCRIBER PROCESS       │
│ Oban / RabbitMQ        │  │ receive • validate       │
│ retry • ack • replay   │  │ coalesce • render        │
└────────────┬───────────┘  └────────────┬─────────────┘
             │                            │
             v                            v
   external durable effect      ephemeral local reaction

Invariant: notification may point to state; it must not impersonate durable state.

Theory Primer

Theory Chapter 1: BEAM Processes, Signals, and Mailboxes

Fundamentals

An Elixir process is an isolated BEAM execution context with its own heap, reduction budget, links, monitors, and mailbox. send/2 places an asynchronous signal toward a destination PID and immediately returns the message term; it does not wait for receipt, pattern matching, successful handling, or a downstream effect. A receiving process examines its mailbox through receive or an OTP behavior callback. Messages from one sender to the same destination preserve signal order, but messages from different senders may interleave. The travel time is unspecified, a dead destination does not process the message, and a distribution connection failure can lose in-flight signals. These rules are the physical substrate beneath every Elixir Pub/Sub abstraction.

Deep Dive

Pub/Sub multiplies ordinary process messaging. A publisher resolves a subscriber set, then a dispatcher sends a term to each PID. The destination mailboxes—not the topic registry—hold the pending work. This distinction explains why a fast broadcast function can coexist with a severely delayed subscriber: enqueueing finished quickly, but consumption did not. Each subscriber owns its receive loop and therefore its failure, latency, selective-receive scan cost, and overload behavior.

The strongest practical ordering guarantee is narrow: signals sent in sequence by one entity to the same destination arrive in that order. There is no universal ordering across publishers. Even a single logical publisher may restart with a new PID, creating a new sender identity. If an application needs a domain order, the event envelope must carry a sequence, stream revision, or causality marker and receivers must define what to do with gaps, duplicates, and stale generations. A timestamp is not a total order; wall clocks can move and different nodes disagree.

Selective receive searches for the first message matching the available clauses. Unmatched messages remain. Repeatedly searching past a large unmatched prefix increases work and lets forgotten protocol messages accumulate. OTP behaviors reduce arbitrary receive-loop mistakes by defining protocol shapes, but they do not make the mailbox bounded. A process handling GenServer.call traffic, Pub/Sub broadcasts, timers, monitor messages, and internal work still has one mailbox.

Message isolation is semantically more important than implementation folklore about copying. Small terms are usually copied between process heaps; reference-counted binaries can be shared by the VM. Either way, the receiver cannot mutate the sender’s value. Large payload fanout can still create allocation, reference, serialization, and garbage-collection pressure. Prefer compact identifiers or summaries when subscribers can load authoritative data, and measure before assuming that millions of lightweight processes make a huge payload free.

Process death defines another boundary. When a subscriber exits, its mailbox disappears. Registry-owned subscriptions tied to that PID can be removed automatically, yet any messages not handled are gone. A supervisor may start a replacement PID, but the replacement has neither the old mailbox nor magical knowledge of missed broadcasts. This is why Phoenix.PubSub is appropriate for UI freshness and cache invalidation but insufficient by itself for “this payment must eventually be captured.”

Links and monitors carry asynchronous signals too. A Pub/Sub consumer that depends on another process should monitor it when lifecycle evidence matters. The race between “look up PID” and “send” never disappears: the PID can die after lookup. send/2 to a dead local PID behaves like a no-op from the caller’s perspective. Correctness must come from idempotency, durable state, acknowledgement, or re-reading—not from assuming the process stayed alive for one more microsecond.

Timers and replies share the same mailbox, which adds another subtlety. A timeout message may be queued behind ordinary work, so “timeout after 1,000 ms” does not always mean the timeout clause executes at exactly that wall-clock instant. Correlated request/reply protocols need unique references and must decide what to do with a late reply after the caller has timed out. Under Pub/Sub fanout, avoid synchronous reply collection from an unbounded audience: the subscriber set can change during collection, one receiver may never answer, and the collector itself becomes a mailbox hotspot. Use bounded cohorts, deadlines, and a durable progress model when acknowledgement from many recipients is a real requirement.

How this fits on projects: Projects 1–3 expose raw sends, receive protocols, restart identity, and mailbox inspection. Projects 5–12 rely on these mechanics when framework subscribers lag or disappear.

Definitions and key terms

• PID: Identity of one process incarnation.
• Mailbox: Per-process queue of unhandled messages.
• Signal: Asynchronous communication unit; ordinary messages are the most common signal.
• Selective receive: Pattern-based scan that leaves unmatched messages queued.
• Reduction: Approximate unit of BEAM scheduler work.
• Monitor: One-way lifecycle observation producing a DOWN message.

Mental model diagram

publisher A ── A1 ── A2 ─────────────┐
                                      v
publisher B ── B1 ── B2 ──▶ [ subscriber mailbox ] ──▶ receive loop
                                      │ A1 before A2        │
                                      │ B1 before B2        ├─ validate
                                      │ interleaving varies ├─ update state
                                      │ unmatched stays     └─ side effect
                                      v
                            queue length and oldest age

How it works

1. The publisher creates an immutable message term.
2. The runtime resolves the PID and queues an asynchronous signal.
3. Signals from the same sender to that receiver retain order; other senders interleave.
4. The receiver becomes runnable, scans for a matching clause, and executes the handler.
5. A crash discards remaining mailbox contents; supervision starts a new incarnation if configured.

Invariants: process state is isolated; handling is receiver-driven; sender-specific order is narrower than domain order.

Failure modes: dead destination, mailbox growth, selective-receive starvation, stale messages after logical restart, large-payload pressure, and lost in-flight distributed signals.

Minimal concrete example

SPAWN subscriber S
SEND S {:reading, producer=A, seq=1}
SEND S {:reading, producer=A, seq=2}
SEND S {:reading, producer=B, seq=9}

VALID receive trace: A1, B9, A2
INVALID receive trace: A2, A1, B9

Common misconceptions

• “Send success means processing success.” It means only that the send expression completed.
• “Mailboxes provide automatic backpressure.” They are unbounded queues unless your protocol adds a bound.
• “Pub/Sub has one global order.” The BEAM guarantee is per sender–receiver signal order.
• “A restarted process resumes its mailbox.” It starts with a new PID and empty mailbox.

Check-your-understanding questions

1. Why can two subscribers observe different interleavings of messages from two publishers?
2. What observable metrics distinguish a healthy busy subscriber from a permanently falling-behind subscriber?
3. Why is a logical producer generation useful after restart?

Check-your-understanding answers

1. Ordering is preserved per sender–receiver pair, not globally across independent senders and destinations.
2. Queue length trend, oldest-message age, processing latency, arrival rate, completion rate, and drop/coalesce counters.
3. It lets receivers reject delayed messages from an obsolete process incarnation even when the logical ID is unchanged.

Real-world applications: WebSocket sessions, device gateways, stateful workers, LiveView processes, GenServer routers, and supervisor lifecycle monitors.

Where you will apply it: Project 1, Project 7, and every later project.

References: Erlang/OTP 29 “Communication in Erlang” and “Processes”; Elixir Process and GenServer documentation; Fred Hébert, Erlang in Anger; Designing for Scalability with Erlang/OTP process chapters.

Key insight: Every Pub/Sub guarantee eventually lands in an ordinary process mailbox, so subscriber behavior—not broadcast syntax—determines end-to-end health.

Summary: BEAM messaging gives isolation and narrow ordering, but not processing acknowledgement, durability, or flow control.

Homework/exercises

1. Draw three valid interleavings for two publishers sending two events each to one subscriber.
2. Define a five-field event identity that survives producer restart.
3. Write an overload policy for a dashboard that only needs the newest temperature per sensor.

Solutions

1. Preserve A1 before A2 and B1 before B2; any merge satisfying those constraints is valid.
2. One answer is {event_id, stream_id, stream_revision, producer_generation, occurred_at}.
3. Keep one latest value per sensor, replace older pending values, render on a fixed tick, and count coalesced updates.

Theory Chapter 2: Pub/Sub Semantics, Topics, and Event Contracts

Fundamentals

Pub/Sub separates publishers from the current set of subscribers through a routing key, usually called a topic. A publisher expresses “something happened under this topic”; a router finds interested processes; each subscriber independently interprets the message. This is different from a work queue, where one unit of work is normally claimed by one consumer, and from request/reply, where the caller expects a correlated answer. A useful event contract specifies identity, type, version, subject, ordering scope, timestamp semantics, payload, and tenant or authorization context. Without that contract, a topic name is only a string and a broadcast is merely an undocumented term sent to many processes.

Deep Dive

The central benefit of Pub/Sub is temporal and spatial decoupling at the routing layer. Publishers do not enumerate receivers, and subscribers can join or leave without publisher code changing. Temporal decoupling is limited, however: an ephemeral subscriber must be alive and subscribed when a broadcast occurs. A durable broker can retain messages for later, but Phoenix.PubSub does not become durable simply because it spans nodes.

Topic design is part API design. Topics should describe a bounded audience or domain stream, not act as a covert database query language. tenant:42:order:913 can provide a clear authorization boundary; all_updates creates unnecessary fanout and pushes filtering into every mailbox. Highly cardinal topics can be appropriate when lifecycle is automatic, while unbounded atom creation from user input is never appropriate because atoms are not garbage collected. Phoenix topics are binaries, which helps avoid that particular risk.

Separate event type from topic. The topic selects candidate recipients; the event type tells them what occurred. One topic can carry a small family of versioned messages, but every subscriber should reject unknown versions deliberately rather than crash unpredictably. A stable envelope might include event_id, event_type, schema_version, tenant_id, subject, stream_revision, producer, producer_generation, occurred_at, correlation_id, and a compact payload. Not every project needs every field. The design exercise is to justify omissions.

Commands and events have different tense and ownership. {:reserve_inventory, ...} is a command directed at a responsible handler; {:inventory_reserved, ...} is a fact notification. Broadcasting a command to many subscribers creates ambiguity about who should execute it. Conversely, treating a transient “inventory reserved” notification as the authoritative record makes late subscribers wrong. The state owner commits the fact; Pub/Sub tells current observers to refresh or react.

Delivery labels require precision. At-most-once means no automatic retry; messages may be lost. At-least-once means retry can create duplicates, so effects must be idempotent or deduplicated. Exactly-once delivery is often a misleading phrase: a broker may deduplicate transport while an external side effect still happens twice after an uncertain timeout. State the boundary—enqueue, handler completion, database transaction, or external effect—and the failure assumptions.

Ordering is scoped similarly. One sender’s sends to one PID retain order, a broker partition may preserve record order within that partition, and a database stream revision can define domain order for one aggregate. None implies a single order across every topic, publisher, node, and subscriber. If a subscriber sees revision 12 after revision 10, it needs a gap policy: wait briefly, reload current state, fetch missing durable events, or mark itself stale. A UI can often reload; a financial projection may require replay.

Topic subscriptions also carry security meaning. Do not allow untrusted clients to construct arbitrary tenant topics and subscribe directly. The boundary that authenticates a socket or process must authorize the subject, derive the topic server-side, and avoid placing secrets or sensitive personal data in topic names, logs, metrics labels, or broadcast payloads. Subscriber-side filtering is not access control because the message already reached the unauthorized process.

Contracts should also specify deprecation evidence: which publisher versions remain active, which subscriber versions still receive traffic, and when an old schema can be removed. Count unknown and legacy versions with bounded labels, rehearse mixed-version rollouts, and retain representative fixtures. Compatibility is an operational transition, not merely a decoder function.

How this fits on projects: Project 2 introduces topic routing; Project 3 formalizes event envelopes and compatibility; Projects 4–12 reuse the contract across UI, broker, and cluster boundaries.

Definitions and key terms

• Topic: Logical routing key used to select subscribers.
• Event envelope: Stable metadata wrapped around a versioned payload.
• Fanout: One publication producing deliveries to multiple subscribers.
• Command: Request for an owner to attempt an action.
• Event: Record or notification that something has occurred.
• Idempotency key: Stable identity used to make retries safe.

Mental model diagram

                          topic selects audience
publisher ── envelope ──▶ tenant:42:orders ─┬─▶ LiveView: refresh summary
                                            ├─▶ cache: invalidate order 913
                                            └─▶ audit observer: record metric

envelope:
┌──────────┬────────────┬─────────┬──────────────┬───────────────┐
│ event_id │ type + ver │ tenant  │ stream + rev │ compact data  │
└──────────┴────────────┴─────────┴──────────────┴───────────────┘
          identity        auth          order          meaning

How it works

1. The state owner validates a command and commits a fact.
2. It constructs a versioned envelope with stable identity and ordering scope.
3. Server-side policy derives the authorized topic.
4. The router selects current subscribers and sends the envelope.
5. Each subscriber validates type/version, checks freshness/idempotency, and performs a bounded reaction.

Invariants: topic names do not prove durability; subscribers do not define authoritative truth; authorization happens before subscription and publication.

Failure modes: topic explosion, accidental global fanout, incompatible payload versions, command broadcasts, missing correlation, unauthorized cross-tenant subscription, and duplicate side effects.

Minimal concrete example

TOPIC: tenant:42:order:913
EVENT: {
  event_id: "evt_01...",
  event_type: "order.status_changed",
  schema_version: 2,
  stream_revision: 18,
  payload: {from: "paid", to: "packed"}
}

Common misconceptions

• “A topic is an event type.” It chooses an audience; the payload still needs a type.
• “Pub/Sub means subscribers can be offline.” Only a retained/durable system supports that.
• “Exactly once is a property of one API call.” It must be defined through the external effect boundary.
• “Filtering in the subscriber protects tenant data.” Delivery has already occurred.

Check-your-understanding questions

1. When should tenant:42:orders be preferred over all_orders?
2. Why include both event_id and stream_revision?
3. What should a UI do after detecting a revision gap?

Check-your-understanding answers

1. When tenant isolation and bounded fanout are part of the routing contract.
2. The ID deduplicates an event; the revision orders facts within a specific stream.
3. Mark its projection stale and reload authoritative state or fetch missing durable events.

Real-world applications: cache invalidation, UI freshness, collaboration cursors, domain notifications, audit signals, and integration-event bridges.

Where you will apply it: Project 3, Project 9, and Project 12.

References: Enterprise Integration Patterns chapters on Publish-Subscribe Channel and Message; Designing Data-Intensive Applications, 2nd Edition stream-processing chapters; CloudEvents 1.0 specification; Phoenix.PubSub official documentation.

Key insight: A Pub/Sub system is only as trustworthy as its explicit topic, envelope, ordering, delivery, and authorization contracts.

Summary: Topics decouple routing, while event contracts preserve meaning across time, restarts, versions, and consumers.

Homework/exercises

1. Classify six messages from an e-commerce system as command, event, query, or reply.
2. Design topics for tenant, order, and operator-wide status without leaking tenant data.
3. Define a gap policy for a cache and a stricter one for an accounting projection.

Solutions

1. Imperative requests are commands; completed facts are events; reads are queries; correlated results are replies.
2. Use server-derived tenant/order topics and a separately authorized operator topic—not a client-chosen wildcard.
3. Cache: invalidate/reload current state. Accounting: stop projection, fetch/replay missing durable revisions, then resume.

Theory Chapter 3: Local Routing with Registry and Custom Dispatch

Fundamentals

Elixir Registry is a local, decentralized process registry backed by ETS tables. With keys: :duplicate, many processes may register under the same key, which makes a key behave like a local Pub/Sub topic. Registry.dispatch/3 runs a callback in the dispatching process for each non-empty registry partition, passing {pid, value} entries. The callback decides how to send, filter, batch, or annotate messages. Registrations belong to the registering process and are removed when it dies, though removal visibility can be delayed and a PID may die immediately after lookup. Registry is node-local even when distributed Erlang is enabled; clustering does not turn it into a global topic directory.

Deep Dive

Registry is the best bridge between raw messaging and Phoenix.PubSub because it exposes subscription mechanics without requiring you to build lifecycle cleanup tables yourself. A subscriber calls Registry.register/3 for a topic and optional metadata. The registry links ownership of that entry to the process. A publisher calls dispatch/3; inside the callback it sends the event to each PID. The registered value can carry preferences such as accepted event versions, priority class, or a precomputed filter—but it must remain small and should not contain secrets that diagnostics might reveal.

The dispatch callback executes in the publisher/caller context, not in subscriber processes. Expensive filtering, serialization, logging, or network calls inside dispatch increase publication latency and can make the caller a hotspot. The callback should perform bounded routing work and enqueue compact messages. Business logic belongs in subscribers or a dedicated pipeline. If custom dispatch needs per-subscriber authorization, prefer pre-authorized registrations or topic segmentation rather than running a database query per PID during every broadcast.

Partitioning changes invocation shape. Registry can use several partitions for concurrent registration and lookup. dispatch/3 may call the callback once per non-empty partition; with parallel: true, callbacks may run in spawned tasks. Parallel dispatch can improve throughput, but it changes scheduling and makes any assumed cross-partition send order invalid. Start with one or a measured default, benchmark representative fanout and churn, then tune. Partition count is an operational choice, not a magical “number of cores” checkbox.

Duplicate registration is literal. A process may register the same key more than once, and a naive dispatcher may send duplicate messages to the same PID. Decide whether multiple registrations represent separate logical subscriptions or should be rejected by your wrapper. unregister/2 removes all entries for the key owned by the caller. Subscription APIs should document whether calling subscribe twice is idempotent.

Lifecycle automation narrows but does not eliminate races. Registry removes entries after a subscriber dies, yet a dispatch can observe a PID that is already dead or dies before send. Sending locally to a dead PID does not give the dispatcher a processing error. Do not build acknowledgement semantics out of registry membership. If the publisher needs proof, add a correlated reply and timeout for a bounded set of receivers, or move required work to a durable queue.

Registry metadata and select/2 can support introspection, but scanning every entry for dashboards can be expensive and may expose tenant boundaries. Provide aggregated metrics such as topic count, subscription count, fanout size, and dispatch duration. Avoid labels containing raw high-cardinality topics. Subscriber-specific queue length inspection is useful in a lab, but production monitoring should sample or have subscribers report their own health.

Finally, Registry is local. A process registered on node_a is absent from node_b’s Registry. You can deliberately build a two-level design—local Registry for fast fanout, plus a cluster adapter that forwards one copy per node—rather than pretending there is one global ETS table. Phoenix.PubSub packages that architecture and migration logic; Project 2 teaches why it exists.

How this fits on projects: Project 2 builds a Registry bus; Project 3 adds contracts and custom dispatch; Project 4 compares the result with Phoenix.PubSub; Project 10 exposes the local/distributed split.

Definitions and key terms

• Duplicate registry: Registry mode allowing several entries per key.
• Dispatch callback: Caller-executed function receiving subscription entries.
• Partition: One shard of Registry ETS tables and ownership processes.
• Registration value: Metadata associated with one PID/key entry.
• Stale PID race: Destination dies between discovery and use.

Mental model diagram

subscribers                    local Registry partitions
S1 register(topic, v1) ──▶ ┌──────────────┐
S2 register(topic, v2) ──▶ │ P0: S1, S3   │ ─┐
S3 register(topic, v3) ──▶ │ P1: S2       │  ├─ dispatch callbacks in publisher
                            └──────────────┘ ─┘          │
publisher dispatch(topic) ──────────────────────────────┼─ send event to S1
                                                        ├─ send event to S2
                                                        └─ send event to S3

No entry from another BEAM node appears here.

How it works

1. Start a supervised Registry with duplicate keys and an explicit partition count.
2. A subscriber registers itself under a binary topic and optional metadata.
3. A publisher dispatches the topic; Registry invokes the callback per non-empty partition.
4. The callback applies bounded routing rules and sends to each destination PID.
5. Subscriber exit triggers eventual automatic registration cleanup.

Invariants: subscriptions are process-owned and local; callback work charges the publisher path; observed PIDs can already be dead.

Failure modes: duplicate subscriptions, blocking dispatch callbacks, accidental cross-tenant fanout, unmeasured partition changes, local-only assumptions in a cluster, and false acknowledgement claims.

Minimal concrete example

START Registry(keys=:duplicate, partitions=4)
SUBSCRIBE current_pid TO "sensor:west" WITH {format=v2}
DISPATCH "sensor:west":
  FOR EACH {pid, metadata} IN partition_entries
    SEND pid {:sensor_event, v2, compact_payload}

Common misconceptions

• “Registry becomes distributed when nodes connect.” It remains node-local.
• “Dispatch runs in subscribers.” The dispatch callback executes on the caller path.
• “Process death makes every concurrent lookup fresh.” Cleanup and lookup/use races still exist.
• “Duplicate keys deduplicate PIDs.” One PID can register the same key multiple times.

Check-your-understanding questions

1. Why can a four-partition Registry invoke a dispatch callback several times for one publication?
2. Where should expensive per-event transformation occur?
3. How would you prove that subscribe is idempotent?

Check-your-understanding answers

1. Each non-empty partition supplies its own entry batch; parallel mode may invoke them concurrently.
2. Before dispatch if shared, or in subscriber/pipeline processes—not repeatedly inside the caller’s routing loop.
3. Subscribe twice, publish once, and assert one delivery while introspection shows one logical entry for that PID/topic.

Real-world applications: local cache invalidation, plugin notifications, test event buses, local telemetry subscribers, and per-node fanout layers.

Where you will apply it: Project 2, Project 3, and Project 10.

References: Elixir 1.20 Registry documentation, especially duplicate keys, dispatch, registrations, and partitions; Elixir Process.monitor/1; Phoenix.PubSub source and adapter documentation.

Key insight: Registry gives you fast local membership and customizable fanout, but the caller still owns dispatch cost and no delivery acknowledgement is created.

Summary: A duplicate Registry is a robust local Pub/Sub primitive and the clearest stepping stone to Phoenix.PubSub architecture.

Homework/exercises

1. Design metadata for subscribers accepting event schema versions 1 and 2.
2. Predict deliveries when one PID registers the same topic twice.
3. Sketch a two-level local Registry plus one-relay-per-node topology.

Solutions

1. Store a compact accepted-version set and route only compatible envelopes; unknown versions remain observable.
2. A naive dispatch sends twice. Make wrapper subscription idempotent or deduplicate destinations intentionally.
3. Publish once to remote node relays, then each relay dispatches through its own node-local Registry.

Theory Chapter 4: Phoenix.PubSub Architecture and Dispatch APIs

Fundamentals

Phoenix.PubSub is a supervised real-time publish/subscribe service that combines local subscription storage with an adapter for cross-node forwarding. In Phoenix.PubSub 2.2, local subscriptions are stored in a duplicate Registry, local delivery becomes ordinary send operations, and the default Phoenix.PubSub.PG2 adapter forwards one broadcast between connected BEAM nodes using :pg groups before each node performs its own local fanout. The public API distinguishes cluster-wide broadcast, node-local local_broadcast, sender-excluding broadcast_from, and node-targeted direct_broadcast. These choices encode meaningful routing scope, not stylistic preferences. A return of :ok means the fanout routine completed its sends; it is not subscriber receipt, handler completion, replay, or durability.

Deep Dive

The architecture is intentionally two-level. Each node owns a Registry containing only its local subscribers. The adapter avoids shipping one cross-node copy per remote subscriber; it forwards a publication to a PubSub worker on each relevant node, and that worker dispatches locally. This reduces distribution traffic and preserves process lifecycle locality. It also makes the boundaries visible: node discovery and connectivity happen elsewhere, cross-node forwarding is transient, and slow subscribers accumulate local mailbox work.

subscribe/3 always subscribes the calling process. This makes lifecycle ownership natural: a LiveView, Channel, GenServer, or task joins a topic and its registration disappears when that process exits. Subscription calls can be duplicated; duplicate PID/topic registrations cause duplicate delivery, and one unsubscribe removes the caller’s registrations for that topic. Application wrappers should usually make repeated subscribe calls idempotent, especially around reconnect and callback re-entry.

Choose the broadcast API by semantics. broadcast/4 includes the publisher if it is subscribed. broadcast_from/5 excludes the given PID and is useful when a socket already applied an optimistic local change. local_broadcast/4 deliberately avoids cluster forwarding for node-specific cache or process events. direct_broadcast/5 targets one node and belongs in controlled routing or diagnostic logic, not a general “more reliable” path. The raising variants surface adapter errors differently; neither supplies end-to-end acknowledgement.

The optional dispatcher module can filter or transform delivery using subscription metadata. This is powerful for fastlane-style encoding or protocol-specific routing, but it inherits Registry constraints. Dispatch code runs on the fanout path, must handle partition batches, and should remain bounded. A dispatcher exception can fail publication. Do not perform blocking I/O or unbounded tenant checks inside it.

Sizing has separate local and cluster-facing knobs in 2.2. registry_size controls local subscription Registry partitions and can be tuned independently per node. pool_size controls PubSub adapter workers, while broadcast_pool_size selects the shard range used during cross-node broadcast; those broadcast settings must remain compatible across nodes. A rolling change from pool size 1 to 2 uses two deployments: first deploy pool_size: 2, broadcast_pool_size: 1 everywhere, then deploy pool_size: 2 without the compatibility override. Reverse the sequence when shrinking. This migration prevents mixed broadcast pools from missing broadcasts solely because nodes hash into different shard sets; it does not prevent loss during partitions or add persistence.

Topic and envelope design remain application responsibilities. Phoenix.PubSub accepts a binary topic and arbitrary term. That flexibility makes it easy to send database structs containing fields a subscriber should never see or to create one catch-all topic. Wrap the library behind domain-specific functions that derive topics server-side, emit stable envelopes, instrument attempts, and document whether echo is included. A wrapper is also the right place to propagate correlation metadata and forbid huge payloads.

Testing should avoid sleeping. Subscribe the test process, trigger a publication, and assert the exact envelope. Test sender inclusion/exclusion explicitly. For cross-node tests, start separate named nodes or isolated OS processes and prove both positive delivery and behavior under disconnect. A test that calls publish and immediately asserts :ok proves almost nothing about a consumer.

Finally, do not confuse Phoenix.PubSub with Phoenix Channels or LiveView. Channels manage client transports and socket topics; LiveView manages server-rendered UI processes. Both commonly use PubSub internally or explicitly, but PubSub itself is transport-agnostic process fanout. A CLI worker and a LiveView can subscribe to the same server-side topic without either knowing the other exists.

How this fits on projects: Project 4 replaces the custom local bus with Phoenix.PubSub; Projects 5–6 expose browser and presence lifecycles; Projects 10–12 use cluster forwarding and safe operational configuration.

Definitions and key terms

• Adapter: Component responsible for cross-node PubSub forwarding.
• PG2 adapter: Compatibility-named default adapter using OTP :pg on current runtimes.
• Dispatcher: Module deciding local per-subscriber delivery.
• Pool size: Number of adapter broadcast shards/workers.
• Registry size: Number of local subscription Registry partitions.
• Broadcast from: Publication that excludes a specified publisher PID.

Mental model diagram

node A                                                     node B
┌─────────────────────────────┐                           ┌─────────────────────────────┐
│ publisher                   │                           │ PG2 worker / :pg member     │
│      │ broadcast            │ one cross-node forward   │             │               │
│      v                      ├──────────────────────────▶│             v               │
│ PG2 worker / :pg member     │                           │ local Registry.dispatch     │
│      │                      │                           │        ├─▶ subscriber B1    │
│      v                      │                           │        └─▶ subscriber B2    │
│ local Registry.dispatch     │                           └─────────────────────────────┘
│   ├─▶ subscriber A1         │
│   └─▶ subscriber A2         │
└─────────────────────────────┘

No acknowledgement arrow returns from subscriber handlers.

How it works

1. Start Phoenix.PubSub under the application supervisor with an explicit name and measured sizing.
2. Each subscriber registers its PID and optional metadata under a binary topic.
3. A publisher invokes a local, cluster, sender-excluding, or direct broadcast function.
4. The adapter forwards one copy through the selected cluster shard when required.
5. Every receiving node dispatches to its node-local subscriber Registry.

Invariants: subscriptions are PID-owned; local fanout uses process messages; cluster forwarding needs already-connected nodes; :ok is not consumption acknowledgement.

Failure modes: duplicate subscriptions, unsafe custom dispatch, incompatible rolling pool sizes, node partitions, large payload fanout, accidental echo, and unauthorized topic construction.

Minimal concrete example

SUBSCRIBE "tenant:42:alerts"
BROADCAST_FROM publisher_pid, "tenant:42:alerts", alert_v2

publisher mailbox: no echo
subscriber A: receives alert_v2
subscriber B on connected node: receives alert_v2
offline subscriber: receives nothing and has no replay

Common misconceptions

• “PG2 uses the removed :pg2 module.” Current Phoenix.PubSub uses OTP :pg where available.
• “broadcast discovers nodes.” Node connectivity must already exist.
• “Safe pool migration makes PubSub durable.” It only bridges mixed shard counts during rolling configuration changes.
• “The adapter stores subscriptions cluster-wide.” Subscription registries remain local.

Check-your-understanding questions

1. Why forward once per remote node rather than once per remote subscriber?
2. When is local_broadcast semantically safer than broadcast?
3. What does a two-stage pool-size migration protect against?

Check-your-understanding answers

1. It reduces distribution encoding/network fanout and lets the destination node perform local sends.
2. When the event is meaningful only inside one node, such as local cache/process coordination.
3. Nodes hashing broadcasts across incompatible shard ranges while different releases coexist.

Real-world applications: Phoenix Channels, LiveView refresh notifications, cache invalidation, collaborative tools, presence transport, and node-local operational signals.

Where you will apply it: Project 4, Project 5, and Project 10.

References: Phoenix.PubSub 2.2 module, PG2 adapter, adapter behavior, and source documentation; Elixir Registry documentation; OTP 29 :pg documentation.

Key insight: Phoenix.PubSub scales a local Registry fanout pattern across connected nodes without pretending transient sends are durable deliveries.

Summary: The API gives deliberate local, cluster, sender-excluding, and direct paths; the application must still own meaning, security, load, and recovery.

Homework/exercises

1. Choose the correct API for local cache eviction, cluster UI refresh, and optimistic sender echo suppression.
2. Write the two deployment configurations for a safe pool change from two to four.
3. List what remains unprotected during that migration.

Solutions

1. Local eviction: local_broadcast; cluster refresh: broadcast; suppress publisher: broadcast_from.
2. First deploy pool 4 with broadcast pool 2, then deploy pool 4 with broadcast pool 4/default; reverse to shrink.
3. Network partitions, dead subscribers, slow mailboxes, handler failures, missed offline events, and application schema incompatibility.

Theory Chapter 5: Distributed Erlang, :pg, Topology, and Partitions

Fundamentals

Distributed Erlang connects named BEAM nodes so PIDs, registered names, links, monitors, and terms can cross machine boundaries. OTP :pg adds named process groups whose membership can span directly connected nodes. It does not discover nodes, elect a leader, persist events, send application messages, or provide an instantaneous global view. A process joins its local node’s group; clients obtain members and decide whom to message. Membership is strongly eventually consistent and explicitly non-transitive: if two leaf nodes each connect only to a hub, their views need not include one another. Network partitions produce component-local membership and transient Pub/Sub delivery has no replay after reconnection.

Deep Dive

Distribution adds serialization, network delay, connection lifecycle, authentication, topology, and partial failure to the local process model. A remote PID still looks like a PID, which is productive but can make network failure feel less real than an HTTP client error. Remote sends remain asynchronous. If the distribution channel goes down, signals may be lost; a later reconnect does not restore the old channel’s in-flight application messages.

Node discovery is separate from Pub/Sub. libcluster, DNS, Kubernetes, platform-private DNS, or explicit Node.connect logic identifies candidate nodes and establishes connections. Phoenix.PubSub relies on the resulting topology. A healthy PubSub process on an isolated node can only fan out within its connected component. Therefore monitor both application metrics and node connectivity; “publish returned :ok” is compatible with a split cluster.

OTP :pg represents group membership, not a broker. Only local processes can be joined by a node, and dead members are removed automatically. A caller may read local members or its current distributed view and send one, some, or all. Membership views converge among directly connected peers, but can temporarily differ after join, leave, crash, or reconnect. Do not use a :pg snapshot as a lease proving a process remains alive.

Topology matters. Full mesh gives every node direct visibility but grows connection count roughly with the square of node count. Alternative topologies reduce connections but violate assumptions made by libraries expecting a fully connected mesh. OTP :pg membership is non-transitive: a hub seeing both leaves does not make the leaves see each other. Before deploying a topology, verify the adapter’s requirements and run the exact connection graph under failure.

Partitions force an application choice. For ephemeral UI notifications, each component may continue and clients reload authoritative state after reconnect. For a singleton business action, continuing independently on both sides may be unsafe; Pub/Sub is not consensus and :pg is not a distributed lock. Use a database constraint, quorum-backed service, or explicitly chosen leader mechanism where mutual exclusion matters.

Node restart changes identities and membership. A restarted process has a new PID and rejoins. A restarted :pg scope loses its local memberships until processes join again. Subscribers should subscribe in their own supervised initialization and treat reconnect as a new incarnation. Cross-node event envelopes should carry stable logical IDs rather than using PIDs as durable identity.

Distribution security begins with the Erlang cookie, which authenticates nodes but is not a substitute for a complete network security posture. Keep distribution on trusted private networks or configure TLS distribution, restrict ports and node membership, rotate secrets deliberately, and never connect arbitrary customer-controlled nodes. A connected node has powerful runtime capabilities. Application tenant authorization must still occur above node authentication.

Rolling deploys combine topology and compatibility. Nodes running different event schema versions coexist. Subscribers must accept a supported compatibility window or publishers must version topics/payloads. PubSub pool-size changes require the documented staged settings. A chaos test should cover node kill, graceful shutdown, one-way or symmetric disconnect, reconnection, and mixed-version publication—not only a three-node happy path.

How this fits on projects: Project 10 builds a three-node laboratory; Project 11 bridges outside the BEAM cluster; Project 12 requires explicit partition and reconnection behavior.

Definitions and key terms

• Node discovery: Finding candidates to connect; independent of Pub/Sub.
• Distribution channel: BEAM connection carrying encoded signals between nodes.
• Process group: Named :pg membership set.
• Strong eventual consistency: Views converge after updates stop and communication resumes, while temporary divergence is allowed.
• Non-transitive membership: Indirectly connected nodes do not automatically share the same group view.
• Partition: Network split creating independently connected components.

Mental model diagram

full mesh                            hub-and-spoke
A────────B                           A       B
│╲      ╱│                            ╲     ╱
│ ╲    ╱ │                             ╲   ╱
│  ╲  ╱  │                              HUB
│   ╲╱   │                               │
C────────D                               C

All peers directly connected          HUB may see A+B+C
compatible with shared :pg view       A need not see B or C membership

partition:
[A──B]  X  [C──D]  -> two valid local views, no replay of missed PubSub events

How it works

1. Start named nodes with controlled authentication and distribution ports.
2. Discovery establishes the intended connection topology.
3. Local PubSub workers join adapter :pg groups.
4. A cluster broadcast reads its current membership view and forwards to reachable workers.
5. On partition, components continue with partial views; reconnection later converges membership, not missed application messages.

Invariants: connectivity precedes cluster fanout; group views can diverge; remote PID syntax does not imply reliability; Pub/Sub cannot provide consensus.

Failure modes: incomplete topology, cookie or TLS mismatch, oversized remote payloads, silent component-local fanout, stale PIDs, mixed-schema nodes, and split-brain business actions.

Minimal concrete example

INITIAL: node_a <-> node_b <-> node_c, with full mesh verified
EVENT 41: received on A, B, C
PARTITION: {A,B} separated from {C}
EVENT 42 from A: received on A, B; absent on C
RECONNECT: membership converges
EVENT 43: received on A, B, C
EVENT 42: still absent on C unless C reloads/replays elsewhere

Common misconceptions

• “:pg broadcasts messages.” It returns members; clients or adapters send.
• “A hub makes membership transitive.” Direct connectivity rules still apply.
• “Reconnection replays missed PubSub.” Membership converges; transient application messages do not.
• “The cookie encrypts and authorizes application traffic.” It is node authentication, not tenant policy or automatically secure transport.

Check-your-understanding questions

1. Why can node A and node B have different group views while both are correct under the model?
2. What state should a LiveView reload after a partition heals?
3. When is continuing on both sides of a partition unsafe?

Check-your-understanding answers

1. Membership is eventually consistent and non-transitive; current direct connections and propagation timing differ.
2. Authoritative current state and any durable revision needed to detect gaps.
3. When operations require a global singleton, uniqueness, or irreversible side effect without a quorum-backed authority.

Real-world applications: multi-node Phoenix applications, distributed Presence, regional real-time dashboards, multiplayer rooms, and clustered cache invalidation.

Where you will apply it: Project 10 and Project 12.

References: OTP 29 :pg; Erlang “Communication in Erlang” and distribution documentation; Phoenix.PubSub PG2 adapter; libcluster documentation; Designing for Scalability with Erlang/OTP distribution chapters.

Key insight: Distributed Pub/Sub gives low-friction fanout inside the current connected component, not a single durable or consensus-backed global bus.

Summary: Treat node discovery, topology, membership convergence, security, schema compatibility, and partition recovery as separate design obligations.

Homework/exercises

1. Draw membership views for three nodes connected only through B.
2. Define UI and payment behavior during the same partition.
3. Write a five-step rolling-deploy verification checklist.

Solutions

1. B may see A and C; A and C need not see one another’s group members.
2. UI may continue with stale markers and reload later; payment uniqueness must rely on a durable authoritative constraint/quorum.
3. Verify topology, schema window, PubSub pool compatibility, component-local behavior, then convergence and state reload after reconnection.

Theory Chapter 6: Flow Control, Mailbox Safety, GenStage, and Broadway

Fundamentals

Plain BEAM sends and Phoenix.PubSub are push mechanisms: publishers can enqueue messages faster than subscribers process them. There is no implicit demand signal, bounded mailbox, or rejection when a subscriber falls behind. Flow control is therefore an application contract. Options include slowing producers, bounding concurrency, coalescing superseded updates, sampling, dropping by priority, sharding hot subscribers, or moving work to demand-aware abstractions. GenStage models producer–consumer demand per subscription. Its default dispatcher distributes events rather than broadcasting them; BroadcastDispatcher sends every event to every consumer but progress is constrained by the slowest consumer’s demand. Broadway builds concurrent broker-ingestion pipelines with demand, processors, optional batching, and acknowledgement callbacks.

Deep Dive

The first overload metric is not a one-time mailbox length; it is the relationship among arrival rate, completion rate, queue-length trend, and oldest-message age. A queue of 1,000 tiny messages draining faster than arrival may be healthy. A queue of 20 growing every minute can be an outage in slow motion. Instrument handler latency and event age alongside sampled message_queue_len. Queue length alone does not reveal expensive messages already executing or selective-receive scan cost.

Overload policy must match semantics. UI counters and sensor readings often tolerate coalescing: retain only the newest value per key. Analytics may tolerate sampling with explicit rate metadata. Security alerts may require priority and a durable spill path. Business commands should not be silently dropped; reject at admission, apply client backpressure, or enqueue durably. “Unlimited mailbox” is not a neutral policy—it converts overload into growing latency and memory use.

Fanout magnifies cost. One event sent to 50,000 subscribers creates 50,000 enqueues and potentially large term references/copies. A single publisher process can become CPU-bound performing sends, as large real-time systems have discovered. Shard audiences, group remote recipients by node, precompute bounded routing, and move formatting to shared representations where safe. Measure scheduler utilization and reductions; do not merely count subscribers.

GenStage introduces explicit demand. A producer should emit no more than requested for each subscription, while dispatchers determine how events satisfy demand. The default demand dispatcher sends each event to one consumer, fitting work distribution. BroadcastDispatcher sends each event to all consumers and waits for demand from all; one slow subscriber throttles the group. This is genuine backpressure but may create unacceptable head-of-line coupling. Separate consumers into policy classes or use independent streams if one observer must not slow another.

GenStage can still buffer if a producer emits beyond demand, and events being processed may be lost when a consumer crashes. Demand bounds flow; it does not automatically make state durable. Manual demand is useful for asynchronous work only when the consumer asks again after its real capacity is free.

Broadway packages a robust ingestion topology. Producers feed processor stages according to demand; processors transform concurrently; optional batchers group by count/time; acknowledgers tell the upstream connector which messages succeeded or failed. Broadway itself does not implement retries. RabbitMQ, SQS, Kafka, or a custom producer determines redelivery, offsets, dead-letter handling, and persistence. Processing is concurrent and unordered by default. partition_by routes the same key to one stage, but a failed item can be retried later after subsequent items unless the broker and application enforce stricter ordering.

Bridging Phoenix.PubSub into Broadway requires honesty. Pushing raw messages with Broadway.push_messages/2 ignores configured demand at that entry point; it cannot send a backpressure signal to the original PubSub publisher. A bounded adapter must choose what happens when full—reject, coalesce, drop, or spill durably. If the source must wait for capacity, use a demand-aware producer protocol rather than an unacknowledged broadcast.

Testing overload requires sustained load and deterministic slow consumers. Establish a baseline, overload one subscriber, verify healthy subscribers remain responsive, then apply the declared policy. Acceptance evidence includes maximum queue length, p95/p99 event age, throughput, dropped/coalesced counts, recovery time after load stops, and memory trend. A system that eventually drains but violates freshness SLOs still failed.

How this fits on projects: Project 7 makes mailbox failure visible; Project 8 replaces unbounded work push with a demand-aware bridge; Projects 9, 11, and 12 combine demand with durable upstream acknowledgement.

Definitions and key terms

• Backpressure: Feedback that prevents upstream from exceeding downstream capacity.
• Demand: Explicit number of events a GenStage consumer is ready to receive.
• Coalescing: Replacing several superseded events with the newest meaningful state.
• Admission control: Rejecting or deferring work before overload enters the system.
• Acknowledgement: Signal to an upstream source about completed or failed processing.
• Dead-letter queue: Durable destination for messages that cannot be processed under retry policy.

Mental model diagram

push PubSub                         demand-aware pipeline

publisher ──▶ subscriber mailbox   broker/producer ◀── demand=20 ─ consumer
publisher ──▶ [grows without bound]      │                          │
publisher ──▶ slow handler               └── at most 20 events ────▶│

policy choices when capacity is full:
REJECT | COALESCE BY KEY | SAMPLE | DROP WITH COUNTER | DURABLE SPILL

How it works

1. Measure input rate, handler service time, event age, and mailbox trend.
2. Classify messages by loss, freshness, ordering, and durability requirements.
3. Choose admission, coalescing, sharding, demand, or durable-queue policy.
4. Configure demand/concurrency/batches from measured capacity, not CPU count alone.
5. Inject overload and verify bounds, degradation visibility, and recovery.

Invariants: PubSub push is not backpressure; demand is not persistence; acknowledgement semantics belong to the upstream connector and effect boundary.

Failure modes: unbounded mailboxes, slowest-subscriber coupling, hidden buffer growth, unordered concurrent effects, retry duplicates, blocking acknowledgers, and false backpressure claims around a push bridge.

Minimal concrete example

BEFORE: arrival=5,000/s, completion=1,200/s, queue=84,000 and rising
POLICY: coalesce dashboard updates by device_id; durable commands go to broker
AFTER: dashboard queue<=2,000, coalesced=61,400, command_lag=240ms, lost_commands=0

Common misconceptions

• “The BEAM mailbox absorbs bursts safely forever.” It converts excess rate into memory and stale work.
• “GenStage broadcast scales independent consumers.” The slowest demand can constrain BroadcastDispatcher.
• “Broadway retries failed messages.” Retry behavior belongs to the producer/broker.
• “Pushing PubSub into Broadway backpressures publishers.” A push entry cannot retroactively create upstream demand.

Check-your-understanding questions

1. Which two rates determine whether a mailbox grows over time?
2. When is coalescing correct and when is it data loss?
3. Why can partitioning by key still permit retry reordering?

Check-your-understanding answers

1. Arrival rate and sustained completion/service rate.
2. Correct when intermediate states are superseded and only freshness matters; data loss when every fact/effect must be processed.
3. A failed message may be retried after later messages have already completed unless the upstream and handler block that key.

Real-world applications: telemetry dashboards, webhook ingestion, broker consumers, bulk notification delivery, device streams, and large real-time fanout.

Where you will apply it: Project 7, Project 8, and Project 11.

References: OTP efficiency guide; Elixir Process.info/2; GenStage 1.3 dispatchers; Broadway 1.3 behavior, acknowledger, and Telemetry docs; Discord engineering posts on Elixir fanout.

Key insight: A reliable Pub/Sub design declares what happens at capacity; otherwise the mailbox silently becomes the least observable queue in the architecture.

Summary: Use push for bounded transient fanout, demand for capacity-aware work, and durable brokers for retryable responsibilities.

Homework/exercises

1. Classify chat typing, payment capture, sensor gauge, and audit fact into overload policies.
2. Explain why BroadcastDispatcher may be wrong for an optional slow auditor.
3. Design an experiment that proves recovery after a 60-second overload.

Solutions

1. Typing: drop/coalesce; payment: durable admission; sensor gauge: coalesce/sample; audit fact: durable append.
2. Its demand can throttle every consumer; isolate the auditor or feed it from a durable independent stream.
3. Record baseline, apply fixed excess rate, enforce bounds, stop load, and measure time until queue/event age/SLO return to baseline.

Theory Chapter 7: Ephemeral Notifications, Durable Work, and Transactional Outbox

Fundamentals

An ephemeral Pub/Sub message exists only while it is being routed and waiting in current subscriber mailboxes. If a subscriber is offline, a node is partitioned, or a process crashes with messages queued, Phoenix.PubSub has no replay log from which to recover. Durable messaging adds storage, acknowledgement, retry or offset state, and an explicit retention policy. A transactional outbox solves a specific dual-write problem: commit the domain state change and an outbox row in one database transaction, then let a relay publish the committed row later. The relay may publish twice if it crashes after broker confirmation but before marking the row complete, so consumers need immutable event IDs and idempotent effects. Pub/Sub can still provide low-latency UI refresh after state is safe.

Deep Dive

Start by naming the fact owner. A database row, event-store append, or stateful service with a documented persistence model owns business truth. Phoenix.PubSub is excellent at saying “order 913 changed; reload it” to currently connected processes. It should not be the only record that the order was paid, because late and recovering consumers cannot reconstruct that fact.

The dual-write problem appears whenever one request must update a database and publish elsewhere. Database commit followed by broker publish can crash in between, leaving committed state without an event. Broker publish followed by database commit can announce a change that rolls back. Publishing inside Ecto.Multi.run does not make the broker participate in the database transaction; a rollback cannot retract a message already observed. The transactional outbox stores the event beside the domain change under the same transaction.

An outbox row should be immutable enough to audit and replay. Useful fields include stable event ID, aggregate/subject ID, aggregate revision, tenant ID, event type, schema version, payload or payload reference, creation time, available time, attempt count, and publication state. Avoid serializing arbitrary internal structs that will not decode after deployments. Define retention and sensitive-data policy; a durable outbox becomes a data store with compliance consequences.

A polling relay claims a bounded batch, commonly with row locks and SKIP LOCKED so several relay workers can divide queue-like work. It publishes to the durable broker, waits for publisher confirmation if required, then marks the row published. A wake-up through PostgreSQL NOTIFY or local Pub/Sub can reduce latency, but it must remain only a hint. Periodic polling is the recovery path because notifications are not replayable and may be folded or missed.

The failure window after publish is essential. Suppose the broker durably accepts event E, but the relay dies before updating the row. The next relay publishes E again. This is correct at-least-once behavior, not an implementation accident. The consumer uses event_id in an inbox/deduplication table or makes the target operation naturally idempotent. External APIs may accept an idempotency key; if not, the consumer needs a domain-specific reconciliation strategy.

RabbitMQ reliability is assembled from several features. Durable or replicated queues and persistent messages protect stored data under the broker’s documented assumptions. Publisher confirms tell the producer the broker accepted responsibility. Consumer acknowledgements tell the broker processing reached the application’s chosen boundary. Redelivery creates duplicates. Dead-letter exchanges preserve exhausted failures for diagnosis. None of these automatically makes a database update and external email one atomic exactly-once action.

Oban provides durable PostgreSQL-backed jobs and can be inserted through an Ecto.Multi, which is often enough when the only asynchronous consumer is the same application. Unique job configuration reduces duplicate enqueue patterns but must be scoped to business semantics. Broadway is appropriate when ingesting an external broker with demand, concurrency, batching, and acknowledgement. Choose the smallest durable mechanism that meets replay, throughput, isolation, and operational requirements.

The UI fast path can coexist with durable handoff. After the transaction commits, publish a transient “changed” notification and let LiveViews reload. Independently, the outbox/Oban path guarantees eventual external work. If the transient publish is missed, the next reload still sees correct state. If the durable worker retries, its effect remains idempotent. This separation produces both low latency and honest recovery semantics.

How this fits on projects: Project 9 builds the outbox and durable worker path; Project 11 consumes a broker; Project 12 combines transient UI fanout with durable domain effects.

Definitions and key terms

• Outbox: Database table committed atomically with domain state and relayed later.
• Publisher confirm: Broker acknowledgement that a publication reached its responsibility boundary.
• Consumer acknowledgement: Signal that processing reached the consumer’s chosen completion boundary.
• Inbox/deduplication table: Durable record of handled event IDs.
• Redelivery: Same message delivered again after failure or missing acknowledgement.
• Replay: Reprocessing retained messages or events from a known position.

Mental model diagram

request
  │
  v                 ONE DATABASE TRANSACTION
┌──────────────────────────────────────────────────────────┐
│ update order state + insert immutable outbox event E     │
└───────────────────────┬──────────────────────────────────┘
                        │ commit
              ┌─────────┴──────────┐
              v                    v
  Phoenix.PubSub changed      relay claims E
  current UI reloads          │ publish + confirm
                              v
                       durable broker ──▶ consumer
                              ▲              │ idempotent effect
                              │ retry E      └─ ack
                              └── relay crash may duplicate

How it works

1. Validate a command and build a versioned event envelope.
2. Commit domain state and the outbox row in the same database transaction.
3. Optionally broadcast an ephemeral refresh only after commit.
4. A relay claims rows, publishes durably, waits for confirmation, and records progress.
5. Consumers deduplicate or apply idempotent effects before acknowledging.

Invariants: no broker call is atomic with an ordinary database transaction; relays can duplicate; transient notification is not the recovery log.

Failure modes: publish-before-commit ghosts, commit-before-publish gaps, stuck rows after missed wake-up, premature acknowledgement, poison-message loops, unbounded retention, and non-idempotent side effects.

Minimal concrete example

TX 881 COMMITTED: order=913 revision=18; outbox event=evt_77 status=pending
relay publish evt_77 -> broker confirm received
relay crashes before marking published
relay restarts -> publishes evt_77 again
consumer inbox contains evt_77 -> skips duplicate effect, acknowledges safely

Common misconceptions

• “Calling a broker inside a database transaction is atomic.” The broker cannot roll back with the database.
• “Publisher confirms mean the consumer completed.” They concern broker acceptance, not downstream handling.
• “Unique jobs prove exactly once.” Uniqueness scope and external effect uncertainty still matter.
• “A NOTIFY wake-up replaces polling.” Missed transient hints must not strand durable rows.

Check-your-understanding questions

1. Which crash window creates duplicate outbox publication?
2. Why must an event ID be stable across relay retries?
3. When is Oban simpler than RabbitMQ?

Check-your-understanding answers

1. Broker acceptance occurs, then the relay crashes before recording publication progress.
2. Consumers can recognize that repeated deliveries represent the same logical fact.
3. When PostgreSQL is already authoritative, consumers are inside the application boundary, and broker-specific scale/isolation is unnecessary.

Real-world applications: emails after transactions, webhooks, search indexing, audit export, billing integration, and cache/UI refresh beside durable work.

Where you will apply it: Project 9, Project 11, and Project 12.

References: Ecto.Multi and Oban 2.23 documentation; PostgreSQL NOTIFY and SKIP LOCKED; RabbitMQ reliability, confirms, and acknowledgement guides; Debezium outbox event router; Designing Data-Intensive Applications, 2nd Edition.

Key insight: Durable correctness comes from atomic state capture plus idempotent retry, while Pub/Sub remains the fast transient notification path.

Summary: Use an outbox to close the database–broker gap, then assume at-least-once publication and make every important effect safe to repeat.

Homework/exercises

1. Enumerate every crash point from command receipt through consumer acknowledgement.
2. Design a deduplication key for a webhook and for a projection.
3. Decide whether three example responsibilities need PubSub, Oban, or RabbitMQ.

Solutions

1. Before transaction, during rollback, after commit before wake-up, after claim, before/after broker confirm, before progress mark, during handler, after effect before ack.
2. Webhook: immutable event ID as provider idempotency key; projection: event ID plus stream revision constraint.
3. Live UI refresh: PubSub; local retryable email: Oban; independently scaled cross-service ingestion with acknowledgements: RabbitMQ/Broadway.

Theory Chapter 8: Presence, Tracking, and Eventual Convergence

Fundamentals

Pub/Sub answers “which current processes subscribed to this topic?” but does not by itself maintain a user-facing model of who is online, how many devices they use, or how membership converges after node failure. Phoenix.Tracker and Phoenix.Presence provide distributed presence tracking. Tracker shards use heartbeat exchange and a conflict-free replicated data type (CRDT) to replicate deltas without one global authority. Presence keys can have multiple metadata entries, such as several browser tabs or devices, each with a reference. Views are eventually consistent. An abrupt node failure is not an immediate leave event, and reconnect may briefly overlap old and new entries. Presence is an ephemeral liveness projection, never durable identity, authorization, billing state, or audit truth.

Deep Dive

Presence looks simple in a browser—an online list—but its semantics are distributed. A connected process tracks a key on a topic with small metadata. Local changes become diffs; tracker replicas exchange heartbeats and deltas; clients begin with a full presence state and then apply join/leave diffs. CRDT convergence means independently received changes can merge without a central lock and replicas converge after communication resumes.

The key represents a logical identity such as user ID, while metas represent concurrent presences. One user may have a phone, laptop, and two tabs. Collapsing the list to one boolean loses useful lifecycle information. The UI can display “online on three devices” or derive online if any meta remains. Each meta has a phx_ref that distinguishes the tracked instance and supports correct diff application.

Metadata should be compact, nonsensitive, and ephemeral: device class, coarse status, or last activity class may fit. User names, avatars, roles, and database details should be fetched in a bounded batch through Presence’s fetch/2 hook rather than copied into every heartbeat update. Secrets, access tokens, and high-churn payloads never belong in presence metadata. Presence replication cost grows with metadata and churn.

Timing matters. Tracker defaults are tuned for real systems, not instantaneous laboratory expectations: heartbeat broadcasts and down detection mean abrupt failure can leave a temporary ghost. Graceful permanent-down signaling can accelerate clean shutdown, but rolling deployments may create large leave/rejoin waves. A user-visible interface should use language like “online recently” when exactness is impossible and should not trigger irreversible actions from one join or leave.

At the raw Phoenix.Tracker layer, callback behavior is part of the control path. Its handle_diff/2 executes inside the tracker server; blocking I/O delays convergence and an exception can crash the tracker. Phoenix.Presence instead exposes the optional handle_metas/4 callback for Elixir-side presence clients. Keep either callback bounded, and offload optional external work to supervised tasks or, if it must survive, a durable job. Do not send billing emails directly from a presence leave callback.

Client synchronization should follow the provided algorithm: apply initial presence_state, then presence_diff using the JavaScript helpers where applicable. Ad hoc set mutation often mishandles reconnects, multiple metas, and out-of-order network observations. A LiveView-only interface may subscribe to presence diffs server-side but still needs to render from the current tracked projection after mount/reconnect.

Authorization is orthogonal. A user allowed to track presence in one room is not automatically allowed to list all room members or subscribe to another tenant’s presence topic. Authenticate the connection, authorize the room, derive the topic server-side, minimize metadata, and re-evaluate authorization when membership changes. Presence tells you what has been tracked, not what should be allowed.

Tests should use controlled multiple processes and nodes. Verify one key with several metas, process exit cleanup, graceful shutdown, abrupt node loss, reconnect overlap, and final convergence. Avoid asserting an immediate leave after hard failure. Instead define a convergence window and expose intermediate state honestly. The acceptance test is not that every replica agrees at every instant; it is that disagreement is bounded, visible, and converges under restored communication.

Presence product semantics deserve their own language. “Online,” “away,” “reconnecting,” and “last seen” may combine Tracker evidence with durable account data, but the UI must communicate uncertainty rather than manufacture precision. If compliance or safety depends on a user being present, require an explicit acknowledged action or durable lease maintained by an authoritative service. A green dot alone cannot carry that responsibility.

How this fits on projects: Project 6 builds a collaborative workspace; Project 10 observes Tracker during partitions; Project 12 uses presence only for operator liveness, never authorization or audit.

Definitions and key terms

• Presence key: Logical identity tracked on a topic.
• Meta: One concurrent presence instance and its small metadata.
• CRDT: Data type whose independently applied updates converge without central coordination.
• Heartbeat: Periodic evidence used for replication and failure detection.
• Presence state: Full current projection sent to initialize a client.
• Presence diff: Incremental joins and leaves applied after initial state.

Mental model diagram

user 42
 ├─ laptop tab ── meta ref=A ──▶ tracker shard node A ─┐
 ├─ phone       ── meta ref=B ──▶ tracker shard node B ─┼─ CRDT deltas ─▶ converged view
 └─ second tab  ── meta ref=C ──▶ tracker shard node A ─┘

client initialization: presence_state -> syncState
ongoing changes:       presence_diff  -> syncDiff

hard partition: temporary ghost/divergence -> heartbeat timeout -> later convergence

How it works

1. Authorize a connection for a server-derived presence topic.
2. Track the connection PID under a stable key with compact metadata.
3. Tracker shards replicate deltas and detect lifecycle changes through heartbeats.
4. Clients initialize from full state, then apply diffs by unique meta reference.
5. On failure/reconnect, replicas temporarily diverge and later converge.

Invariants: presence is multi-meta and eventually consistent; callbacks must be fast; tracked membership is not authorization or durable truth.

Failure modes: ghost entries, reconnect double presence, oversized metadata, blocking diff handlers, false offline business actions, and cross-tenant listing.

Minimal concrete example

presence key=user:42 metas=[{ref:A, device:laptop}, {ref:B, device:phone}]
phone disconnects -> leave ref:B
render user:42 ONLINE with one remaining device
node partition -> ref:A may remain visible temporarily
reconnect -> ref:C joins; convergence removes stale ref:A later

Common misconceptions

• “One user equals one presence row.” A key may have several metas.
• “Leave is immediate after machine loss.” Failure detection and convergence take time.
• “Presence is a session database.” It is an ephemeral replicated projection.
• “Presence membership authorizes access.” Authorization must be checked separately.

Check-your-understanding questions

1. Why should a UI keep the user online after one of two metas leaves?
2. Why is blocking work unsafe in raw Tracker handle_diff/2, and which callback does Phoenix.Presence expose instead?
3. Why does graceful shutdown policy affect rolling deploy UX?

Check-your-understanding answers

1. Another tracked device/tab remains active for the same logical key.
2. Blocking network/database calls, expensive computation, and irreversible undurable effects.
3. Immediate permanent-down can create mass leave/rejoin churn; retaining old entries can create temporary overlap.

Real-world applications: collaboration cursors, chat rosters, incident responders, multiplayer lobbies, device online status, and operator dashboards.

Where you will apply it: Project 6, Project 10, and Project 12.

References: Phoenix.Tracker 2.2; Phoenix.Presence and JavaScript Presence docs; CRDT background in Designing Data-Intensive Applications, 2nd Edition; Phoenix Channels documentation.

Key insight: Presence is a convergent liveness hint with multi-device semantics, not a globally instantaneous or authoritative user record.

Summary: Build presence UIs around state-plus-diff synchronization, temporary uncertainty, metadata restraint, and independent authorization.

Homework/exercises

1. Model a user with three tabs and trace two closes plus one reconnect.
2. Divide ten candidate metadata fields into safe, fetch-later, and forbidden.
3. Define a convergence SLO after abrupt node loss.

Solutions

1. Remove metas by ref; logical user stays online until the last active meta leaves, with reconnect creating a new ref.
2. Safe: coarse device/status; fetch later: profile/role; forbidden: tokens, secrets, sensitive free text.
3. Example: divergent entries visible with “reconnecting,” 99% converged within the measured failure-detection window after connectivity returns.

Theory Chapter 9: Observability, Testing, Chaos, and Tenant Safety

Fundamentals

Asynchronous systems require evidence that crosses the publish/handle gap. Useful signals include publish attempts and failures, fanout size, handler latency and exceptions, event age, sequence gaps, duplicate count, sampled mailbox length, node connectivity, presence convergence, broker lag, acknowledgement latency, retries, and dead letters. Telemetry handlers execute synchronously in the process calling :telemetry.execute, so they must be fast and nonblocking. Tests should assert messages and invariants rather than sleep, while cluster tests deliberately inject node and network faults. Security begins before subscription: authenticate the caller, authorize the resource, derive topics server-side, minimize payloads, rate-limit publication, and prevent tenant identifiers or raw topics from becoming unbounded metric labels.

Deep Dive

A publish call and a subscriber handler are two operations separated by an asynchronous queue. One synchronous trace span around broadcast cannot represent end-to-end latency. Put a stable event ID, correlation ID, causation ID, schema version, stream revision, and trace context in the envelope where justified. Instrument a publish span around routing and a separate handler span when each subscriber begins processing. Event age is the difference between monotonic-compatible enqueue/occurrence evidence and handler time; distributed wall clocks require care.

Telemetry’s dynamic dispatch is deliberately lightweight, but handlers run synchronously. A handler that logs over the network or performs a database write directly increases publisher or consumer latency. Handlers should transform bounded metadata and hand it to an efficient reporter. Metric dimensions must be bounded: event type, result class, node role, and handler class are safer than raw topic, event ID, user ID, or tenant ID. High-cardinality identities belong in sampled logs or traces with access controls.

Phoenix.PubSub 2.2 does not document a built-in Telemetry event surface for every broadcast. Wrap domain publish functions and instrument subscriber handling. Broadway already emits topology, processor, message, batcher, and failure events; reuse rather than double-count. Sample process queue length and oldest event age from the subscriber side. A central process that synchronously inspects every PID can become the outage it is measuring.

Define SLOs from user-visible behavior. Examples: 99% of online dashboard updates applied within 500 ms under normal load; no unauthorized tenant delivery in any test; mailbox queue below a sustained threshold; outbox oldest-pending age below 30 seconds; presence convergence within the documented failure-detection window; zero silent dead letters. SLOs need load shape and failure assumptions.

Unit tests validate topic derivation, envelope versions, idempotency, and overload policy. Integration tests subscribe the test process, trigger a committed state change, and assert the exact message plus durable row. Property-based tests generate event sequences with duplicates, gaps, and version changes, checking that projections never regress. Avoid arbitrary sleeps; use monitors, barriers, eventually helpers with deadlines, or broker acknowledgement fixtures.

Cluster tests require separate nodes and a controllable network. Cover normal cross-node fanout, graceful node stop, hard kill, symmetric partition, reconnect, mixed release versions, and PubSub pool-size migration. Record which messages are expected to be missed. Chaos is useful only when attached to hypotheses: “during isolation, connected components update locally; after reconnect, clients reload revision and converge without cross-tenant leakage.” Random killing without an invariant is theater.

Tenant safety starts at the ingress. A browser-supplied tenant_id is untrusted even if it appears inside a signed-in socket. Load authorization from server state, derive the topic, and subscribe only after permission. Publish functions accept domain subjects, not arbitrary topic strings. Re-check authorization on membership or role changes. Payloads should include the least data every recipient needs; subscribers can fetch authorized detail separately.

Rate limits and quotas belong per authenticated principal, tenant, topic class, and payload size. A malicious or buggy publisher can create fanout amplification. Enforce maximum envelope size, known schema versions, bounded topic length, and publication rate. Distribution connections are a privileged control boundary; do not expose them to customers. External broker credentials need least privilege to exchanges/queues and rotation.

An incident runbook ties evidence to action. If event age rises while publish duration is flat, inspect subscriber capacity and mailbox trend. If only remote delivery fails, inspect node topology and adapter membership. If outbox age rises, inspect relay claim/confirm progress. If duplicate effects rise, inspect idempotency and acknowledgement timing. Every project should leave a failure signature table rather than a vague “check logs” instruction.

How this fits on projects: Every project uses assertion-driven verification. Projects 7–12 add load, cluster, broker, security, and chaos evidence, culminating in tenant isolation and recovery SLOs.

Definitions and key terms

• Event age: Time from occurrence/enqueue to handler observation.
• Correlation ID: Identity tying operations in one request or workflow.
• Causation ID: Identity of the event/command that caused another event.
• SLO: Measurable reliability or latency objective.
• Cardinality: Number of distinct values a metric label can take.
• Fault injection: Controlled failure used to test a stated recovery hypothesis.

Mental model diagram

publish span                 async boundary                 handle span
┌─────────────────┐        [topic + mailbox]       ┌─────────────────────┐
│ validate/auth   │ ─evt─▶ queue age grows here ─▶│ version/dedupe/work │
│ route duration  │                                │ effect/result       │
└─────────────────┘                                └─────────────────────┘
        │ event_id / correlation / trace context carried in envelope │
        └──────────────────── logs + traces ──────────────────────────┘

metrics: bounded labels         logs/traces: sampled high-cardinality context

How it works

1. Define invariants and SLOs before adding metrics.
2. Instrument domain publish wrappers and subscriber handler boundaries.
3. Propagate stable identity/correlation through envelopes.
4. Test local, durable, and distributed boundaries with deterministic fault fixtures.
5. Authorize server-side and continuously verify zero cross-tenant delivery.

Invariants: observability must not block the hot path; async work needs separate spans; topic possession is not authorization; chaos has a predicted result.

Failure modes: synchronous slow Telemetry handlers, high-cardinality metric explosion, sleep-based flaky tests, trace loss across messages, unauthorized wildcard topics, payload leaks, and fault injection without cleanup.

Minimal concrete example

ALERT: pubsub_handler_event_age_p99=2.8s > 0.5s
publish_route_p99=4ms (healthy)
subscriber_mailbox=18,420 and rising
handler_completion=900/s < arrival=2,700/s
diagnosis: slow subscriber capacity, not adapter routing
action: activate coalescing policy; queue bounded in 12s; no durable commands dropped

Common misconceptions

• “Broadcast duration measures user-visible latency.” It stops before subscriber handling.
• “Telemetry handlers are asynchronous.” They execute synchronously on the emitting path.
• “A topic string is a capability.” It is only a routing identifier unless access is enforced.
• “Chaos tests prove resilience by crashing things.” They prove a specific invariant under a controlled fault.

Check-your-understanding questions

1. Why should event ID not be a Prometheus label?
2. Which metric separates routing slowness from handler backlog?
3. What should a partition test assert after reconnection?

Check-your-understanding answers

1. It is nearly unique and creates unbounded series cardinality; keep it in logs/traces.
2. Compare publish-route latency with handler event age, queue trend, and completion rate.
3. Membership convergence plus authoritative state reload/replay, while explicitly accepting missed ephemeral messages.

Real-world applications: on-call dashboards, SLO alerting, tenant-safe real-time APIs, broker lag monitoring, and resilience certification.

Where you will apply it: Project 7, Project 10, Project 11, and Project 12.

References: Telemetry 1.4 and span/3; Broadway Telemetry docs; Phoenix security guidance; OWASP WebSocket Security Cheat Sheet; Release It!, 2nd Edition; Designing Data-Intensive Applications, 2nd Edition.

Key insight: You cannot operate or secure asynchronous fanout by observing the publisher alone; evidence and authorization must follow the message to every boundary.

Summary: Instrument routing and handling separately, test failures against explicit invariants, and make server-derived tenant policy part of every subscription and publication.

Homework/exercises

1. Design bounded metric labels and rich trace fields for an order event.
2. Write three partition hypotheses and their expected observations.
3. Threat-model arbitrary client topic selection.

Solutions

1. Metrics: event type/version/result/node role; trace: event ID, tenant, order ID, revision, correlation, causation.
2. Local components continue, remote transient messages are missed, then membership and state projections converge after reconnect.
3. Risks include cross-tenant reads/writes, existence probing, amplification, and label leakage; mitigate with server-derived topics, authorization, quotas, and audits.

Glossary

• Acknowledgement: Evidence that responsibility reached a stated boundary, such as broker acceptance or consumer completion.
• Adapter: Phoenix.PubSub component that forwards publications between nodes.
• At-least-once: Delivery policy that retries and may therefore produce duplicates.
• At-most-once: Delivery policy with no redelivery; loss is possible.
• Backpressure: Capacity feedback that limits upstream production.
• Broadcast: One publication fanned out to all current matching subscribers.
• Causation ID: Identity of the command/event that caused another message.
• Coalescing: Replacing superseded updates with a newer representative update.
• Command: Directed request asking one responsible owner to attempt an action.
• Correlation ID: Identity tying messages into one workflow or request.
• CRDT: Replicated data type whose concurrent updates converge under its merge rules.
• Dead letter: Message removed from normal retry flow for diagnosis or manual recovery.
• Demand: Explicit downstream capacity in GenStage/Broadway.
• Dispatcher: Logic that selects or transforms local deliveries from subscription entries.
• Durable subscriber: Consumer whose position/messages survive disconnection under a retained system.
• Event: Versioned fact or notification that something occurred.
• Event age: Time between occurrence/enqueue and current handling.
• Event envelope: Stable metadata surrounding an event payload.
• Fanout: Number and work of deliveries produced by one publication.
• Idempotency: Property allowing a repeated operation to have the same effective outcome as one execution.
• Mailbox: One process’s queue of unhandled messages.
• Outbox: Transactionally stored event record relayed after domain commit.
• Partition: Network split creating independently connected components.
• PG2 adapter: Phoenix.PubSub default adapter; on current OTP it uses :pg, despite the compatibility name.
• Presence meta: One tracked connection/device instance for a logical presence key.
• Process group: Named set of local or distributed PIDs managed by :pg.
• Publisher confirm: Broker evidence that it accepted responsibility for a publication.
• Redelivery: Repeated delivery after failure or missing acknowledgement.
• Registry partition: Shard of Registry storage and dispatch ownership.
• Replay: Reprocessing retained events/messages from a known position.
• Selective receive: Pattern-based mailbox scan that leaves unmatched messages queued.
• Stream revision: Monotonic domain position within one aggregate or ordered stream.
• Subscriber: Process currently registered to receive matching publications.
• Topic: Logical binary routing key selecting a candidate audience.

Why Elixir Pub/Sub Matters

Modern products are expected to update without refresh, show who is online, distribute device and operational signals, invalidate caches quickly, and coordinate many independent processes. Elixir makes the small-scale version unusually direct because every process already has a mailbox, while Phoenix.PubSub packages local and cluster fanout into a supervised component. The attraction is not merely speed; process isolation makes subscriber failures local and lets application structure mirror millions of independent connections or entities.

The pattern is proven at high fanout, but production stories also reveal its limits. Discord described five million concurrent users and millions of events per second in 2017, then documented guilds approaching two million concurrent online members in 2023. Its engineers had to shard fanout and measure mailbox pressure because one event’s work grows with its recipient set. In March 2026, Discord again described tracing Elixir systems serving guilds with millions of members—and how excessive observability could itself overload them. Those cases make the guide’s central lesson concrete: BEAM concurrency makes the architecture possible, while explicit fanout, mailbox, and instrumentation policies make it operable.

The 2025 Stack Overflow Developer Survey reported Elixir as the third most admired programming language at 66%, and Phoenix as the most admired web framework at 79%. These are satisfaction statistics rather than market-share claims, but they show sustained confidence among people using the stack. Elixir 1.20, released in June 2026, also completed the first milestone of gradual type inference/checking across Elixir programs, improving the feedback available for ordinary message and data-shape code without changing Pub/Sub’s runtime semantics.

Phoenix.PubSub 2.2, GenStage 1.3, Broadway 1.3, Telemetry 1.4, and OTP 29 form a mature set of composable tools. The important architectural choice is not “which one replaces all the others?” It is which boundary needs transient fanout, eventual presence, demand-aware work, transactional durability, broker acknowledgements, or replay.

Older instinct: one tool called “the event bus”

publisher ──▶ GLOBAL BUS ──▶ everything
                  │
                  └─ hidden ordering, auth, durability, and overload assumptions

Explicit Elixir architecture:

state commit ──▶ PubSub refresh ──▶ online UI/processes
      │
      └────────▶ outbox/broker ──▶ retryable durable effects

presence tracker ────────────────▶ convergent liveness projection
demand pipeline  ────────────────▶ bounded high-volume work

Context and evolution: The publish/subscribe pattern predates modern brokers and appears in classic messaging and observer designs. Erlang made asynchronous process communication and supervision runtime primitives. Elixir’s Registry added a scalable local registration/dispatch building block; Phoenix.PubSub evolved into an independent adapter-based library; OTP 23 introduced the current :pg module used by modern cluster membership; Phoenix.Tracker applied CRDT convergence to presence; GenStage and Broadway separated push notification from demand-aware ingestion.

Sources for the statistics and current versions: Elixir 1.20 release announcement; Stack Overflow 2025 Technology Survey; Phoenix.PubSub 2.2, GenStage 1.3.2, Broadway 1.3.0, Telemetry 1.4.2, and OTP 29 official documentation; Discord engineering posts from 2017, 2023, and 2026.

Concept Summary Table

Concept Cluster	What You Need to Internalize
BEAM Processes, Signals, and Mailboxes	Send is asynchronous, ordering is per sender–destination, mailboxes are receiver-owned and unbounded, and restart discards pending messages.
Pub/Sub Semantics, Topics, and Event Contracts	Topics route; envelopes preserve meaning; delivery, ordering, compatibility, and authorization scopes must be explicit.
Local Routing with Registry and Custom Dispatch	Duplicate Registry keys provide lifecycle-aware local membership while dispatch cost and custom logic remain on the caller path.
Phoenix.PubSub Architecture and Dispatch APIs	Local Registry fanout and adapter-based cross-node forwarding remain transient; API choice, sizing, and migration matter.
Distributed Erlang, :pg, Topology, and Partitions	Connectivity precedes fanout, membership is eventually consistent/non-transitive, and reconnection does not replay missed events.
Flow Control, Mailbox Safety, GenStage, and Broadway	Push is not backpressure; choose bounded loss/coalescing or explicit demand, with acknowledgements defined by the upstream source.
Ephemeral Notifications, Durable Work, and Transactional Outbox	Commit state and outbox atomically, then publish at least once and make consumers idempotent.
Presence, Tracking, and Eventual Convergence	Presence is multi-meta, replicated liveness with temporary uncertainty—not identity, authorization, or durable history.
Observability, Testing, Chaos, and Tenant Safety	Instrument publish and handle separately, use bounded metrics, test failure hypotheses, and authorize before topic derivation.

Project-to-Concept Map

Project	Concepts Applied
1. Mailbox Broadcast Laboratory	BEAM processes, signals, mailboxes; basic fanout semantics
2. Registry Topic Bus	Local Registry routing; subscriber lifecycle; mailbox semantics
3. Contract-Aware Domain Event Hub	Topic taxonomy; event envelopes; compatibility; tenant-safe dispatch
4. Phoenix.PubSub Notification Hub	Phoenix.PubSub APIs; local fanout; echo; subscription lifecycle
5. LiveView Operations Board	PubSub UI freshness; reconnect; authoritative state reload; observability
6. Distributed Presence Workspace	Tracker/Presence; multi-meta identities; CRDT convergence; authorization
7. Mailbox Pressure Laboratory	Mailbox safety; fanout cost; coalescing; sampling; SLO evidence
8. Demand-Aware Fanout Bridge	GenStage demand; Broadway topology; batching; acknowledgement boundaries
9. Durable Notification Outbox	Transactional outbox; Oban; at-least-once; idempotency; PubSub fast path
10. Three-Node PubSub Laboratory	Distributed Erlang; :pg; topology; partitions; safe pool migration
11. Tenant-Safe Broker Edge Bridge	RabbitMQ durability; Broadway acknowledgement; tenant policy; chaos/telemetry
12. Multi-Tenant Real-Time Event Platform	All nine concept clusters integrated under measurable production contracts

Deep Dive Reading by Concept

Concept	Book and Chapter	Why This Matters
Process messaging and failure	“Designing for Scalability with Erlang/OTP” by Cesarini & Vinoski — chapters on processes, supervision, and distributed architectures	Connects raw mailboxes to OTP reliability boundaries.
Pub/Sub channels	“Enterprise Integration Patterns” by Hohpe & Woolf — Ch. 2 Messaging Systems, Ch. 3 Messaging Channels	Defines Publish-Subscribe Channel, Point-to-Point Channel, Guaranteed Delivery, and Dead Letter Channel precisely.
Message contracts	“Enterprise Integration Patterns” — Ch. 4 Message Construction	Grounds event, command, correlation, format indicator, and versioning decisions.
Data and event semantics	“Designing Data-Intensive Applications, 2nd Edition” by Kleppmann et al. — chapters on encoding/evolution and event streams	Explains schema evolution, ordering, replay, and log-derived state.
Distribution and partitions	“Designing Data-Intensive Applications, 2nd Edition” — chapters on distributed-system trouble and consistency	Prevents local-process intuition from becoming unsafe cluster claims.
Capacity and stability	“Release It!, 2nd Edition” by Michael Nygard — Stability Patterns and Stability Antipatterns	Frames bounded queues, bulkheads, timeouts, and cascading failure.
Durable integration	“Enterprise Integration Patterns” — Messaging Channels and Message Routing chapters	Covers durable subscribers, idempotent receivers, competing consumers, bridges, and dead letters.
Service boundaries	“Building Microservices, 2nd Edition” by Sam Newman — chapters on communication, reliability, and asynchronous integration	Helps decide when a local PubSub message should cross into a broker/service contract.
Architecture observability	“Software Architecture in Practice, 4th Edition” by Bass et al. — availability and modifiability quality-attribute scenarios	Turns vague resilience goals into testable scenarios and trade-offs.

Quick Start: Your First 48 Hours

Day 1

1. Read Theory Chapters 1–3 and write the guarantees of send, Registry.dispatch, and a topic envelope in your own words.
2. Start Project 1. Produce two publishers, three subscribers, a monitored crash, and a queue-length observation.
3. Prove one valid cross-sender interleaving and reject one invalid same-sender reordering claim.

Day 2

1. Complete Project 1’s Definition of Done and write a one-page failure report.
2. Read Theory Chapter 4 and start Project 2’s duplicate Registry topic bus.
3. Subscribe twice, kill a subscriber, publish under load, and document exactly what Registry cleans up and what it does not acknowledge.

Recommended Learning Paths

Path 1: The Elixir/OTP Developer (Recommended)

• Project 1 -> 2 -> 3 -> 4 -> 7 -> 9 -> 10 -> 12

This path starts locally, makes semantics explicit, then adds load, durability, and distribution.

Path 2: The Phoenix Product Engineer

• Project 1 -> 4 -> 5 -> 6 -> 7 -> 12

This path reaches visible LiveView and Presence outcomes quickly, then confronts reconnect, overload, and tenant safety.

Path 3: The Integration Engineer

• Project 1 -> 3 -> 7 -> 8 -> 9 -> 11 -> 12

This path emphasizes contracts, demand, acknowledgement, outbox, and broker boundaries.

Path 4: The Distributed Systems / SRE Engineer

• Project 1 -> 2 -> 4 -> 7 -> 10 -> 11 -> 12

This path prioritizes mailboxes, cluster topology, Telemetry, failure injection, and operational recovery.

Success Metrics

• You can state the exact ordering and loss assumptions of every message path without using the phrase “BEAM guarantees order” unqualified.
• You can inspect a slow subscriber and distinguish routing latency from mailbox/handler backlog.
• You can choose among direct messaging, Registry, Phoenix.PubSub, Presence, GenStage, Broadway, Oban, and RabbitMQ from requirements rather than familiarity.
• Every important event has a documented topic, schema version, identity, tenant scope, and gap/duplicate policy.
• You can prove that a node partition loses only explicitly ephemeral updates and that clients reload or replay durable truth after reconnection.
• You can inject duplicate outbox/broker delivery and demonstrate one effective external result.
• You can show zero cross-tenant delivery under forged-topic, stale-authorization, and reconnect tests.
• You can defend a measured SLO for event age, mailbox bound, outbox lag, broker acknowledgement, and presence convergence.

Pub/Sub Debugging and Verification Appendix

Failure signature table

Symptom	Likely Boundary	Evidence to Collect First
Publish is fast but UI is seconds late	Subscriber mailbox/handler	Event age, message_queue_len, handler duration, render cadence
Local subscribers update; remote do not	Distribution/topology	Node.list, node up/down events, adapter group membership, pool settings
One process receives duplicates	Subscription or retry	Registry duplicate entries, event ID, delivery count, broker redelivery flag
Presence user remains after crash	Tracker failure detection	Node status, tracker heartbeat/convergence window, meta refs
Committed data never triggers integration	Outbox relay	Pending-row oldest age, claim locks, publisher confirms, periodic scan
Side effect happens twice	Idempotency/ack boundary	Event ID, inbox constraint, effect idempotency key, crash timing
Metrics system increases latency	Telemetry handler	Handler duration, reporter queue, synchronous work in callback
Tenant sees another tenant’s event	Authorization/topic derivation	Auth decision, derived topic, payload scope, subscription audit trail

Useful runtime probes

Process.info(pid, [:message_queue_len, :memory, :reductions, :current_function])
Registry.lookup(registry, topic)
:pg.get_members(scope, group)
Node.list()
Phoenix.Tracker.list(tracker, topic)
outbox oldest pending age / attempts / last error
broker ready / unacked / redelivered / dead-letter counts

Use these as bounded diagnostics. Do not continuously enumerate every process or high-cardinality topic in production.

Delivery decision table

Requirement	Preferred Starting Mechanism
Online UI freshness or cache invalidation	Phoenix.PubSub
Local plugin/process fanout with custom metadata	Duplicate Registry
Online/offline collaborative projection	Phoenix.Tracker / Presence
Capacity-aware local work distribution	GenStage
Broker ingestion, batching, acknowledgements	Broadway plus a durable connector
Retryable work inside a PostgreSQL-backed Elixir app	Oban
Atomic database state plus later integration	Transactional outbox
Offline subscribers, replay, cross-service isolation	Durable broker or event log
Irreversible business truth	Database/event store, never PubSub or Presence alone

Project Overview Table

#	Project	Difficulty	Time	Primary Focus
1	Mailbox Broadcast Laboratory	Level 1	6–8h	Raw sends, receive, ordering, monitors
2	Registry Topic Bus	Level 2	8–12h	Duplicate Registry, lifecycle, dispatch
3	Contract-Aware Domain Event Hub	Level 2	10–14h	Topics, envelopes, versions, tenant routing
4	Phoenix.PubSub Notification Hub	Level 2	10–14h	Framework APIs, echo, subscriptions
5	LiveView Operations Board	Level 3	14–20h	UI freshness, reconnect, state reload
6	Distributed Presence Workspace	Level 3	16–22h	Tracker CRDT, multi-meta, convergence
7	Mailbox Pressure Laboratory	Level 3	16–24h	Load, queue bounds, coalescing, SLOs
8	Demand-Aware Fanout Bridge	Level 3	18–26h	GenStage/Broadway demand and batching
9	Durable Notification Outbox	Level 4	22–32h	Ecto transaction, Oban, idempotency
10	Three-Node PubSub Laboratory	Level 4	20–30h	Distribution, :pg, partitions, migration
11	Tenant-Safe Broker Edge Bridge	Level 4	24–36h	RabbitMQ, Broadway ack, DLQ, tenant policy
12	Multi-Tenant Real-Time Event Platform	Level 5	45–60h	Production synthesis and chaos recovery

Project List

The projects progress from observable mailbox mechanics to an independently scalable, durable, secure, and operable real-time platform. Build the first three even if you already use Phoenix; they make later failure behavior legible.

Project 1: Mailbox Broadcast Laboratory

• File: P01-mailbox-broadcast-lab.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Gleam
• Coolness Level: Level 3 (Genuinely Clever)
• Business Potential: Level 1 (Resume Gold)
• Difficulty: Level 1 (Beginner)
• Knowledge Area: BEAM concurrency, message passing, process lifecycle
• Software or Tool: IEx, ExUnit, Observer or :observer_cli
• Main Book: “Designing for Scalability with Erlang/OTP” by Cesarini & Vinoski

What you will build: A terminal laboratory with several publishers and subscribers that makes mailbox order, selective receive, timeouts, death, and queue growth visible.

Why it teaches Pub/Sub: It removes every framework layer so you can see that fanout is repeated asynchronous sends into independent mailboxes.

Core challenges you will face:

• Per-sender order vs cross-sender interleaving -> Theory Chapter 1.
• Subscriber death without acknowledgement -> process lifecycle and monitors.
• Slow receiver and unmatched-message buildup -> mailbox safety.

Real World Outcome

Run the experiment and see one timestamped transcript containing three subscribers. One handles immediately, one sleeps for a controlled interval, and one selectively matches only priority events. Queue samples and event age make the difference visible. Kill the slow subscriber and see a DOWN record; publishing continues, but no result falsely claims its queued events were processed.

$ mix run lab/mailbox_broadcast.exs --events 12 --slow-ms 250
[00.000] publish source=A seq=1 recipients=3
[00.001] fast handled A:1 queue=0 age_ms=1
[00.002] priority skipped normal A:1 queue=1
[00.252] slow handled A:1 queue=8 age_ms=252
[00.300] inject_exit subscriber=slow reason=simulated_failure
[00.301] DOWN subscriber=slow pending_before_exit=8
[00.302] publish source=A seq=12 recipients=2 send_result=completed

Guarantee check: A sequence preserved per destination = PASS
Cross-sender interleaving differs between subscribers = OBSERVED
Processing acknowledgement from send alone = NONE

The Core Question You Are Answering

“What actually happens between send returning and a subscriber completing work?”

The answer must account for queueing, scheduling, pattern matching, process death, and the absence of an implicit reply.

Concepts You Must Understand First

1. PID and process incarnation — Why is a restarted PID not the same subscriber? Book Reference: “Designing for Scalability with Erlang/OTP” — process architecture chapters.
2. Selective receive — What happens to unmatched messages? Reference: Elixir receive/1 and OTP efficiency guide.
3. Monitors — What evidence does a DOWN message provide? Book Reference: “Erlang in Anger” — runtime diagnostics sections.
4. Per-pair ordering — What is and is not preserved? Reference: OTP “Communication in Erlang.”

Questions to Guide Your Design

1. How will every event carry publisher identity, sequence, and monotonic timestamp?
2. How will you release all publishers from one barrier so interleavings are meaningful?
3. Which subscriber behaviors make queue growth and selective receive observable?
4. How will you avoid treating a dead-process send as successful processing?

Thinking Exercise

Draw two publishers sending A1,A2 and B1,B2 to two subscribers. List four valid per-subscriber traces, then mark which constraints must never be violated. Predict what remains after a receive clause matches only :priority messages.

The Interview Questions They Will Ask

1. “What ordering does Erlang message passing guarantee?”
2. “Does send/2 fail when the destination PID is dead?”
3. “Why can selective receive become expensive?”
4. “What happens to a mailbox after process crash?”
5. “How would you detect a subscriber falling behind?”

Hints in Layers

Hint 1: One sender, one receiver — Prove same-sender order before adding fanout.

Hint 2: Add identity — Include publisher, sequence, event ID, and monotonic send time in every term.

Hint 3: Synchronize starts — Use a barrier protocol so concurrent publishers begin from the same explicit signal.

Hint 4: Observe, do not sleep in tests — Use acknowledgements only in the lab protocol, monitors, and deadline-based assertions; reserve sleeps for the intentionally slow fixture.

Books That Will Help

Topic	Book	Chapter
Process communication	“Designing for Scalability with Erlang/OTP”	Processes and message passing
Failure observation	“Erlang in Anger”	Process debugging and overload
Queue stability	“Release It!, 2nd Edition”	Stability Antipatterns

Common Pitfalls and Debugging

Problem 1: “Every run has the same order”

• Why: Publishers are accidentally serialized by startup logic.
• Fix: Use a start barrier and independent publisher processes.
• Quick test: Ten runs should preserve each sender’s order while producing at least two valid cross-sender interleavings.

Problem 2: “Slow subscriber death looks successful”

• Why: The report equates completed sends with handled events.
• Fix: Track handler acknowledgements separately and label missing ones after DOWN.
• Quick test: Kill with a nonzero queue and verify processed count stays below sent count.

Definition of Done

• Three subscriber behaviors are independently observable.
• Same-sender order is proven without claiming global order.
• Queue length and event age expose one slow receiver.
• Subscriber death discards queued work and produces monitor evidence.
• Tests use message assertions/deadlines rather than arbitrary timing assumptions.

Project 2: Registry Topic Bus

• File: P02-registry-topic-bus.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Gleam
• Coolness Level: Level 3 (Genuinely Clever)
• Business Potential: Level 2 (Micro-SaaS / Pro Tool)
• Difficulty: Level 2 (Intermediate)
• Knowledge Area: Process registry, local routing, supervision
• Software or Tool: Elixir Registry, Supervisor, ExUnit
• Main Book: “Enterprise Integration Patterns” by Hohpe & Woolf

What you will build: A supervised local topic bus whose subscriptions are process-owned duplicate Registry entries, with introspection and custom bounded dispatch.

Why it teaches Pub/Sub: You implement the local membership and fanout layer that Phoenix.PubSub builds on, while Registry handles dead-process cleanup.

Core challenges you will face:

• Duplicate registration semantics -> idempotent subscription API.
• Partitioned dispatch callbacks -> no assumption of one complete ordered list.
• Caller-path work -> bounded filtering and instrumentation.

Real World Outcome

Open three IEx sessions attached to one node. Two subscribe to sensor:west; one subscribes to sensor:east. Publishing west reaches exactly the two west subscribers. Subscribe one process twice and observe the chosen wrapper policy. Kill a subscriber and confirm its entry disappears without a custom cleanup GenServer. Change Registry partition count in a benchmark profile and compare dispatch duration without claiming cross-partition order.

$ mix run --no-halt
bus=LocalTopics registry_partitions=4 status=ready
subscribe pid=<0.181.0> topic=sensor:west result=new
subscribe pid=<0.182.0> topic=sensor:west result=new
subscribe pid=<0.181.0> topic=sensor:west result=already_subscribed
publish topic=sensor:west entries=2 partitions_touched=2 duration_us=41
delivery event=evt-7 recipients=[<0.181.0>,<0.182.0>]
DOWN pid=<0.182.0>
publish topic=sensor:west entries=1 delivery=completed acknowledgement=none

The Core Question You Are Answering

“How can process-owned registrations replace a centralized subscription broker without pretending local membership is delivery?”

Concepts You Must Understand First

1. Duplicate Registry keys — Can one PID register one key several times? Reference: Elixir Registry duplicate-key docs.
2. Dispatch execution — Which process pays for custom callback work? Reference: Registry.dispatch/4.
3. Lifecycle cleanup — Why can a lookup still return a PID that just died? Book Reference: “Designing for Scalability with Erlang/OTP” — process lifecycle.
4. Partitions — Why can the callback run more than once? Reference: Registry partitions documentation.

Questions to Guide Your Design

1. Is repeated subscribe idempotent or a distinct logical subscription?
2. What small metadata is safe to store per registration?
3. How will the dispatcher behave if one metadata entry is malformed?
4. Which metrics avoid raw topic labels and full-registry scans?

Thinking Exercise

Compare a central GenServer holding %{topic => pids} with duplicate Registry. Draw the bottleneck, crash-state, cleanup, and concurrency consequences. Decide which features would justify a wrapper GenServer and which belong in Registry metadata.

The Interview Questions They Will Ask

1. “How does Registry differ from a GenServer-owned map?”
2. “Why is duplicate Registry useful for Pub/Sub?”
3. “Where does dispatch execute?”
4. “Is Registry distributed across nodes?”
5. “What happens when a registered process exits?”

Hints in Layers

Hint 1: Supervise Registry first — Treat it as infrastructure under the application supervisor.

Hint 2: Wrap subscription semantics — Prevent accidental duplicate delivery at the domain API.

Hint 3: Keep dispatch tiny — Validate compact metadata and send; move business work elsewhere.

Hint 4: Benchmark representative churn — Measure subscribe/unsubscribe plus fanout, not only a static lookup loop.

Books That Will Help

Topic	Book	Chapter
Publish-subscribe channel	“Enterprise Integration Patterns”	Ch. 3 Messaging Channels
Process architecture	“Designing for Scalability with Erlang/OTP”	Process and supervision chapters
Stability	“Release It!, 2nd Edition”	Stability Patterns

Common Pitfalls and Debugging

Problem 1: “One subscriber receives two copies”

• Why: It registered the same duplicate key twice.
• Fix: Make wrapper subscription idempotent or document multiple logical registrations.
• Quick test: Subscribe twice, publish once, and assert the intended delivery count.

Problem 2: “Publishing stalls unpredictably”

• Why: Custom dispatch performs blocking or expensive work on the publisher path.
• Fix: Precompute routing metadata and enqueue compact messages only.
• Quick test: Inject a slow metadata path and ensure the wrapper rejects it instead of blocking fanout.

Definition of Done

• Topic membership uses a supervised duplicate Registry.
• Repeated subscription semantics are explicit and tested.
• Dead subscribers disappear without application cleanup tables.
• Partitioned dispatch makes no global-order or complete-list assumption.
• Publish metrics are bounded and :ok is not labeled processing success.

Project 3: Contract-Aware Domain Event Hub

• File: P03-domain-event-hub.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, F#
• Coolness Level: Level 3 (Genuinely Clever)
• Business Potential: Level 3 (Service & Support)
• Difficulty: Level 2 (Intermediate)
• Knowledge Area: Domain events, schema evolution, authorization-aware routing
• Software or Tool: Registry, structs/typespecs, ExUnit, StreamData (optional)
• Main Book: “Enterprise Integration Patterns” by Hohpe & Woolf

What you will build: A local event hub with server-derived topics, stable envelopes, schema compatibility rules, sender exclusion, filters, and revision-gap detection.

Why it teaches Pub/Sub: It turns a routing primitive into a documented domain protocol without confusing notification with authoritative state.

Core challenges you will face:

• Command vs event boundaries -> one owner for actions, many observers for facts.
• Schema evolution -> compatible v1/v2 subscribers during rollout.
• Tenant/topic derivation -> authorization before routing.

Real World Outcome

Run an order scenario with v1 and v2 subscribers. A committed status change produces a stable envelope, derived tenant/order topics, and separate audit/refresh reactions. A deliberately unknown v3 event is quarantined and counted rather than crashing a subscriber. Publish revisions 10 then 12 and see the projection mark itself stale and request a reload.

$ mix hub.demo
command reserve_order tenant=42 order=913 -> accepted by OrderOwner
state commit order=913 revision=18 status=packed
event evt_01J... type=order.status_changed schema=2
topics [tenant:42:orders, tenant:42:order:913]
subscriber ui_v2 action=reload order=913 result=ok
subscriber audit_v1 action=compat_decode result=ok
inject schema=3 -> quarantine reason=unsupported_version count=1
inject revision=20 after=18 -> gap_detected missing=19 projection=STALE

The Core Question You Are Answering

“What must travel with an event so independent subscribers can remain correct across versions, restarts, gaps, and tenants?”

Concepts You Must Understand First

1. Command vs event — Who owns the action and who observes the fact? Book Reference: “Enterprise Integration Patterns” — Ch. 4 Message Construction.
2. Correlation and causation — How do messages form a workflow? Book Reference: same chapter.
3. Schema compatibility — What can an older subscriber safely ignore? Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — encoding/evolution chapter.
4. Authorization before topics — Why is subscriber-side filtering too late? Reference: OWASP WebSocket guidance.

Questions to Guide Your Design

1. Which envelope fields are required for every domain event?
2. Is order revision scoped globally, by tenant, or by order aggregate?
3. What is the exact policy for duplicate, stale, gapped, and unknown-version events?
4. Which data should be an identifier so subscribers reload authorized detail?

Thinking Exercise

Take send_email, email_sent, order_status, and get_order. Classify each as command, event, state/document, or query. Then design one incorrect catch-all topic and replace it with bounded server-derived topics.

The Interview Questions They Will Ask

1. “How is an event different from a command?”
2. “Why do you need both event ID and stream revision?”
3. “How do you roll out a new event schema safely?”
4. “What should a subscriber do when it detects a revision gap?”
5. “Why isn’t a topic name an authorization boundary by itself?”

Hints in Layers

Hint 1: Begin with one aggregate — Give an order a monotonic revision and stable event identity.

Hint 2: Derive topics internally — Accept domain IDs at the API, never an arbitrary caller topic.

Hint 3: Write a compatibility matrix — Specify v1/v2 publisher and subscriber combinations before implementation.

Hint 4: Prefer reload over payload bloat — For UI freshness, broadcast identity/revision and fetch authoritative detail.

Books That Will Help

Topic	Book	Chapter
Message construction	“Enterprise Integration Patterns”	Ch. 4 Message Construction
Channel design	“Enterprise Integration Patterns”	Ch. 3 Messaging Channels
Schema evolution	“Designing Data-Intensive Applications, 2nd Edition”	Encoding and Evolution

Common Pitfalls and Debugging

Problem 1: “Subscribers crash after rollout”

• Why: Payload shape changed without versioning or compatibility policy.
• Fix: Version envelopes and support an explicit overlap window.
• Quick test: Run old and new subscriber fixtures against both supported publisher versions.

Problem 2: “Tenant B receives Tenant A metadata”

• Why: A global topic relies on subscriber-side filtering.
• Fix: Authorize first and derive tenant-scoped topics/payloads server-side.
• Quick test: Forge Tenant A IDs under a Tenant B principal and assert zero delivery plus an audit record.

Definition of Done

• Commands have one responsible owner; events describe committed facts.
• Envelopes carry stable identity, version, tenant, subject, and ordering scope.
• Duplicate/stale/gap/unknown-version policies are tested.
• Topics are server-derived after authorization.
• UI notifications point to authoritative state instead of impersonating it.

Project 4: Phoenix.PubSub Notification Hub

• File: P04-phoenix-pubsub-notification-hub.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Gleam
• Coolness Level: Level 4 (Hardcore Tech Flex)
• Business Potential: Level 2 (Micro-SaaS / Pro Tool)
• Difficulty: Level 2 (Intermediate)
• Knowledge Area: Framework Pub/Sub, supervision, API design
• Software or Tool: Phoenix.PubSub 2.2, ExUnit
• Main Book: “Real-Time Phoenix” by Stephen Bussey

What you will build: A framework-backed notification hub exposing cluster, local, and sender-excluding publication through a contract-aware wrapper.

Why it teaches Pub/Sub: You replace the custom bus with Phoenix.PubSub while proving what the abstraction adds and which guarantees remain absent.

Core challenges you will face:

• API selection -> local, cluster, direct, and sender-excluding semantics.
• Subscription lifecycle -> duplicate subscribe and process restart behavior.
• Adapter sizing -> separate local Registry and broadcast pool concerns.

Real World Outcome

Start the supervised hub and three terminal clients: publisher/operator, subscribed UI, and node-local cache observer. Publish one alert with echo, one without sender echo, and one local-only cache event. Restart the UI subscriber and show that it receives future events but not the missed alert. The diagnostic report labels fanout completion separately from handler receipts.

$ mix pubsub.demo
pubsub=Demo.PubSub pool_size=2 registry_size=4 status=ready
subscribe pid=<0.210.0> topic=tenant:42:alerts result=new
broadcast event=evt-31 scope=cluster include_sender=true route_us=38
handled pid=<0.210.0> event=evt-31 age_us=74
broadcast_from event=evt-32 sender=<0.210.0> sender_echo=excluded
local_broadcast event=cache-9 recipients_on_this_node=1
restart subscriber old=<0.210.0> new=<0.225.0> missed_replay=not_available

The Core Question You Are Answering

“Which parts of a production Pub/Sub system does Phoenix.PubSub provide, and which contracts still belong to my application?”

Concepts You Must Understand First

1. Local Registry fanout — What did Project 2 already implement? Reference: Phoenix.PubSub source architecture.
2. Adapter boundary — Which work crosses nodes? Reference: Phoenix.PubSub.Adapter and PG2 docs.
3. Echo semantics — When should the sender receive its event? Book Reference: “Real-Time Phoenix” — PubSub/Channels chapters.
4. Ephemeral lifecycle — What does a restarted subscriber miss? Book Reference: “Enterprise Integration Patterns” — Durable Subscriber pattern.

Questions to Guide Your Design

1. Which domain functions expose publication without accepting arbitrary topic strings?
2. When is local-only delivery a correctness requirement rather than an optimization?
3. How will you make subscribe idempotent around reconnect?
4. What compatibility settings are needed before a pool-size change?

Thinking Exercise

For cache eviction, chat echo, LiveView refresh, and node diagnostic signals, choose broadcast, broadcast_from, or local_broadcast. Explain the harm caused by each wrong choice.

The Interview Questions They Will Ask

1. “How does Phoenix.PubSub deliver locally?”
2. “What does broadcast/4 returning :ok prove?”
3. “What is the difference between broadcast and broadcast_from?”
4. “Why is the default adapter still named PG2?”
5. “How do you safely change PubSub pool size?”

Hints in Layers

Hint 1: Wrap, do not scatter — Centralize topic derivation, envelope validation, and instrumentation.

Hint 2: Test sender behavior — Subscribe the publisher and assert inclusion/exclusion deliberately.

Hint 3: Restart the subscriber — Prove subscription is tied to one PID incarnation and no replay exists.

Hint 4: Configure sizes explicitly — Benchmark and document pool/registry choices instead of relying on ambiguous intuition.

Books That Will Help

Topic	Book	Chapter
Phoenix real-time	“Real-Time Phoenix”	PubSub and Channels chapters
Messaging contracts	“Enterprise Integration Patterns”	Ch. 3–4
Stability	“Release It!, 2nd Edition”	Stability Patterns

Common Pitfalls and Debugging

Problem 1: “The publisher gets duplicate UI updates”

• Why: Optimistic local state plus ordinary broadcast echoes to the sender.
• Fix: Use sender-excluding publication where the local change is already applied.
• Quick test: Assert sender receives zero copies and peer receives exactly one.

Problem 2: “A rolling deploy misses some cluster events”

• Why: Nodes use incompatible pool shard ranges.
• Fix: Apply the documented intermediate broadcast_pool_size deployment.
• Quick test: Run mixed-release nodes and continuously publish during each rollout stage.

Definition of Done

• Local, cluster, and sender-excluding semantics are individually demonstrated.
• Subscription is idempotent and tied to process lifecycle.
• Restart proves no offline replay exists.
• Domain wrapper owns topics, envelopes, and Telemetry.
• Pool-size migration has a tested two-stage runbook.

Project 5: LiveView Operations Board

• File: P05-liveview-operations-board.md
• Main Programming Language: Elixir
• Alternative Programming Languages: JavaScript/TypeScript client, Erlang backend
• Coolness Level: Level 4 (Hardcore Tech Flex)
• Business Potential: Level 3 (Service & Support)
• Difficulty: Level 3 (Advanced)
• Knowledge Area: LiveView lifecycle, real-time projections, reconnect recovery
• Software or Tool: Phoenix 1.8, LiveView, Phoenix.PubSub
• Main Book: “Programming Phoenix LiveView” by Tate & DeBenedetto

What you will build: A two-browser operational incident board that receives transient change notifications but renders from authoritative incident state.

Why it teaches Pub/Sub: The UI makes delivery visible and forces correct mount, reconnect, resubscribe, stale-state, and render-cadence decisions.

Core challenges you will face:

• Connected lifecycle -> subscribe once when the LiveView is connected.
• Notification vs state -> reload facts instead of treating PubSub as history.
• Render pressure -> coalesce high-rate updates into bounded UI cadence.

Real World Outcome

Open an operator window and a wallboard window. Changing an incident in either window updates both within the target SLO without refresh. Disconnect the wallboard, change three incidents, then reconnect: it loads current authoritative state and marks the transient revision gap rather than expecting PubSub replay. A diagnostics drawer shows last applied revision, event age, reconnect count, and coalesced render count.

Browser A — Operator                 Browser B — Wallboard
INC-104 API latency [MITIGATING]     revision=22 age=84ms LIVE
action: resolve -> committed r23     update r23 applied without refresh

[Browser B network offline]
commits r24, r25, r26                no transient delivery
[Browser B reconnect]
mount revision=26 source=database    gap 23->26 recovered_by=reload PASS

The Core Question You Are Answering

“How does a real-time UI remain correct after it inevitably misses transient notifications?”

Concepts You Must Understand First

1. LiveView mount phases — When is a persistent process connected? Book Reference: “Programming Phoenix LiveView” — lifecycle/state chapters.
2. handle_info — How do external process messages enter LiveView state? Reference: Phoenix LiveView docs.
3. State projection — Why should the database/repository own incident truth? Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — derived data chapters.
4. Render cadence — Which events can be coalesced? Book Reference: “Release It!, 2nd Edition” — backpressure/stability sections.

Questions to Guide Your Design

1. Does the notification carry full state or subject ID plus revision?
2. How does the LiveView detect duplicate, stale, and gapped revisions?
3. What is the maximum update/render rate for a human-facing board?
4. How does authorization change when an operator’s role is revoked while connected?

Thinking Exercise

Trace mount, connected mount, subscribe, event handling, disconnect, three missed events, reconnect, and state reload. Mark which state exists in the socket, process mailbox, PubSub registry, and database at every step.

The Interview Questions They Will Ask

1. “Why subscribe only when a LiveView is connected?”
2. “How should LiveView handle missed PubSub messages?”
3. “What belongs in handle_info versus the domain layer?”
4. “How do you prevent high-rate PubSub from overwhelming rendering?”
5. “How do you test two clients receiving one committed update?”

Hints in Layers

Hint 1: Load before listen — Fetch current state/revision, then subscribe under a documented race policy.

Hint 2: Broadcast identifiers — Send incident ID and revision; reload authorized current detail.

Hint 3: Treat gaps as stale — Stop incrementally applying and refresh from the source of truth.

Hint 4: Separate ingest/render cadence — Accumulate current state and render on a bounded tick if events are frequent.

Books That Will Help

Topic	Book	Chapter
LiveView lifecycle	“Programming Phoenix LiveView”	Lifecycle and state management
Real-time Phoenix	“Real-Time Phoenix”	PubSub and Presence chapters
Derived views	“Designing Data-Intensive Applications, 2nd Edition”	Derived Data / Stream Processing

Common Pitfalls and Debugging

Problem 1: “Each event arrives twice after navigation”

• Why: The same process subscribed more than once or lifecycle boundaries are misunderstood.
• Fix: Subscribe once in connected mount and make wrapper subscription idempotent.
• Quick test: Navigate/reconnect repeatedly, then assert one handler invocation per event ID.

Problem 2: “Wallboard is permanently stale after reconnect”

• Why: It expects PubSub to replay missed events.
• Fix: Reload current state and revision on mount/reconnect.
• Quick test: Disconnect, commit three changes, reconnect, and compare with database revision.

Definition of Done

• Two browser sessions update from one committed change.
• Reconnect restores current state without PubSub replay assumptions.
• Duplicate/stale/gap revisions have explicit UI behavior.
• Render cadence and event age remain within measured SLOs.
• Role revocation prevents future topic access and detail reload.

Project 6: Distributed Presence Workspace

• File: P06-presence-workspace.md
• Main Programming Language: Elixir
• Alternative Programming Languages: JavaScript/TypeScript client, Erlang backend
• Coolness Level: Level 4 (Hardcore Tech Flex)
• Business Potential: Level 2 (Micro-SaaS / Pro Tool)
• Difficulty: Level 3 (Advanced)
• Knowledge Area: Presence CRDTs, multi-device lifecycle, eventual consistency
• Software or Tool: Phoenix.Presence, Phoenix.Tracker, LiveView or Channels
• Main Book: “Real-Time Phoenix” by Stephen Bussey

What you will build: A collaborative incident room showing responders, multiple devices, cursor/activity hints, reconnect overlap, and eventual convergence.

Why it teaches Pub/Sub: Presence adds replicated membership state over PubSub and exposes why liveness is not durable identity or authorization.

Core challenges you will face:

• One key, many metas -> tabs/devices without false offline transitions.
• Hard failure detection -> temporary ghosts and convergence windows.
• Safe metadata and callbacks -> small state, batched fetch, no blocking diff handler.

Real World Outcome

Open a laptop tab, second tab, and phone session for the same user plus a second responder. The roster groups three metas under one identity, displays device count, and keeps the user online as individual tabs close. Abruptly isolate one node; the UI shows an uncertain/reconnecting state until the tracker convergence window. Restore connectivity and see one final consistent roster without using presence changes as audit events.

Room INC-104 — Responders
Douglas  ONLINE  devices=3 [laptop:A, tab:C, phone:B]
Margarita ONLINE devices=1 [laptop:D]

leave ref=C -> Douglas remains ONLINE devices=2
partition node_b -> phone ref=B status=UNCERTAIN
reconnect phone ref=E -> temporary metas=[B,E]
convergence complete -> metas=[A,E] consistent_on=3/3 nodes
audit truth changed by presence diff=false

The Core Question You Are Answering

“How do I represent ‘online’ honestly when users have many connections and the cluster cannot know failure instantly?”

Concepts You Must Understand First

1. Presence state and diff — Why initialize then apply refs? Reference: Phoenix Presence client/server docs.
2. Tracker CRDT — What does eventual conflict-free convergence guarantee? Reference: Phoenix.Tracker docs.
3. Failure detection — Why is hard node loss not immediate? Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — distributed failure chapter.
4. Metadata minimization — Which fields belong in fetch/2? Reference: Phoenix.Presence docs.

Questions to Guide Your Design

1. What stable key represents a user and what identifies each meta?
2. Which UI language represents temporary uncertainty?
3. How will you batch-fetch profile data without blocking Tracker callbacks?
4. What authorization check precedes track and list operations?

Thinking Exercise

Draw one user with three metas across two nodes. Trace graceful tab close, hard node loss, reconnect with a new ref, and delayed removal of the old ref. Decide when the UI says online, offline, and uncertain.

The Interview Questions They Will Ask

1. “How does Phoenix Presence differ from PubSub subscription membership?”
2. “Why can a user have multiple metas?”
3. “What consistency model does Phoenix.Tracker use?”
4. “How do Presence handle_metas/4 and raw Tracker handle_diff/2 differ, and what work should stay out of both?”
5. “Why isn’t presence suitable for billing or audit state?”

Hints in Layers

Hint 1: Model metas explicitly — Do not flatten a user to one boolean too early.

Hint 2: Use state then diff — Reuse supported sync behavior rather than inventing set mutation.

Hint 3: Keep metadata tiny — Store IDs/coarse status and batch-fetch display details.

Hint 4: Test graceful and abrupt loss separately — Their timing and deploy trade-offs differ.

Books That Will Help

Topic	Book	Chapter
Phoenix Presence	“Real-Time Phoenix”	Presence and Channels chapters
Convergence	“Designing Data-Intensive Applications, 2nd Edition”	Replication and CRDT sections
Distributed failure	“Release It!, 2nd Edition”	Stability Patterns

Common Pitfalls and Debugging

Problem 1: “User appears offline when one tab closes”

• Why: UI removed the key instead of one meta reference.
• Fix: Keep the logical key while any meta remains.
• Quick test: Close one of three tabs and assert device count becomes two, not offline.

Problem 2: “Presence callback processing stalls”

• Why: Presence handle_metas/4 performs blocking external work, or a lower-level raw Tracker handle_diff/2 blocks the tracker server.
• Fix: Keep callbacks bounded and offload optional/durable work appropriately; remember that raw Tracker callbacks run in the tracker server context.
• Quick test: Inject slow external dependency and confirm tracker heartbeat/diff latency stays bounded.

Definition of Done

• One logical user supports several visible metas.
• State-plus-diff synchronization survives reconnect.
• Hard loss shows uncertainty and converges within a measured window.
• Presence metadata is compact and safe; profile detail is batch-fetched.
• Presence never controls authorization, billing, audit, or irreversible actions.

Project 7: Mailbox Pressure Laboratory

• File: P07-mailbox-pressure-lab.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Gleam
• Coolness Level: Level 5 (Pure Magic)
• Business Potential: Level 4 (Open Core Infrastructure)
• Difficulty: Level 3 (Advanced)
• Knowledge Area: Performance, overload control, BEAM scheduling
• Software or Tool: Phoenix.PubSub, Telemetry, Benchee (optional), Observer
• Main Book: “Release It!, 2nd Edition” by Michael Nygard

What you will build: A reproducible load laboratory that overwhelms one subscriber, captures the failure signature, and compares coalescing, sampling, admission, and sharding policies.

Why it teaches Pub/Sub: It confronts the hardest omitted guarantee—PubSub can enqueue faster than any mailbox can drain.

Core challenges you will face:

• Measure event age, not only queue length -> freshness SLO.
• Loss policy by message class -> safe coalescing vs forbidden drops.
• Fanout amplification -> publisher reductions, memory, and subscriber isolation.

Real World Outcome

Run a staged load plan: baseline, burst, sustained overload, policy activation, and recovery. Healthy subscribers stay within SLO while one deliberately slow subscriber’s queue and age rise. Activate latest-value coalescing for dashboard gauges and durable admission for commands. The final report states exactly what was preserved, replaced, rejected, or delayed.

$ mix pubsub.pressure --profile sustained-60s
phase=baseline arrival=800/s completion=805/s queue_p99=8 age_p99_ms=18
phase=overload arrival=5000/s slow_completion=920/s queue=121440 age_p99_ms=22184
alert=SUBSCRIBER_FALLING_BEHIND
policy dashboard=coalesce_by_device command=durable_admission
phase=controlled queue_max=1950 age_p99_ms=438 coalesced=184220 rejected=0
healthy_subscriber_age_p99_ms=31
recovery_to_slo_ms=8240 memory_returns_to_baseline=true

The Core Question You Are Answering

“When publications exceed capacity, which information may be compressed or refused, and how will the system prove it stayed safe?”

Concepts You Must Understand First

1. Queueing rates — How do arrival and service rates predict growth? Book Reference: “Release It!, 2nd Edition” — Stability Antipatterns.
2. Event age — Why can a short queue still violate freshness? Reference: Telemetry spans and process metrics.
3. Coalescing semantics — Which states supersede earlier states? Book Reference: “Enterprise Integration Patterns” — Message Filter/Aggregator patterns.
4. BEAM scheduling — Why can fanout cost affect the publisher? Reference: OTP efficiency guide.

Questions to Guide Your Design

1. What exact load shape represents steady state, burst, and pathological overload?
2. Which message classes may be dropped, sampled, or coalesced?
3. How will you keep the measurement process from becoming a bottleneck?
4. What threshold activates and deactivates degraded mode without flapping?

Thinking Exercise

Classify typing indicators, device gauges, fraud alerts, payment commands, and audit facts. For each, choose loss, coalescing, rejection, or durable queueing, then defend the consequence of a subscriber being offline for one minute.

The Interview Questions They Will Ask

1. “Does Phoenix.PubSub implement backpressure?”
2. “How do you know a mailbox is falling behind?”
3. “When is dropping messages correct?”
4. “How can one slow subscriber affect a publisher?”
5. “What metrics prove recovery after overload?”

Hints in Layers

Hint 1: Establish equilibrium — Measure arrival and completion at a safe rate first.

Hint 2: Add one deterministic slow consumer — Keep other subscribers normal to test isolation.

Hint 3: Separate message classes — A universal drop policy is almost always wrong.

Hint 4: Measure the tail and recovery — Report p95/p99 age, maximum queue, memory trend, and time back to SLO.

Books That Will Help

Topic	Book	Chapter
Stability	“Release It!, 2nd Edition”	Stability Antipatterns and Patterns
Queueing/fanout	“Designing Data-Intensive Applications, 2nd Edition”	Stream Processing
Message filtering	“Enterprise Integration Patterns”	Message Routing chapter

Common Pitfalls and Debugging

Problem 1: “Queue length seems fine but users see stale data”

• Why: Events are expensive or old despite a moderate count.
• Fix: Measure oldest/event age and handler duration.
• Quick test: Inject a few very slow events and verify age alert fires before queue-count threshold.

Problem 2: “Metrics worsen the overload”

• Why: Every event triggers blocking/high-cardinality Telemetry work.
• Fix: Use bounded metadata, sampling, and asynchronous reporting.
• Quick test: Compare load with reporting enabled/disabled and cap observer overhead.

Definition of Done

• Baseline, overload, controlled degradation, and recovery phases are reproducible.
• Event age, queue trend, throughput, reductions, and memory are reported.
• Healthy subscribers remain isolated from one slow subscriber.
• Each message class has an explicit overload policy and counter.
• Recovery returns within SLO without silently losing durable responsibilities.

Project 8: Demand-Aware Fanout Bridge

• File: P08-demand-aware-fanout-bridge.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Gleam
• Coolness Level: Level 4 (Hardcore Tech Flex)
• Business Potential: Level 4 (Open Core Infrastructure)
• Difficulty: Level 3 (Advanced)
• Knowledge Area: Reactive streams, demand, batching, acknowledgements
• Software or Tool: GenStage 1.3, Broadway 1.3, Telemetry
• Main Book: “Designing Data-Intensive Applications, 2nd Edition” by Kleppmann et al.

What you will build: A capacity-aware telemetry pipeline that contrasts default work distribution, broadcast demand, batching, and an explicitly bounded PubSub edge.

Why it teaches Pub/Sub: It shows where push fanout ends and demand-aware processing begins without claiming that a push bridge can backpressure its original publisher.

Core challenges you will face:

• Dispatcher semantics -> distribute once vs broadcast to all.
• Demand propagation -> ask only when real capacity is free.
• Boundary policy -> reject/coalesce/spill when PubSub pushes into a bounded edge.

Real World Outcome

Run the same 50,000 telemetry events through three modes: raw PubSub to slow workers, GenStage demand distribution, and BroadcastDispatcher. The report shows raw mailbox growth, bounded work distribution, and slowest-consumer coupling in broadcast mode. Add Broadway processors/batches and a caller acknowledger test so completion is observable. Then push into a deliberately full edge and see the declared coalescing/rejection counter rather than hidden buffering.

$ mix flow.compare --events 50000
MODE pubsub_push      queue_max=38112 age_p99_ms=9440 backpressure=false
MODE genstage_demand  in_flight_max=160 age_p99_ms=218 distributed_once=true
MODE broadcast_demand consumers=4 throughput_limited_by=slow_consumer
MODE broadway_batch   processed=50000 ack_success=49992 ack_failed=8 batch_p99=100
EDGE full policy=coalesce_by_device coalesced=6840 durable_events_dropped=0

The Core Question You Are Answering

“How can downstream capacity control work flow, and where does that control stop when the source is transient push Pub/Sub?”

Concepts You Must Understand First

1. GenStage demand — What does a consumer request? Reference: GenStage behavior docs.
2. Dispatcher choice — Why does BroadcastDispatcher wait for every consumer’s demand? Reference: GenStage dispatchers.
3. Broadway acknowledgement — Who decides retry after failure? Reference: Broadway.Acknowledger.
4. Ordering/concurrency — What does partition_by preserve and not preserve? Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — stream processing.

Questions to Guide Your Design

1. Is the workload competing-consumer work or true all-consumer fanout?
2. What are min/max demand, concurrency, batch size, and batch timeout based on?
3. What happens when the transient input edge is full?
4. At which point does acknowledgement mean the effect is safe?

Thinking Exercise

For four consumers—billing, metrics, search, and UI—decide whether one GenStage stream, BroadcastDispatcher, separate streams, PubSub, or a durable broker is appropriate. Mark where a slow consumer should and should not throttle others.

The Interview Questions They Will Ask

1. “What is the difference between GenStage DemandDispatcher and BroadcastDispatcher?”
2. “Does demand make GenStage durable?”
3. “Does Broadway retry failed messages?”
4. “Why doesn’t push_messages backpressure Phoenix.PubSub?”
5. “How do batching and partitioning affect order?”

Hints in Layers

Hint 1: Start with one producer/consumer — Prove demand never exceeds the budget.

Hint 2: Compare dispatchers — Observe one-event-to-one-consumer versus one-event-to-all.

Hint 3: Add real completion acknowledgement — Request more only after asynchronous capacity returns.

Hint 4: Label the push edge honestly — It needs a bounded policy because upstream PubSub cannot be paused by Broadway demand.

Books That Will Help

Topic	Book	Chapter
Stream processing	“Designing Data-Intensive Applications, 2nd Edition”	Stream Processing
Competing consumers	“Enterprise Integration Patterns”	Message Routing chapter
Stability	“Release It!, 2nd Edition”	Backpressure and Bulkheads

Common Pitfalls and Debugging

Problem 1: “One optional consumer stalls every broadcast”

• Why: BroadcastDispatcher honors the slowest consumer’s demand.
• Fix: Separate policy classes/streams or use ephemeral PubSub for optional observation.
• Quick test: Pause one consumer and verify the expected stream, not unrelated work, stops.

Problem 2: “Failed Broadway messages never retry”

• Why: Broadway reports failure; the producer/broker owns retry/DLQ semantics.
• Fix: Configure and test the upstream connector’s redelivery policy.
• Quick test: Fail one message and assert its documented ack/redelivery/DLQ outcome.

Definition of Done

• Work distribution and true broadcast modes are compared accurately.
• In-flight work stays within measured demand budgets.
• PubSub edge overflow is bounded and counted.
• Broadway success/failure acknowledgement is observable.
• Ordering, partitioning, retry, and durability claims name their exact boundary.

Project 9: Durable Notification Outbox

• File: P09-durable-notification-outbox.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Java
• Coolness Level: Level 4 (Hardcore Tech Flex)
• Business Potential: Level 4 (Open Core Infrastructure)
• Difficulty: Level 4 (Expert)
• Knowledge Area: Database transactions, durable jobs, idempotency
• Software or Tool: Ecto, PostgreSQL, Oban 2.23, Phoenix.PubSub
• Main Book: “Designing Data-Intensive Applications, 2nd Edition” by Kleppmann et al.

What you will build: An order workflow that atomically commits domain state and a durable notification job/outbox, while PubSub supplies fast online UI refresh.

Why it teaches Pub/Sub: It proves why ephemeral fanout and durable responsibility are complementary, not interchangeable.

Core challenges you will face:

• Database–message dual write -> one Ecto transaction for state plus durable handoff.
• Crash after effect/before acknowledgement -> duplicate delivery and idempotency.
• Missed wake-up -> periodic durable scan rather than notification dependence.

Real World Outcome

Commit an order transition and see its row plus outbox/Oban job appear atomically. The LiveView updates immediately via PubSub. Pause the worker, restart the app, and confirm the durable job remains. Inject a crash after the mock webhook accepts the event but before job completion; retry uses the same event ID and the receiver records one effective action. Roll back an invalid order change and prove neither state nor job exists.

$ mix outbox.scenario
TX begin order=913 expected_revision=17
TX commit order_revision=18 outbox_event=evt-77 oban_job=551
pubsub refresh tenant:42:order:913 online_views=2
worker paused -> app restart -> job=551 state=available PASS
attempt=1 webhook accepted idempotency_key=evt-77 then injected_crash
attempt=2 webhook response=already_processed effective_calls=1
invalid_transition rollback order_revision=18 new_outbox_rows=0

The Core Question You Are Answering

“How do I guarantee that committed state eventually triggers required work without pretending the database and message system share one transaction?”

Concepts You Must Understand First

1. Transactional outbox — Which records commit together? Reference: Debezium outbox documentation.
2. Ecto.Multi/Oban insert — How does a job join a DB transaction? Reference: Ecto.Multi and Oban docs.
3. At-least-once effects — Which crash window creates duplicates? Book Reference: “Enterprise Integration Patterns” — Idempotent Receiver.
4. Notification vs fact — Why can UI reload after missed PubSub? Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — derived data.

Questions to Guide Your Design

1. What immutable event ID and stream revision enter both durable job and PubSub notification?
2. Where is the idempotency record stored relative to the external effect?
3. Which errors retry, discard, or dead-letter?
4. How are old outbox rows retained, redacted, and audited?

Thinking Exercise

Draw every crash point between request receipt, database commit, transient refresh, worker claim, webhook acceptance, and acknowledgement. For each, state the recovery source and whether a duplicate is possible.

The Interview Questions They Will Ask

1. “What problem does the transactional outbox solve?”
2. “Why isn’t a broker publish inside Ecto.Multi.run atomic?”
3. “How do you handle a crash after the side effect but before acknowledgement?”
4. “Why keep a periodic scan if PubSub/NOTIFY wakes the relay?”
5. “When would you choose Oban over RabbitMQ?”

Hints in Layers

Hint 1: Commit an immutable handoff — Domain row and job/outbox row share one transaction.

Hint 2: Publish refresh only after commit — UI can reload by ID/revision.

Hint 3: Reuse the same event ID — Retries must not invent new logical identity.

Hint 4: Inject the dangerous crash — Fail after external acceptance but before completion is persisted.

Books That Will Help

Topic	Book	Chapter
Reliable messaging	“Enterprise Integration Patterns”	Guaranteed Delivery and Idempotent Receiver
Transactions/streams	“Designing Data-Intensive Applications, 2nd Edition”	Transactions and Stream Processing
Service integration	“Building Microservices, 2nd Edition”	Asynchronous Communication

Common Pitfalls and Debugging

Problem 1: “Committed orders have no job”

• Why: State and handoff were inserted in separate transactions.
• Fix: Insert both through one Ecto transaction/Multi.
• Quick test: Kill at every transaction boundary and assert state/job consistency.

Problem 2: “Webhook fires twice”

• Why: Retry after uncertain completion lacks stable idempotency.
• Fix: Use immutable event ID at receiver or a local inbox/effect ledger.
• Quick test: Crash after acceptance and assert one effective external result after retry.

Definition of Done

• Domain state and durable handoff commit or roll back together.
• PubSub refresh occurs after commit and is safe to miss.
• App restart preserves pending work.
• Dangerous crash produces duplicate delivery but one effective result.
• Retry, discard, dead-letter, retention, and observability policies are documented.

Project 10: Three-Node PubSub Laboratory

• File: P10-three-node-pubsub-lab.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Gleam
• Coolness Level: Level 5 (Pure Magic)
• Business Potential: Level 4 (Open Core Infrastructure)
• Difficulty: Level 4 (Expert)
• Knowledge Area: Distributed systems, topology, partitions, rolling deploys
• Software or Tool: Distributed Erlang, Phoenix.PubSub.PG2, :pg, libcluster (optional)
• Main Book: “Designing for Scalability with Erlang/OTP” by Cesarini & Vinoski

What you will build: A repeatable three-node laboratory that visualizes connectivity, adapter group membership, cross-node fanout, partitions, reconnection, and safe pool migration.

Why it teaches Pub/Sub: It replaces “works on my cluster” with direct evidence about current connected components and missed transient events.

Core challenges you will face:

• Discovery vs PubSub -> establish and observe topology separately.
• Partition semantics -> component-local fanout and explicit missed revisions.
• Rolling configuration -> mixed pool sizes with compatibility stage.

Real World Outcome

Launch alpha, beta, and gamma in separate terminals. Each node hosts a subscriber and topology panel. Broadcast revision 41 from alpha and see all three. Partition gamma, broadcast 42, and see only alpha/beta. Reconnect; membership converges but 42 remains missing until gamma reloads authoritative revision. Then execute a pool 1→2 rolling migration while continuously publishing and prove no migration-induced gap.

$ ./lab cluster status
alpha@lab <-> beta@lab <-> gamma@lab  topology=full_mesh PASS
pg_group Demo.PubSub shard=0 members=3
publish revision=41 observed=[alpha,beta,gamma]
partition component_1=[alpha,beta] component_2=[gamma]
publish revision=42 observed=[alpha,beta] gamma=MISS_EXPECTED
reconnect membership_converged=true gamma_reload_revision=42
pool_migration stage=compat old_broadcast_pool=1 new_pool=2 gaps=0
pool_migration stage=final pool=2 gaps=0

The Core Question You Are Answering

“What does cluster Pub/Sub guarantee when node views diverge, and how does the application recover facts after connectivity returns?”

Concepts You Must Understand First

1. Named nodes and distribution — What establishes a connection? Book Reference: “Designing for Scalability with Erlang/OTP” — distribution chapters.
2. :pg membership — Why are views eventual and non-transitive? Reference: OTP 29 :pg.
3. Adapter forwarding — Why one remote worker per node/shard? Reference: Phoenix.PubSub PG2 source/docs.
4. Partition recovery — Why does membership converge without message replay? Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — distributed failure.

Questions to Guide Your Design

1. How will the lab prove full mesh rather than assume it?
2. Which events may be missed and which durable revision enables reload?
3. How will you create, heal, and clean up a deterministic partition?
4. How will continuous sequence probes validate pool migration?

Thinking Exercise

Draw full mesh and hub-and-spoke for three nodes. List each node’s possible :pg view. Then split {alpha,beta} from {gamma} and predict delivery, node monitor signals, presence uncertainty, and post-heal recovery.

The Interview Questions They Will Ask

1. “Does Phoenix.PubSub discover nodes?”
2. “What consistency model does :pg provide?”
3. “Are :pg views transitive through a hub?”
4. “What happens to PubSub messages during a partition?”
5. “How do you safely change Phoenix.PubSub pool size?”

Hints in Layers

Hint 1: Prove topology first — Show Node.list and symmetric connectivity from every node.

Hint 2: Number every probe — A sequence reveals exactly which component missed what.

Hint 3: Separate convergence from replay — After heal, query membership and reload authoritative state independently.

Hint 4: Publish throughout rollout — A quiet migration cannot prove shard compatibility.

Books That Will Help

Topic	Book	Chapter
Distributed OTP	“Designing for Scalability with Erlang/OTP”	Distribution and distributed architectures
Partitions	“Designing Data-Intensive Applications, 2nd Edition”	Distributed-System Trouble
Operations	“Release It!, 2nd Edition”	Stability Patterns

Common Pitfalls and Debugging

Problem 1: “All nodes connect to beta, but alpha misses gamma”

• Why: Membership is non-transitive and the topology is not full mesh.
• Fix: Configure/verify the topology expected by the adapter.
• Quick test: Query connectivity and group members from each node, not only the hub.

Problem 2: “Gamma never sees revision 42 after reconnect”

• Why: PubSub is transient and does not replay partition-missed events.
• Fix: Detect revision gap and reload/replay from durable state.
• Quick test: Verify membership converges, then separately verify state catch-up.

Definition of Done

• Three nodes prove the intended symmetric topology.
• Cross-node fanout and component-local partition behavior match predictions.
• Reconnection converges membership but recovery reloads durable facts.
• Node security assumptions and privileged network boundary are documented.
• Pool-size rolling migration passes continuous sequence probes.

Project 11: Tenant-Safe Broker Edge Bridge

• File: P11-broker-edge-bridge.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang, Java
• Coolness Level: Level 5 (Pure Magic)
• Business Potential: Level 4 (Open Core Infrastructure)
• Difficulty: Level 4 (Expert)
• Knowledge Area: Durable brokers, acknowledgements, security, chaos observability
• Software or Tool: RabbitMQ, Broadway, Phoenix.PubSub, Telemetry
• Main Book: “Enterprise Integration Patterns” by Hohpe & Woolf

What you will build: A RabbitMQ/Broadway ingestion edge that validates tenant envelopes, applies idempotent effects, dead-letters poison messages, and republishes safe transient UI updates.

Why it teaches Pub/Sub: It connects durable cross-service messaging to ephemeral Phoenix fanout while making acknowledgement, replay, tenant, and observability boundaries explicit.

Core challenges you will face:

• Publisher confirm vs consumer ack -> different responsibility boundaries.
• At-least-once redelivery -> immutable identity and deduplication.
• Tenant and payload validation -> least-privilege queues/exchanges and server-derived PubSub topics.

Real World Outcome

Publish tenant-scoped integration events with confirms. Broadway consumes under demand, validates schema/tenant, performs an idempotent projection, acknowledges, and emits a compact Phoenix.PubSub refresh. Inject handler crash to force redelivery, malformed version to reach DLQ, forged tenant routing to fail closed, and RabbitMQ outage to build visible lag without unbounded BEAM mailboxes. A dashboard shows ready/unacked/redelivered/DLQ counts and end-to-end age.

$ mix broker.edge.drill
publish evt-901 tenant=42 confirm=ack
consume evt-901 redelivered=false projection=applied pubsub_refresh=sent ack=success
inject crash_after_projection evt-902
redelivery evt-902 dedupe=already_applied pubsub_refresh=coalesced ack=success
schema=99 -> rejected reason=unsupported_version dlq_count=1
forged tenant=77 with credential=tenant42 -> authorization=DENY delivery_to_pubsub=0
broker outage ready_lag=12400 beam_mailbox_max=72 alert=UPSTREAM_LAG

The Core Question You Are Answering

“How do durable broker guarantees terminate safely at an Elixir application and become transient UI fanout without leaking tenants or duplicating effects?”

Concepts You Must Understand First

1. RabbitMQ reliability — What do durable queues, persistence, confirms, and acks each cover? Reference: RabbitMQ reliability/confirm docs.
2. Broadway demand and acknowledger — When is success reported? Reference: Broadway behavior and acknowledger.
3. Idempotent Receiver — How do duplicate deliveries become one effect? Book Reference: “Enterprise Integration Patterns” — Message Routing.
4. Least privilege and topic derivation — Why validate tenant twice? Reference: OWASP and RabbitMQ access-control docs.

Questions to Guide Your Design

1. Which exchange/queue topology bounds tenant and event-class fanout?
2. At what durable point is a consumer acknowledgement safe?
3. Which failures retry, dead-letter, or reject without requeue loops?
4. Which trace context crosses broker and Phoenix.PubSub boundaries?

Thinking Exercise

Trace one event through publisher confirm, ready queue, Broadway demand, handler, projection transaction, PubSub refresh, and consumer acknowledgement. Inject a crash before and after every boundary and predict redelivery/effect/UI behavior.

The Interview Questions They Will Ask

1. “What is the difference between publisher confirms and consumer acknowledgements?”
2. “Why does RabbitMQ at-least-once require idempotency?”
3. “Does Broadway implement retries?”
4. “What belongs in a dead-letter queue?”
5. “How do you prevent a tenant from forging a PubSub topic?”
6. “Which metrics reveal broker lag versus subscriber lag?”

Hints in Layers

Hint 1: Build the failure table first — Name broker, handler, database, and PubSub boundaries.

Hint 2: Acknowledge last — Only after the chosen durable effect and idempotency record are safe.

Hint 3: Keep UI refresh compact — The browser reloads authorized current projection.

Hint 4: Drill outages deliberately — Pause broker, kill processor, poison message, forge tenant, and recover under metrics.

Books That Will Help

Topic	Book	Chapter
Messaging reliability	“Enterprise Integration Patterns”	Ch. 3 Messaging Channels and Message Routing
Stream processing	“Designing Data-Intensive Applications, 2nd Edition”	Stream Processing
Operational stability	“Release It!, 2nd Edition”	Stability Patterns

Common Pitfalls and Debugging

Problem 1: “Poison message retries forever”

• Why: All failures requeue without classification or attempt bound.
• Fix: Reject permanent schema/auth failures and dead-letter after documented transient retries.
• Quick test: Publish unsupported schema and assert one DLQ outcome, not a hot loop.

Problem 2: “Tenant event reaches global UI topic”

• Why: Broker routing key or payload tenant is trusted directly.
• Fix: Bind credentials/queue policy, validate envelope against authorized context, then derive PubSub topic server-side.
• Quick test: Mismatch credential, routing key, and payload tenant in every combination.

Definition of Done

• Publisher confirms and consumer acknowledgements are separately measured.
• Redelivery produces one effective durable result.
• Retry, reject, and DLQ paths are deterministic and observable.
• Forged tenant/schema/payload attempts fail closed with zero PubSub leakage.
• Broker outage creates bounded upstream lag, not hidden mailbox explosion.

Project 12: Multi-Tenant Real-Time Event Platform

• File: P12-multitenant-event-platform.md
• Main Programming Language: Elixir
• Alternative Programming Languages: Erlang services, TypeScript clients
• Coolness Level: Level 5 (Pure Magic)
• Business Potential: Level 5 (Industry Disruptor)
• Difficulty: Level 5 (Master)
• Knowledge Area: Distributed architecture, real-time UX, durable integration, security, SRE
• Software or Tool: Phoenix, LiveView, PubSub, Presence, Ecto, Oban/RabbitMQ, Broadway, Telemetry
• Main Book: “Software Architecture in Practice, 4th Edition” by Bass et al.

What you will build: A multi-node incident/fleet operations platform with tenant-safe real-time boards, responder Presence, bounded telemetry, transactional domain events, durable integrations, and chaos-tested recovery.

Why it teaches Pub/Sub: Every earlier mechanism must be placed at its correct boundary and defended through production-like failures.

Core challenges you will face:

• Four-plane architecture -> state, notification, work, and control planes.
• Tenant isolation at every hop -> socket, topic, database, broker, metrics, logs.
• Measured resilience -> partitions, overload, duplicates, reconnects, and recovery SLOs.

Real World Outcome

Deploy three application nodes and a durable data/broker layer. Tenant 42 and Tenant 77 each have operators, devices/incidents, responder presence, and LiveView boards. Domain changes commit with outbox identity, online boards refresh through PubSub, integrations run under acknowledgements, and high-rate gauges flow through bounded/coalesced paths. A chaos console executes a slow subscriber, node partition, app kill, broker pause, duplicate delivery, and role revocation while invariant cards remain green or enter an explicitly predicted degraded state.

EVENT PLATFORM — CHAOS CERTIFICATION
nodes=3 connected_components=1 tenants_active=2
ui_event_age_p99=182ms target<500ms PASS
mailbox_p99=44 bound<1000 PASS coalesced=228401
outbox_oldest=1.8s target<30s PASS
broker_redeliveries=17 duplicate_effects=0 PASS
presence_convergence_p99=6.2s target<35s PASS
cross_tenant_delivery_attempts=240 observed_leaks=0 PASS

drill node_partition:
  transient revisions missed=expected
  durable facts lost=0
  reconnect reload/replay=PASS
overall certification=PASS_WITH_DOCUMENTED_EPHEMERAL_LOSS

The Core Question You Are Answering

“Can I justify, observe, and recover every event boundary without using Pub/Sub as a universal substitute for state, work queues, authorization, or consensus?”

Concepts You Must Understand First

1. All nine theory chapters — Explain the contract of each plane before building.
2. Quality attribute scenarios — What stimulus, environment, response, and measure define success? Book Reference: “Software Architecture in Practice, 4th Edition” — Quality Attributes.
3. Evolutionary boundaries — How can mechanisms change without breaking domain contracts? Book Reference: “Building Evolutionary Architectures, 2nd Edition” — fitness functions.
4. Operational safety — How do stability patterns prevent cascade? Book Reference: “Release It!, 2nd Edition” — Stability Patterns.

Questions to Guide Your Design

1. Which facts are authoritative, which notifications are transient, and which work is durable?
2. What event identity and revision follow a fact through outbox, broker, projection, and PubSub?
3. Where are tenant/auth decisions cached, revoked, audited, and rechecked?
4. What overload mode preserves critical work and degrades optional freshness?
5. Which partition behaviors continue and which fail closed?

Thinking Exercise

Create a complete event journey for “incident severity raised”: command, database transaction, outbox, PubSub refresh, broker integration, presence notification, metrics, reconnect, and replay. For every arrow, name ordering, loss, duplicate, authorization, capacity, and recovery semantics.

The Interview Questions They Will Ask

1. “How do you choose between Phoenix.PubSub, Oban, GenStage, and RabbitMQ?”
2. “How does your platform recover after missing PubSub during a partition?”
3. “Where can duplicates occur and how are effects idempotent?”
4. “How do you prevent cross-tenant leakage through topics and observability?”
5. “What happens when one subscriber cannot keep up?”
6. “Which evidence proves the system meets its SLOs under chaos?”

Hints in Layers

Hint 1: Draw four planes — Do not start with a framework component diagram.

Hint 2: Build one tenant end to end — Add a second only after identity and policy are explicit.

Hint 3: Add failure before scale — Crash and replay one event before increasing throughput/nodes.

Hint 4: Certification is the product — Automate invariants and publish a chaos evidence report.

Books That Will Help

Topic	Book	Chapter
Quality attributes	“Software Architecture in Practice, 4th Edition”	Quality Attribute Scenarios
Messaging	“Enterprise Integration Patterns”	Ch. 2–4 and Message Routing
Data/distribution	“Designing Data-Intensive Applications, 2nd Edition”	Transactions, Distribution, Streams
Stability	“Release It!, 2nd Edition”	Stability Patterns and Production

Common Pitfalls and Debugging

Problem 1: “One generic event bus owns everything”

• Why: Notification, work, state, and control contracts were collapsed.
• Fix: Assign every flow to one plane with explicit recovery and authority.
• Quick test: Remove PubSub entirely for one minute; durable facts/work must remain recoverable.

Problem 2: “Chaos report is green but no hypothesis exists”

• Why: Tests only prove processes restarted, not user/business invariants.
• Fix: Define stimulus, expected degraded behavior, measurable response, and recovery deadline.
• Quick test: Every drill must fail when its recovery mechanism is deliberately disabled.

Definition of Done

• Three-node tenant-safe real-time UI and Presence work under normal load.
• Durable state/outbox/broker paths survive restart and duplicate delivery.
• Optional high-rate updates remain bounded under overload.
• Partition recovery reloads/replays facts and documents missed transient events.
• Cross-tenant tests cover subscription, publication, broker, payload, metrics, and logs.
• Automated chaos evidence proves latency, capacity, durability, convergence, and isolation SLOs.

Project Comparison Table

Project	Difficulty	Time	Depth of Understanding	Fun Factor
1. Mailbox Broadcast Laboratory	Level 1	6–8h	Foundational	★★★☆☆
2. Registry Topic Bus	Level 2	8–12h	Foundational	★★★☆☆
3. Contract-Aware Domain Event Hub	Level 2	10–14h	High	★★★☆☆
4. Phoenix.PubSub Notification Hub	Level 2	10–14h	High	★★★★☆
5. LiveView Operations Board	Level 3	14–20h	High	★★★★☆
6. Distributed Presence Workspace	Level 3	16–22h	High	★★★★☆
7. Mailbox Pressure Laboratory	Level 3	16–24h	Very High	★★★★★
8. Demand-Aware Fanout Bridge	Level 3	18–26h	Very High	★★★★☆
9. Durable Notification Outbox	Level 4	22–32h	Very High	★★★★☆
10. Three-Node PubSub Laboratory	Level 4	20–30h	Very High	★★★★★
11. Tenant-Safe Broker Edge Bridge	Level 4	24–36h	Expert	★★★★★
12. Multi-Tenant Real-Time Event Platform	Level 5	45–60h	Mastery	★★★★★

Recommendation

If you are new to Elixir concurrency: Start with Project 1, then complete 2 and 3 before touching Phoenix. You will debug framework behavior much faster once mailboxes and Registry are concrete.

If you already build Phoenix applications: Complete Projects 1, 4, 5, and 7 first. Project 7 is the corrective for the common assumption that lightweight processes make subscriber overload irrelevant.

If you integrate services or brokers: Focus on Projects 3, 8, 9, and 11. They separate contracts, demand, atomic handoff, and acknowledgement.

If you operate clustered BEAM systems: Focus on Projects 7, 10, 11, and 12. Require partition and rolling-migration evidence before calling the cluster production-ready.

Final Overall Project: Real-Time Operations Mesh

The Goal: Combine Projects 3, 5–12 into a deployable operations mesh for incidents or device fleets.

1. Commit versioned tenant facts and outbox rows atomically.
2. Refresh connected LiveViews through Phoenix.PubSub and show responders through Presence.
3. Route high-rate gauges through an explicit bounded/coalesced path.
4. Deliver required integrations through Oban or RabbitMQ/Broadway with acknowledgements and idempotency.
5. Operate on three nodes with secure topology, safe PubSub pool migration, partition reload, and schema compatibility.
6. Certify the platform with automated load, duplicate, outage, partition, reconnect, revocation, and cross-tenant drills.

Success Criteria: The certification report shows no lost durable fact or required effect, no cross-tenant delivery, bounded mailboxes, documented transient loss, state convergence after reconnect, and every SLO measured at publisher, queue, handler, and effect boundaries.

From Learning to Production: What Is Next

Your Project	Production Equivalent	Gap to Fill
Registry Topic Bus	Library-internal/local event dispatcher	API compatibility, performance matrix, lifecycle documentation
Phoenix.PubSub Hub	Cluster real-time notification plane	topology automation, secure distribution, rolling runbooks
Presence Workspace	Collaboration/online projection	privacy, region behavior, deploy churn policy, product semantics
Mailbox Pressure Lab	Capacity and SLO program	production sampling, admission controls, autoscaling signals
Demand-Aware Bridge	Broker ingestion pipeline	connector-specific retry, DLQ, partition/order, cost controls
Durable Outbox	Integration event relay	retention, backfill, schema registry, reconciliation, CDC option
Three-Node Lab	Multi-region/node Phoenix runtime	secure discovery, latency budgets, failure-domain design
Broker Edge Bridge	Cross-service event platform	governance, ACL automation, replay tooling, disaster recovery
Event Platform Capstone	Managed real-time control plane	SSO/RBAC, audit, multi-region data, compliance, tenancy economics

Before production, add explicit ownership, threat modeling, data classification, capacity tests on representative hardware, backup/restore drills, schema governance, secret rotation, and an on-call runbook. Do not move irreversible work onto transient PubSub merely to reduce infrastructure count.

Summary

This learning path covers Elixir Pub/Sub through twelve projects, beginning with raw BEAM mailboxes and ending with a secure, durable, observable multi-node event platform.

#	Project Name	Main Language	Difficulty	Time Estimate
1	Mailbox Broadcast Laboratory	Elixir	Level 1	6–8h
2	Registry Topic Bus	Elixir	Level 2	8–12h
3	Contract-Aware Domain Event Hub	Elixir	Level 2	10–14h
4	Phoenix.PubSub Notification Hub	Elixir	Level 2	10–14h
5	LiveView Operations Board	Elixir	Level 3	14–20h
6	Distributed Presence Workspace	Elixir	Level 3	16–22h
7	Mailbox Pressure Laboratory	Elixir	Level 3	16–24h
8	Demand-Aware Fanout Bridge	Elixir	Level 3	18–26h
9	Durable Notification Outbox	Elixir	Level 4	22–32h
10	Three-Node PubSub Laboratory	Elixir	Level 4	20–30h
11	Tenant-Safe Broker Edge Bridge	Elixir	Level 4	24–36h
12	Multi-Tenant Real-Time Event Platform	Elixir	Level 5	45–60h

Expected Outcomes

• Explain BEAM and Phoenix.PubSub delivery from first principles.
• Design topic/envelope contracts that survive version, restart, and tenant boundaries.
• Keep slow subscribers and high fanout from silently exhausting mailboxes.
• Separate transient notification from demand-aware and durable work.
• Operate distributed PubSub honestly under partitions and rolling changes.
• Prove idempotency, convergence, observability, and tenant isolation under controlled failure.

Additional Resources and References

Official Elixir, Erlang, and Phoenix documentation

• Elixir Registry — duplicate keys, dispatch, partitions, and lifecycle.
• Phoenix.PubSub 2.2 — APIs, adapters, custom dispatch, and safe pool migration.
• Phoenix.Tracker and Phoenix.Presence — convergent presence state and diffs.
• OTP 29 :pg — distributed group semantics and topology constraints.
• Communication in Erlang — signal ordering and process rules.
• GenStage, Broadway, and Telemetry — demand, acknowledgements, and instrumentation.
• Ecto.Multi and Oban — atomic database work and durable jobs.

Messaging and durability specifications/guides

• CloudEvents 1.0 — interoperable event-envelope vocabulary.
• RabbitMQ Reliability Guide and Confirms/Acknowledgements — broker responsibility boundaries.
• Debezium Outbox Event Router — stable identity and aggregate-key routing.
• PostgreSQL NOTIFY and SKIP LOCKED — wake-up hints and queue-like relay claiming.

Industry analysis and case studies

• Discord: How Discord Scaled Elixir to 5,000,000 Concurrent Users — fanout and scheduling bottlenecks.
• Discord: Maxjourney — relays and million-user guild fanout.
• Discord: Tracing Elixir Systems Without Melting Everything — 2026 observability lessons.
• Stack Overflow 2025 Technology Survey — Elixir and Phoenix admiration statistics.
• Elixir 1.20 Release — current language baseline.

Books

• “Enterprise Integration Patterns” by Gregor Hohpe and Bobby Woolf — the messaging vocabulary behind topics, durable subscribers, routing, dead letters, and idempotent receivers.
• “Designing Data-Intensive Applications, 2nd Edition” by Martin Kleppmann et al. — distributed failure, event streams, consistency, and replay.
• “Designing for Scalability with Erlang/OTP” by Francesco Cesarini and Steve Vinoski — OTP-native process and distribution architecture.
• “Release It!, 2nd Edition” by Michael Nygard — overload, stability, and production failure patterns.
• “Real-Time Phoenix” by Stephen Bussey — Phoenix Channels, PubSub, and Presence in real-time systems.