Project 4: Phoenix.PubSub Notification Hub

Replace the custom local bus with Phoenix.PubSub while preserving domain contracts and proving local, cluster, sender-excluding, restart, and pool-migration semantics.

Quick Reference

Attribute Value
Difficulty Level 2: Intermediate
Time Estimate 10–14 hours
Language Elixir (alternatives: Erlang, Gleam)
Prerequisites Projects 1–3, OTP supervision, Registry semantics, domain event contracts, basic distributed Erlang vocabulary
Key Topics Phoenix.PubSub 2.2, subscriptions, broadcasts, sender exclusion, local scope, adapter boundary, duplicate subscriptions, pool sizing and migration
Primary Tools Phoenix.PubSub 2.2, Mix, IEx, ExUnit
Produces A contract-aware notification hub and a tested pool-size migration runbook

1. Learning Objectives

By completing this project, you will:

  1. Explain which responsibilities Phoenix.PubSub provides and which remain application responsibilities.
  2. Compare cluster broadcast, cluster sender-excluding broadcast, node-local broadcast, and node-local sender-excluding broadcast.
  3. Explain that subscribe attaches membership to the calling PID and that process restart requires resubscription.
  4. Prevent accidental duplicate subscriptions, which Phoenix.PubSub explicitly permits and which cause duplicate messages.
  5. Distinguish adapter routing success from subscriber mailbox handling or business acknowledgement.
  6. Describe local Registry fanout and the adapter boundary used to exchange notifications between nodes.
  7. Configure broadcast pool size and local Registry partition count explicitly and for different reasons.
  8. Produce a two-stage rolling migration runbook using broadcast_pool_size when changing PG2 pool size.
  9. Preserve Project 3’s topic derivation, envelope validation, tenant boundary, and instrumentation in a domain wrapper.
  10. Demonstrate that Pub/Sub provides no offline replay after a subscriber restarts.

2. Theoretical Foundation

2.1 Core Concepts

Phoenix.PubSub as infrastructure

Phoenix.PubSub is a real-time publisher/subscriber service that manages process subscriptions and fanout. A process subscribes itself to a string topic. A publisher broadcasts an Elixir term to a topic. Local subscribers receive that term as an ordinary mailbox message. When nodes are connected through a supported adapter, a cluster broadcast is exchanged with other nodes and fanned out to local subscribers there.

The library solves infrastructure concerns: supervised Pub/Sub processes, local registration, local dispatch, adapter integration, broadcast pooling, and well-defined APIs. It does not invent domain topics, validate tenant authorization, version event schemas, store event history, acknowledge handler completion, bound subscriber mailboxes, or reload missed state. Those responsibilities remain in the application layer designed in Project 3.

Subscription lifecycle and duplicates

Phoenix.PubSub.subscribe subscribes the calling process. Membership therefore belongs to one PID incarnation. If a LiveView, GenServer, or fixture restarts, its new PID must subscribe during initialization. Events broadcast while it is absent are not replayed.

The official API permits duplicate subscriptions for one PID/topic pair. Each duplicate causes another message to be sent. Unsubscribe drops all duplicate subscriptions for that PID/topic. A domain wrapper should make ordinary subscriptions idempotent unless multiplicity is an intentional, documented feature.

Broadcast variants

The core variants differ along two axes:

Operation Scope Sender receives?
broadcast Whole connected PubSub cluster Yes, if subscribed
broadcast_from Whole connected PubSub cluster No for the supplied sender PID
local_broadcast Current node only Yes, if subscribed
local_broadcast_from Current node only No for the supplied sender PID

Sender exclusion is useful when a process already applied an optimistic local update and should not apply its own notification twice. It is not a security feature. The supplied sender PID and wrapper contract must be trusted.

Local-only delivery is a semantic choice when an event concerns node-local state, such as local cache bookkeeping or diagnostics. It should not be selected merely because it is faster if correctness requires every node to invalidate state.

Adapter boundary

The default Phoenix.PubSub.PG2 adapter uses Distributed Erlang and is described by the official documentation as based on pg/pg2. The PG2 module name is historical and retained as the adapter’s public identity. The adapter exchanges notifications between nodes; each destination node then performs local fanout. Node discovery, network reachability, distribution security, and partition recovery are separate concerns developed in Project 10.

No processing acknowledgement

Phoenix.PubSub.broadcast can return ok after the library and adapter complete their routing operation, but that result is not a receipt from subscribers. The message may be waiting in a mailbox, the process may exit, or the handler may fail. If the domain needs an acknowledgement, define a separate protocol or use a durable workflow mechanism.

2.2 Local and Cluster Delivery Path

Publisher on Node A
       |
       | NotificationHub.broadcast(envelope)
       v
+---------------- Phoenix.PubSub on Node A ----------------+
| validate wrapper contract                                |
| adapter broadcast shard                                  |
| local Registry subscriptions                             |
+------------+---------------------------+-----------------+
             |                           |
             v                           | Distributed Erlang adapter
   local subscriber mailbox              v
                              +---------------------------+
                              | Phoenix.PubSub on Node B  |
                              | local Registry fanout     |
                              +-------------+-------------+
                                            |
                                            v
                                  remote subscriber mailbox

For local_broadcast, the adapter-crossing path is intentionally absent. For broadcast_from, the dispatcher excludes the supplied sender PID from local delivery wherever that subscription is represented.

The message is still just an Elixir term entering a process mailbox. Project 1’s ordering, queue, death, and acknowledgement lessons remain in force.

2.3 Pool Size and Registry Size

Phoenix.PubSub 2.2 exposes related but distinct controls:

  • pool_size: Number of Pub/Sub broadcast partitions. With the PG2 adapter, nodes in one cluster must use compatible pool sizing.
  • registry_size: Number of local Registry partitions used for subscriptions. It defaults to pool_size but can be tuned independently.
  • broadcast_pool_size: Number of shards used for outbound broadcasts during a safe pool-size migration; it can temporarily remain at the old size while all nodes learn the new receive-side pool.

Do not compute a cluster-wide PG2 pool size independently from local CPU count on heterogeneous machines. The official adapter guidance requires consistent pool size across the cluster. Choose and configure it as deployment topology, not an incidental machine default.

Safe increase from one broadcast shard to two:

Initial release on every node
  pool_size = 1

Migration release on every node
  pool_size = 2
  broadcast_pool_size = 1

Final release after every node runs migration release
  pool_size = 2
  broadcast_pool_size omitted, therefore 2

During the middle stage, nodes can receive on both new partitions while publishers continue to choose the old compatible broadcast range. Decreasing follows the documented process in reverse. This project produces a runbook and a mixed-release test fixture; Project 9 applies it to a real multi-node cluster.

2.4 Domain Wrapper Responsibilities

The NotificationHub wrapper preserves the prior project’s contracts:

  1. Accept authenticated/domain context, not arbitrary topic strings from untrusted callers.
  2. Authorize and derive bounded topics.
  3. Validate the domain event envelope and supported schema.
  4. Choose cluster, local, or sender-excluding semantics from an explicit internal operation.
  5. Instrument routing duration and result with bounded labels.
  6. Return a routing result, never a “processed by all subscribers” result.
  7. Keep subscriber handlers outside the wrapper.

Scattering direct Phoenix.PubSub calls through contexts, LiveViews, and workers makes it difficult to change topic naming, authorization, envelope versions, or instrumentation. A thin wrapper creates one reviewable boundary without hiding what Phoenix.PubSub actually does.

2.5 Duplicate and Restart Timeline

PID <0.210.0> subscribes once
   |
   +-- accidental second subscribe
   |
broadcast evt-1
   |
   +-- mailbox receives evt-1
   +-- mailbox receives evt-1 again

wrapper policy:
   first subscribe  -> NEW
   second subscribe -> ALREADY_SUBSCRIBED
   one broadcast    -> one mailbox copy

process exits:
   membership removed
   queued messages disappear

replacement PID <0.225.0> starts:
   subscribe again
   future events arrive
   missed events are not replayed

2.6 Definitions and Key Terms

  • PubSub system name: The supervised name identifying one Phoenix.PubSub instance.
  • Topic: String routing key used to associate subscribers and broadcasts.
  • Adapter: Component transporting broadcasts beyond local fanout.
  • PG2 adapter: Default Distributed Erlang adapter, historically named and based on pg/pg2.
  • Dispatcher: Component performing local delivery to subscription entries.
  • Broadcast pool: Set of partitions used to exchange broadcasts.
  • Registry size: Number of local subscription Registry partitions.
  • Sender exclusion: Omitting one specified PID from local delivery.
  • Local broadcast: Fanout only to subscribers on the current node.
  • Cluster broadcast: Adapter-mediated fanout across connected PubSub nodes.
  • Routing result: Evidence that the PubSub operation completed, not that subscribers handled the message.
  • Safe pool migration: Staged configuration allowing mixed releases to receive compatible broadcast shards.

2.7 Minimal Concrete Example

Configuration outline:

PubSub child:
  name: Demo.PubSub
  adapter: Phoenix.PubSub.PG2
  pool_size: 2
  registry_size: 4

Wrapper protocol:

subscribe_alerts(principal, tenant_id):
  authorize principal
  derive tenant alerts topic
  subscribe current PID once

publish_alert(context, envelope, mode):
  authorize and validate
  derive topic
  choose:
    CLUSTER_INCLUDE_SENDER
    CLUSTER_EXCLUDE_SENDER
    LOCAL_INCLUDE_SENDER
    LOCAL_EXCLUDE_SENDER
  instrument routing result

Subscriber protocol:

receive validated envelope
compare revision
handle or reload according to Project 3

This is architecture pseudocode, not runnable Elixir.

2.8 Common Misconceptions

  • “Phoenix.PubSub is a durable message broker.” It stores no replayable event history.
  • “broadcast returning ok means every subscriber handled the message.” It only reports routing-level completion.
  • “Subscribe is automatically idempotent.” Duplicate PID/topic subscriptions are permitted and cause duplicate messages.
  • “A restarted subscriber keeps its membership.” The replacement PID must subscribe again.
  • “local_broadcast is a harmless optimization.” It changes which nodes can receive and is therefore semantic.
  • “broadcast_from authenticates the sender.” It merely excludes the supplied PID from delivery.
  • “pool_size and registry_size are the same tuning knob.” One concerns broadcast partitions; the other local subscription Registry partitions.
  • “Changing pool_size in one deployment step is always safe.” PG2 rolling migrations need the documented broadcast_pool_size transition.
  • “The PG2 adapter requires the removed pg2 runtime API on modern OTP.” The adapter is officially described as pg/pg2-based; its historical module name remains.

2.9 Check Your Understanding

  1. What does Phoenix.PubSub provide beyond Project 2’s custom bus?
  2. Which operation excludes a subscribed publisher across the cluster?
  3. When is local_broadcast correct?
  4. Why must a replacement process subscribe again?
  5. What does broadcast returning ok not prove?
  6. Why separate registry_size from pool_size?
  7. How does broadcast_pool_size protect a rolling migration?

Answers

  1. A supported framework API, supervised infrastructure, adapter boundary, broadcast pooling, and cluster-aware operations.
  2. broadcast_from with the intended sender PID.
  3. Only when the event is semantically node-local.
  4. Subscription belongs to the old PID incarnation and there is no durable consumer identity.
  5. It does not prove mailbox receive, handler completion, state commit, or user-visible update.
  6. Local membership contention and cluster broadcast partitioning can have different tuning needs.
  7. It lets nodes receive the new shard layout while broadcasts continue using the old compatible range during the mixed-release stage.

2.10 Historical Context and Evolution

Phoenix extracted Pub/Sub into an independent library so Channels, LiveView, Presence, and application code could share a transport-neutral real-time substrate. The default adapter’s PG2 name reflects its historical lineage; current documentation describes it as based on pg/pg2 and running over Distributed Erlang. The library’s stable API is more important to application code than the runtime membership mechanism beneath the adapter.

2.11 Why This Matters

Phoenix makes a broadcast easy enough to write anywhere. That convenience can spread inconsistent topics, duplicate subscriptions, false delivery claims, and unsafe rolling configuration. This project establishes a narrow, observable boundary so later LiveView and Presence projects inherit one coherent contract. It also teaches where to stop: Phoenix.PubSub is excellent at transient real-time fanout, while durable workflows and broker acknowledgements belong elsewhere.

2.12 References

  • Phoenix.PubSub 2.2 — APIs, duplicate subscriptions, options, and safe pool migration.
  • Phoenix.PubSub.PG2 — default adapter and pg/pg2 lineage.
  • Phoenix.PubSub.Adapter — adapter contract.
  • Elixir Registry — local subscription primitive.
  • “Real-Time Phoenix” by Stephen Bussey — PubSub, Channels, and real-time architecture.
  • “Enterprise Integration Patterns” — Publish-Subscribe Channel and Durable Subscriber contrast.

3. Project Specification

3.1 What You Will Build

Build a supervised Demo.PubSub instance and a NotificationHub wrapper carrying the Project 3 order/alert envelope. Three terminal clients demonstrate:

  • An operator process that publishes and is also subscribed.
  • A UI subscriber receiving tenant alert events.
  • A node-local cache observer receiving local invalidation events.

The scenario sends one ordinary cluster broadcast, one sender-excluding cluster broadcast, and one local-only cache notification. It then restarts the UI process and proves the replacement receives future events but not the event broadcast while it was absent.

The project also includes explicit pool and Registry configuration, idempotent subscription semantics, wrapper Telemetry, API contract tests, and a two-stage pool-size migration runbook with a mixed-release configuration matrix.

3.2 Functional Requirements

  1. Supervised PubSub: Demo.PubSub starts before subscribers.
  2. Explicit configuration: pool_size and registry_size are documented.
  3. Domain wrapper: Application code does not scatter raw PubSub calls.
  4. Authorized topic derivation: Caller supplies principal and domain IDs, not arbitrary topic.
  5. Envelope validation: Only supported event contracts are broadcast.
  6. Idempotent logical subscribe: Repeated wrapper subscribe yields one delivery.
  7. Cluster include-sender mode: A subscribed publisher receives its own event.
  8. Cluster exclude-sender mode: The supplied sender PID receives zero copies while peers receive one.
  9. Local-only mode: Only current-node subscribers are targeted.
  10. Local exclude-sender mode: Supported and separately tested.
  11. Restart demonstration: Replacement subscriber explicitly resubscribes.
  12. No replay: Missed event remains absent after restart.
  13. Routing instrumentation: Duration, operation class, topic family, and result are emitted.
  14. No handling claim: Subscriber receipts are recorded separately in the demo.
  15. Migration runbook: Initial, transition, and final configurations are documented and tested.
  16. Error policy: Non-raising and raising broadcast variants have a deliberate wrapper policy.

3.3 Non-Functional Requirements

  • Correctness: Every mode has an independent test; no behavior is inferred from naming.
  • Security: Authorization and topic derivation occur before PubSub API invocation.
  • Compatibility: PG2 pool size is consistent across nodes in every simulated rollout stage.
  • Observability: Metrics use bounded operation and topic-family labels.
  • Performance: Wrapper instrumentation and validation remain bounded.
  • Maintainability: PubSub configuration has an owner and a deployment runbook.
  • Resilience: Subscriber restart does not crash publishers or imply replay.

3.4 Example Usage and Exact Output

$ mix pubsub.demo
pubsub=Demo.PubSub adapter=PG2 pool_size=2 registry_size=4 status=ready
subscribe pid=<0.210.0> topic=tenant:42:alerts result=new
subscribe pid=<0.211.0> topic=tenant:42:alerts result=new
subscribe pid=<0.210.0> topic=tenant:42:alerts result=already_subscribed
broadcast event=evt-31 scope=cluster include_sender=true route_us=38 result=ok
handled pid=<0.210.0> event=evt-31 age_us=74
handled pid=<0.211.0> event=evt-31 age_us=81
broadcast_from event=evt-32 sender=<0.210.0> sender_echo=excluded result=ok
handled pid=<0.211.0> event=evt-32 copies=1
local_broadcast event=cache-9 scope=node_local recipients_observed=1
stop subscriber old=<0.211.0>
broadcast event=evt-33 while_subscriber_absent result=ok
restart subscriber old=<0.211.0> new=<0.225.0> subscribe=result_new
replay event=evt-33 available=false
broadcast event=evt-34 handled_by=<0.225.0>

3.5 Real World Outcome

Run the demo and observe each API decision as a distinct line. The first broadcast reaches the subscribed operator and UI. The sender-excluding publication reaches the UI once while the operator receives zero echo. The cache event is explicitly labeled node_local and reaches only the local cache fixture.

After the UI subscriber exits, an event is broadcast successfully while it is absent. The replacement PID subscribes and receives the next event, but the report shows replay available=false for the missed one. The final diagnostic separates PubSub routing result from explicit handler receipts.

The accompanying migration report shows three configuration stages. A validation task rejects a simulated mixed cluster whose PG2 pool sizes cannot interoperate safely and accepts the documented transition using broadcast_pool_size.

4. Solution Architecture

4.1 High-Level Design

+---------------------- application layer ----------------------+
| authenticated context + domain IDs + event envelope           |
+-------------------------------+-------------------------------+
                                |
                                v
                     +----------------------+
                     | NotificationHub      |
                     | authorize            |
                     | derive topic         |
                     | validate envelope    |
                     | select delivery mode |
                     | instrument result    |
                     +----------+-----------+
                                |
                                v
                     +----------------------+
                     | Demo.PubSub          |
                     | broadcast pool       |
                     | local Registry       |
                     | PG2 adapter          |
                     +---+--------------+---+
                         |              |
              local send |              | adapter exchange
                         v              v
                  local mailbox    other-node PubSub
                                        |
                                        v
                                  remote mailbox

4.2 Key Components

Component Responsibility Key Decisions
Demo.PubSub child Run supervised PubSub infrastructure Explicit adapter, pool_size, registry_size
NotificationHub Own domain-facing publish/subscribe API No arbitrary raw topics
Topic derivation Reuse tenant/subject rules from Project 3 Bounded topic families
Envelope validator Enforce event catalog Reject before broadcast
Mode selector Map domain intent to exact PubSub operation Scope and echo are explicit
Subscription helper Prevent repeated PID/topic registration Idempotent ordinary subscription
Telemetry boundary Measure routing operation Separate handler receipts
Terminal fixtures Demonstrate operator, UI, and cache behavior New PID subscribes on restart
Migration validator Check staged configuration matrix Two-stage PG2 change

4.3 Data Structures

DeliveryMode:
  CLUSTER_INCLUDE_SENDER
  CLUSTER_EXCLUDE_SENDER
  LOCAL_INCLUDE_SENDER
  LOCAL_EXCLUDE_SENDER

NotificationRequest:
  authenticated_context
  tenant_id
  subject_reference
  event_envelope
  delivery_mode
  sender_pid when exclusion is requested

RoutingObservation:
  event_id
  operation_class
  topic_family
  scope
  sender_policy
  started_monotonic
  duration_microseconds
  result
  processing_acknowledgement: none

PubSubDeploymentConfig:
  release_label
  pool_size
  registry_size
  broadcast_pool_size or omitted

4.4 Algorithm Overview

Subscribe once

  1. Validate authenticated context and domain identifiers.
  2. Derive topic.
  3. Check the current process’s logical subscription state.
  4. If already subscribed, return ALREADY_SUBSCRIBED.
  5. Otherwise call Phoenix.PubSub subscribe and record local helper state.

Publish

  1. Authorize context and validate envelope.
  2. Derive topic and bounded metric family.
  3. Start monotonic timing.
  4. Select the exact PubSub operation from delivery mode.
  5. Record routing result.
  6. Do not wait for or infer subscriber handling.

Pool-size migration validation

  1. Read desired configuration for every release stage.
  2. Verify all nodes in the initial stage broadcast and receive on old pool.
  3. Verify transition stage receives with new pool but broadcasts on old pool.
  4. Verify final stage broadcasts and receives on new pool.
  5. Reject a stage that introduces new broadcast shards before all nodes can receive them.

Complexity

  • Subscription: approximately local Registry insertion cost.
  • Local fanout: O(local matching subscribers).
  • Cluster broadcast: local fanout plus adapter work proportional to participating node topology and serialization.
  • Wrapper validation/topic derivation: bounded O(1) for fixed envelope/topic shapes.

4.5 Operation Selection Matrix

Use case Required scope Sender already applied? Operation class
Tenant alert refresh Cluster No CLUSTER_INCLUDE_SENDER
Optimistic chat/UI update Cluster Yes CLUSTER_EXCLUDE_SENDER
Node cache bookkeeping Local No LOCAL_INCLUDE_SENDER
Node diagnostic echo suppression Local Yes LOCAL_EXCLUDE_SENDER

The matrix is application policy. Do not choose based only on convenience or microbenchmark results.

4.6 Architectural Invariants

Invariant Enforcement
Direct PubSub use does not leak across application Wrapper is sole domain entry point
One logical subscription yields one copy Subscribe-once helper and tests
Delivery scope is explicit Closed delivery-mode vocabulary
Topic is authorized and derived Project 3 policy retained
Routing result is not handler acknowledgement Observation schema
Restart does not imply replay Lifecycle scenario
Pool migration is staged Versioned runbook and validator

5. Implementation Guide

5.1 Development Environment Setup

$ elixir --version
Erlang/OTP 27 [...]
Elixir 1.18.[...]

$ mix new phoenix_pubsub_notification_hub --sup
* creating mix.exs
* creating lib/...
* creating test/...

Dependency outline:

dependency: phoenix_pubsub
compatible series: 2.2

Resolve the current supported patch release in the project lockfile. The guide targets the Phoenix.PubSub 2.2 API documented in the official HexDocs.

5.2 Suggested Project Structure

phoenix_pubsub_notification_hub/
├── lib/
│   ├── notification_hub/application.ex
│   └── notification_hub/
│       ├── pub_sub.ex
│       ├── topics.ex
│       ├── envelopes.ex
│       ├── subscription_state.ex
│       ├── telemetry.ex
│       ├── migration_config.ex
│       └── fixtures/
│           ├── operator.ex
│           ├── ui_subscriber.ex
│           └── cache_observer.ex
├── config/
│   ├── config.exs
│   └── migration_profiles/
├── lab/
│   └── pubsub_demo.exs
├── test/
│   ├── delivery_modes_test.exs
│   ├── duplicate_subscription_test.exs
│   ├── lifecycle_test.exs
│   └── pool_migration_test.exs
└── README.md

5.3 The Core Question You Are Answering

“Which parts of a production Pub/Sub system does Phoenix.PubSub provide, and which contracts still belong to my application?”

List the concerns before implementation. Place local membership, adapter exchange, and fanout under Phoenix.PubSub. Place authorization, topic derivation, event schema, revision policy, backpressure strategy, durability, and business acknowledgement under application or other infrastructure.

5.4 Concepts You Must Understand First

  1. Local Registry fanout
    • Which ideas from Project 2 remain underneath?
    • Why does subscribe belong to the calling PID?
    • Reference: Phoenix.PubSub and Elixir Registry docs.
  2. Adapter boundary
    • Which work crosses nodes?
    • Which discovery/network concerns are outside PubSub?
    • Reference: Phoenix.PubSub.Adapter and PG2 docs.
  3. Echo semantics
    • Has the publisher already applied the change locally?
    • Which supplied PID is excluded?
    • Book: “Real-Time Phoenix,” PubSub and Channels.
  4. Ephemeral lifecycle
    • What does a restarted subscriber miss?
    • How does it recover current state?
    • Book: “Enterprise Integration Patterns,” Durable Subscriber contrast.
  5. Pool compatibility
    • Why must cluster broadcast partitions be compatible?
    • What does broadcast_pool_size do during migration?
    • Reference: official Phoenix.PubSub safe pool migration.
  6. Routing versus handling
    • What exactly does an ok result prove?
    • Which protocol would produce business acknowledgement?

5.5 Questions to Guide Your Design

  1. Which application module owns all PubSub calls?
  2. Which functions accept domain IDs rather than topics?
  3. When is sender inclusion required?
  4. When is local-only delivery a correctness property?
  5. How does each subscriber resubscribe after restart?
  6. How does the wrapper detect repeated logical subscription?
  7. Which labels are safe for Telemetry metrics?
  8. How are raising and non-raising broadcast variants handled?
  9. Who owns the pool-size configuration and runbook?
  10. How will mixed-release compatibility be validated before deployment?

5.6 Thinking Exercise

For each event, choose a delivery mode and explain the consequence of every wrong choice:

  • Local cache table was cleared.
  • Tenant order status changed.
  • Chat composer already inserted its own optimistic message.
  • A node-specific diagnostic sampler changed configuration.
  • A fleet-wide authorization policy changed.

Then draw a rolling deployment from pool_size one to two. Mark which shards each release can receive from and which shards it may broadcast on.

5.7 The Interview Questions They Will Ask

  1. How does Phoenix.PubSub deliver locally?
  2. What does broadcast returning ok prove?
  3. What is the difference between broadcast and broadcast_from?
  4. When should local_broadcast be used?
  5. Why is the default adapter still named PG2?
  6. How do you safely change PubSub pool size?

5.8 Hints in Layers

Hint 1: Wrap, do not scatter

Centralize topic derivation, envelope validation, mode selection, and instrumentation.

Hint 2: Subscribe the publisher in tests

You cannot prove echo inclusion or exclusion if the sender is not subscribed.

Hint 3: Restart the actual subscriber process

Use a new PID and publish once while it is absent. Do not simulate restart with local state reset.

Hint 4: Treat migration as topology

Represent every release stage explicitly and verify that send shards are understood by all receiving nodes.

5.9 Books That Will Help

Topic Book Chapter or Section
Phoenix real-time architecture “Real-Time Phoenix” PubSub and Channels
Pub/Sub contracts “Enterprise Integration Patterns” Ch. 3–4
Durable subscriber contrast “Enterprise Integration Patterns” Messaging endpoint patterns
Stability and rollout “Release It!, 2nd Edition” Stability Patterns and Operations
Distributed assumptions “Designing Data-Intensive Applications” Distributed Systems chapters

5.10 Implementation Phases

Phase 1: Supervised PubSub — 2 hours

  • Add Phoenix.PubSub as an explicitly configured child.
  • Start one subscriber and one publisher.
  • Record a routing observation separately from handler receipt.

Checkpoint: One cluster-class broadcast works on the local node and output labels its limits.

Phase 2: Domain Wrapper — 3–4 hours

  • Reuse Project 3 envelope and topic rules.
  • Add closed delivery-mode selection.
  • Instrument the wrapper.

Checkpoint: Application fixtures do not call Phoenix.PubSub directly.

Phase 3: Duplicate and Echo Semantics — 3 hours

  • Make ordinary subscription idempotent.
  • Test include-sender and exclude-sender modes.
  • Test all duplicates are removed by explicit unsubscribe behavior.

Checkpoint: Each operation’s sender copy count matches the matrix exactly.

Phase 4: Local Scope and Restart — 2–3 hours

  • Add cache observer and local modes.
  • Restart UI PID and resubscribe.
  • Publish during absence to prove no replay.

Checkpoint: Replacement receives the next event but never the missed event.

Phase 5: Pool Migration Runbook — 2–4 hours

  • Document initial, transition, and final configurations.
  • Add a compatibility validator.
  • Include reverse procedure for decreasing pool size.

Checkpoint: Unsafe single-stage mixed release is rejected; staged transition is accepted.

Phase 6: Quality Pass — 1–2 hours

  • Add failure-path tests and bounded metric labels.
  • Confirm exact CLI transcript.
  • Verify official 2.2 API references.

Checkpoint: The full suite and documentation distinguish infrastructure guarantees from domain guarantees.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
PubSub usage Scattered calls, domain wrapper Domain wrapper Central contract and instrumentation
Subscription Raw repeated calls, idempotent helper Idempotent helper Prevents duplicate messages
Default scope Implicit local, explicit modes Explicit closed modes Scope is correctness
Sender behavior Always echo, explicit include/exclude Explicit Avoids double apply
Pool sizing Machine-derived per node, deployment value Deployment value PG2 cluster compatibility
Registry partitions Always equal pool, tune separately Explicit independent choice Different local contention concern
Error style Raising everywhere, deliberate boundary Non-raising wrapper with explicit escalation Makes routing failures observable
Missed events Pretend replay, reload authority Reload authority PubSub is ephemeral

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Subscription Validate PID/topic lifecycle Subscribe once, duplicate attempt, unsubscribe
Delivery mode Validate scope and echo Four-operation matrix
Contract Validate wrapper boundary Unauthorized ID and malformed envelope
Lifecycle Validate restart and no replay Publish during subscriber absence
Instrumentation Validate routing observations Bounded labels and result
Configuration Validate explicit pool/Registry values Startup report
Migration Validate mixed-release compatibility Initial, transition, final stages

6.2 Critical Test Cases

  1. Two subscriber PIDs receive one cluster broadcast.
  2. Subscribed publisher receives ordinary broadcast.
  3. Supplied sender receives zero copies from broadcast_from; peer receives one.
  4. local_broadcast reaches local subscriber and is marked node-local.
  5. local_broadcast_from excludes the supplied local sender.
  6. Repeated wrapper subscribe produces one copy.
  7. Raw duplicate behavior is demonstrated in an isolated educational test and hidden from application API.
  8. Unsubscribe removes the current PID’s topic subscriptions.
  9. Dead subscriber membership disappears.
  10. Replacement PID resubscribes and receives future events.
  11. Replacement does not receive event broadcast while absent.
  12. Routing ok and handler receipt are different evidence records.
  13. Unauthorized topic derivation yields no PubSub call.
  14. Unsafe pool-size rollout stage fails validation.
  15. Two-stage pool-size migration passes validation.

6.3 Test Data

topic:
  tenant:42:alerts

subscribers:
  operator: <0.210.0>, subscribed
  ui:       <0.211.0>, subscribed
  cache:    <0.212.0>, local cache topic

events:
  evt-31: cluster include sender
  evt-32: cluster exclude operator
  cache-9: node-local include sender
  evt-33: broadcast while UI absent
  evt-34: broadcast after replacement subscribes

expected copies:
  evt-31 operator=1 ui=1
  evt-32 operator=0 ui=1
  cache-9 cache=1 remote_fixture=0
  evt-33 replacement_ui=0
  evt-34 replacement_ui=1

6.4 Migration Matrix

stage 0:
  every node pool_size=1
  every node broadcast_pool_size=1

stage 1:
  every upgraded node pool_size=2
  every upgraded node broadcast_pool_size=1
  old nodes remain pool_size=1 during rollout
  outbound broadcasts stay in old compatible range

stage 2:
  only after every node can receive pool_size=2
  remove broadcast_pool_size override
  outbound broadcasts use pool_size=2

Test the configuration model separately from message delivery tests so a runbook regression fails with a precise explanation.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Duplicate subscribe One PID receives two copies Make wrapper subscribe idempotent
Ordinary broadcast after optimistic update Publisher applies update twice Use sender-excluding mode deliberately
local_broadcast used for shared cache Other nodes stay stale Use cluster scope when correctness requires
Routing ok called delivered Lost work after process crash surprises team Track handler evidence separately
Subscriber restarts without subscribe New PID receives nothing Subscribe during process initialization
Replay assumed UI remains stale after downtime Reload authoritative state
Pool size changed in one rolling step Mixed nodes miss shard traffic Use broadcast_pool_size transition
Per-node CPU-derived pool Heterogeneous nodes disagree Configure cluster-wide value
Raw tenant topic accepted Forged cross-tenant route Authorize and derive internally
Raw IDs in metrics Cardinality explosion Use operation and topic-family labels

7.2 Debugging Strategies

  • Inspect the subscriber PID’s mailbox and confirm the exact event copy count.
  • Log operation class, scope, sender policy, and event ID at the wrapper.
  • Check whether a restarted process actually called subscribe.
  • Reproduce duplicate delivery by counting raw subscribe calls for the same PID/topic.
  • Compare PubSub config across every node before investigating random cluster misses.
  • During migration, print both pool_size and broadcast_pool_size for every release.
  • Reduce cluster-class behavior to one node to isolate domain wrapper bugs from distribution bugs.

7.3 Performance Traps

Custom dispatchers can perform specialized local delivery, but expensive dispatch logic remains on a critical path. Large event terms cost copying and cross-node serialization. More shards and Registry partitions add overhead as well as concurrency. Measure representative subscriber counts, broadcast rates, and churn; do not maximize knobs by default. Keep validation and Telemetry bounded and move subscriber work out of dispatch.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a diagnostic command listing the current PID’s logical topics.
  • Add explicit unsubscribe and resubscribe output.
  • Add a mode-selection explanation to each CLI line.

8.2 Intermediate Extensions

  • Add subscription metadata and a tiny custom dispatcher in an isolated branch of the lab.
  • Compare local_broadcast with direct_broadcast to the current node and explain why official docs recommend local delivery for the current node.
  • Add a property test ensuring the wrapper maps each delivery mode to exactly one PubSub operation.

8.3 Advanced Extensions

  • Run the migration matrix against two actual nodes before Project 9.
  • Add a Redis adapter comparison and document which operational dependency moves outside Distributed Erlang.
  • Benchmark pool_size and registry_size independently with representative membership shapes.

9. Real-World Connections

9.1 Industry Applications

  • LiveView refresh notifications: Domain events enter LiveView process mailboxes and trigger authorized reloads.
  • Phoenix Channels: PubSub distributes messages to channel processes and supports specialized dispatch metadata.
  • Presence: Presence diffs rely on PubSub-style propagation while representing ephemeral membership.
  • Cache invalidation: Scope choice determines whether one node or the cluster becomes consistent.
  • Operational alerts: Tenant-scoped events fan out to several connected interfaces.
  • Rolling deployment: Pool compatibility is an operational correctness concern, not only configuration hygiene.

9.3 Interview Relevance

You can explain Phoenix.PubSub without presenting it as magic or Kafka. You understand local Registry fanout, adapter exchange, echo control, process lifecycle, duplicate subscription behavior, and the operational steps required to change pool size safely.

10. Resources

10.1 Essential Reading

10.2 Video and Talk Topics

  • PhoenixConf and ElixirConf talks on PubSub internals.
  • Talks on rolling BEAM releases and heterogeneous clusters.
  • LiveView lifecycle talks that distinguish disconnected and connected mount.
  • Production incidents involving duplicate subscriptions or growing subscriber mailboxes.

10.3 Tools and Documentation

  • IEx attached sessions: Observe several subscribers on one node.
  • ExUnit: Assert exact mailbox copy counts and process lifecycle.
  • Telemetry: Instrument the application wrapper.
  • Mix release configuration: Model staged pool-size settings.
  • Distributed Erlang tools: Deferred to the multi-node project, but useful for the optional extension.
  • Previous: Project 3 — Contract-Aware Domain Event Hub supplies topics, envelopes, and subscriber recovery policy.
  • Next: Project 5 — LiveView Operations Board turns this notification hub into visible browser updates.
  • Project 10 — Three-Node PubSub Laboratory applies the adapter and migration model under real node failure.

11. Self-Assessment Checklist

11.1 Understanding

  • I can list what Phoenix.PubSub provides and what it does not.
  • I can select among the four local/cluster and include/exclude modes.
  • I can explain duplicate subscription behavior.
  • I can explain why restart requires resubscription and reload.
  • I can distinguish pool_size, registry_size, and broadcast_pool_size.
  • I can explain the safe two-stage PG2 migration.
  • I can explain why routing ok is not processing acknowledgement.

11.2 Implementation

  • Demo.PubSub is supervised with explicit configuration.
  • Application code uses the NotificationHub wrapper.
  • Topics are authorized and derived.
  • Envelopes are validated.
  • Logical subscription is idempotent.
  • All four delivery-mode semantics have tests.
  • Restart proves no replay.
  • Routing and handler evidence are separate.
  • Migration configuration has automated validation.

11.3 Growth

  • I can review a direct PubSub call and identify missing contract concerns.
  • I can explain when local-only delivery is correct.
  • I can diagnose duplicate message copies.
  • I can present the pool migration runbook to an operations team.
  • I documented one responsibility that belongs outside PubSub.

12. Submission / Completion Criteria

Minimum Viable Completion

  • A supervised Phoenix.PubSub instance has two subscribers.
  • Ordinary broadcast reaches both.
  • Sender-excluding broadcast omits the supplied subscribed PID.

Full Completion

  • Domain wrapper owns authorization, topics, envelopes, mode selection, and Telemetry.
  • Repeated logical subscription produces one copy.
  • Cluster, local, and both sender-excluding modes are demonstrated.
  • Subscriber restart proves no offline replay.
  • Output distinguishes routing completion from handler receipt.
  • pool_size and registry_size are explicit.
  • Safe pool-size migration has a tested initial, transition, and final runbook.

Excellence

  • A mixed-release fixture validates the migration matrix.
  • Representative profiles compare broadcast and Registry sizing independently.
  • Documentation maps every application use case to an explicit delivery mode and recovery strategy.

Return to the Pub/Sub in Elixir mastery guide.

See the expanded project index.