Project 10: Three-Node PubSub Laboratory

Build a repeatable three-node laboratory that makes topology, process-group membership, cross-node fanout, partitions, reconnection, and safe PubSub pool migration directly observable.

Quick Reference

Attribute Value
Difficulty Level 4 — Expert
Time Estimate 20–30 hours
Language Elixir (alternatives: Erlang, Gleam)
Prerequisites OTP supervision, distributed Erlang basics, Phoenix.PubSub, node monitoring, revision-based recovery
Key Topics Full-mesh topology, :pg, Phoenix.PubSub.PG2, partitions, missed transient events, pool-size migration

1. Learning Objectives

By completing this project, you will be able to:

  1. Separate node discovery, node connectivity, process-group membership, and application message delivery as distinct layers.
  2. Prove a cluster topology from every node rather than inferring it from one successful connection.
  3. Explain :pg strong eventual consistency, temporary membership divergence, and non-transitive views.
  4. Trace how Phoenix.PubSub.PG2 performs local subscription and cross-node forwarding.
  5. Predict and observe component-local delivery during a network partition.
  6. Demonstrate that healed membership does not replay transient messages missed during disconnection.
  7. Repair application state using an authoritative revision rather than inventing replay semantics for PubSub.
  8. Perform a safe pool_size migration with a compatibility stage and continuous sequence probes.
  9. Capture node security, cleanup, and failure evidence in a reproducible lab report.

2. Theoretical Foundation

2.1 Core Concepts

Distribution has layers

A distributed Elixir application can fail even when every component works individually because several independent layers are often collapsed into the word “cluster.” Discovery produces candidate node names or addresses. Distributed Erlang establishes authenticated connections. Topology determines which nodes are directly connected. :pg propagates process-group membership over those direct connections. Phoenix.PubSub uses adapter processes and those membership views to forward broadcasts. Finally, the application interprets delivered messages and repairs missed state.

No layer automatically supplies the layer below it. Phoenix.PubSub does not discover nodes. A discovery library does not guarantee all expected distribution connections remain healthy. A full Node.list/0 snapshot does not prove every PubSub shard has converged. Converged membership does not replay a message missed during a partition.

:pg process groups

OTP’s :pg module implements distributed named process groups. Processes join locally, and membership changes propagate across directly connected nodes. Official OTP documentation describes the model as strong eventual consistency: replicas may temporarily observe updates in different orders, but connected replicas converge after communication stabilizes.

Membership is non-transitive. If alpha and gamma are both connected to beta but not to each other, beta may see both sets of members while alpha and gamma do not see one another’s group membership. A hub-and-spoke drawing is therefore not equivalent to a full mesh for group propagation. Group membership is also ephemeral. If a local scope process restarts, its local membership is not preserved automatically; supervised adapter processes must rejoin.

Phoenix.PubSub.PG2 forwarding

Phoenix.PubSub maintains local subscriber registrations and uses an adapter for cluster forwarding. With the PG2 adapter, pool shards distribute forwarding work. A broadcast enters a local shard, is sent to corresponding remote adapter members visible through the process group, and each remote node dispatches locally. The application subscriber does not receive a durability guarantee or broker acknowledgement. A successful call means the broadcast operation was accepted according to the current adapter view, not that every subscriber processed it.

Pool size matters because the shard used to broadcast must be understood by old and new nodes during a rolling deployment. Official Phoenix.PubSub documentation provides a compatibility mechanism: deploy the new pool_size while temporarily retaining the old broadcast_pool_size, then deploy again with the final broadcast size. Increasing from 1 to 2 without the compatibility stage can create migration-induced gaps even in an otherwise healthy cluster.

Partitions and transient loss

When gamma is isolated from alpha and beta, each connected component continues local work. Alpha and beta can exchange PubSub messages; gamma can deliver only within its component. This is availability, not global consistency. Reconnecting restores distribution and allows membership to converge, but Phoenix.PubSub has no historical log from which to replay revision 42.

Applications therefore distinguish notifications from facts. A notification says, “revision 42 may now exist.” Each subscriber tracks the last applied revision. A gap, reconnect, or node-up event triggers an authorized reload from the authoritative database or event log. The durable system decides what catch-up means; PubSub merely lowers latency while connectivity is healthy.

Topology evidence and security

Distribution is privileged remote-code infrastructure, not a generic public transport. A shared cookie authenticates nodes but is not a complete production security design. Network isolation, TLS distribution where appropriate, certificate and cookie rotation, restricted node names and ports, least-privilege runtime identities, and explicit discovery boundaries are part of the system.

The lab must be disposable. It should use isolated names, deterministic ports or containers, training credentials, and a teardown check that finds no surviving nodes, listeners, proxy rules, or partition hooks.

2.2 Why This Matters

Cluster tests often prove only the happy path: start two nodes on one laptop, publish once, and celebrate when both print a line. That test does not expose partial topology, temporary membership disagreement, node churn, partition behavior, or rolling configuration compatibility. Production incidents happen in those omitted states.

A three-node lab is the smallest topology that reveals non-transitivity. With two nodes, connected and disconnected are the only shapes. With three, alpha and gamma can both see beta while remaining disconnected from each other. Each node can then hold a different but locally reasonable view. This makes abstract distributed-systems language concrete.

The sequence probe turns visibility into evidence. Publishing monotonically numbered revisions before, during, and after a partition shows exactly which component observed each transient message. Comparing that stream with an authoritative revision demonstrates recovery without pretending the missing message appears later.

2.3 Historical Context / Background

Distributed Erlang was designed to make sending to a remote PID syntactically similar to local messaging. That transparency is productive, but latency, connectivity, and failure semantics remain distributed. Earlier Phoenix.PubSub versions supported multiple adapters and Erlang process-group mechanisms; current PG2 naming is historical while its implementation is based on modern :pg/:pg2 compatibility according to the package documentation.

The operational lesson has remained stable: an API may be location-transparent, but failures are not. Modern deployment platforms automate DNS discovery and rolling releases, yet the application must still validate topology, mixed-version compatibility, and state repair.

2.4 Common Misconceptions

  • “Phoenix.PubSub discovers application nodes.” Discovery and connection happen elsewhere.
  • “If alpha and gamma both connect to beta, they see each other through beta.” :pg membership is non-transitive.
  • “Strong eventual consistency means every read is current.” Views may temporarily diverge before convergence.
  • “A healed partition replays missed broadcasts.” PubSub is transient; only current membership converges.
  • “Node.list/0 on alpha proves the full topology.” Every node needs a symmetric connectivity snapshot.
  • “A shared cookie makes distribution safe on an untrusted network.” It is only one credential in a larger security boundary.
  • “Changing pool_size is a normal one-step rolling config update.” Safe migration requires the documented compatibility phase.

3. Project Specification

3.1 What You Will Build

Build a local laboratory with nodes alpha@lab, beta@lab, and gamma@lab. Each node runs the same supervised application, one Phoenix.PubSub instance, a probe subscriber, a topology observer, a revision tracker, and a small status endpoint or terminal panel.

The lab controller starts and stops nodes, requests connectivity snapshots from each one, waits for adapter group convergence, publishes numbered probes, injects a deterministic partition, heals it, and validates application catch-up.

The authoritative revision store may be a small disposable PostgreSQL table or one deliberately centralized training service. It must be clearly separate from PubSub. When alpha commits revision 42 during gamma’s isolation, gamma misses the transient probe. After reconnection, gamma queries the authority and updates from 41 to 42.

The final drill changes PubSub from pool size 1 to 2. A continuous publisher emits sequence probes throughout both deployment stages. The first stage runs the new pool while broadcasting compatibly with the old size; the second removes compatibility after all nodes are upgraded.

3.2 Functional Requirements

  1. Three named nodes: Start three isolated BEAM nodes with predictable logical identities.
  2. Topology snapshots: Collect each node’s direct connections and render a matrix.
  3. Membership snapshots: Query every PubSub shard/process group from every node.
  4. Sequence probes: Publish a monotonic sequence with publisher node, event ID, and authoritative revision.
  5. Observation ledger: Record which node observed which sequence and when.
  6. Deterministic partition: Isolate gamma from alpha/beta without crashing the application.
  7. Component-local assertions: Prove alpha/beta share revision 42 while gamma misses it.
  8. Heal and converge: Restore connectivity and wait for node and membership convergence separately.
  9. Authoritative repair: Detect gamma’s revision gap and reload revision 42.
  10. Pool migration: Execute the documented 1→2 compatibility rollout under continuous publication.
  11. Cleanup: Remove network rules and stop all nodes even when a drill fails.
  12. Security report: Document credentials, network boundary, naming, and why the setup is training-only.

3.3 Non-Functional Requirements

  • Reproducibility: A fresh checkout can run the full lab with deterministic node names and steps.
  • Isolation: The lab cannot connect to unrelated local or production clusters.
  • Observability: Every delivery includes sequence, node, component, topology generation, and timestamps.
  • Boundedness: Observation buffers, log retention, and probe rates are explicitly limited.
  • Convergence budget: Full-mesh and group convergence have measured deadlines rather than unbounded sleeps.
  • Recovery: Missed transient messages are labeled, not hidden; durable revision catch-up is independently verified.
  • Migration safety: Continuous probes show zero pool-migration-induced gaps while the network remains healthy.
  • Cleanup safety: Teardown is idempotent and reports residual nodes or partition controls.

3.4 Example Usage / Output

$ ./lab cluster status
node matrix:
  alpha -> [beta,gamma]
  beta  -> [alpha,gamma]
  gamma -> [alpha,beta]
topology=full_mesh PASS
pubsub shard=0 members alpha=3 beta=3 gamma=3 converged=true

publish sequence=41 revision=41 from=alpha
observed alpha=12ms beta=18ms gamma=21ms PASS

partition applied component_A=[alpha,beta] component_B=[gamma]
publish sequence=42 revision=42 from=alpha
observed=[alpha,beta] gamma=MISS_EXPECTED

partition healed
node_views_converged=true pg_views_converged=true
gamma last_observed=41 authoritative=42 gap=1
gamma reload revision=42 source=authoritative_store PASS

pool_migration stage=compat pool_size=2 broadcast_pool_size=1 probes=500 gaps=0
pool_migration stage=final  pool_size=2 broadcast_pool_size=2 probes=500 gaps=0
cleanup residual_nodes=0 residual_partition_rules=0
lab result=PASS

3.5 Real World Outcome

Run three terminals or open the lab’s topology page. The normal-state view shows a 3×3 adjacency matrix with symmetric direct connections, one PubSub shard containing one forwarding member per node, and three subscribers on the selected topic. Publish revision 41 and watch all three node timelines add the same event with different arrival latencies.

Activate “Partition Gamma.” The topology view splits into two colored components. Alpha and beta show node-down for gamma; gamma shows isolation from both. Publishing revision 42 produces green delivery markers only on alpha and beta and an explicit gray MISS_EXPECTED marker on gamma. The system must never paint gamma green merely because membership later heals.

Heal the partition. Node and process-group views converge. Gamma’s revision card remains 41 until its recovery path compares against the authoritative revision. It then reloads 42 and labels the transition recovered_from_authority, not replayed_by_pubsub.

Finally, start a continuous probe and perform the pool-size migration. The dashboard overlays rollout stage, node version, configured pool size, broadcast pool size, and any gap. Completion evidence shows no gaps during healthy-network migration, while the earlier partition gap remains documented as expected transient loss.

4. Solution Architecture

4.1 High-Level Design

                         LAB CONTROLLER
         start | inspect | partition | heal | migrate | teardown
                               │
          ┌────────────────────┼────────────────────┐
          v                    v                    v
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│ alpha@lab        │  │ beta@lab         │  │ gamma@lab        │
│ PubSub shards    │◀▶│ PubSub shards    │◀▶│ PubSub shards    │
│ probe subscriber │  │ probe subscriber │  │ probe subscriber │
│ topology observer│◀▶│ topology observer│◀▶│ topology observer│
│ revision tracker │  │ revision tracker │  │ revision tracker │
└────────┬─────────┘  └────────┬─────────┘  └────────┬─────────┘
         │                     │                     │
         └─────────────────────┼─────────────────────┘
                               v
                    ┌───────────────────────┐
                    │ Authoritative revision│
                    │ store / API           │
                    └───────────────────────┘

Partition drill:
  {alpha,beta}  X  {gamma}

After heal:
  connectivity converges -> :pg membership converges
  missed messages do not return -> revision tracker reloads facts

4.2 Key Components

Component Responsibility Key Decisions
Node launcher Start isolated named runtimes Stable lab names and per-run credential
Topology observer Report direct peers and node up/down Snapshot from every node
Membership observer Report adapter shard group members Poll with deadline; retain divergent views
Probe publisher Emit monotonic sequence/revision events Fixed rate and bounded run length
Probe subscriber Record observed events Bounded ETS/ring buffer plus export
Revision tracker Detect gaps and reload facts PubSub never treated as replay log
Partition controller Apply and remove deterministic isolation Idempotent cleanup in failure path
Migration orchestrator Roll nodes through compatibility/final stages Continuous probes during every restart
Evidence aggregator Produce matrix, timeline, and verdicts Correlate monotonic and wall-clock time

4.3 Data Structures

Probe
  run_id                 isolates repeated lab runs
  sequence               monotonic integer
  event_id               immutable logical ID
  revision               authoritative fact revision
  publisher_node         alpha/beta/gamma
  published_monotonic    latency baseline

Observation
  run_id + sequence + observer_node   unique evidence key
  received_monotonic
  topology_generation
  pool_stage

NodeView
  observer_node
  captured_at
  direct_peers
  connected_component

GroupView
  observer_node
  scope
  shard
  visible_members
  captured_at

RevisionCursor
  tenant_or_stream
  node
  last_observed_revision
  last_authoritative_revision
  recovery_reason

4.4 Algorithm Overview

Algorithm: Prove full mesh

FOR each expected node N:
  request direct peer snapshot from N
  assert peers(N) = expected_nodes - N
  assert every edge is symmetric
THEN request group view from every N and shard
WAIT until expected membership appears everywhere or deadline expires
PRESERVE intermediate divergent snapshots as evidence

Algorithm: Partition and recover

publish revision 41 and await all observations
isolate gamma from alpha and beta
wait for topology to show components {alpha,beta} and {gamma}
commit authoritative revision 42
publish transient probe 42 from alpha
assert alpha and beta observe; gamma does not
heal partition
wait separately for node and group convergence
compare gamma cursor 41 with authority 42
reload current fact or durable events
set gamma cursor 42 with recovery_reason=authoritative_reload

Algorithm: Safe pool migration 1→2

start continuous sequence probes
stage A: roll every node to pool_size=2, broadcast_pool_size=1
assert each node healthy and probes remain gap-free
stage B: roll every node to pool_size=2, broadcast_pool_size=2/final
assert each node healthy and probes remain gap-free
stop probes and compare published/observed ranges per node

Complexity analysis

  • Full topology validation is O(n²); intentional for a small lab and necessary for symmetric evidence.
  • Membership evidence is O(n × s × m), for nodes, shards, and visible members.
  • Probe storage is O(n × p) for p bounded probes observed across n nodes.
  • The lab uses three nodes; production observability should aggregate rather than render a quadratic matrix at very large scale.

5. Implementation Guide

5.1 Development Environment Setup

Use containers or isolated local loopback addresses. Do not reuse a production cookie or open distribution ports to an untrusted network.

$ elixir --version
Elixir 1.20.x (compiled with Erlang/OTP 29)

$ ./lab doctor
loopback aliases=available
distribution ports=closed_before_start
expected nodes=[alpha@lab,beta@lab,gamma@lab]
training credential=generated_for_run
partition backend=container_network
doctor result=PASS

The controller may wrap mix release, containers, or three explicit iex --name commands. Commands are setup interfaces; do not hide the actual node names, cookie source, port restrictions, or cleanup behavior.

5.2 Project Structure

three_node_pubsub_lab/
├── lib/
│   ├── lab/topology_observer.ex
│   ├── lab/membership_observer.ex
│   ├── lab/probe_publisher.ex
│   ├── lab/probe_subscriber.ex
│   ├── lab/revision_tracker.ex
│   └── lab/evidence_reporter.ex
├── config/
│   ├── runtime.exs              # node/pool stage inputs
│   └── lab_topology.example     # non-secret outline
├── scripts/
│   ├── lab                     # lifecycle interface
│   └── partition_backends/     # platform-specific isolation
├── test/
│   ├── distributed/
│   └── migration/
├── docs/
│   ├── topology-contract.md
│   ├── security-boundary.md
│   └── failure-predictions.md
└── evidence/

5.3 The Core Question You’re Answering

“What does cluster Pub/Sub guarantee when node views diverge, and how does the application recover facts after connectivity returns?”

Your answer must distinguish current connected component, current membership view, transient message delivery, and authoritative application state. “The cluster heals” is incomplete unless it states what converges and what does not replay.

5.4 Concepts You Must Understand First

  1. Distributed Erlang nodes
    • What makes a node name resolvable and a connection authenticated?
    • Which failures appear as nodedown and when?
    • Book Reference: “Designing for Scalability with Erlang/OTP” — distributed architecture chapters.
  2. Topology and discovery
    • Does discovery return candidates or guarantee live full-mesh connectivity?
    • How will every node prove its direct peers?
    • Reference: Erlang distributed-system documentation and chosen discovery adapter docs.
  3. :pg membership
    • What does strong eventual consistency permit during convergence?
    • Why is membership non-transitive?
    • Reference: OTP 29 pg documentation.
  4. Phoenix.PubSub adapter behavior
    • What is local subscription versus cluster forwarding?
    • How do pool shards and remote members relate?
    • Reference: Phoenix.PubSub and Phoenix.PubSub.PG2 documentation.
  5. Revision-based recovery
    • Which system stores authoritative facts?
    • How does a subscriber detect and repair a gap?
    • Book Reference: “Designing Data-Intensive Applications, 2nd Edition” — distributed-system trouble.

5.5 Questions to Guide Your Design

  1. Proof quality
    • Are topology and membership sampled from all three nodes?
    • Are intermediate divergent views preserved instead of waiting silently?
  2. Partition mechanism
    • Can isolation be applied and removed deterministically?
    • What cleanup runs after an assertion failure or interrupted shell?
  3. Clock and ordering
    • Which results use monotonic time, and which display wall-clock time?
    • Does sequence, not timestamp, determine missing messages?
  4. Recovery
    • Is the source a current-state reload or durable event replay?
    • How are stale and missing revisions labeled differently?
  5. Migration
    • Which config is deployed in stage A and stage B?
    • How does a probe distinguish network loss from migration loss?

5.6 Thinking Exercise

Draw both topologies:

FULL MESH                         HUB AND SPOKE
 alpha ───── beta                 alpha ───── beta ───── gamma
   │  ╲       │                  alpha X gamma direct connection
   │    ╲     │
   └──── gamma

For each node, list Node.list and possible :pg members. Then partition {alpha,beta} from {gamma} and predict:

  • node monitor signals;
  • adapter membership visible from each component;
  • delivery of sequences 41, 42, and 43;
  • what happens to a subscriber that joins during isolation;
  • what membership converges after heal;
  • why sequence 42 remains absent on gamma;
  • which authoritative operation repairs gamma.

Record predictions before running the lab and compare every mismatch.

5.7 The Interview Questions They’ll Ask

  1. “Does Phoenix.PubSub discover or connect Erlang nodes?”
  2. “What consistency model does :pg provide?”
  3. “Are :pg membership views transitive through a hub?”
  4. “What happens to PubSub broadcasts during a partition?”
  5. “How does an application repair state after the partition heals?”
  6. “How do you safely increase Phoenix.PubSub pool size during a rolling deploy?”

5.8 Hints in Layers

Hint 1: Prove the transport before the abstraction

Render the direct-peer matrix before inspecting PubSub. A wrong matrix invalidates later assumptions.

Hint 2: Number every probe

Use one bounded stream of monotonically increasing sequence numbers. Do not infer loss from wall-clock gaps.

Hint 3: Keep convergence and recovery separate

Produce two verdicts after heal:

membership_converged=true
application_revision_recovered=true source=authoritative_reload

Hint 4: Migrate while noisy

A quiet rolling deployment proves only startup. Continuous probes reveal compatibility gaps.

5.9 Books That Will Help

Topic Book Chapter / Section
Distributed OTP “Designing for Scalability with Erlang/OTP” Distribution and distributed architectures
Network partitions “Designing Data-Intensive Applications, 2nd Edition” Distributed-System Trouble
Rollouts and stability “Release It!, 2nd Edition” Stability Patterns
Architecture evidence “Software Architecture in Practice, 4th Edition” Quality Attribute Scenarios

5.10 Implementation Phases

Phase 1: Topology and Membership Observatory (8–12 hours)

Goals

  • Start three isolated nodes.
  • Prove full mesh and adapter membership from every node.

Tasks

  1. Create lifecycle and teardown interfaces.
  2. Add node monitoring and symmetric peer snapshots.
  3. Add per-shard group snapshots with deadlines.
  4. Add bounded sequence publisher/subscribers.
  5. Export a normal-state topology report.

Checkpoint: Revision 41 reaches all nodes, every node reports two peers, every shard view converges, and teardown leaves no residual runtime.

Phase 2: Partition and Authoritative Recovery (8–14 hours)

Goals

  • Observe component-local fanout.
  • Repair missed facts after heal.

Tasks

  1. Implement deterministic gamma isolation and cleanup.
  2. Capture intermediate node/group views.
  3. Commit and publish revision 42 during isolation.
  4. Heal connectivity and wait for convergence.
  5. Detect gamma’s gap and reload authoritative state.

Checkpoint: Gamma explicitly misses transient 42, later converges membership, and recovers revision 42 only through the authority.

Phase 3: Safe Pool Migration and Certification (8–14 hours)

Goals

  • Exercise mixed configuration safely.
  • Produce repeatable migration evidence.

Tasks

  1. Define stage-A and stage-B runtime configurations.
  2. Start a fixed-rate sequence probe.
  3. Roll alpha, beta, and gamma one at a time through compatibility.
  4. Roll them again to final configuration.
  5. Attribute gaps, validate cleanup, and publish the lab report.

Checkpoint: At least 1,000 healthy-network probes across the two stages show zero migration gap, and the report still preserves the expected partition loss separately.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Node environment Host processes; containers Containers or isolated loopback with explicit teardown Deterministic partition and port boundary
Topology Full mesh; hub/spoke Full mesh for reference lab; demonstrate hub/spoke failure separately Matches adapter membership expectations
Observation storage Unbounded process list; bounded ETS/ring buffer Bounded store plus export Avoids evidence causing mailbox/memory failure
Catch-up Assume replay; query authority Query/replay from durable authority PubSub has no replay log
Timing Wall clock only; sequence + monotonic time Sequence + monotonic time Correct gap and latency evidence
Pool rollout One stage; compatibility then final Two-stage documented migration Supports mixed old/new nodes

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Local unit Validate matrices, gaps, and verdicts Symmetric edges, missing sequence ranges
Multi-node integration Verify real distribution Full mesh, shard membership, cross-node probe
Partition Verify component-local behavior Gamma isolation and heal
Recovery Verify durable catch-up Cursor 41 to authority 42
Churn Verify process/node lifecycle Subscriber crash, adapter restart, node restart
Migration Verify mixed pool configuration Stage A and B under continuous probes
Security/cleanup Verify isolation Wrong credential denial, no residual ports/rules

6.2 Critical Test Cases

  1. Full mesh: Every node sees the other two directly and group views converge.
  2. Hub/spoke counterexample: Alpha and gamma both see beta but not each other; non-transitive membership is exposed.
  3. Normal fanout: Sequence 41 appears once on all expected subscribers.
  4. Partition delivery: Sequence 42 remains inside {alpha,beta}; gamma’s miss is expected.
  5. Local gamma publish: During isolation, gamma’s publish remains in its component and does not reach alpha/beta.
  6. Heal convergence: Node and group views converge within measured budgets.
  7. No replay: Sequence 42 does not spontaneously arrive on gamma after heal.
  8. Revision repair: Gamma reloads authoritative revision 42 and records the source.
  9. Adapter restart: Local subscriptions/members recover through supervision; temporary divergence is measured.
  10. Pool migration: Continuous healthy-network probes show no stage-induced gaps.
  11. Wrong credential: An unauthorized fourth node cannot join.
  12. Cleanup: Interrupted drill still removes partition controls and terminates nodes.

6.3 Test Data

run_id: lab-2026-07-15-01
nodes: [alpha@lab, beta@lab, gamma@lab]
topic: lab:tenant42:orders
initial_pool_size: 1
target_pool_size: 2

probes:
  41 -> before partition, expected [alpha,beta,gamma]
  42 -> during partition, publisher alpha, expected [alpha,beta]
  43 -> during partition, publisher gamma, expected [gamma]
  44 -> after heal, expected [alpha,beta,gamma]

authoritative revisions:
  before partition: 41
  during partition: 42
  gamma recovery target: 42

migration:
  stage A: pool_size=2 broadcast_pool_size=1 probes=1000
  stage B: pool_size=2 broadcast_pool_size=2 probes=1000

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Assuming discovery equals topology Candidates exist but probes miss nodes Capture direct-peer matrix from every node
Checking only beta Hub appears healthy while edges disagree Require symmetric all-node snapshots
Waiting with arbitrary sleep Flaky convergence test Poll explicit conditions with deadline and evidence
Expecting replay after heal Gamma never sees 42 Reload/replay from durable authority
Hiding expected loss Dashboard reports false success Label expected transient misses and repair separately
One-step pool increase Gaps during mixed rollout Use broadcast_pool_size compatibility stage
Shared production cookie Lab can join privileged cluster Generate isolated per-run credential and network
Failed teardown Later runs start in a partitioned state Make cleanup idempotent and verify residual state

7.2 Debugging Strategies

  • Work bottom-up: node names/resolution → direct connections → group views → adapter forwarding → subscriber receipt.
  • Inspect every node: a global-looking view from one node may be uniquely complete.
  • Preserve divergence: log membership snapshots with observer identity and time.
  • Compare components: classify each delivery against the component graph at publication time.
  • Use sequence ranges: calculate missing sets per node rather than reading interleaved terminal logs.
  • Separate restart from reconnect: a restarted process has a new PID and may need to rejoin even if the node never disconnected.
  • Run cleanup doctor: before a new drill, confirm no old nodes, ports, or network filters remain.

7.3 Performance Traps

  • Excessive topology polling adds distribution traffic and can distort convergence measurements.
  • Broadcasting probes too quickly can make subscriber mailboxes the bottleneck instead of the network fault under study.
  • Storing every observation forever creates quadratic evidence growth.
  • Large PubSub pool sizes add processes and group membership traffic; bigger is not automatically faster.
  • Blocking report generation in subscriber processes delays receipt and produces misleading gaps.
  • DNS or discovery retry storms can hide the original partition under control-plane load.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a live adjacency matrix and node-event timeline.
  • Add a fourth unauthorized node and prove join rejection.
  • Compare a subscriber restart with a full node restart.

8.2 Intermediate Extensions

  • Replace static seeds with one discovery mechanism and keep topology proof unchanged.
  • Add asymmetric packet loss or latency rather than complete isolation.
  • Track Phoenix.Tracker/Presence convergence alongside PubSub delivery.
  • Compare state reload with durable event-log replay.

8.3 Advanced Extensions

  • Run nodes across three local networks and route only explicit distribution ports.
  • Configure TLS distribution and automate certificate rotation in the lab.
  • Test a rolling OTP/Elixir/Phoenix.PubSub version upgrade under probes.
  • Build a Jepsen-inspired history checker for published, observed, and recovered revisions.

9. Real-World Connections

9.1 Industry Applications

  • Realtime SaaS: Multiple Phoenix nodes fan out tenant updates while clients reconnect across deploys.
  • Fleet and IoT control: Regional partitions require local operation plus later authoritative catch-up.
  • Collaboration tools: Presence and editing notifications tolerate transient loss but documents require durable state.
  • Operations dashboards: Cluster health must distinguish stale views from durable business loss.
  • Rolling platform upgrades: Mixed configuration must be explicitly compatible under live traffic.

9.3 Interview Relevance

This lab equips you to reject vague claims such as “Erlang distribution is transparent” or “PubSub works across the cluster.” You can draw the exact topology, state the membership consistency model, predict delivery per connected component, explain non-replay after heal, and describe a safe rolling configuration change with evidence.

10. Resources

10.1 Essential Reading

  • “Designing for Scalability with Erlang/OTP” by Francesco Cesarini and Steve Vinoski — distribution and scalable architecture chapters.
  • “Designing Data-Intensive Applications, 2nd Edition” by Martin Kleppmann et al. — distributed failure, consistency, and recovery.
  • “Release It!, 2nd Edition” by Michael Nygard — stability and deployment failure patterns.
  • “Software Architecture in Practice, 4th Edition” by Bass et al. — measurable quality-attribute scenarios.

10.2 Video Resources

  • Official Erlang/OTP conference talks on distribution, partitions, and process groups.
  • Phoenix conference talks that instrument multi-node PubSub and Presence behavior.

10.3 Tools & Documentation

  • Project 2: Registry Topic Bus introduces local membership without distribution.
  • Project 6: Distributed Presence Workspace introduces eventually consistent presence semantics.
  • Project 9: Durable Notification Outbox supplies the authoritative revision recovery path.
  • Project 11: Tenant-Safe Broker Edge Bridge adds durable cross-service delivery beyond a BEAM cluster.
  • Project 12: Multi-Tenant Real-Time Event Platform turns this lab into a production-style chaos certification.

11. Self-Assessment Checklist

11.1 Understanding

  • I can separate discovery, connectivity, membership, forwarding, and application recovery.
  • I can explain strong eventual consistency without claiming immediate agreement.
  • I can draw a non-transitive membership example with three nodes.
  • I can explain why membership convergence does not replay sequence 42.
  • I can describe the two stages of a safe pool-size increase.
  • I can defend the distribution security boundary.

11.2 Implementation

  • Three nodes start with isolated names and credentials.
  • Every node reports the expected direct peers and shard members.
  • Sequence 41 reaches all nodes before partition.
  • Sequences 42 and 43 remain in their expected components.
  • Heal converges node and group views within measured budgets.
  • Gamma recovers revision 42 from the authority.
  • Continuous migration probes show zero healthy-network gaps.
  • Teardown reports no residual nodes or partition controls.

11.3 Growth

  • I documented at least one prediction that the lab disproved.
  • I can diagnose a hub-and-spoke topology from per-node evidence.
  • I can explain when a durable broker or event log is required instead of PubSub.
  • I can present the partition timeline and migration evidence in an interview.

12. Submission / Completion Criteria

Minimum Viable Completion

  • Three nodes form and prove a symmetric full mesh.
  • A numbered probe reaches all nodes in normal state.
  • Gamma isolation produces the predicted component-local delivery.
  • Healing restores topology and membership visibility.
  • Gamma catches up from an authoritative revision source.

Full Completion

  • All minimum criteria plus captured intermediate membership divergence, bounded evidence, security documentation, and idempotent teardown.
  • The lab distinguishes node convergence, group convergence, transient non-replay, and durable state recovery.
  • A two-stage 1→2 pool migration runs under at least 1,000 continuous probes with zero migration-induced gaps.
  • The final report includes matrices, timelines, sequence sets, convergence durations, and residual-state checks.

Excellence (Going Above & Beyond)

  • The lab supports static and dynamic discovery while retaining the same evidence contract.
  • TLS distribution and credential rotation are automated and verified.
  • Asymmetric latency/loss tests produce explainable convergence and delivery histories.
  • A history checker automatically validates component-aware delivery and eventual revision recovery.

This guide was expanded from LEARN_ELIXIR_PUBSUB_DEEP_DIVE.md. For the complete expanded project index, see README.md.