Project 6: Distributed Presence Workspace

Build a collaborative incident room that represents users, devices, reconnect overlap, and uncertain failure honestly while Phoenix.Tracker converges across a cluster.

Quick Reference

Attribute	Value
Difficulty	Level 3 — Advanced
Time Estimate	16–22 hours
Language	Elixir (alternatives: TypeScript presence client, Erlang tracker)
Primary Tools	Phoenix.Presence, Phoenix.Tracker, Phoenix.PubSub, LiveView or Channels
Prerequisites	PubSub topics, process monitoring, distributed failure, client state/diff synchronization
Key Topics	One key/many metas, CRDT convergence, heartbeat failure detection, reconnect overlap, compact metadata
Observable Deliverable	Multi-device responder roster with uncertainty states and a repeatable two-node partition drill

1. Learning Objectives

By completing this project, you will be able to:

1. Distinguish PubSub subscription membership from replicated application presence.
2. Model one logical responder with multiple concurrent connection metadata records.
3. Explain Phoenix.Tracker’s heartbeat and CRDT-based eventual convergence without claiming instant global truth.
4. Apply an initial presence state followed by join/leave diffs using stable metadata references.
5. Represent hard failure and reconnect overlap honestly in the user interface.
6. Keep presence metadata small, ephemeral, and safe while batch-fetching durable profile detail.
7. Keep Presence handle_metas/4 bounded, and understand the stricter control-path constraints of raw Tracker handle_diff/2.
8. Prove that authorization, billing, audit, and incident participation history do not depend on presence.

2. Theoretical Foundation

2.1 Presence Is a Replicated Liveness Projection

Phoenix.Presence builds on Phoenix.Tracker, which uses heartbeat exchange and a conflict-free replicated data structure to merge presence changes across nodes. Each node owns a local view. There is no single process that can answer “who is online everywhere right now?” with instantaneous certainty. Local joins appear immediately, remote joins appear after replication, and hard failure requires time to distinguish from temporary delay or network partition.

That model is appropriate for a responder roster because temporary disagreement is acceptable and convergence is useful. It is not appropriate for access control, billing, compliance evidence, or an audit trail. Those concerns require a durable, authoritative store and explicit transactions.

2.2 One Key, Many Metas

A stable presence key typically represents a logical resource such as a user ID. Each connection—browser tab, mobile socket, desktop application—adds a metadata entry. Phoenix assigns each entry a phx_ref; an update may also include phx_ref_prev. Therefore “Douglas is online” is a derived statement: at least one valid meta remains for Douglas’s key.

user key: 42
│
├── meta ref=A  node_a  laptop tab  status=active
├── meta ref=B  node_b  phone       status=away
└── meta ref=C  node_a  second tab  status=active

close ref=C
│
└── key 42 remains online because A and B remain

Flattening this structure to a boolean too early produces false offline transitions and loses device-level diagnostics.

2.3 State Then Diff

A client begins with a complete presence state and then applies diffs. The state is grouped by logical key and contains a metas collection. A diff has joins and leaves, each carrying the same keyed/meta structure. Supported synchronization logic uses references to add or remove exact metas.

INITIAL STATE
  42 -> [A, B]
  77 -> [D]

DIFF
  joins: 42 -> [E]
  leaves: 42 -> [B]

RESULT
  42 -> [A, E]
  77 -> [D]

The join and leave may describe reconnect overlap, not a human deliberately leaving and re-entering. The UI should avoid turning every diff into a durable activity event.

2.4 Failure Detection and Uncertainty

Phoenix.Tracker periodically broadcasts deltas and heartbeats. In Phoenix.PubSub 2.2, documented defaults include a 1.5-second broadcast period, a heartbeat after ten silent periods, approximately 30-second down detection, and a 20-minute permanent-down period. These are operational defaults, not instant-failure guarantees.

On graceful shutdown, permdown_on_shutdown can tell replicas that a tracker is permanently gone. It is safe to set this option to true only when the Tracker is mounted at the root of the supervision tree under a :one_for_one strategy. With that prerequisite satisfied, true removes presence quickly but may generate mass leave/rejoin churn during a rolling deploy; false allows reconnect overlap but leaves temporary ghosts. The correct policy depends on product behavior and deployment topology.

2.5 Convergence Mental Model

                         Phoenix.PubSub replication
        ┌──────────────────────────┬──────────────────────────┐
        │                          │                          │
        v                          v                          v
┌────────────────┐        ┌────────────────┐        ┌────────────────┐
│ Tracker node A │        │ Tracker node B │        │ Tracker node C │
│ refs A,C,D     │        │ refs A,B,C,D   │        │ refs A,B,D     │
└───────┬────────┘        └───────┬────────┘        └───────┬────────┘
        │ temporary views         │                          │
        └──────────────────────────┴──────────────────────────┘
                          heartbeat + CRDT merge
                                      │
                                      v
                       converged refs A,D,E after reconnect

Convergence means replicas that can communicate and continue exchanging state eventually agree. It does not mean every intermediate read is identical, nor does it replay application PubSub events missed during a partition.

2.6 Metadata Discipline

Presence metadata is replicated and broadcast. Store only small ephemeral fields needed for liveness presentation: device class, coarse activity state, connection timestamp, and opaque identifiers. Durable name, avatar, team, incident role, or clearance should be fetched in one batch through fetch/2 or an application query.

fetch/2 runs on list and updates. It should avoid N+1 queries by collecting all keys, querying once, and enriching the grouped result. A Phoenix.Presence module may implement handle_metas/4 to consume presence metadata changes on the Elixir side; keep that optional callback bounded. At the lower-level Phoenix.Tracker boundary, handle_diff/2 executes in the tracker server context, so blocking I/O or an unhandled exception can delay or crash tracking. Optional work belongs in a supervised task or downstream durable pipeline.

2.7 Common Misconceptions

• “Presence says the user is logged in.” It says tracked processes are currently believed alive; authentication is separate.
• “A leave diff is an audit event.” It may represent timeout, partition, deploy, tab close, or tracker policy.
• “Each user has one presence.” One key can have many metas.
• “CRDT means every node is always consistent.” It means mergeable eventual convergence, not synchronous agreement.
• “Hard failure is immediate.” Failure detectors trade detection speed against false positives.

3. Project Specification

3.1 What You Will Build

Create a collaborative incident workspace with:

• A responder roster grouped by logical user.
• Expandable device details for every active meta.
• Coarse activity states such as active, away, and reconnecting.
• A visible UNCERTAIN state when a node or connection is suspected but convergence is pending.
• A cluster diagnostics panel showing local node, tracker replica, last diff, known metas, and convergence drill timing.
• Development controls for graceful tab close, abrupt process loss, node isolation, reconnect, and delayed profile fetching.

The durable incident membership table determines who may enter the room and preserves incident participation history. Presence only answers who appears connected now.

3.2 Functional Requirements

1. Authorization before tracking: Only authorized incident responders may track or list room presence.
2. Stable key: Use one opaque stable key per logical user.
3. Many metas: Preserve each connection with its own ref and device metadata.
4. Grouped roster: A user remains online until the last meta leaves.
5. Initial synchronization: Apply full state before diffs and tolerate diffs that race with initial loading.
6. Reconnect overlap: Show old and new refs without duplicating the logical user.
7. Failure uncertainty: Distinguish graceful leave from suspected hard loss.
8. Compact replication: Keep tracked metadata bounded and non-sensitive.
9. Batch enrichment: Resolve durable display details in one query per state/diff cycle.
10. Bounded callbacks: Presence handle_metas/4 performs no blocking external work; any optional raw Tracker exercise keeps handle_diff/2 equally bounded.
11. Diagnostics: Measure time from isolation to uncertainty, leave, reconnect, and convergence.
12. Non-authoritative behavior: No business transaction is triggered solely by presence join/leave.

3.3 Non-Functional Requirements

• Convergence target: Under the local three-node drill, all connected nodes report the same final refs within 10 seconds after connectivity is restored.
• Callback budget: p99 diff callback duration under 25 ms without profile-fetch latency included.
• Metadata budget: At most 512 encoded bytes per meta in the reference scenario.
• Privacy: No email, token, role secret, or incident narrative in replicated metadata.
• Resilience: A failed enrichment query may degrade display detail but must not stop tracker convergence.
• Observability: Node, topic class, and outcome are bounded dimensions; raw user IDs remain in traces/logs rather than metric labels.

3.4 Presence Data Outline

Presence key: opaque user identifier

Meta:
  phx_ref: tracker-generated unique reference
  phx_ref_prev: previous ref when updated, optional
  device_id: opaque connection/device identifier
  device_class: laptop | phone | tablet
  activity: active | away
  connected_at: UTC timestamp
  tracker_node: bounded node alias

Fetched profile detail:
  display_name, avatar_url, incident_role
  source: authorized database query

3.5 Real World Outcome

Run three application nodes, join incident room INC-104 from two laptop tabs and one phone as Douglas, then join from one laptop as Margarita. The roster groups the three Douglas metas and shows one logical user. Close the second laptop tab: Douglas remains online with two devices.

Next, isolate the node hosting the phone. The UI must not instantly assert that the phone definitely left. It marks that connection uncertain according to your failure policy. Reconnect the phone on a new node, producing a new ref while the old ref is still known. After Tracker converges, all three nodes show the same final refs.

The reference transcript is:

Room INC-104 — Responders
Douglas   ONLINE devices=3 [laptop:A, tab:C, phone:B]
Margarita ONLINE devices=1 [laptop:D]

leave ref=C -> Douglas remains ONLINE devices=2
partition node_b -> phone ref=B status=UNCERTAIN
reconnect phone ref=E -> temporary metas=[B,E]
convergence complete -> metas=[A,E] consistent_on=3/3 nodes
audit truth changed by presence diff=false

4. Solution Architecture

4.1 High-Level Design

┌─────────────────────┐  authorize  ┌──────────────────────┐
│ Channel / LiveView  │────────────>│ Incident membership │
└──────────┬──────────┘             │ durable authority    │
           │ track authorized user  └──────────────────────┘
           v
┌─────────────────────┐   deltas    ┌─────────────────────┐
│ Presence / Tracker A│<----------->│ Presence / Tracker B│
│ local refs A,C,D    │   PubSub    │ local ref B         │
└──────────┬──────────┘             └──────────┬──────────┘
           │ state + diff                      │ state + diff
           v                                   v
┌─────────────────────┐             ┌─────────────────────┐
│ Client sync reducer │             │ Client sync reducer │
└──────────┬──────────┘             └──────────┬──────────┘
           └──────────────────┬────────────────┘
                              v
                    grouped responder roster

4.2 Key Components

Component	Responsibility	Key Decision
Incident membership service	Durable authorization and participation history	Never derive membership from Presence
Presence module	Track/untrack/update and enrich list	Compact meta, batched fetch
Phoenix.Tracker	Replicate liveness deltas	Accept eventual consistency
Client sync reducer	Apply state then refs-based diffs	Never remove whole key for one meta leave
Roster projection	Group metas and derive online/uncertain	Display uncertainty explicitly
Failure drill controller	Isolate/reconnect nodes in development	Deterministic timestamps and refs
Telemetry handlers	Measure callbacks and convergence	No blocking handler work

4.3 Invariants

1. A logical user appears once in the roster regardless of meta count.
2. A user is not displayed offline while any accepted meta remains.
3. Presence cannot grant room access.
4. Every leave removes only the matching ref.
5. Durable audit rows are never created from unverified presence diffs.
6. Tracker callbacks remain bounded even when enrichment or audit dependencies fail.

4.4 Convergence Algorithm Overview

ON initial state:
  replace local presence map using supported state synchronization
  batch-fetch authorized display details
  render grouped roster

ON presence diff:
  apply joins and leaves by exact ref
  derive users affected by the diff
  batch-fetch missing durable display details
  update grouped roster

ON suspected replica loss:
  preserve uncertainty marker until Tracker reports the applicable change

ON reconnect:
  accept new ref; do not manually delete old ref
  allow Tracker merge/down policy to converge

5. Implementation Guide

5.1 Development Environment Setup

Verify versions and create three named development nodes through your preferred local release, terminal, or container workflow. The exact node-launch mechanism may vary, but the observable check should resemble:

$ epmd -names
epmd: up and running on port 4369 with data:
name room_a at port 9101
name room_b at port 9102
name room_c at port 9103

All nodes need the same cookie, compatible Phoenix.PubSub configuration, and direct connectivity. Record the topology you actually test.

5.2 Suggested Project Structure

presence_workspace/
├── lib/presence_workspace/
│   ├── incidents/membership.ex       # durable authority
│   ├── presence.ex                   # Phoenix.Presence behaviour
│   ├── presence_projection.ex        # grouped keys/metas
│   └── telemetry.ex
├── lib/presence_workspace_web/
│   ├── channels/incident_channel.ex
│   └── live/incident_room_live.ex
├── test/presence_workspace/
│   ├── presence_projection_test.exs
│   └── presence_cluster_test.exs
└── scripts/
    └── presence_partition_drill.md    # commands and expected observations

5.3 The Core Question You’re Answering

“How do I represent ‘online’ honestly when users have many connections and the cluster cannot know failure instantly?”

Your written answer should define online, offline, uncertain, reconnecting, and converged in terms of observable refs and tracker state—not human intent.

5.4 Concepts You Must Understand First

1. State and diff synchronization
   • Why must initial state precede ordinary diffs?
   • How does a phx_ref identify one meta?
   • Reference: Phoenix Presence server and JavaScript client docs.
2. CRDT convergence
   • What does conflict-free merge guarantee after communication resumes?
   • What does it not guarantee during a partition?
   • Book: “Designing Data-Intensive Applications, 2nd Edition,” replication and distributed systems chapters.
3. Failure detection
   • Why is silence ambiguous?
   • How do down and permanent-down windows affect UX?
4. Process ownership
   • What happens to tracked metas when their owning process exits?
   • How does abrupt node loss differ from orderly untracking?
5. Callback isolation
   • Why should Presence handle_metas/4 remain bounded, and why is blocking raw Tracker handle_diff/2 even more dangerous?
   • Which work belongs in a supervised task or durable pipeline?

5.5 Questions to Guide Your Design

1. Which stable identifier groups a user across devices without exposing private data?
2. Which metadata is truly ephemeral and necessary on every node?
3. How does the UI distinguish one meta leave from the final meta leave?
4. When does the UI show UNCERTAIN, and what clears it?
5. How will a failed batch profile query affect the roster?
6. Is the Tracker at the supervision-tree root under :one_for_one, and what does a rolling deploy look like under the safe permdown_on_shutdown policy available to that topology?

5.6 Thinking Exercise

Draw two nodes and one user with refs A and C on node A and ref B on node B. Trace:

1. Graceful close of C.
2. Network isolation of B.
3. Reconnect on node C as ref E.
4. Delayed observation of B’s down status.
5. Connectivity restoration and convergence.

For each step, list each node’s possible local view and the safest UI label. Then explain why none of those steps should charge a bill or create a compliance attendance record.

5.7 The Interview Questions They’ll Ask

1. “How is Phoenix Presence different from PubSub subscriptions?”
2. “Why can one presence key contain multiple metas?”
3. “What consistency model does Phoenix.Tracker provide?”
4. “Why can a disconnected user remain visible temporarily?”
5. “How do Presence handle_metas/4 and raw Tracker handle_diff/2 differ, and what work is dangerous inside either callback?”
6. “Why isn’t presence a valid authorization or audit source?”

5.8 Hints in Layers

Hint 1 — Render the raw shape first: Inspect grouped keys and metas before inventing an online boolean.

Hint 2 — Let refs drive mutation: Remove exact refs using supported sync behavior; never infer removal by device name.

Hint 3 — Degrade enrichment, not tracking: If profile data fails, show an opaque responder label and preserve liveness convergence.

Hint 4 — Separate graceful from abrupt drills: A tab close, process kill, node halt, and network partition have different timing.

Hint 5 — Measure convergence from a shared scenario ID: Compare final ref sets across nodes without using raw user IDs as metrics.

5.9 Books That Will Help

Topic	Book	Focus
Phoenix Presence	“Real-Time Phoenix” by Stephen Bussey	Presence and Channels chapters
Replication and convergence	“Designing Data-Intensive Applications, 2nd Edition”	Replication, distributed systems, and CRDT discussion
Operational failure	“Release It!, 2nd Edition”	Stability patterns and failure isolation
Integration boundaries	“Enterprise Integration Patterns”	Message channels and event messages

5.10 Implementation Phases

Phase 1 — Durable membership and single-node presence (4–6 hours)

• Define authorized incident membership.
• Track one connection with minimal metadata.
• Render grouped state.

Checkpoint: An unauthorized user cannot track or list; one authorized user appears once.

Phase 2 — Multi-device state and diff (4–5 hours)

• Open three connections under one key.
• Apply state and diffs by ref.
• Keep the user online as individual refs leave.

Checkpoint: Closing one of three tabs yields devices=2, not offline.

Phase 3 — Batch enrichment and callback safety (3–5 hours)

• Add compact meta validation.
• Batch-fetch display details.
• Isolate optional failure-prone work.

Checkpoint: A delayed profile source degrades names/avatars without blocking Tracker callbacks.

Phase 4 — Cluster and hard-failure drill (5–8 hours)

• Run three connected nodes.
• Isolate one node and record local views.
• Reconnect with a new ref and observe convergence.

Checkpoint: All connected nodes reach the same final refs within the measured target.

Phase 5 — Deploy policy and tests (4–6 hours)

• Compare graceful permanent-down enabled/disabled behavior.
• Test rolling restart, abrupt halt, and partition.
• Document the chosen product language for uncertainty.

Checkpoint: Test reports distinguish all failure modes and prove no business truth depends on diffs.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Logical key	Connection ID / user ID	Opaque stable user ID	Groups devices without exposing details
Meta detail	Full profile / compact ephemeral fields	Compact fields	Limits replication and privacy risk
User online rule	Latest tab / any meta	Any accepted meta	Avoids false offline transitions
Profile enrichment	Per meta query / batched fetch	Batched fetch	Prevents N+1 work
Shutdown behavior	Immediate permdown / convergence window	First verify root-level :one_for_one, then choose from measured deploy UX	Immediate permdown is safe only with the documented supervision topology
Audit handling	Presence diff / explicit durable event	Explicit durable event	Presence is eventual and ephemeral

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Projection unit	Verify grouping and exact-ref mutation	One key with A/B/C; remove C
Authorization	Separate presence from access	Revoked responder cannot track
Callback performance	Keep Tracker work bounded	Slow profile source isolated
Cluster integration	Verify eventual convergence	Partition B, reconnect as E
Lifecycle	Compare graceful and abrupt loss	Tab close vs node halt
Privacy	Enforce metadata schema and size	Reject secret or oversized meta

6.2 Critical Test Cases

1. One user with three metas renders one roster row and devices=3.
2. Removing one ref preserves the other refs and online status.
3. Removing the final ref changes the user to offline according to UI policy.
4. A duplicate join ref does not duplicate a device.
5. A leave for an unknown ref does not remove the logical key.
6. Initial state plus queued diff produces the correct final refs.
7. Profile fetch uses one batch for all affected keys.
8. Profile fetch failure does not crash the tracker path.
9. Abrupt node loss shows uncertainty before convergence.
10. Reconnect overlap temporarily permits old and new refs.
11. Restored connectivity converges all nodes to the same set.
12. Presence diffs never write durable attendance/audit rows.

6.3 Deterministic Test Data

Room: INC-104
Authorized keys: user-42, user-77

Initial metas:
  user-42 -> A(node_a,laptop), B(node_b,phone), C(node_a,tab)
  user-77 -> D(node_c,laptop)

Sequence:
  leave C
  isolate node_b
  join E(node_c,phone)
  mark B down after configured window
  restore node_b connectivity

Expected converged refs:
  user-42 -> [A,E]
  user-77 -> [D]

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Correction
Flatten key to boolean	One tab close makes user offline	Preserve the meta collection
Store profile in meta	Large replicated payload and leaks	Batch-fetch durable detail
Block callback work	Presence handling stalls; raw Tracker work can stall heartbeats/diffs	Bound handle_metas/4; offload optional work from either layer
Treat leave as intent	False audit or workflow events	Use explicit durable commands
Expect instant hard-loss detection	“Ghost” users reported as bug	Model and explain uncertainty window
Assume transitive cluster membership	Nodes see different rosters	Verify direct connectivity and topology

7.2 Debugging Strategies

• Log scenario ID, tracker node, ref, operation, and monotonic timestamp.
• Compare Presence.list results on every node at controlled checkpoints.
• Inspect node connectivity before blaming CRDT merge.
• Measure Presence handle_metas/4 duration; if you run the optional raw Tracker experiment, also measure handle_diff/2 duration and tracker mailbox length during injected latency.
• Capture both the raw ref set and the grouped roster; grouping bugs often masquerade as replication bugs.

7.3 Performance Traps

Frequent activity updates create high diff volume. Large metadata multiplies cluster traffic. fetch/2 on every change can overload the database if it performs one query per meta. Avoid broadcasting cursor pixels or keystrokes as Presence metadata; use a separate ephemeral channel with sampling/coalescing.

8. Extensions & Challenges

8.1 Beginner Extensions

• Display device count and last connected time.
• Add a tooltip explaining UNCERTAIN versus OFFLINE.
• Show raw refs in a development-only diagnostics drawer.

8.2 Intermediate Extensions

• Add coarse active/away updates with a minimum update interval.
• Compare Channel and LiveView clients using the same Presence module.
• Add tenant-scoped rooms and cross-tenant authorization tests.

8.3 Advanced Extensions

• Measure convergence across three geographically separated development nodes.
• Simulate a rolling deploy under both permanent-down policies and compare churn.
• Build a replayable, durable incident attendance log from explicit join/leave commands—not Presence diffs.

9. Real-World Connections

9.1 Industry Applications

• Collaborative editors: Presence shows cursors/devices while the document log stores durable edits.
• Incident response: Presence shows current responders while the incident record stores assigned roles and actions.
• Customer support: Agent availability is a hint for routing, not the authoritative staffing schedule.
• Gaming and social systems: Online status tolerates convergence; entitlements and purchases do not.

9.2 Related Open Source Projects

• Phoenix.Presence: Server behavior and client-oriented state/diff representation.
• Phoenix.Tracker: Heartbeat and CRDT-based distributed tracking.
• Phoenix.PubSub: Transport used to replicate tracker changes and broadcast Presence diffs.

9.3 Interview Relevance

The project provides a concrete answer to “what does online mean in a distributed system?” It demonstrates failure-detector uncertainty, replicated-state convergence, multi-device modeling, callback isolation, and the boundary between ephemeral liveness and durable business truth.

10. Resources

10.1 Essential Reading

• Phoenix.Presence — server behavior, metadata shape, and fetch/2 guidance.
• Phoenix.Tracker — CRDT, heartbeat, callback, and shutdown semantics.
• Phoenix JavaScript Presence — supported state and diff synchronization.
• OTP :pg — strong eventual membership and non-transitive views.
• “Real-Time Phoenix” by Stephen Bussey — Presence and Channels chapters.

10.2 Tools & Documentation

• Phoenix.PubSub — distributed transport configuration.
• Telemetry — bounded instrumentation handlers.
• epmd, remote IEx, and network fault tooling for the local cluster drill.

10.3 Related Projects in This Series

• Project 4: Establishes the distributed PubSub substrate.
• Project 5: Provides the LiveView incident workspace that renders the roster.
• Project 7: Supplies the mailbox and callback pressure techniques used to validate Presence safety.

11. Self-Assessment Checklist

11.1 Understanding

• I can define key, meta, phx_ref, state, diff, and convergence.
• I can explain why hard failure is not detected instantly.
• I can explain how one user remains online after one tab closes.
• I can defend the metadata boundary.
• I can state why Presence is not authorization or audit truth.

11.2 Implementation

• Multi-device grouping and exact-ref removal work.
• Initial state plus diffs survives reconnect.
• Profile detail is batch-fetched and failure-isolated.
• The three-node partition drill converges.
• Callback duration and metadata size meet budgets.

11.3 Growth

• I documented graceful close, process kill, node halt, and partition separately.
• I can explain the shutdown policy trade-off.
• I can reproduce and diagnose a temporary ghost meta.
• I can explain this consistency model in an interview.

12. Submission / Completion Criteria

Minimum Viable Completion

• One logical user supports at least three visible metas.
• Closing one meta does not create a false offline state.
• State and diff synchronization produces the expected final set.
• Presence is gated by durable authorization.

Full Completion

• All functional requirements and critical tests pass.
• A three-node partition/reconnect drill is documented with timestamps.
• Metadata and callback budgets are verified.
• Durable business behavior is proven independent of Presence diffs.

Excellence

• Compare both shutdown policies during a rolling deployment.
• Produce a convergence timeline visualization for every node.
• Demonstrate degraded profile enrichment without tracker delay or crash.

Return to the Pub/Sub in Elixir deep-dive guide or browse the expanded project index.