Project 1: Mailbox Broadcast Laboratory

Build a terminal laboratory that makes raw BEAM fanout, mailbox ordering, selective receive, queue growth, and process death directly observable.

Quick Reference

Attribute Value
Difficulty Level 1: Beginner
Time Estimate 6–8 hours
Language Elixir (alternatives: Erlang, Gleam)
Prerequisites Basic Elixir syntax, pattern matching, functions, IEx, elementary ExUnit
Key Topics Processes, PIDs, asynchronous send, receive, mailbox ordering, selective receive, monitors, overload
Primary Tools Elixir, Mix, IEx, ExUnit, Observer or observer_cli
Produces A repeatable CLI experiment and evidence report, not a reusable production broker

1. Learning Objectives

By completing this project, you will:

  1. Explain Pub/Sub fanout as repeated asynchronous sends to independent process mailboxes.
  2. Distinguish message acceptance by the runtime from mailbox arrival, receive matching, handler completion, and business acknowledgement.
  3. Demonstrate the BEAM ordering guarantee: signals from one sender to one destination preserve order, while messages from different senders may interleave.
  4. Explain how selective receive scans and skips unmatched messages without moving them out of the mailbox.
  5. Use process monitors to observe subscriber termination without coupling the observer’s lifecycle to the subscriber.
  6. Measure message queue length and event age to identify a subscriber that is falling behind.
  7. Design deterministic concurrency tests around barriers, references, acknowledgements, and deadlines rather than arbitrary sleeps.
  8. State clearly what happens to queued work when a process terminates.

2. Theoretical Foundation

2.1 Core Concepts

BEAM processes and isolation

A BEAM process is an isolated unit of execution with its own mailbox, heap, stack, process dictionary, and scheduling state. It is not an operating-system thread. Processes communicate primarily by sending immutable Elixir terms. Isolation is the first important Pub/Sub property: one subscriber’s handler state is not shared with another subscriber, and a slow subscriber accumulates work in its own mailbox rather than directly blocking a fast subscriber. Isolation does not remove resource coupling, however. Every queued message consumes memory, and a sufficiently overloaded process can contribute to node-wide memory pressure.

Asynchronous send

A sender places a message signal on the path to a destination PID and continues. The return value of send is not a handler receipt. It does not mean that the subscriber executed a receive, finished a callback, committed a database transaction, or emitted a user-visible result. If the destination no longer exists, an ordinary send does not turn that fact into a business acknowledgement. Applications needing processing evidence must define a separate protocol, usually with an event identifier, a reply destination, and a correlation reference.

Mailboxes and receive

Each receiving process owns a mailbox. Arrival and processing are separate events: messages may already be queued while the process is doing other work. A receive expression examines queued messages in arrival order and selects the first message matching one of its clauses. Earlier unmatched messages remain queued. This selective receive capability is powerful for protocols, but repeatedly searching past a large unmatched prefix increases work and can starve messages that never match.

Ordering

The language guarantee is scoped to a sender-destination pair. If Publisher A sends A1 and then A2 to Subscriber S, S cannot observe A2 arriving before A1. The guarantee does not define a global order between Publisher A and Publisher B. Two subscribers can therefore observe different valid interleavings while each still preserves A1 before A2 and B1 before B2. Pub/Sub designs that require one total order need an explicit sequencer or ordered log; fanout alone does not create one.

Monitors and lifecycle

A monitor is one-way failure observation. The monitoring process receives a DOWN message containing a unique reference, the monitored PID, and a reason when the target terminates. A monitor does not restart the process, preserve its mailbox, or confirm which messages completed. When the target dies, its mailbox and process-local state disappear. The laboratory therefore records handler acknowledgements separately from sends and labels unacknowledged events as unknown or lost for this volatile experiment.

2.2 How Fanout Actually Behaves

The laboratory separates a broadcast into stages that are often incorrectly collapsed into “delivery”:

Publisher                    Runtime                    Subscriber
    |                           |                           |
    |  construct event         |                           |
    |-------------------------->| enqueue signal           |
    |  send returns             |-------------------------->|
    |<--------------------------| message enters mailbox    |
    |                           |                           |
    |                           |          scheduled -------|
    |                           |                           | scan receive clauses
    |                           |                           | execute handler
    |<------------------------------------------------------| explicit lab receipt
    |                           |                           |

The critical boundaries are:

  1. The publisher creates an immutable event with identity, source sequence, and a monotonic send timestamp.
  2. Fanout enumerates the current subscriber snapshot.
  3. One asynchronous send is issued for each destination.
  4. Each destination accumulates and selects messages independently.
  5. The handler records its own completion.
  6. Only the explicit laboratory receipt provides evidence of handling.

The slow subscriber intentionally spends longer handling each event than the publisher spends creating events. Its queue therefore follows the rough relationship:

queue change over an interval
  = events arriving
  - events handled
  - events discarded because the process died

If arrival rate remains above service rate, queue length grows without a natural bound. The mailbox is not a demand protocol and does not push back on the publisher merely because the subscriber is slow.

The priority subscriber demonstrates selective receive. It accepts only priority events during one phase. Normal events remain in front of later priority events from the mailbox’s perspective, so finding a match can require scanning. After the priority phase ends, the subscriber changes protocol state and drains normal events. The report must distinguish “skipped while searching” from “discarded”; receive does not silently delete unmatched messages.

2.3 Mental Model

Two independent publishers

  Publisher A                Publisher B
   A1 -> A2                   B1 -> B2
      \                         /
       \ repeated send calls  /
        v                     v
  +------------------------------------------------+
  |             subscriber snapshot                |
  +----------------+---------------+---------------+
                   |               |
      +------------+-------+  +----+---------------+
      |                    |  |                    |
      v                    v  v                    v
+-----------+        +-----------+          +-------------+
| fast PID  |        | slow PID  |          | priority PID|
| mailbox   |        | mailbox   |          | mailbox     |
| A1 B1 ... |        | A1 B1 ... |          | N1 P1 N2... |
+-----+-----+        +-----+-----+          +------+------+
      |                    |                       |
   immediate           250 ms/item          selective scan
   receipts            queue growth         unmatched remain

Valid observations:
  fast:     A1, B1, B2, A2
  slow:     B1, A1, A2, B2

Required constraints:
  A1 occurs before A2 at every destination
  B1 occurs before B2 at every destination
  no required order exists between A and B

2.4 Definitions and Key Terms

  • PID: A process identifier naming one process incarnation.
  • Mailbox: The ordered queue of messages available to a process’s receive expressions.
  • Fanout: Sending one logical event to multiple destinations.
  • Selective receive: Searching the mailbox for the first message matching a receive clause while retaining unmatched messages.
  • Signal ordering: The BEAM guarantee that signals sent from one entity to the same destination preserve their send order.
  • Handler receipt: An application-level message emitted by the laboratory after a subscriber finishes handling an event.
  • Monitor reference: A unique value connecting one monitor request to its eventual DOWN message.
  • Event age: Monotonic time at handler completion minus monotonic time at publication.
  • Queue length: The number of messages currently visible in a process mailbox; a useful but momentary overload signal.
  • Backpressure: A protocol by which demand or capacity constrains production. Ordinary send does not provide it.

2.5 Minimal Concrete Example

Use a protocol outline rather than implementation code:

EVENT:
  id: evt-A-0001
  source: publisher-A
  source_sequence: 1
  class: normal
  sent_at_monotonic: T0
  payload: small immutable term

BROADCAST:
  snapshot current subscriber PIDs
  for each subscriber:
    asynchronously send EVENT
  record sends_issued

SUBSCRIBER:
  receive matching EVENT
  record queue length before handling
  apply configured behavior
  send HANDLED(event_id, subscriber_id, finished_at) to coordinator

COORDINATOR:
  correlate HANDLED records by event_id
  correlate DOWN records by monitor reference
  never infer HANDLED from sends_issued

2.6 Common Misconceptions

  • “Messages from all publishers have one arrival order.” Only each sender-destination stream has an ordering guarantee.
  • “send returning means the handler ran.” It only means the send operation completed from the caller’s perspective.
  • “Selective receive moves skipped messages to the back.” Unmatched messages remain queued; matching searches past them.
  • “A PID represents a logical subscriber forever.” A restarted process has a new PID and a new mailbox.
  • “A monitor preserves queued work.” It reports termination; it does not recover process-local state.
  • “A slow subscriber only hurts itself.” Its handler is isolated, but unchecked mailbox memory can eventually hurt the whole node.

2.7 Check Your Understanding

  1. Can Subscriber S observe A2 before A1 if both were sent by Publisher A in that order?
  2. Can Subscriber X and Subscriber Y observe different interleavings of A and B?
  3. What does a normal send return tell you about handler completion?
  4. Where are unmatched messages after a selective receive finds a later match?
  5. What happens to queued messages when the subscriber process dies?
  6. Which measurement reveals backlog, and which reveals user-perceived staleness?

Answers

  1. No, assuming the same sender and same destination incarnation; their relative signal order is preserved.
  2. Yes. Each destination has an independent mailbox and scheduling history.
  3. Nothing. Handler completion requires a separate application protocol.
  4. They remain in the mailbox in their existing relative order.
  5. They disappear with that process and are not replayed into its replacement.
  6. Queue length reveals pending volume; event age reveals how stale completed work has become.

2.8 Why This Matters

Every later abstraction in the learning path eventually causes messages to enter BEAM mailboxes. Phoenix.PubSub makes membership and fanout convenient, but it does not repeal mailbox behavior. LiveView updates, Presence diffs, and cluster broadcasts can all produce queue growth or be lost when a volatile subscriber dies. This laboratory gives you a truthful vocabulary for diagnosing those systems.

2.9 References

3. Project Specification

3.1 What You Will Build

Build a Mix project containing a coordinator, two synchronized publishers, three subscriber fixtures, an observation sampler, and a report formatter. The fast subscriber handles immediately. The slow subscriber uses a configurable delay. The priority subscriber temporarily accepts only priority events and later drains normal events. The coordinator monitors all subscribers and separately records sends, completed handler receipts, queue samples, and DOWN evidence.

The deliverable is a deterministic experiment harness, not a production Pub/Sub library. It should be easy to rerun with several event counts and slow-handler delays and should produce a human-readable guarantee report.

3.2 Functional Requirements

  1. Event identity: Every event includes a unique ID, publisher identity, per-publisher sequence, class, and monotonic send time.
  2. Start barrier: Publishers wait for one explicit release so their streams can interleave.
  3. Three subscriber behaviors: Fast, delayed, and selective-receive behaviors are independently visible.
  4. Fanout snapshot: Each broadcast records the intended recipient PIDs at send time.
  5. Separate receipts: A handler completion receipt is recorded only after subscriber work finishes.
  6. Queue sampling: Queue length and oldest observed event age are sampled during the run.
  7. Failure injection: The slow subscriber can be terminated while it has queued events.
  8. Lifecycle evidence: The coordinator records a monitor DOWN message and the last pre-failure queue sample.
  9. Ordering analysis: The report verifies per-source order separately for each subscriber.
  10. Interleaving report: Cross-source differences are described as observed behavior, not correctness failures.
  11. No replay claim: The restarted or replacement process is explicitly shown to lack the old mailbox.

3.3 Non-Functional Requirements

  • Reproducibility: Tests synchronize with messages and bounded deadlines; only the slow fixture uses an intentional delay.
  • Safety: Event payloads remain small so the lab studies queueing rather than giant-term copying.
  • Clarity: Each output line includes relative time, event ID or source sequence, subscriber identity, queue sample, and event age where relevant.
  • Bounded execution: Every experiment has a maximum deadline and terminates all fixture processes.
  • Truthful terminology: Use “send issued,” “mailbox observed,” “handled,” and “monitor DOWN” as distinct states.

3.4 Example Usage and Exact Output

$ mix run lab/mailbox_broadcast.exs --events 12 --slow-ms 250
[00.000] barrier release publishers=[A,B] subscribers=[fast,slow,priority]
[00.000] publish source=A seq=1 class=normal recipients=3
[00.001] fast handled A:1 queue_before=0 age_ms=1
[00.002] priority awaiting class=priority unmatched_normal=1
[00.004] publish source=B seq=1 class=priority recipients=3
[00.006] priority handled B:1 queue_before=1 age_ms=2
[00.252] slow handled A:1 queue_before=8 age_ms=252
[00.300] inject_exit subscriber=slow reason=simulated_failure
[00.301] DOWN subscriber=slow pending_before_exit=8 handled=1
[00.302] publish source=A seq=6 recipients=2 sends_issued=2

Guarantee report
A sequence preserved at fast                         PASS
A sequence preserved at priority                     PASS
B sequence preserved at every surviving destination  PASS
Different cross-sender interleavings                 OBSERVED
Handler acknowledgement inferred from send           NONE
Slow queued events recovered after exit              NO

3.5 Real World Outcome

The learner runs the command in a normal terminal and watches three stories unfold in the same timestamped transcript. Fast receipts remain close to publish time. Slow receipts become progressively older while its queue sample rises. Priority receipts can appear ahead of older normal messages because the process is intentionally searching for a later matching class; the output calls those older messages unmatched, not dropped.

At the injected failure, the slow process stops. The monitor report shows why it stopped and how many messages were last observed in its mailbox. The final report does not count those messages as processed and does not pretend a replacement PID inherited them. A second run may produce a different A/B interleaving, but all runs preserve A’s and B’s individual sequence at every destination.

4. Solution Architecture

4.1 High-Level Design

+-------------------- experiment supervisor --------------------+
|                                                               |
|  +-------------+      release      +-------------+            |
|  | publisher A |<------------------| coordinator |            |
|  +------+------+                   +------+------+            |
|         |                                  | monitors          |
|  +------+------+                           |                   |
|  | publisher B |                           v                   |
|  +------+------+      +----------+  +----------+  +----------+|
|         +------------>| fast     |  | slow     |  | priority ||
|         +------------>| mailbox  |  | mailbox  |  | mailbox  ||
|         +------------>|          |  |          |  |          ||
|                       +----+-----+  +----+-----+  +----+-----+|
|                            \           |             /         |
|                             \ HANDLED  |            /          |
|                              +---------v-----------+           |
|                              | coordinator/report  |           |
|  sampler ------------------->| queue observations  |           |
+---------------------------------------------------------------+

4.2 Key Components

Component Responsibility Key Decisions
Experiment coordinator Starts fixtures, creates barrier, monitors subscribers, correlates evidence Must never equate send count with handled count
Publisher fixture Emits ordered source stream after release Per-source sequence is monotonic
Fast subscriber Establishes healthy baseline No artificial work
Slow subscriber Creates controlled service-rate deficit Delay is configurable and isolated to fixture
Priority subscriber Demonstrates selective receive Protocol changes phase before draining normal work
Sampler Reads queue metrics at bounded intervals Samples are observations, not transactional facts
Analyzer Checks ordering and receipt invariants Evaluates each destination independently
Reporter Produces exact transcript and summary Uses monotonic relative time

4.3 Data Structures

Use conceptual records:

Event:
  id
  source
  source_sequence
  class
  sent_at_monotonic
  payload_digest

DeliveryIntent:
  event_id
  subscriber_pid
  send_issued_at

HandlerReceipt:
  event_id
  subscriber_id
  handled_at
  queue_before

QueueSample:
  subscriber_id
  sampled_at
  message_queue_len

DownEvidence:
  monitor_reference
  subscriber_id
  pid
  reason
  last_queue_sample

4.4 Algorithm Overview

Barrier-controlled publication

  1. Start subscribers and monitor each PID.
  2. Start publishers in a waiting state.
  3. Confirm every fixture reports ready.
  4. Release both publishers with one barrier reference.
  5. For each event, snapshot destination PIDs and send once to each.
  6. Continue sampling queues while receipts arrive.
  7. Inject subscriber death at the configured condition.
  8. Close the run at completion or deadline and analyze the evidence.

Ordering analysis

  1. Group receipts by subscriber, then by source.
  2. Extract source sequences in handled order.
  3. Verify each extracted list is strictly increasing.
  4. Compare complete source-label traces across subscribers.
  5. Report different valid interleavings without requiring them.

Complexity

  • Fanout time: O(S) send operations for S subscribers.
  • Receipt storage: O(E × S) in the evidence-heavy lab.
  • Ordering analysis: O(R), where R is the number of receipts.
  • Selective receive search: up to O(Q) for a matching attempt across a mailbox prefix of Q messages.
  • Mailbox space: O(arrival rate minus service rate over time) until handled or process termination.

4.5 Invariants and Failure Modes

Invariant Evidence
Same source sequence never reverses at one destination Per-subscriber ordering analysis
A send is not a handler receipt Separate intent and receipt stores
One subscriber’s delay does not serialize another subscriber’s handler Fast age stays low while slow queue rises
Termination removes queued process-local work DOWN plus missing receipts
All fixtures terminate at experiment end Supervisor/test teardown report

Primary failure modes are accidental publisher serialization, ambiguous clock usage, report races, monitoring after the target already died, and a test process consuming messages intended for a fixture.

5. Implementation Guide

5.1 Development Environment Setup

$ elixir --version
Erlang/OTP 27 [...]
Elixir 1.18.[...]

$ mix new mailbox_broadcast_lab --sup
* creating README.md
* creating mix.exs
* creating lib/...
* creating test/...

Observer is optional. If the graphical observer is unavailable, use process inspection from IEx or observer_cli. Do not make the project depend on a GUI.

5.2 Suggested Project Structure

mailbox_broadcast_lab/
├── lib/
│   └── mailbox_lab/
│       ├── experiment.ex
│       ├── publisher_fixture.ex
│       ├── subscriber_fixture.ex
│       ├── sampler.ex
│       ├── analyzer.ex
│       └── reporter.ex
├── lab/
│   └── mailbox_broadcast.exs
├── test/
│   ├── ordering_test.exs
│   ├── selective_receive_test.exs
│   └── lifecycle_test.exs
└── README.md

5.3 The Core Question You Are Answering

“What actually happens between send returning and a subscriber completing work?”

Before building the reporter, write down every stage in between. If the answer uses the word “delivered,” replace it with a more precise state: send issued, arrived in mailbox, matched by receive, handler started, handler completed, or explicitly acknowledged.

5.4 Concepts You Must Understand First

  1. Process incarnation
    • Why does a restarted process have a different PID?
    • What state and queued work survive termination?
    • Book: “Designing for Scalability with Erlang/OTP,” process architecture chapters.
  2. Signal ordering
    • Which sender and destination define the guarantee?
    • What changes when there are two senders?
    • Reference: Erlang System Documentation, Processes.
  3. Selective receive
    • What happens to unmatched messages?
    • Why can a large unmatched prefix be expensive?
    • Reference: Erlang Efficiency Guide, Processes.
  4. Monitors
    • Why is DOWN evidence not a processing receipt?
    • What role does the monitor reference play?
    • Book: “Erlang in Anger,” runtime diagnosis.
  5. Queue stability
    • What relationship between arrival and service rates causes growth?
    • Why is queue length alone insufficient without age?
    • Book: “Release It!, 2nd Edition,” stability antipatterns.

5.5 Questions to Guide Your Design

  1. How will the coordinator know all publishers and subscribers are ready?
  2. Which process owns the subscriber list, and when is the fanout snapshot taken?
  3. How will timestamps remain monotonic and comparable?
  4. How will the priority fixture reveal unmatched messages without introspecting private mailbox contents as a correctness mechanism?
  5. What run condition guarantees the slow subscriber has backlog before termination?
  6. Which state will distinguish missing, unknown, and handled events?
  7. How will teardown prevent fixture messages leaking into later tests?

5.6 Thinking Exercise

On paper, draw Publisher A sending A1 then A2 and Publisher B sending B1 then B2 to Subscribers X and Y.

  1. Write four valid complete traces.
  2. Cross out any trace that places A2 before A1 or B2 before B1.
  3. Explain why X and Y need not have identical traces.
  4. Add normal N1, priority P1, and normal N2 to one mailbox.
  5. Trace a receive clause that matches only priority.
  6. Mark the location of N1 and N2 after P1 is handled.
  7. Terminate the process and mark which items survive.

5.7 The Interview Questions They Will Ask

  1. What ordering does Erlang message passing guarantee?
  2. Does send fail if an ordinary destination PID has just died?
  3. Why can selective receive become expensive?
  4. What happens to a process mailbox after a crash?
  5. What is the difference between a link and a monitor?
  6. How would you detect a subscriber falling behind?

5.8 Hints in Layers

Hint 1: Prove one stream first

Start with one publisher, one subscriber, and five sequenced messages. Make the order assertion reliable before adding fanout.

Hint 2: Make the event self-describing

Carry event ID, source, source sequence, class, and monotonic send time. Do not infer these from log line order.

Hint 3: Synchronize with protocol messages

Use READY, RELEASE(reference), HANDLED(reference, event), and FINISHED messages. The protocol is part of the experiment.

Hint 4: Keep observation separate

The sampler may inspect queue length, but the analyzer should establish correctness from explicit receipts and DOWN evidence.

5.9 Books That Will Help

Topic Book Chapter or Section
Process communication “Designing for Scalability with Erlang/OTP” Process architecture and communication
Failure observation “Erlang in Anger” Processes, tracing, and overload
Queue stability “Release It!, 2nd Edition” Stability Antipatterns
Messaging vocabulary “Enterprise Integration Patterns” Ch. 3 Messaging Channels

5.10 Implementation Phases

Phase 1: Ordering Baseline — 90 minutes

  • Create event and receipt representations.
  • Start one publisher and one fast subscriber.
  • Assert one source’s sequence and exact receipt correlation.

Checkpoint: Fifty events are handled in sequence without timing sleeps in the test.

Phase 2: Fanout and Interleaving — 90 minutes

  • Add two more subscribers.
  • Add the barrier-controlled second publisher.
  • Analyze order independently for every subscriber and source.

Checkpoint: Ten runs preserve per-source order; at least one run or a deterministic scheduler fixture demonstrates a permitted cross-source difference.

Phase 3: Selective Receive and Pressure — 2 hours

  • Add the priority fixture and phase transition.
  • Add slow service behavior.
  • Sample queue length and compute event age.

Checkpoint: Output visibly separates unmatched priority behavior from slow-consumer backlog.

Phase 4: Death and Evidence — 90 minutes

  • Monitor subscribers.
  • Inject failure while backlog is nonzero.
  • Report unhandled events without claiming they were recovered.

Checkpoint: DOWN evidence and receipt totals agree with the final guarantee report.

Phase 5: Tests and Documentation — 60 minutes

  • Add deadline-based failure messages.
  • Document every guarantee and non-guarantee.
  • Ensure teardown leaves no fixture alive.

Checkpoint: The complete suite passes repeatedly and the README can be used to interpret every output field.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Clock Wall clock, monotonic clock Monotonic clock for durations Avoids clock adjustments corrupting age
Completion evidence Send count, explicit receipt Explicit receipt Separates routing from handling
Concurrency start Sequential spawn, barrier Barrier Makes concurrent streams meaningful
Failure observation Link, monitor Monitor Observer should not die with fixture
Slow behavior Global sleep, one fixture delay One fixture delay Preserves independent subscriber behavior
Test synchronization Sleeps, references/deadlines References and deadlines Reduces flakiness

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Validate analyzer and report logic Sequence checks, event-age calculation
Protocol Validate fixture messages READY/RELEASE/HANDLED correlation
Concurrency Validate ordering boundaries Two senders to several destinations
Lifecycle Validate monitors and mailbox loss Kill slow subscriber with backlog
Edge case Validate empty and small runs Zero subscribers, one event, subscriber dies before send
Performance observation Reveal trend, not benchmark hardware Queue growth as service rate falls

6.2 Critical Test Cases

  1. One publisher sends sequences 1–100; every surviving subscriber records them in order.
  2. Two publishers produce interleaved receipts without violating either source’s order.
  3. Priority receive handles a later priority message while earlier normal messages remain pending.
  4. Slow subscriber queue rises when production rate exceeds service rate.
  5. Fast subscriber remains low-latency during the same run.
  6. Killing the slow PID emits exactly the expected monitor evidence.
  7. Events without handler receipts are never counted as handled.
  8. A replacement subscriber receives only future messages.
  9. A deadline failure prints outstanding event IDs and subscriber identities.
  10. Experiment teardown leaves no monitored fixture process alive.

6.3 Test Data

publishers:
  A: [A1 normal, A2 normal, A3 priority, A4 normal]
  B: [B1 priority, B2 normal, B3 normal, B4 priority]

subscribers:
  fast:     service_time = immediate
  slow:     service_time = 250 ms
  priority: first_phase = priority_only

required constraints:
  every observed A sequence is increasing
  every observed B sequence is increasing
  fast age remains below slow age trend
  slow queue is greater than zero before injected exit
  replacement gets no event published before its subscription

6.4 Test Discipline

  • Give every test its own correlation reference.
  • Match only messages containing that reference.
  • Use a generous but finite deadline and report missing evidence.
  • Do not assert one cross-sender order.
  • Do not depend on queue length being exact after another process runs; test a trend or a synchronized sample.
  • Keep intentional delay out of unit tests that do not study slowness.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Publishers start sequentially Every run has one source followed by the other Use a readiness barrier and one release reference
Send count called delivery count Report claims dead subscriber handled queued events Store handler receipts separately
Global ordering assertion Valid runs fail intermittently Check order per source and destination
Priority messages tagged incorrectly Selective fixture appears broken Print event class and inspect protocol fixture
Wall-clock age Negative or discontinuous durations Use monotonic timestamps
Monitor created too late Immediate or missing lifecycle story Monitor as part of fixture registration
Queue sample treated as exact history Flaky exact-count tests Synchronize sampling or assert directional trends
Test mailbox contamination Unexpected messages in later test Correlate references and terminate fixtures

7.2 Debugging Strategies

  • Inspect one PID: Query current function, status, and message queue length for the slow fixture.
  • Trace receive events sparingly: Trace only selected fixture PIDs and a small event count.
  • Print correlation references: A mismatched reference often reveals cross-test leakage.
  • Freeze the topology: Return to one publisher and one subscriber before diagnosing fanout.
  • Compare intent and receipt sets: Set difference reveals exactly which event-subscriber pairs remain unresolved.
  • Record phase changes: The priority subscriber should log when it switches from priority-only to drain mode.

7.3 Performance Traps

Selective receive can repeatedly scan a long unmatched prefix. Excessive logging can also dominate the experiment and accidentally serialize publishers through one logger path. Keep payloads small, use modest event counts for trace mode, and provide a quiet metrics mode for queue-growth observations. Never interpret the lab as a universal throughput benchmark; its purpose is causal visibility.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a fourth subscriber that handles only one publisher.
  • Show queue length and event-age sparklines in the final text report.
  • Add a run mode with no subscribers and explain the result.

8.2 Intermediate Extensions

  • Compare selective receive with a process that explicitly routes priority and normal work into separate workers.
  • Add a bounded coalescing policy and document which events may be replaced.
  • Compare links and monitors in an isolated failure experiment.

8.3 Advanced Extensions

  • Repeat the experiment across two distributed nodes and document what changes when the connection drops.
  • Use runtime tracing to measure receive-scan pressure without logging every event.
  • Model an explicit acknowledgement protocol and prove why it still needs timeout and duplicate handling.

9. Real-World Connections

9.1 Industry Applications

  • LiveView processes: UI updates arrive as mailbox messages; a slow handler can make the browser state stale.
  • GenServers: Calls and casts ultimately interact with a process mailbox and must respect service capacity.
  • Telemetry consumers: High-cardinality or expensive handlers can become slow subscribers.
  • Device gateways: Many producers often feed per-device or per-tenant subscriber processes.
  • Background orchestration: Volatile process messages are useful for coordination but not a durable job record.
  • Elixir — language-level process and receive abstractions.
  • Erlang/OTP — runtime, signal, scheduler, and monitor semantics.
  • observer_cli — terminal inspection of process and queue behavior.
  • Phoenix.PubSub — later framework abstraction built over local process membership and fanout.

9.3 Interview Relevance

This project prepares you to answer concurrency questions with explicit guarantee scopes. You can explain why BEAM isolation helps responsiveness, why asynchronous messaging is not processing confirmation, why selective receive has a cost, and how monitors differ from recovery. Those explanations are more valuable than merely recalling send and receive syntax.

10. Resources

10.1 Essential Reading

10.2 Video and Talk Topics

  • Search Erlang Solutions conference archives for talks on BEAM scheduling and process mailboxes.
  • Review ElixirConf talks on tracing slow GenServers and mailbox overload.
  • Use talks only after reading the official signal-ordering specification.

10.3 Tools and Documentation

  • IEx: Interactive fixture startup and process inspection.
  • ExUnit: Message assertions, monitored-process tests, and deadline control.
  • Observer: Visual process and queue inspection when a GUI is available.
  • observer_cli: Terminal alternative for node inspection.
  • Next: Project 2 — Registry Topic Bus replaces the hand-maintained subscriber snapshot with process-owned duplicate registrations.
  • Project 7 — Mailbox Pressure Laboratory returns to these mechanics with production-oriented flow-control policies.

11. Self-Assessment Checklist

11.1 Understanding

  • I can state the sender-destination scope of BEAM signal ordering.
  • I can explain why two subscribers may observe different valid interleavings.
  • I can explain every stage between send and completed handling.
  • I can explain where unmatched messages remain after selective receive.
  • I can explain what DOWN proves and what it does not prove.
  • I can explain why mailbox queueing is not backpressure.

11.2 Implementation

  • The experiment has a real readiness barrier.
  • All events have identity, source sequence, class, and monotonic time.
  • Three subscriber behaviors are independently visible.
  • Intent, receipt, queue sample, and DOWN evidence are stored separately.
  • Per-source order is analyzed per destination.
  • Slow-subscriber death occurs with verified backlog.
  • Tests use correlation references and deadlines.
  • All fixture processes terminate after each run.

11.3 Growth

  • I can diagnose a growing mailbox with queue length and event age.
  • I can identify an invalid global-order assumption in a design review.
  • I can explain this laboratory without relying on framework vocabulary.
  • I documented one observation that surprised me.

12. Submission / Completion Criteria

Minimum Viable Completion

  • The CLI shows one publisher, two subscribers, and distinct send versus handled evidence.
  • Tests prove same-sender ordering.
  • One subscriber can be monitored and terminated.

Full Completion

  • All three subscriber behaviors are implemented.
  • Two publishers begin through a readiness barrier.
  • Queue length and event age expose slow-consumer pressure.
  • Selective receive leaves unmatched work visible.
  • The final guarantee report separates guarantees, observations, and non-guarantees.
  • The test suite is repeatable without arbitrary synchronization sleeps.

Excellence

  • The analyzer produces a machine-readable evidence artifact in addition to the terminal report.
  • A trace mode explains selective-receive search cost without changing correctness.
  • Documentation includes a concise failure timeline and a comparison of monitor evidence with business acknowledgement.

Return to the Pub/Sub in Elixir mastery guide.

See the expanded project index.