Project 4: Fault-Tolerant Telemetry Lab

Build a supervised sensor simulation whose Livebook dashboard visibly proves crash detection, restart semantics, stream continuity, bounded load, and cleanup.

Quick Reference

Attribute Value
Difficulty Level 3 — Advanced
Time Estimate 16–24 hours
Language Elixir (alternative: Erlang)
Prerequisites Projects 1–3 or equivalent; pattern matching; recursion; processes and message passing; basic Kino controls/frames
Key Topics BEAM processes, mailboxes, links, monitors, supervision, restart identity, backpressure, bounded windows, lifecycle cleanup

1. Learning Objectives

By completing this project, you will:

  1. Design a process tree in which every process has one clear responsibility and state owner.
  2. Explain the behavioral difference among links, monitors, trapping exits, and supervision.
  3. Define versioned message contracts that identify sensor, producer generation, sequence, measurement time, and unit.
  4. Inject a producer crash and prove what the supervisor restarts, what state is lost, and what remains continuous.
  5. Reject delayed messages from an obsolete producer PID or generation after restart.
  6. Separate high-rate measurement ingestion from a slower UI rendering cadence.
  7. Bound mailbox pressure, aggregate history, frame data, and event logs while reporting coalesced/dropped counts.
  8. Make repeated notebook startup and orderly shutdown idempotent so re-evaluation does not duplicate systems.
  9. Verify fault recovery, overload behavior, restart intensity, client reconnect, and clean-runtime replay with observable evidence.

2. Theoretical Foundation

2.1 Core Concepts

BEAM processes and isolation. An Elixir process is a lightweight isolated execution context with its own mailbox and heap. Processes share no mutable application memory. They communicate by copying or sharing immutable message terms according to VM implementation details while preserving isolation semantics. A process should own the state it changes; callers interact through a protocol rather than reaching into that state.

Mailboxes and selective receive. Messages arrive asynchronously and wait in a process mailbox. The receiver decides which patterns to handle. A sender does not automatically know when work has completed, and messages can arrive after the world that produced them has changed. If producers outrun an aggregator, mailbox length and processing lag grow. Selective receive can also leave unmatched messages accumulating. “The BEAM can handle many processes” does not mean any mailbox is safe without a load policy.

Links. A link is bidirectional failure coupling. Under default semantics, an abnormal exit signal from one linked process causes the other to exit. Links are appropriate when components should fail together or when a supervisor needs child lifecycle signals. Trapping exits converts eligible exit signals into messages, but changes the process’s responsibility; it is not a generic protection switch.

Monitors. A monitor is unidirectional observation. The monitoring process receives a DOWN message when the target exits, but is not automatically terminated. A dashboard observer can monitor the supervised system to display lifecycle events without sharing its failure fate. Monitors may also report normal termination, and the observer must handle race conditions between replies, DOWN messages, and demonitoring.

Supervision. A supervisor starts children according to child specifications and restart strategy. It provides structured recovery, not persistence. After a sensor crashes, the restarted process has a new PID and initializes state from its start arguments or an external durable source. Its old sequence counter, random-generator state, and in-memory measurements do not magically return unless the design explicitly reconstructs them.

Restart strategies and intensity. one_for_one restarts only the failed child; one_for_all restarts all children when one fails; rest_for_one restarts the failed child and later siblings in start order. A maximum restart intensity prevents an endlessly crashing subtree from consuming resources. Exceeding it terminates the supervisor, escalating failure to its parent or monitor. The lab must make this terminal state visible rather than silently respawning forever.

Identity across restart. A logical sensor ID such as sensor-2 is stable, while a PID identifies one process incarnation. A producer generation or epoch distinguishes messages from successive incarnations. The aggregator should accept a reading only if its {sensor_id, generation} matches the current registration. This prevents a delayed message from an old PID from contaminating the new stream.

Aggregation and rendering cadence. Sensors may produce every 100 ms, while a human-facing dashboard needs updates only every second. The aggregator can ingest, validate, coalesce, and maintain a bounded window at the fast cadence, then publish a compact snapshot at the slower cadence. Rendering every reading increases browser traffic and frame history without improving comprehension.

Backpressure and overload policy. Bare process messaging has no automatic demand control. The lab must declare what happens when input exceeds processing: reject, sample, coalesce by sensor, drop oldest, slow producers, or move to a demand-driven abstraction. For a teaching dashboard, a bounded latest-value map plus fixed history window and reported coalesced count is often clearer than pretending no data was lost.

Notebook lifecycle. Livebook cells can be re-evaluated, runtimes can reconnect, and controls/listeners can outlive a binding while the current runtime remains alive. Startup needs a system handle that can identify supervisor, observer/listener, timers, and frames. Starting again must stop the old instance or return it deliberately. Shutdown must be safe to invoke multiple times.

2.2 Why This Matters

Fault tolerance is frequently described as “let it crash,” but that slogan omits the design work: choose failure boundaries, preserve the right state elsewhere, expose recovery, and prevent retry storms. This lab makes those choices visible. A sensor crash produces an old PID, exit reason, restart event, new PID, recovery interval, and continuity evidence. The learner sees exactly what supervision did and did not preserve.

The same architecture appears in ingestion systems, device gateways, job processors, websocket hubs, and monitoring agents. The load lessons generalize too: an unbounded mailbox is a hidden queue; a dashboard that appends forever is a memory leak; and a restarted producer can emit stale messages unless logical identity and incarnation are distinguished.

2.3 Historical Context / Background

Erlang/OTP was developed for telecommunications systems that needed massive concurrency, isolation, hot operation, and recovery from localized failure. Supervisors, generic servers, applications, and release structure turned recurring reliability patterns into reusable behaviors. Elixir exposes the same BEAM and OTP foundation with different language syntax, tooling, and metaprogramming.

The actor model predates Erlang and models independent entities communicating through messages. OTP adds pragmatic conventions around actors: standard lifecycle callbacks, child specifications, restart strategies, and hierarchical escalation. Modern observability systems add another concern: measuring recovery and overload without the observer becoming part of the failure path. Livebook and Kino provide an unusually direct way to visualize this behavior while learning it.

2.4 Common Misconceptions

  • “Supervision preserves process state.” It restarts a child from its specification. In-memory state is gone unless reconstructed from arguments or durable storage.
  • “A new PID means a new logical sensor.” PID identity is an incarnation; the logical sensor ID can remain stable across restarts.
  • “Message order is globally guaranteed.” Messages from one sender to one receiver preserve order, but multiple senders and restart incarnations create interleavings.
  • “Links and monitors are interchangeable.” Links couple failure; monitors observe it without automatic reciprocal failure.
  • “Mailbox size is just a performance metric.” Sustained mailbox growth is a correctness and availability risk because results become stale and memory grows.
  • “Dropping is always wrong.” Under overload, an explicit bounded coalescing/drop policy with counters may be safer than unbounded delay.
  • “Re-evaluating the cell replaces the old system.” It can start a second system unless startup/cleanup is deliberately idempotent.
  • “The dashboard surviving proves the whole system recovered.” You also need producer identity, accepted readings, lag, continuity, and restart-intensity evidence.

3. Project Specification

3.1 What You Will Build

Create telemetry-supervision-lab.livemd, a supervised simulation with four logical sensors. Each producer emits synthetic temperature readings. A registry/coordinator tracks the current incarnation of every sensor. One aggregator owns latest values, bounded history, and load counters. A dashboard observer renders snapshots to Kino frames and records lifecycle events.

The notebook provides:

  • A supervisor-tree diagram and failure matrix before startup.
  • Start, orderly stop, injection-target, and fault-injection controls.
  • Four sensor cards showing logical ID, current PID/generation, reading, sequence, age, and status.
  • A bounded time-series view or tabular frame containing the last 120 accepted points.
  • Mailbox, lag, received, accepted, coalesced/dropped, and stale-generation counters.
  • A bounded event transcript showing startup, injected exit, DOWN, restart, new identity, and recovery.
  • A high-rate overload fixture and a restart-intensity fixture.
  • An idempotent lifecycle handle suitable for notebook re-evaluation.

3.2 Functional Requirements

  1. Supervised producers: Start four independently restartable sensor processes under a documented strategy.
  2. Stable logical identity: Keep sensor-1 through sensor-4 stable while PID/generation changes on restart.
  3. Versioned messages: Every reading includes sensor ID, generation, sequence, monotonic measurement time, value, and unit.
  4. Single state owner: Exactly one aggregator owns the current window and load counters.
  5. Current-incarnation gate: Reject and count messages whose PID/generation is no longer current.
  6. Fault injection: A control terminates one selected producer with a deliberate abnormal reason.
  7. Visible recovery: Record old PID, reason, new PID, and first accepted post-restart reading/recovery duration.
  8. Continuity check: Other sensors continue producing and the aggregate/dashboard remain available during a single producer restart.
  9. Bounded history: Retain at most 120 accepted time-series points or an equivalently documented fixed window.
  10. Separated cadence: Ingest faster than UI rendering and publish snapshots on a fixed slower interval.
  11. Overload evidence: Display mailbox/lag plus received, accepted, coalesced/dropped, and stale-generation counts.
  12. Restart-intensity behavior: Repeated crashes eventually produce a visible escalated/terminal state according to policy.
  13. Idempotent lifecycle: Starting twice never creates duplicate producers/listeners; stopping twice is safe.
  14. Replay: A clean runtime rebuilds the whole system from source and leaves no reliance on prior PIDs.

3.3 Non-Functional Requirements

  • Fault isolation: One producer crash does not terminate the aggregator, observer, UI listener, or other producers.
  • Boundedness: Mailbox alert thresholds, history length, event log, frame dataset, and per-snapshot work are explicit.
  • Observability: Recovery and dropping/coalescing are measured, not inferred from a smooth-looking chart.
  • Responsiveness: Fault injection acknowledgement appears within one rendering interval; the restarted sensor produces within the declared recovery objective.
  • Determinism where useful: Training random sequences can be seeded per generation for repeatable tests, while monotonic timing remains measured evidence.
  • Cleanup: No producer, observer, listener, or timer survives an orderly stop in the same runtime.
  • Reviewability: Protocols, restart policy, failure matrix, thresholds, and tests are written before the implementation cells.

3.4 Example Usage / Output

Select sensor-2 and activate the fault control. The bounded event transcript shows one example:

14:32:10 sensor-2 PID<0.241.0> generation=1 temperature=21.8°C seq=43
14:32:12 action=inject_fault target=sensor-2 generation=1
14:32:12 DOWN PID<0.241.0> reason=simulated_failure
14:32:12 supervisor restart sensor-2 -> PID<0.255.0> generation=2
14:32:13 sensor-2 PID<0.255.0> generation=2 temperature=21.7°C seq=1

Stream continuity: PASS
Window: last 120 points
Dropped/coalesced events: 34 (reported)
Stale-generation events rejected: 1
Recovery time: 684 ms

The overload fixture produces an explicit degraded state rather than hiding lag:

LOAD STATUS: DEGRADED — COALESCING ACTIVE
received_per_second=400
rendered_snapshots_per_second=1
aggregator_mailbox=18 threshold=50
accepted_readings=2,410
coalesced_readings=1,936
dropped_readings=0
oldest_visible_age_ms=812
history_points=120/120

3.5 Real World Outcome

Open the notebook and study the supervisor tree and failure matrix before running the lab. Evaluate the startup section once. Four sensor cards become ready, each with a logical ID and current PID/generation. A time-series frame begins replacing its bounded dataset at the render cadence. A metrics strip reveals ingestion rate, render rate, mailbox size, lag, history occupancy, and loss/coalescing counters.

Choose sensor-2 and inject a fault. Its card changes to RECOVERING; the other three continue changing. The event transcript records the intentional action, old process exit, supervisor restart, new PID/generation, and first accepted new reading. The continuity and recovery checks move to PASS. A deliberately delayed generation-1 message is rejected after generation 2 registers, proving that logical identity alone is insufficient.

Increase the producer rate. The chart still displays at its human-scale cadence; counters show how many values were coalesced or dropped under the declared policy. History remains exactly bounded. Trigger the rapid-crash fixture and observe the supervisor exceed restart intensity, after which the lab renders an escalated terminal state and requires deliberate restart rather than looping invisibly.

Activate “Orderly stop.” Cards become stopped, timers cease, and a process audit confirms no lab-owned process remains. Evaluate startup twice and confirm it returns the existing system or replaces it cleanly—never two copies. Reconnect to a new runtime, replay from the top, and observe entirely new PIDs with the same logical architecture and tests.

4. Solution Architecture

4.1 High-Level Design

                              Livebook / Kino controls
                     start • stop • target • inject • load mode
                                       │
                                       v
┌──────────────────────────────────────────────────────────────────────┐
│ Lab lifecycle coordinator                                           │
│ owns system handle • idempotent start/stop • control event routing  │
└───────────────────────┬──────────────────────────────┬───────────────┘
                        │ start/stop                   │ monitor/snapshot
                        v                              v
             ┌──────────────────────┐       ┌────────────────────────┐
             │ Lab Supervisor       │       │ Dashboard observer     │
             │ strategy=one_for_one │       │ not failure-coupled to │
             │ restart intensity=N/T│       │ an individual producer │
             └──────┬───────────────┘       └────────────┬───────────┘
                    │                                     │ replace bounded frames
        ┌───────────┼───────────────┐                     v
        v           v               v          ┌────────────────────────┐
┌────────────┐ ┌────────────┐ ┌──────────────┐ │ Sensor cards + chart   │
│ sensor-1   │ │ sensor-2   │ │ sensor-3/4   │ │ metrics + event log    │
│ PID/gen/seq│ │ PID/gen/seq│ │ PID/gen/seq  │ └────────────────────────┘
└──────┬─────┘ └──────┬─────┘ └──────┬───────┘
       └──────────────┼───────────────┘
                      │ versioned reading messages
                      v
             ┌────────────────────────────────────────┐
             │ Aggregator — sole owner of stream state│
             │ current incarnations • latest values   │
             │ bounded 120-point window • counters    │
             │ coalescing/drop policy • snapshot tick │
             └──────────────────┬─────────────────────┘
                                │ compact snapshot at slower cadence
                                └──────────────────────────▶ observer

Failure path for sensor-2:

sensor-2 gen=1 exits
        │
        ├── supervisor receives linked child exit -> starts gen=2/new PID
        ├── lifecycle/observer receives monitor evidence -> renders recovery
        ├── aggregator updates accepted incarnation -> rejects delayed gen=1
        └── other producers, aggregator, and dashboard remain alive

4.2 Key Components

Component Responsibility Key Decisions
Lifecycle coordinator Start/stop whole lab and route controls One durable handle per runtime; idempotent operations
Lab supervisor Start and restart child processes one_for_one for independent sensors; documented intensity
Sensor producer Emit one sensor’s readings and generation/sequence Own only ephemeral producer state
Incarnation registry Map logical sensor ID to current PID/generation Update atomically around restart registration
Aggregator Validate messages and own bounded stream/window/counters Single writer; reject obsolete generations
Snapshot timer Trigger compact render snapshots Cadence independent of measurement rate
Dashboard observer Monitor lifecycle and render frames Observe without being linked to individual producer failure
Fault injector Target current sensor incarnation with deliberate abnormal exit Training-only, narrow action, explicit audit event
Process auditor Verify startup/stop ownership and counts Detect duplicates and cleanup leaks

4.3 Data Structures

SensorIncarnation = {
  sensor_id,
  pid,
  generation: positive_integer,
  started_monotonic_time,
  status: STARTING | RUNNING | RECOVERING | STOPPED
}

ReadingMessage = {
  protocol_version,
  sensor_id,
  producer_pid,
  generation,
  sequence,
  measured_monotonic_time,
  metric: TEMPERATURE,
  unit: CELSIUS,
  value
}

AggregatorState = {
  current_incarnations: map_sensor_to_pid_generation,
  latest_values: map_sensor_to_reading,
  history: ring_buffer_at_most_120,
  received_count,
  accepted_count,
  coalesced_count,
  dropped_count,
  stale_generation_count,
  invalid_message_count,
  last_snapshot_time
}

RecoveryEvent = {
  sensor_id,
  old_pid,
  old_generation,
  exit_reason_class,
  detected_time,
  new_pid,
  new_generation,
  first_reading_time,
  recovery_duration_ms
}

LabHandle = {
  supervisor_pid,
  coordinator_pid,
  observer_pid,
  listener_refs,
  timer_refs,
  frame_refs,
  protocol_revision
}

4.4 Algorithm Overview

Key Algorithm: Incarnation-Aware Bounded Aggregation

  1. On producer registration, record {sensor_id, pid, generation} as current and mark the sensor starting/running.
  2. On each reading, validate protocol version, sensor identity, unit, finite value, and sequence shape.
  3. Compare producer PID/generation with the current incarnation for that logical sensor.
  4. If obsolete, increment the stale-generation counter and discard the message.
  5. If current, update the sensor’s latest reading and accepted counter.
  6. Apply the overload policy: retain a fixed-size ring buffer, coalesce intermediate latest values, or drop by documented rule.
  7. On snapshot tick, create a compact immutable snapshot containing latest values, bounded history, mailbox/lag indicators, and counters.
  8. Send the snapshot to the dashboard observer and reset only interval-scoped counters if the policy requires it.
  9. On a monitored exit, correlate the old incarnation with supervisor restart registration and first new reading to calculate recovery.
  10. On orderly stop, stop timers/listeners, terminate the supervisor tree, clear frames/state, and make repeated stop a no-op success.

Complexity Analysis:

  • Per reading: expected O(1) validation, incarnation lookup, latest-value update, and ring-buffer operation.
  • Per snapshot: O(s + w) for s sensors and history window w ≤ 120.
  • Space: O(s + w + e) for sensors, bounded history, and bounded event log e.
  • Fault recovery: supervisor work is bounded by child start cost and restart policy, not history size.

5. Implementation Guide

5.1 Development Environment Setup

Use a current supported baseline such as Elixir 1.20.2, OTP 29.0.3, Livebook 0.19.8, and Kino 0.19.0, or record the actual compatible versions installed.

$ elixir --version
$ livebook --version
$ git --version
$ livebook server

Run the lab only with synthetic sensors. Do not connect fault-injection controls to real devices or production processes. Keep the notebook’s package setup and module definitions above the startup cell so clean replay reconstructs them.

5.2 Project Structure

telemetry-supervision-lab/
├── telemetry-supervision-lab.livemd
├── fixtures/
│   ├── deterministic-readings.fixture
│   ├── delayed-old-generation.fixture
│   ├── overload-profile.fixture
│   └── restart-intensity.fixture
├── evidence/
│   ├── fault-recovery-transcript.txt
│   ├── ten-minute-soak.txt
│   └── cleanup-audit.txt
└── README.md

As the domain core grows, extract process modules and ExUnit tests to a small Mix project; keep the notebook as the visual experiment and narrative.

5.3 The Core Question You’re Answering

“What does fault tolerance actually preserve, restart, discard, and reveal?”

Do not answer “the supervisor restarts the process.” Name the logical identity, PID, generation, in-memory state, downstream window, observer state, pending messages, counters, and durable evidence. For each, say whether it survives, is rebuilt, is deliberately lost, or must be rejected.

5.4 Concepts You Must Understand First

  1. Processes and mailboxes
    • Who owns aggregate state, and why should there be one writer?
    • What makes mailbox growth a form of unbounded queueing?
    • Book Reference: Elixir in Action, 3rd ed., Chapters 5–6.
  2. Links, monitors, and exit signals
    • Which relation observes failure and which couples it?
    • What changes when a process traps exits?
    • Primary References: Process and GenServer.
  3. Supervisors and restart strategy
    • What does one_for_one preserve compared with one_for_all?
    • What happens when restart intensity is exceeded?
    • Primary Reference: Supervisor.
  4. Identity and protocol design
    • Why are sensor ID, PID, generation, and sequence all different?
    • Which delayed messages must be rejected after restart?
  5. Kino event and frame lifecycle
    • What happens when production is faster than rendering?
    • How do you stop a listener created by a re-evaluated cell?
    • Primary References: Kino.Control and Kino.Frame.

5.5 Questions to Guide Your Design

  1. Which process owns current incarnation mappings?
  2. Which child processes belong under the same supervisor, and why?
  3. Should an aggregator crash restart producers, or remain independently recoverable?
  4. What exact reading message shape is accepted?
  5. How will the aggregator reject a valid-shaped but obsolete message?
  6. Which state is derived and safe to lose on aggregator restart?
  7. Which state, if any, would need durable persistence in a real system?
  8. What input rate, render rate, mailbox threshold, window bound, and drop/coalescing policy are declared?
  9. How does the UI learn about child exit without sharing the child’s failure fate?
  10. What happens after restart intensity is exceeded?
  11. How will startup discover and stop an older lab instance?
  12. What exact process evidence proves orderly stop completed?

5.6 Thinking Exercise

Draw the supervisor tree and complete this failure matrix before implementation:

process            crash result       restarted?   state preserved?   UI evidence
sensor-2           learner predicts   predict      predict            predict
aggregator         learner predicts   predict      predict            predict
dashboard observer learner predicts   predict      predict            predict
lab supervisor     learner predicts   predict      predict            predict

Then trace this sequence:

  1. sensor-2 generation 1 sends sequence 43.
  2. Fault injection terminates its PID.
  3. A delayed generation-1 sequence 44 is already in transit.
  4. The supervisor starts generation 2 with sequence 1.
  5. The delayed old message reaches the aggregator.
  6. Generation 2’s first reading arrives.

Predict accepted/stale counters, card state, history entries, PIDs, and recovery duration at every step.

5.7 The Interview Questions They’ll Ask

  1. “How do links and monitors differ?”
  2. “Why does supervision not equal persistence?”
  3. “What causes mailbox growth, and how would you detect it?”
  4. “Why distinguish logical identity from PID?”
  5. “How do you prevent delayed messages from an old process incarnation?”
  6. “How do you avoid duplicated notebook listeners?”
  7. “What should happen after restart intensity is exceeded?”
  8. “When is dropping or coalescing telemetry preferable to unbounded buffering?”

5.8 Hints in Layers

Hint 1: One state owner. Make the aggregator the sole owner of latest readings, bounded history, and stream counters.

Hint 2: Start with one sensor. Prove protocol, normal stop, abnormal exit, and restart identity before scaling to four.

Hint 3: Version producers. Register each sensor incarnation and include both generation and PID in messages.

Hint 4: Separate measurement and rendering. Aggregate fast readings and render compact snapshots less often.

Hint 5: Make loss visible. A coalesced or dropped value must increment a named counter.

Hint 6: Preserve a complete handle. Startup should return everything required for idempotent orderly stop.

Hint 7: Test the terminal path. Repeated fault injection must eventually exercise restart-intensity escalation rather than only happy recovery.

5.9 Books That Will Help

Topic Book Chapter
Processes and servers Elixir in Action, 3rd ed. Chapters 5–6
Fault tolerance and supervision Elixir in Action, 3rd ed. Chapters 7–9
Supervision-oriented design Designing Elixir Systems with OTP Chapters 3–8
Load and stability Release It!, 2nd ed. Stability, queues, timeouts, and capacity chapters
Distributed/actor reasoning Programming Erlang, 2nd ed. Concurrency and OTP chapters

5.10 Implementation Phases

Phase 1: Foundation (4–6 hours)

Goals: Define topology, protocols, state ownership, and one supervised sensor.

Tasks:

  1. Draw the supervisor tree and complete the failure matrix.
  2. Define reading, registration, snapshot, lifecycle, and recovery schemas.
  3. Build one producer and aggregator with a fixed bounded window.
  4. Inject one crash and observe old/new PID without Kino rendering.

Checkpoint: A transcript proves new process identity, explicit state loss, and one accepted post-restart reading.

Phase 2: Core Functionality (7–10 hours)

Goals: Add four sensors, generation gating, snapshots, dashboard, and controls.

Tasks:

  1. Scale to four supervised logical sensors with independent generations.
  2. Add delayed-old-generation rejection and counters.
  3. Add slower snapshot cadence and bounded Kino frames.
  4. Add client controls for target selection, fault injection, rate, start, and stop.
  5. Correlate exit/restart/first-reading evidence into recovery events.

Checkpoint: One injected producer fault shows continuity, old/new identities, stale-message rejection, and bounded frames.

Phase 3: Polish & Edge Cases (5–8 hours)

Goals: Prove overload, terminal escalation, lifecycle, soak stability, and replay.

Tasks:

  1. Add overload fixture, explicit coalescing/drop policy, and mailbox/lag display.
  2. Add restart-intensity fixture and terminal-state rendering.
  3. Make startup/stop idempotent and audit all owned processes/timers.
  4. Run a ten-minute high-rate soak and clean-runtime replay.
  5. Record evidence for duplicate startup, reconnect, and orderly stop.

Checkpoint: All critical tests and Definition of Done checks pass with a saved evidence transcript.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Supervisor strategy one_for_one, rest_for_one, one_for_all one_for_one for independent sensors One sensor failure should not interrupt others
Observer relation Link to producer, monitor system Monitor UI observes lifecycle without sharing producer failure fate
Logical identity PID only, sensor ID only, ID + generation/PID ID + generation/PID Stable identity plus incarnation safety
Stream owner Each frame, multiple consumers, one aggregator One aggregator Makes invariants, bounds, and counters authoritative
Overload policy Unbounded mailbox, block sender, coalesce/sample/drop Bounded coalescing with counters for this lab Preserves current signal and makes loss explicit
Render cadence Every reading, fixed snapshot tick Fixed slower tick Protects browser and communicates meaningful change
Restart state Pretend persistence, reconstruct explicitly Explicit reconstruction/loss Teaches true supervision semantics
Notebook startup Always spawn, idempotent handle Idempotent handle Prevents duplicated systems on re-evaluation

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Protocol tests Validate accepted/rejected messages wrong version, unit, sensor ID, generation
Process tests Verify state ownership and replies sequence, latest value, bounded history
Supervision tests Prove restart behavior abnormal exit, normal stop, restart intensity
Concurrency tests Exercise message timing delayed old generation, interleaved sensors
Load tests Verify bounded behavior high producer rate, mailbox threshold, coalescing counters
UI integration tests Verify observable state recovering card, PID change, continuity, terminal state
Lifecycle tests Detect leaks/duplicates start twice, stop twice, reconnect, process audit

6.2 Critical Test Cases

  1. Normal stream: Four sensors emit accepted readings; sequence/generation and units are valid; window never exceeds 120.
  2. Single injected crash: Old PID exits with simulated_failure, new PID/generation appears, and first new reading meets recovery objective.
  3. Continuity: Other three sensors, aggregator, and dashboard retain PID continuity and keep advancing during one producer restart.
  4. Delayed old message: A generation-1 reading arriving after generation 2 registration is rejected and counted.
  5. High-rate overload: History and frame remain bounded; coalesced/dropped counters increase according to policy; lag remains visible.
  6. Restart intensity: Rapid repeated crashes exceed policy, terminate/escalate the subtree, and render a terminal state without an infinite loop.
  7. Duplicate startup: Two start actions produce one active supervisor tree, one observer, and one event per producer interval.
  8. Orderly stop: Two stop actions are safe; process/timer audit finds no lab-owned survivors.
  9. Clean replay: Reconnect yields new PIDs and empty ephemeral state while reconstructing the same logical topology and tests.

6.3 Test Data

NORMAL
sensors=[sensor-1,sensor-2,sensor-3,sensor-4]
producer_interval_ms=250 snapshot_interval_ms=1000 history_limit=120
expected active_sensors=4 invalid_messages=0

FAULT
target=sensor-2 old_generation=1 exit_reason=simulated_failure
expected new_generation=2 new_pid_different=true
expected other_sensor_sequence_continues=true

STALE
message sensor=sensor-2 generation=1 sequence=44 arrives_after_generation=2_registration
expected accepted=false stale_generation_count_delta=1

OVERLOAD
combined_input_per_second=400 snapshot_per_second=1 duration_seconds=60
expected history_size_max=120
expected coalesced_or_dropped_count_positive=true
expected unbounded_growth=false

LIFECYCLE
actions=[start,start,stop,stop]
expected active_lab_supervisors_after_start=1
expected lab_owned_processes_after_stop=0

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Duplicate startup Every reading or event appears twice Stop prior handle or return the existing initialized lab
Unbounded history/frame Memory and browser cost grow during demo Ring-buffer data, replace frame, cap event transcript
PID treated as logical ID Restart appears as a new sensor or old messages pass Track stable sensor ID plus current generation/PID
UI linked to producer Dashboard dies with injected sensor Observe via monitors and intended supervisor boundaries
Supervision assumed persistence Sequence/window unexpectedly resets Declare ephemeral state and reconstruct only from explicit sources
Rendering every event Listener/browser falls behind Aggregate fast, snapshot slowly, report loss/coalescing
No terminal-state design Crash loop consumes resources Configure/test restart intensity and render escalation

7.2 Debugging Strategies

  • Display identities: Include logical ID, PID, generation, and sequence in the bounded transcript.
  • Inspect mailbox length and lag together: A small mailbox can still contain old work; age reveals freshness.
  • Trace one failure timeline: Record injection, exit detection, restart registration, and first accepted reading with monotonic times.
  • Audit process ownership: Tag/namescope lab processes so startup/stop counts can be checked without scanning unrelated runtime work.
  • Inject deterministic stale messages: Do not wait for a rare race; schedule an old-generation fixture explicitly.
  • Test without UI first: If supervision/protocol tests pass headlessly, investigate Kino routing/render cadence separately.

7.3 Performance Traps

Unbounded mailboxes and frame histories are the central traps. Avoid expensive formatting in the aggregator receive loop; snapshot immutable compact state and render elsewhere. Do not calculate chart history from all events on every tick. A very small render interval can serialize the listener behind browser work. Monitoring every ephemeral process without demonitoring can leak references/messages. During soak tests, observe process count, mailbox length, history size, snapshot duration, and browser responsiveness together.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add humidity as a second metric with explicit unit/protocol validation.
  • Add a card explaining which state resets on each sensor restart.
  • Add pause/resume controls that do not confuse normal stop with failure.

8.2 Intermediate Extensions

  • Compare one_for_one, rest_for_one, and one_for_all with separate failure transcripts.
  • Add aggregator checkpoint/restore and clearly distinguish it from supervisor behavior.
  • Add property tests for ring-buffer bounds and stale-generation rejection.

8.3 Advanced Extensions

  • Replace push messaging with GenStage/Broadway-style demand and compare overload evidence.
  • Distribute producers to a second disposable BEAM node and model disconnect/reconnect generations.
  • Emit standard :telemetry events and attach bounded metrics handlers with correct detach lifecycle.
  • Package the process tree as an OTP application and use Livebook only as an observer/control plane.

9. Real-World Connections

9.1 Industry Applications

  • IoT gateways: Maintain logical device identity through reconnects and process restarts.
  • Telemetry ingestion: Bound queues, coalesce high-rate measurements, and surface freshness/loss.
  • Job processors: Supervise workers while keeping job durability outside ephemeral worker state.
  • Websocket systems: Isolate connection failures and reject messages from obsolete session incarnations.
  • Operational dashboards: Render compact snapshots without joining the observed component’s failure fate.
  • Elixir: Process, GenServer, Supervisor, and DynamicSupervisor abstractions.
  • Livebook: Executable environment used to observe and control the lab.
  • Kino: Event controls and dynamic frames.
  • Telemetry: Standard event/measurement dispatch used across the BEAM ecosystem.
  • Broadway: Concurrent multi-stage data ingestion with demand and acknowledgements for advanced comparison.

9.3 Interview Relevance

  • OTP architecture: Defend supervisor strategy and failure boundaries.
  • Concurrency correctness: Explain mailboxes, message interleaving, and incarnation-aware protocols.
  • Reliability: Distinguish restart from persistence and discuss escalation.
  • Performance: Diagnose unbounded queues and justify backpressure/coalescing policy.
  • Lifecycle: Show how notebook re-evaluation can leak duplicate processes and how to prevent it.

10. Resources

10.1 Essential Reading

  • Elixir in Action, 3rd ed., by Saša Jurić — Chapters 5–9 for processes, servers, supervision, and fault tolerance.
  • Designing Elixir Systems with OTP by James Edward Gray II and Bruce A. Tate — Chapters 3–8 for supervision-oriented design.
  • Release It!, 2nd ed., by Michael T. Nygard — stability, timeouts, queues, and capacity.
  • Programming Erlang, 2nd ed., by Joe Armstrong — concurrency and OTP foundations.

10.2 Video Resources

10.3 Tools & Documentation

11. Self-Assessment Checklist

Before considering this project complete, verify:

11.1 Understanding

  • I can explain links, monitors, exit signals, and supervision without conflating them.
  • I can state exactly which state survives and which disappears after a sensor restart.
  • I can explain logical ID, PID, generation, and sequence as distinct identities.
  • I can explain how mailbox growth becomes stale work and memory pressure.
  • I can defend the selected restart strategy, intensity, and overload policy.

11.2 Implementation

  • Supervisor tree and message contracts are documented.
  • One crash shows old PID, reason, new PID/generation, and recovery duration.
  • Other sensors and aggregate output continue through one producer fault.
  • Delayed obsolete-generation messages are rejected and counted.
  • Queue/history/frame bounds and coalesced/dropped counts are visible.
  • Restart-intensity escalation renders an intentional terminal state.
  • Repeated startup and orderly stop are idempotent.
  • Ten-minute soak, cleanup audit, reconnect, and clean replay pass.

11.3 Growth

  • I documented one incorrect assumption I previously had about supervision.
  • I can explain when this design should move from raw messages to demand-driven ingestion.
  • I can present the failure timeline and evidence in a technical interview.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Four logical sensors run under a documented supervisor and feed one bounded aggregator.
  • Fault injection restarts one producer with a new PID while other producers continue.
  • The dashboard displays current identity, bounded history, and one recovery transcript.
  • Startup and stop are safe to execute repeatedly.

Full Completion:

  • All minimum criteria plus:
  • Reading protocol includes version, logical ID, PID/generation, sequence, time, value, and unit.
  • Delayed old-generation rejection, overload counters, render cadence, and 120-point bound are proven.
  • Restart-intensity terminal behavior, ten-minute soak, cleanup audit, and clean-runtime replay pass.
  • Evidence names what supervision restarted, discarded, preserved, and exposed.

Excellence (Going Above & Beyond):

  • Headless ExUnit/property tests cover process protocols, ring-buffer bounds, and generation safety.
  • An OTP application package separates production-grade process logic from the Livebook observer.
  • A demand-driven ingestion extension compares throughput, latency, and loss with the raw-message design.
  • A disposable distributed-node extension demonstrates node loss, reconnect, and incarnation handling without production access.

This guide was generated from LEARN_LIVEBOOK_ELIXIR_DEEP_DIVE.md. For the complete learning path, see the parent directory.