Sprint: Livebook + Elixir Mastery - Real World Projects

Goal: Learn Elixir by making its data transformations, processes, failures, and numerical workloads visible inside Livebook. You will understand how Livebook turns a Markdown document into a reproducible evaluation graph, how Kino turns BEAM processes into interactive interfaces, and how runtimes define the trust boundary around executed code. Across twelve projects you will progress from a deterministic notebook to data pipelines, concurrent control rooms, custom Smart Cells, numerical models, attached-node diagnostics, and a secured deployed app. The final outcome is not a pile of experiments: it is a portfolio of executable documents that can be replayed, reviewed, operated, and promoted safely.

Introduction

Livebook is an interactive and collaborative notebook environment built for Elixir. A notebook combines narrative, diagrams, code cells, package declarations, inputs, and rich outputs in a Live Markdown file with the extension .livemd. The important difference from a loose REPL transcript is that Livebook tracks cell dependencies and stale state, gives each notebook a runtime, supports real-time collaboration, and can present the same notebook as a code-hidden web application.

This guide uses Livebook as a microscope for Elixir. Pattern matching becomes visible as accepted and rejected rows. Immutable transformations become inspectable stages. BEAM processes become live telemetry producers. Supervision becomes a PID change you can see after an injected crash. Explorer query plans become a measurable data pipeline. Nx shapes and compiled numerical functions become evidence rather than magic.

What you will build

  • Reproducible Live Markdown field reports and validation workbenches
  • A fault-tolerant telemetry lab with visible process recovery
  • Explorer pipelines, safe SQL diagnostics, and honest interactive charts
  • Portable local and S3-compatible data workflows
  • A tested domain-specific Smart Cell
  • A numerical and pretrained-model triage lab
  • A read-only attached-runtime console for a disposable Phoenix node
  • A secured, multi-session operations app deployed from a notebook

In scope

  • Livebook v0.19-era notebook semantics, runtimes, apps, and deployment model
  • Elixir data modeling, tagged results, streams, processes, Tasks, and supervision
  • Kino inputs, controls, frames, layouts, renderers, event streams, and Smart Cells
  • Explorer, VegaLite, Nx, and one carefully bounded Bumblebee workflow
  • Reproducibility, provenance, secrets, stamping, authentication, and audit evidence

Out of scope

  • A complete Elixir language reference or replacement for an OTP-focused guide
  • Unrestricted production shells or mutating production runbooks
  • Training large neural networks, distributed model training, or GPU administration
  • Full Phoenix, Ecto, Nerves, or distributed-systems mastery
                        AUTHORING PLANE
┌──────────────────────────────────────────────────────────────────┐
│ .livemd = prose + setup + code cells + sections + app settings  │
└──────────────────────────────┬───────────────────────────────────┘
                               │ connect
                               v
                         RUNTIME PLANE
┌──────────────────────────────────────────────────────────────────┐
│ standalone VM | attached node | Fly.io machine | Kubernetes pod │
│ bindings + modules + processes + dependencies + external access │
└───────────────┬──────────────────────┬───────────────────────────┘
                │                      │
                v                      v
        COMPUTATION PLANE        EXPERIENCE PLANE
┌──────────────────────────┐  ┌────────────────────────────────────┐
│ Elixir / OTP / Explorer  │  │ Kino outputs / controls / frames  │
│ Nx / HTTP / SQL / files  │  │ collaboration / Livebook Apps     │
└───────────────┬──────────┘  └──────────────────┬─────────────────┘
                └───────────────┬─────────────────┘
                                v
                        GOVERNANCE PLANE
┌──────────────────────────────────────────────────────────────────┐
│ clean replay | provenance | secrets | stamps | auth | audit     │
└──────────────────────────────────────────────────────────────────┘

How to Use This Guide

  1. Read the eight theory chapters before choosing a project. Each later concept table row maps to exactly one chapter.
  2. If Elixir is new, follow Projects 1 through 4 in order. They deliberately move from values to events to supervised processes.
  3. Begin every project by writing its input contract, observable outcome, and failure budget before creating cells.
  4. Use a two-pass notebook workflow: explore freely, then restart the runtime and make the final notebook evaluate cleanly from top to bottom.
  5. Treat screenshots as supplementary evidence. The primary evidence is the saved .livemd file, a clean replay transcript, exported artifacts, and explicit invariant checks.
  6. For live systems, use a disposable local node first. Projects 6, 11, and 12 are read-only by default.
  7. Read hints only after recording your own hypothesis. A hint should unblock reasoning, not replace it.

Every project has three completion levels:

  • Minimum proves the central concept with a bounded dataset and one happy path.
  • Full includes failure states, clean-runtime replay, provenance, and tests.
  • Excellence adds scale, packaging, or deployment without widening privileges carelessly.

Prerequisites & Background Knowledge

Essential Prerequisites (Must Have)

  • Comfort using a terminal, editing Markdown, and reading structured data such as CSV and JSON
  • Basic programming ideas: values, functions, conditionals, collections, and errors
  • Basic Git workflow: diff, commit, and reviewing text changes
  • Recommended Reading: The Pragmatic Programmer, 20th Anniversary Edition, Topic 21 “Text Manipulation” and Topic 37 “Listen to Your Lizard Brain”

Helpful But Not Required

  • Prior Elixir or Erlang experience; Projects 1–4 teach the required subset
  • SQL and data analysis; Projects 5–7 introduce the necessary workflow
  • OTP supervision; Project 4 makes the model observable
  • Machine learning; Project 10 stays focused on shapes, inference contracts, and evaluation
  • Docker and CI/CD; Project 12 introduces the operational minimum

Self-Assessment Questions

  1. Can you explain why reassigning a name is not the same as mutating a value?
  2. Can you distinguish an expected domain failure from an exceptional crash?
  3. Can you identify which external inputs make an analysis non-reproducible?
  4. Can you explain why an HTTP 200 response may still contain unusable data?
  5. Can you describe the difference between a process, an operating-system process, and an Elixir Task?
  6. Can you state why a chart can be technically correct yet misleading?
  7. Can you explain why attaching a notebook to a production node changes its risk profile?

If questions 1–3 are unfamiliar, start with Projects 1 and 2. If questions 4–6 are unfamiliar, do Projects 3, 5, and 7 before the advanced track. If question 7 is unclear, do not skip the runtime and security chapters.

Development Environment Setup

Required Tools

  • Livebook Desktop or a self-hosted Livebook matching the current stable documentation
  • Git
  • A browser with two separate windows or profiles for collaboration tests
  • Docker for Project 12

The official Livebook v0.19.8 documentation states that direct installation needs Elixir 1.18 or later. Desktop and Docker users receive a compatible runtime. Do not pin every package in this guide to the versions shown in July 2026; use Livebook package search and verify compatibility when starting each project.

Verified ecosystem snapshot — July 15, 2026

Component Current stable observed in official release/package sources Role in this guide
Elixir 1.20.2 Language and standard library
Erlang/OTP 29.0.3 BEAM runtime
Livebook 0.19.8 Notebook, runtimes, collaboration, apps
Kino 0.19.0 Rich output and interaction
Explorer 0.12.0 DataFrames and analytical I/O
Nx 0.12.1 Tensors and numerical definitions
Bumblebee 0.7.0 Pretrained-model integration

This table is a research snapshot, not a lockfile. Project manifests must record the actual resolved compatible versions used in each completed notebook. Version evidence: Livebook v0.19.8, Elixir v1.20.2, Erlang/OTP 29.0.3, and the official Hex package registry.

Recommended Tools

  • SQLite or PostgreSQL with a disposable training database
  • A local S3-compatible service for Project 8
  • A disposable Phoenix application for Project 11
  • Optional EXLA-compatible CPU/GPU environment for numerical comparisons

Testing Your Setup

$ livebook server
[Livebook] Running at http://localhost:8080/?token=REDACTED

Browser -> New notebook -> Runtime
Status: Connected
Runtime type: Standalone

Evaluate a cell containing a version query
Expected: an Elixir and OTP version tuple is rendered without an exception

Never copy a real access token into project evidence. Redact tokens, cookies, passwords, connection strings, and personal data.

Time Investment

  • Foundation projects 1–3: 5–10 hours each
  • Interactive and data projects 4–8: 10–20 hours each
  • Extension and runtime projects 9–11: 15–30 hours each
  • Capstone Project 12: 30–50 hours
  • Full sprint: roughly 4–6 months part time

Important Reality Check

Notebook success can be deceptive. A cell may work because an old binding, process, downloaded artifact, or compiled dependency remains in memory. Interactive work also hides concurrency bugs until a second client connects. Your strongest habit in this guide is destructive verification: reconnect the runtime, delete caches when appropriate, use fresh input, and replay from the top. If the outcome cannot be reproduced after the comfortable state is removed, it is not done.

Big Picture / Mental Model

A Livebook notebook has two kinds of state that must never be confused:

  1. Document state: the saved .livemd content, cell order, metadata, app settings, and stamps.
  2. Runtime state: bindings, loaded modules, installed packages, processes, open connections, downloaded files, and caches.

The document may survive while the runtime disappears. The runtime may contain facts not represented in the document. Reproducibility means the saved document plus declared inputs can reconstruct the required runtime state.

                    SAVE                              RECONNECT
          ┌────────────────────┐               ┌────────────────────┐
          │ Document state     │ persists      │ New runtime state  │
          │ .livemd + settings │──────────────▶│ initially empty    │
          └─────────┬──────────┘               └──────────┬─────────┘
                    │                                     │
                    │ setup + dependency graph            │ evaluate
                    └──────────────────┬──────────────────┘
                                       v
                          ┌─────────────────────────┐
                          │ Reconstructed outcome   │
                          │ tables, UI, files, logs │
                          └─────────────────────────┘

Reproducibility invariant:
saved document + declared inputs + compatible environment = verified outcome

The second mental model is a trust map. Livebook is an execution environment, not a static document viewer. Code in a standalone runtime can access what that runtime’s operating-system identity can access. Code in an attached runtime executes inside the external node. A beautiful UI does not reduce authority.

LOWER COUPLING                                             HIGHER COUPLING

standalone notebook ──narrow API/DB──▶ service
        │
        ├── isolated VM and dependencies
        └── explicit credentials and network boundary

attached notebook ───────────────────▶ running service node
        │
        ├── direct access to loaded modules and process state
        └── mistakes execute inside the target application

Preferred production default: standalone + least-privilege read-only interfaces.

Theory Primer

The following eight chapters are the conceptual spine of the sprint. The Concept Summary Table later in this guide uses the same eight names and no additional concept clusters.

Theory Chapter 1: Livebook Execution and Reproducibility

Fundamentals

A Livebook notebook is an executable document, but the file and the execution are separate things. The .livemd file stores Markdown, cell source, section structure, notebook metadata, and app settings. A runtime supplies the evaluator, bindings, loaded modules, installed dependencies, processes, and access to external resources. Ordinary sections form an evaluation path: a cell may use bindings produced earlier on that path. When upstream source changes, Livebook marks dependent work stale because the visible output may no longer correspond to the current program. The implicit Setup section establishes packages and configuration before ordinary work begins. A reproducible notebook can connect to a fresh compatible runtime, obtain declared inputs, evaluate in the intended order, and produce outcomes that satisfy the same invariants.

Deep Dive

Thinking of a notebook as “a script split into boxes” is useful only at the beginning. A more accurate model is a dependency-aware program with a human-readable presentation layer. Livebook records cell order and observes which bindings each evaluated cell produces and consumes. This is why changing a source cell can make later cells stale without immediately running them. Stale is a truthfulness signal: the old output still exists on screen, but it is evidence from an older program state.

The Setup section is the root of the evaluation graph. Package declarations, dependency configuration, deterministic random seeds, and resource initialization belong there when they are prerequisites for most of the document. This does not mean every side effect belongs in Setup. Opening a database transaction, starting a timer, or downloading a huge model there can make every reconnect slow or leave resources alive longer than intended. Setup should declare the environment; later sections should own bounded work.

Sections are more than headings. An ordinary section extends the main evaluation path. A branching section chooses a parent section and then evaluates independently of its siblings. Branching is valuable for long-running alternatives, experiments that should not invalidate the main narrative, and failure isolation. It is not a substitute for modules or processes. A branch still has an explicit ancestry, and a learner must be able to explain which bindings cross the boundary.

Top-level notebook expressions optimize for exploration. Significant reusable work should move into named modules and functions. This improves stack traces, permits direct testing, clarifies inputs and outputs, and avoids repeatedly interpreting heavy logic at the top level. The document should reveal the important orchestration while the module defines the stable behavior. This is the bridge from a notebook prototype to normal Elixir engineering.

Dependencies are part of reproducibility. In a standalone runtime, Mix.install establishes a per-notebook package environment. A package name without a compatible version policy makes tomorrow’s run ambiguous; an exact pin without an update policy makes the notebook brittle. Record the resolved versions in an evidence section, keep a known-good snapshot for important workflows, and deliberately test upgrades. An attached runtime is different: code runs inside an existing Elixir node, so the notebook cannot freely install dependencies and must use what the target already provides.

External state completes the equation. A live HTTP response, database table, local file, current clock, random generator, and environment variable can all change without a notebook diff. Every serious notebook should name its input mode: live, snapshotted, generated with a seed, or user supplied. A manifest should record retrieval time, source identity, schema expectations, checksum where possible, and redaction policy. Reproducing an exact historical result requires the snapshot; reproducing the method against current data requires a clearly labeled live mode.

Testing a notebook means more than evaluating all cells once. First test individual functions with doctest-style examples or an explicit ExUnit suite. Then reconnect the runtime and evaluate from the top. Next test a declared failure: missing file, empty dataset, unavailable service, or incompatible schema. Finally inspect the saved diff and verify that outputs or stamps have not introduced secrets or noisy nondeterminism. A notebook is trustworthy when its success and failure conditions are both visible.

How This Fits on Projects

Projects 1 and 2 establish clean replay and stale-state literacy. Projects 3, 5, 8, 10, and 12 add changing external resources, while Projects 9 and 12 turn the document into a reusable tool.

Definitions and Key Terms

  • Live Markdown: the readable .livemd format used by Livebook.
  • Runtime: the processes and Elixir node responsible for evaluating notebook code.
  • Binding: a name-to-value association available on an evaluation path.
  • Stale cell: a cell whose displayed result may not reflect changed upstream source.
  • Setup section: the implicit root section for dependencies and shared initialization.
  • Branching section: a section evaluated from a chosen parent and independently of sibling branches.
  • Clean replay: reconnecting and evaluating from declared inputs with no hidden prior state.

Mental Model Diagram

                         SETUP
                  packages + shared config
                           │
                ┌──────────┴──────────┐
                v                     v
          Main Section          Branch: benchmark
          input -> clean        inherits setup
                │                     │
                v                     v
          aggregate -> chart     alternative path
                │
                v
            final report

Change input source:
main descendants = STALE
independent branch = unchanged unless it consumed that input

How It Works

  1. Connect a runtime and evaluate Setup.
  2. Livebook evaluates a cell with bindings from its valid ancestors.
  3. The result and new bindings become available to later cells.
  4. Editing an ancestor invalidates the evidence chain and marks descendants stale.
  5. Re-evaluation reconstructs results along the affected path.
  6. Reconnect destroys runtime state; clean replay proves the document declares enough to rebuild it.

Invariants: a cell never silently receives a binding from a later cell; a saved output is not proof that current source produced it; a clean runtime begins without prior notebook processes or bindings.

Failure modes: hidden local files, unpinned incompatible packages, current-time defaults, branch ancestry mistakes, a dependency installed interactively but absent from Setup, and long-lived listeners duplicated after partial re-evaluation.

Minimal Concrete Example

SETUP:
  declare parser and table dependencies
  record compatible versions

SECTION "Input":
  source_mode = snapshot
  source_checksum = abc123...

SECTION "Transform":
  parsed = parse(source)
  assert parsed_count = accepted_count + rejected_count

SECTION "Evidence":
  render manifest
  report clean_replay = PASS

Common Misconceptions

  • “Evaluate all passed, therefore the notebook is reproducible.” It may have reused files, caches, or processes.
  • “Sections are visual only.” Branching sections change evaluation ancestry and failure isolation.
  • “The file contains the runtime.” The file describes how runtime state should be rebuilt.
  • “A green output is current.” Livebook’s stale marker exists precisely because it may not be.

Check-Your-Understanding Questions

  1. Why can two identical .livemd files produce different results?
  2. When should a long-running alternative use a branch rather than the main path?
  3. What evidence distinguishes an exact historical replay from a current-data rerun?

Check-Your-Understanding Answers

  1. Runtime versions, packages, external inputs, secrets, clock, randomness, and hidden files may differ.
  2. Use a branch when it inherits a stable base but should evaluate independently of sibling work or isolate a crash.
  3. An exact replay names a snapshot and checksum; a current-data rerun names retrieval time and live provenance.

Real-World Applications

Executable tutorials, incident reports, compliance evidence, data-quality reports, model cards, package documentation, and repeatable operational diagnostics.

Where You Will Apply It

Project 1, Project 2, Project 3, Project 5, Project 8, Project 10, and Project 12.

References

Key Insight

The saved notebook is a recipe and evidence trail, not a frozen runtime.

Summary

Treat cell order, section ancestry, packages, and input provenance as program structure. Clean replay converts an attractive notebook into a reproducible one.

Homework / Exercises

  1. Draw the evaluation graph for a notebook with one setup, two ordinary sections, and two branches.
  2. List five hidden inputs in a notebook you already use.
  3. Design a replay manifest without writing implementation code.

Solutions

  1. Both ordinary sections form the main ancestry; each branch names the point it inherits and has no dependency on its sibling.
  2. Typical answers: clock, environment variables, local files, HTTP state, database state, random seed, package resolution, locale, and cached model files.
  3. Include notebook revision, runtime versions, dependency versions, source mode, source identity, retrieval time, checksum, seed, output invariants, and replay result.

Theory Chapter 2: Functional Elixir and Data Modeling

Fundamentals

Elixir programs transform immutable terms. Names may be rebound, but existing values do not change in place. Pattern matching describes the shape a value must have and extracts parts of that shape. Guards refine which patterns are valid. Expected outcomes are commonly modeled with tagged tuples such as success-with-value or error-with-reason, while exceptional failures raise and can crash a process. Lists, tuples, maps, structs, and binaries each communicate different constraints. Enum eagerly traverses finite enumerables; Stream builds lazy transformations consumed later. The pipe operator expresses a sequence of transformations when each step has a clear input and output. These ideas are especially visible in Livebook because every stage can render its contract and counterexamples.

Deep Dive

Immutability changes the central design question from “who changed this object?” to “which transformation produced this new value?” A capacity plan can preserve its raw assumptions, normalized inputs, scenario outputs, and final recommendation simultaneously. That property is excellent for notebooks because intermediate evidence remains inspectable. Rebinding is only a new association between a name and a value; another process or earlier binding does not observe a mutated object.

Pattern matching combines validation and decomposition. A function head can accept a map with required keys, distinguish an empty collection from a nonempty one, or separate successful from failed results. However, a pattern is not a complete schema. Matching on a key proves presence and immediate shape, not that a date is real, a decimal is within policy, or a string is safe. Domain validation should return precise error data rather than depend on a MatchError for ordinary bad input.

Tagged results make branching explicit. A transformation pipeline that may fail should preserve the distinction between expected rejection and programmer defect. A malformed customer row is expected data and belongs in a rejected collection with reasons. A misspelled module name or violated internal invariant is a defect and may deserve a crash. Turning every exception into a generic error tuple hides defects; raising on every user mistake makes bulk workflows brittle.

Data structures are design language. A tuple is compact and positional, suitable for fixed tagged records. A map is flexible and useful at boundaries. A struct defines a known domain shape and participates in protocols. Lists excel at sequential construction but have linear random access. MapSet communicates uniqueness. Binaries carry strings and arbitrary bytes; string operations must respect Unicode rather than assume one byte per character. Dates and money should use domain-appropriate representations instead of floats and ad hoc strings.

Enum and Stream express different evaluation contracts. Enum is direct and predictable for bounded in-memory data. Stream delays work and can compose filters and mappings without allocating every intermediate list, but laziness does not guarantee low memory: a final sort, group, or full materialization still needs data. A notebook must make the collection boundary visible. Otherwise a pipeline may appear efficient on a sample and explode on a full dataset.

The pipe operator is most valuable when it reads as a sequence of domain steps: decode, normalize, validate, enrich, summarize. A long pipe that mixes network calls, hidden state, rescue clauses, and presentation is not clarity. Break transformations into named functions with explicit tagged outputs. Then the notebook can render each stage, test it in isolation, and compare counts across boundaries.

Modules move knowledge from cell history into stable interfaces. Define domain functions in modules, keep public functions narrow, and use protocols when behavior should vary by data type without central conditionals. A protocol-based renderer can let different domain results present themselves through Kino while preserving a text fallback. Avoid macros until a repeated syntactic pattern truly requires compile-time transformation; notebooks already add dynamic context, so unnecessary metaprogramming compounds confusion.

Finally, functional design improves concurrent design. Immutable messages avoid shared-memory races, but they do not remove ordering problems, duplicate delivery, stale messages, or unbounded mailboxes. A well-modeled message has a tag, identifiers, version or timestamp when needed, and a receiver that owns state transitions. Projects later in the guide build on this foundation.

How This Fits on Projects

Project 1 teaches transformations and invariants; Projects 2 and 3 model user and data failures; Projects 4 and 7 turn values into messages and UI state; Project 9 generates inspectable source from domain attributes.

Definitions and Key Terms

  • Immutable term: an Elixir value that is never modified in place.
  • Rebinding: associating a name with a new value.
  • Pattern match: asserting and decomposing a value’s shape.
  • Guard: a restricted condition refining a matching clause.
  • Tagged result: a tuple whose first element identifies the outcome category.
  • Protocol: polymorphic behavior dispatched by data type.
  • Collection boundary: the point where lazy or external data becomes materialized.

Mental Model Diagram

RAW VALUE
   │ pattern: shape?
   ├──────── no ───────▶ {:error, :shape, evidence}
   v yes
NORMALIZED VALUE
   │ guards/domain rules?
   ├──────── no ───────▶ {:error, :policy, evidence}
   v yes
VALID DOMAIN VALUE
   │ pure transformations
   v
DERIVED RESULT ────────▶ renderer / file / message

Original values remain available for audit and comparison.

How It Works

  1. Represent boundary data with maps, binaries, or tagged tuples.
  2. Match the broad shape and normalize textual forms.
  3. Apply domain validation that accumulates useful reasons.
  4. Convert accepted values into a struct or other explicit domain form.
  5. Transform through named pure functions.
  6. Materialize lazy work deliberately and render the result separately.

Invariants: transformations do not mutate inputs; expected failures retain reasons; accepted plus rejected counts reconcile with input count.

Failure modes: float money, fail-fast bulk validation, a catch-all rescue hiding defects, assuming Stream means streaming end to end, anonymous maps with drifting keys, and pipes whose steps mix computation with UI side effects.

Minimal Concrete Example

PSEUDOCODE

normalize(row)
  -> validate_required_fields
  -> parse_decimal_money
  -> validate_policy
  -> one of:
       {:ok, subscription}
       {:error, [reason, ...], redacted_row}

invariant:
input_count = accepted_count + rejected_count

Common Misconceptions

  • “Rebinding mutates the old value.” It changes the current name binding only.
  • “A pattern is a schema validator.” It checks immediate structure, not all domain meaning.
  • “Functional means no state.” State exists, but transitions produce new values and ownership is explicit.
  • “Streams are always faster.” They delay work; total cost depends on operations and consumption.

Check-Your-Understanding Questions

  1. Why should a rejected CSV row not normally crash the runtime?
  2. When is a struct more appropriate than a map?
  3. Why can a lazy pipeline still consume large memory?

Check-Your-Understanding Answers

  1. It is expected boundary data and should be represented with an error value and evidence.
  2. Use a struct after validation when the domain has a stable known shape and behavior.
  3. Sorting, grouping, joining, collecting, or retaining all outputs requires materialization.

Real-World Applications

Financial reconciliation, API clients, ETL validation, policy engines, form handling, audit records, message contracts, and domain-specific renderers.

Where You Will Apply It

Project 1, Project 2, Project 3, Project 4, and Project 9.

References

Key Insight

Good Elixir data design makes valid states, expected failures, and transformation boundaries visible.

Summary

Use immutable domain values, pattern matching, tagged results, and named transformations to turn notebook cells into an explainable program instead of a sequence of accidents.

Homework / Exercises

  1. Model the outcomes of parsing a money field without exceptions.
  2. Rewrite a five-stage mixed pipeline as pure computation plus presentation.
  3. Decide whether three example datasets should use Enum or Stream and explain the collection boundary.

Solutions

  1. Return success with a decimal value or error with reason, original redacted field, and row identity.
  2. Separate decode, normalize, validate, calculate, and render; only render performs UI effects.
  3. Small bounded lists suit Enum; large sequential input may use Stream until a necessary join/group; infinite events require a bounded consumer and must never be fully collected.

Theory Chapter 3: BEAM Concurrency and OTP

Fundamentals

BEAM processes are lightweight isolated units with private heaps and mailboxes. They communicate by sending immutable messages. Receiving is selective: a process chooses the first mailbox message matching one of its patterns, which means message shape and ordering matter. Links propagate exits; monitors report termination without automatically propagating it. Tasks wrap concurrent work and preserve result or exit semantics. GenServer formalizes a process-owned state machine with calls, casts, and ordinary messages. Supervisors restart children according to declared strategy. Concurrency does not guarantee parallelism, fairness at the application level, bounded memory, or correct backpressure. Livebook makes these abstractions observable through PIDs, event transcripts, charts, and deliberate fault injection.

Deep Dive

A process is an ownership boundary. If a telemetry aggregator owns the current window, other processes should send it events rather than mutate its state. Its receive loop or GenServer callbacks serialize transitions, making each change individually understandable. The isolation removes shared-memory races, but the system can still suffer from stale events, duplicate identifiers, unexpected order, slow consumers, and mailbox growth.

Message protocols deserve the same care as public APIs. Give every message a clear tag and payload contract. Include a source or correlation identifier when multiple producers exist. Include a monotonic sequence or event time only when the consumer has a defined policy for late and duplicate data. A map full of arbitrary keys postpones errors until runtime; a documented tuple or struct-backed message makes the state machine explicit.

Links and monitors answer different failure questions. A link says the lifecycles are coupled: an abnormal exit may take the linked process down unless exits are trapped. A monitor says “tell me when that process terminates” without coupling survival. Supervisors use links because a child’s failure must be observed and acted upon. A dashboard observer often uses a monitor because it should report a producer crash without crashing with it.

Tasks are appropriate for bounded jobs with a result. Task.async_stream provides concurrency with ordering, timeouts, and a maximum concurrency setting, but only if the caller handles exits and slow work intentionally. Starting one Task per unbounded input can overwhelm remote APIs, connection pools, or memory. Concurrency limits are capacity decisions, not decoration.

GenServer is not a magic performance layer. It serializes access to owned state. A long callback blocks later calls and messages. Synchronous calls propagate backpressure to callers but can create timeout chains. Casts return immediately but provide no delivery acknowledgement and can flood the mailbox. Ordinary messages are appropriate for asynchronous protocols with explicit handling. The design question is where state belongs and how callers learn whether work was accepted.

Supervision separates recovery policy from business logic. A one-for-one strategy restarts only the failed child; other strategies encode stronger coupling. Restart intensity prevents an endlessly crashing child from consuming the system. State held only in the process disappears on restart, which is often correct for derived telemetry and wrong for durable decisions. Recovery design must say which state can be reconstructed, which must persist elsewhere, and what the UI shows while recovery happens.

Livebook introduces lifecycle hazards. Re-evaluating a cell that starts a process may leave the old process alive if it was not named or stopped. A listener launched by Kino.listen runs in the background. Re-running the cell can duplicate handlers and make every click appear twice. A runtime reconnect cleans the whole notebook node, but partial re-evaluation does not. Projects should keep process handles, define idempotent startup, monitor termination, and provide a visible cleanup action.

Backpressure is the hardest concurrency lesson in an interactive notebook. Kino.listen handles events sequentially by default. If a handler is slower than event production, latency and mailbox length grow. Async handling can improve throughput but introduces ordering and resource contention. Bounded windows, sampling, dropping policies, and coalescing are honest choices when displaying high-rate telemetry. The UI should reveal when it has sampled or dropped data rather than pretend every event was processed.

Fault injection completes the mental model. A learner should crash a producer, observe the exit, watch a supervisor start a new PID, and verify the aggregate continues. Then crash repeatedly until restart intensity is reached and explain why the supervisor stops. Correctness is not “nothing crashed”; it is predictable containment and recovery.

How This Fits on Projects

Project 4 is the dedicated process and supervision laboratory. Projects 7 and 12 apply event ownership and backpressure to interactive apps; Project 11 applies monitoring and timeouts at a remote node boundary.

Definitions and Key Terms

  • Mailbox: a process-local queue of received messages.
  • Selective receive: choosing the earliest message that matches a receive clause.
  • Link: a bidirectional lifecycle relationship between processes.
  • Monitor: a one-way termination notification relationship.
  • Backpressure: controlling producers when consumers cannot keep up.
  • Supervision strategy: the policy determining which children restart after failure.
  • Restart intensity: the maximum restart rate before a supervisor gives up.

Mental Model Diagram

 producer A ─┐
 producer B ─┼── readings ──▶ aggregator mailbox ──▶ window state
 producer C ─┘                       │                    │
      ▲                              │ monitor            v
      │ restart                      v                Kino.Frame
┌─────┴────────┐               recovery event        bounded chart
│ Supervisor   │
└──────────────┘

Slow UI without policy:
events/sec > handler/sec -> mailbox grows -> latency grows -> memory grows

How It Works

  1. A supervisor starts named or discoverable children from child specifications.
  2. Producers send versioned readings to an aggregator.
  3. The aggregator alone owns the current bounded window.
  4. A UI listener renders snapshots, not mutable shared state.
  5. A fault kills one producer; its linked supervisor observes the exit and applies restart policy.
  6. Monitors and logs make the old PID, new PID, reason, and recovery time visible.

Invariants: one owner transitions each state; UI history is bounded; every started background process has a cleanup or runtime-lifecycle plan.

Failure modes: duplicate listeners after re-evaluation, unbounded mailboxes, synchronous calls inside long computations, linked observers dying unexpectedly, restart loops, timer messages arriving after replacement, and stale PIDs stored in cells.

Minimal Concrete Example

EVENT TRANSCRIPT

12:00:00 producer sensor-2 pid=PID-A reading=21.8
12:00:05 action inject_fault target=sensor-2
12:00:05 DOWN pid=PID-A reason=simulated_failure
12:00:05 supervisor restart child=sensor-2 pid=PID-B
12:00:06 producer sensor-2 pid=PID-B reading=21.7
invariant aggregate_stream_continued = true

Common Misconceptions

  • “No shared memory means no race conditions.” Message order, timeouts, and duplicated events still race.
  • “A GenServer makes work concurrent.” One server callback runs at a time.
  • “Cast is faster and therefore better.” It removes acknowledgement and can overload the mailbox.
  • “Supervision preserves state.” It restarts processes; reconstruction and durability are separate.

Check-Your-Understanding Questions

  1. When should an observer monitor rather than link to a producer?
  2. Why can a sequential event listener become a memory problem?
  3. What must be true before state may safely disappear on restart?

Check-Your-Understanding Answers

  1. When it needs termination evidence but should survive independently.
  2. If arrival exceeds handling, queued events accumulate and increase latency and memory.
  3. The state is derived or disposable and can be reconstructed from authoritative inputs.

Real-World Applications

Telemetry, job orchestration, API fan-out, dashboards, device gateways, notification systems, background workflows, and fault-tolerant services.

Where You Will Apply It

Project 4, Project 7, Project 11, and Project 12.

References

Key Insight

BEAM concurrency is controlled state ownership plus explicit failure relationships, not merely many processes.

Summary

Use processes to own state, messages to define protocols, Tasks for bounded work, and supervisors for recovery policy. Always design queue bounds and cleanup in interactive notebooks.

Homework / Exercises

  1. Draw link and monitor relationships for three producers, one aggregator, one supervisor, and one observer.
  2. Choose a backpressure policy for 1,000 events per second displayed at 10 frames per second.
  3. Predict the result of re-evaluating a process-starting cell three times without cleanup.

Solutions

  1. Supervisor links to children; aggregator may monitor producers; observer monitors relevant processes; producers send to aggregator.
  2. Aggregate or sample into bounded time windows, report dropped/coalesced counts, and render at a fixed cadence.
  3. Up to three process or listener generations may run, duplicating events and retaining state until explicitly stopped or the runtime reconnects.

Theory Chapter 4: External Data, Contracts, and Persistence

Fundamentals

External data enters a notebook through files, HTTP, databases, user input, environment-backed secrets, and remote nodes. Every boundary needs a contract: source identity, authentication, timeout, schema, units, null policy, size limit, retry policy, redaction rule, and ownership of side effects. HTTP success is a combination of transport, status, headers, and body validation. Database safety depends on parameterized queries, least-privilege credentials, bounded results, and explicit transaction semantics. Livebook secrets keep credentials out of source, while file references and Kino.FS let workflows use local or external resources without assuming one fixed path. Persistence is not process memory; durable decisions need an external store or exported artifact.

Deep Dive

Boundary code should be suspicious by design. A CSV may contain a byte-order mark, changed delimiter, duplicate header, invalid encoding, or a column whose meaning changed. An API may return an HTML error page with a 200 status. A database query may return millions of rows after a filter is removed. A user may upload the correct format at an unsafe size. The notebook must validate before trusting and must preserve enough evidence to explain rejection.

HTTP work begins with a budget. Define connection and total timeouts, maximum retries, concurrency, and acceptable response size. Retry only operations and failures that are safe to repeat. A timeout does not prove the server did nothing, so mutating operations require idempotency keys or an independent status check. Respect Retry-After and rate-limit headers. Preserve partial success when a report fans out to several services, and label missing sources rather than replacing them with zeros.

Schema validation has layers. First validate transport and encoding. Then validate structural fields and types. Next normalize dates, decimals, identifiers, and categorical values. Finally apply domain rules such as nonnegative price or end-after-start. Accumulate row-level errors when the workflow is a triage report; fail fast when continuing would corrupt state. Always reconcile counts across accepted, rejected, and ignored categories.

Money and time are classic traps. Binary floating point cannot represent many decimal amounts exactly, so financial values need a decimal representation and declared rounding policy. Timestamps need timezone semantics. A naive date-time and a UTC instant are not interchangeable. Record the source timezone, conversion rule, and daylight-saving behavior. In dashboards, label whether a period uses event time, ingestion time, or query time.

SQL Smart Cells and KinoDB can accelerate exploration, but UI generation does not eliminate database risk. Queries still need parameter binding. The credential should be read-only for diagnostic projects. Set statement timeouts, row limits, and an allowlist of schemas. Render redacted columns and export only data permitted by policy. A transaction preview against a disposable training database can teach write semantics; production mutation is excluded from this guide’s completion criteria.

Secrets should enter through Livebook’s secret mechanism or deployment environment, never through committed source or output. Stamps remember approved access to Livebook-managed resources, but a stamp is not a sandbox. If a notebook changes or comes from another authority, resource permissions may require approval again. Redact credentials from exceptions, request dumps, SQL logs, and screenshots.

File references decouple the notebook from one local path. Kino.FS can resolve a Livebook file reference to a readable local path, downloading and caching when necessary; treat the resolved path as read-only. Copy to a controlled output location before modification. For large data, prefer formats and query plans that support column pruning, predicate pushdown, and bounded collection. A full download may still occur depending on the source and format, so measure rather than assume.

Persistence requires a source of truth outside transient processes. An app session can keep a draft, but a restart erases it. Durable incident decisions belong in a database, object store, Git repository, or deliberately exported signed report. Choose idempotent identifiers and version each write. For every artifact, record producer version, input provenance, row counts, checksum, and creation time.

How This Fits on Projects

Project 3 handles dirty file contracts. Project 5 adds typed analytical I/O. Project 6 establishes database and redaction safety. Project 8 moves between local and S3-compatible storage. Projects 10 and 12 add model artifacts, secrets, and durable app evidence.

Definitions and Key Terms

  • Provenance: evidence describing where data came from and how it was transformed.
  • Idempotency: repeated application has the same intended effect as one application.
  • Parameterized query: SQL whose values are bound separately from query syntax.
  • Least privilege: granting only the minimum permissions needed.
  • File reference: a Livebook-managed reference to a local or external resource.
  • Stamp: a signature retaining approved Livebook resource permissions and authorship context.
  • Durable state: data that survives process, session, or node restart.

Mental Model Diagram

UNTRUSTED BOUNDARY
HTTP / SQL / file / user / secret / remote node
                  │
                  v
┌──────────────────────────────────────────────┐
│ budget -> authenticate -> bound size/time    │
│ decode -> validate -> normalize -> redact    │
└──────────────────────┬───────────────────────┘
                       │
              ┌────────┴────────┐
              v                 v
        accepted domain      rejection evidence
              │
              v
    transform -> bounded output -> manifest -> durable artifact

How It Works

  1. Identify the source and declare live versus snapshot mode.
  2. Apply authentication without exposing secret values.
  3. Enforce time, size, concurrency, and row budgets.
  4. Decode and validate structure, then domain meaning.
  5. Separate accepted values from redacted rejection evidence.
  6. Perform read-only or idempotent operations.
  7. Persist artifacts with provenance and verify checksums.

Invariants: no secret in source or output; no dynamic SQL interpolation for values; row counts reconcile; production diagnostics are read-only; exported artifacts name their source and transformation version.

Failure modes: infinite retries, retrying unsafe writes, loading an unbounded result, schema drift interpreted as nulls, timezone ambiguity, float money, modifying a cached file path, treating a stamp as a security sandbox, and storing durable decisions only in app memory.

Minimal Concrete Example

DATA CONTRACT
source: s3-compatible://training/mobility/2026-07/
mode: snapshot
expected columns: trip_id, started_at_utc, ended_at_utc, distance_km
maximum rows collected: 100000
null policy: reject missing trip_id; quarantine invalid timestamps
credential: read-only Livebook secret
output: parquet + manifest + checksum

Common Misconceptions

  • “The API returned 200, so the data is valid.” Status does not prove body schema or meaning.
  • “A secret is safe if the cell is hidden in app mode.” Source and logs still require proper secret handling.
  • “An attached runtime is just another API client.” It executes inside the target node.
  • “A cached local file is durable source data.” It may be evicted or stale.

Check-Your-Understanding Questions

  1. Why is a timeout ambiguous for a write request?
  2. What is the minimum safe SQL posture for Project 6?
  3. Why should a resolved Kino.FS path be treated as read-only?

Check-Your-Understanding Answers

  1. The server may have completed the effect even though the response did not arrive.
  2. Read-only credential, parameterized queries, timeout, result bound, schema allowlist, and redaction.
  3. It may refer to a managed cached download; modification can violate ownership and portability assumptions.

Real-World Applications

ETL, support tooling, compliance exports, API health reports, data lakes, incident diagnostics, reproducible research, and internal operations apps.

Where You Will Apply It

Project 3, Project 5, Project 6, Project 8, Project 10, and Project 12.

References

Key Insight

External data is not a value until its authority, bounds, schema, and failure behavior are explicit.

Summary

Treat every external boundary as untrusted, every credential as non-displayable, every query as bounded, and every durable output as a versioned artifact with provenance.

Homework / Exercises

  1. Write a boundary contract for a paginated API.
  2. Threat-model a database Smart Cell used by support engineers.
  3. Design an artifact manifest for a Parquet export.

Solutions

  1. Include base URL, auth source, page size, termination rule, rate-limit behavior, retryable statuses, timeouts, maximum pages, schema, and snapshot policy.
  2. Risks include injection, overbroad credentials, PII exposure, unbounded queries, unsafe exports, secret leakage, and ambiguous writes; controls follow the chapter invariants.
  3. Include source reference, retrieval time, schema version, transformation revision, input/output counts, rejected count, time range, checksum, package/runtime versions, and redaction policy.

Theory Chapter 5: Explorer Analytics and Visualization

Fundamentals

Explorer provides immutable Series and DataFrames for analytical work in Elixir, with a Polars-backed default implementation. A DataFrame has named columns, data types, row count, optional groups, and either eager or lazy execution. Eager operations produce results immediately. Lazy operations build a plan that may combine filters, projections, joins, and aggregations before collection, enabling optimization and reducing unnecessary work. Data quality begins with types, nulls, uniqueness, ranges, and row-conservation checks. Visualization is another data contract: VegaLite encodings map fields to marks, axes, color, size, and facets. A correct dashboard states units, population, aggregation, time basis, missing-data policy, and uncertainty rather than relying on visual polish.

Deep Dive

DataFrame work should begin with a profile, not a chart. Record the physical source, format, byte size, inferred schema, row count, null counts, duplicate-key counts, and time range. Type inference is a hypothesis. An identifier that looks numeric may need to remain text because leading zeros matter. A date column may contain multiple formats. A categorical field may contain whitespace variants or unexpected new levels. Promote inferred types into an explicit contract once the first profile is understood.

Explorer’s immutable API fits Elixir’s transformation model. Each operation returns a new logical DataFrame, so the notebook can name raw, normalized, validated, enriched, and aggregated stages. Those names are evidence, not mandatory copies: with lazy execution they may represent plans rather than materialized tables. The crucial skill is knowing when work actually happens.

A lazy pipeline can push filters and selected columns toward the source, combine operations, and exploit parallel execution. Collection materializes results. Collect too early and you lose optimization and may allocate large intermediates. Collect too late and an unsupported operation or hidden full join may surprise you at the presentation boundary. Place a collection boundary where a bounded result is needed for validation, visualization, export, or an API that requires eager values. Measure the plan on realistic volume.

Joins and groups create semantic risk. A join may multiply rows when keys are nonunique. An inner join silently drops unmatched records; a left join preserves them as nulls. Every join should report left count, right count, matched keys, unmatched keys, and resulting count. Grouped summaries must state the denominator. “Error rate by service” is meaningless without the population, window, and treatment of missing services. Window functions and time bucketing require a defined timezone and boundary convention.

Missing data has multiple meanings: unknown, not applicable, not yet observed, redacted, and parsing failure are not interchangeable. A single null cannot communicate all of them unless another quality column supplies the reason. Dropping null rows can bias analysis. Imputation can make a model run while hiding a broken data source. Project 5 therefore creates a quarantine path and a manifest instead of silently cleaning everything.

Charts are encodings, not decorations. Position on a common scale supports precise comparison; area and color are less precise. A truncated axis can exaggerate a small change. Dual axes can imply relationships that the data does not support. Aggregation can hide distribution shape, and a mean can conceal severe tails. Use a table or distribution view to preserve detail, and provide drill-down from summary to evidence.

VegaLite’s declarative model is useful because the chart specification is reviewable. It says which field maps to x, y, color, tooltip, or facet; which transformation aggregates; and which scale applies. KinoVegaLite can stream or update data, but a streaming chart still needs a bounded window. The absence of old points should be explicit in the UI.

Interactive filtering introduces state. A dashboard filter may update a pre-aggregated frame quickly or recompute from raw data accurately. State the trade-off. Cancel obsolete work when a user changes controls rapidly, and never let an older slow result overwrite a newer request. Render loading, empty, stale, partial, and failed states as deliberate outcomes.

How This Fits on Projects

Project 5 teaches typed, lazy pipelines and manifests. Project 6 applies bounded analytical queries. Project 7 teaches chart contracts and interactive recomputation. Projects 8, 10, and 12 add external storage, tensors, and production dashboards.

Definitions and Key Terms

  • Series: a typed named or unnamed column-like vector.
  • DataFrame: a named collection of same-length Series.
  • Lazy plan: a deferred graph of analytical operations.
  • Collection: executing a lazy plan into an eager result.
  • Predicate pushdown: applying filters near the source to reduce data read.
  • Cardinality: the count of distinct values or relationship multiplicity.
  • Encoding: the mapping of a data field to a visual property.

Mental Model Diagram

source scan
   │ schema contract
   v
lazy select -> filter -> normalize -> join -> group -> aggregate
   │                           │
   │ plan optimization         └── join cardinality evidence
   v
COLLECT at bounded boundary
   ├── quality table
   ├── VegaLite specification -> interactive chart
   └── Parquet + manifest

How It Works

  1. Scan and profile the source.
  2. Override incorrect inferred types.
  3. Build lazy transformations and quality flags.
  4. Check join cardinality and row conservation.
  5. Collect only a bounded result or export stage.
  6. Encode charts with declared units, population, and aggregation.
  7. Test empty, missing, skewed, and high-volume inputs.

Invariants: source rows reconcile with accepted and quarantined rows; join expansion is explained; every chart names its units and population; interactive history is bounded.

Failure modes: identifier type inference, accidental Cartesian joins, naive timestamps, collecting before filtering, silently dropping nulls, misleading axes, stale interactive responses, and unbounded streaming charts.

Minimal Concrete Example

PIPELINE PLAN
scan trips.parquet
  -> select trip_id, started_at, ended_at, distance_km, zone_id
  -> filter declared date partition
  -> derive duration_minutes
  -> quarantine duration <= 0
  -> left join zone reference
  -> group by day and zone
  -> summarize count, median duration, p95 duration
  -> collect bounded summary

Common Misconceptions

  • “A DataFrame printout proves all work ran.” A lazy frame may represent a plan.
  • “Null means zero.” It means absence under a policy that must be defined.
  • “A chart is objective if the data is correct.” Encodings and aggregation shape interpretation.
  • “Filtering after loading is equivalent to pushdown.” Results may match while resource cost differs dramatically.

Check-Your-Understanding Questions

  1. What evidence detects a one-to-many join mistake?
  2. When should a lazy pipeline collect?
  3. Why must a p95 chart state its population and time window?

Check-Your-Understanding Answers

  1. Key uniqueness checks plus before/after row counts and unmatched-key reports.
  2. At a deliberate bounded boundary for validation, rendering, export, or an eager-only operation.
  3. Percentiles depend on which events and interval are included; without them comparison is invalid.

Real-World Applications

Data-quality gates, operational analytics, finance reports, mobility analysis, product dashboards, feature preparation, and compliance evidence.

Where You Will Apply It

Project 5, Project 6, Project 7, Project 8, Project 10, and Project 12.

References

Key Insight

An analytical result is trustworthy only when its types, row lineage, execution boundary, and visual encoding are all inspectable.

Summary

Use lazy plans to control work, manifests to explain lineage, invariants to expose data loss, and declarative charts to make visual choices reviewable.

Homework / Exercises

  1. Design quality checks for a trip dataset.
  2. Explain whether a left or inner join fits a service dashboard.
  3. Critique a chart with a truncated y-axis and no sample size.

Solutions

  1. Check schema, unique trip IDs, null IDs, timestamp ordering, nonnegative distance, timezone, category set, duplicates, and row reconciliation.
  2. A left join often preserves all events and exposes missing service metadata; an inner join is acceptable only when deliberate loss is reported.
  3. The scale may exaggerate change and the missing population prevents judging stability; add full context, sample count, and distribution evidence.

Theory Chapter 6: Kino Interactivity, Apps, and Collaboration

Fundamentals

Kino is Livebook’s rich-output and interaction layer. Kino.Input creates values such as text, number, select, file, image, and audio. Regular inputs are shared across notebook collaborators by default. Kino.Control creates event-producing controls and forms; forms produce user-specific submissions. Kino.listen consumes events sequentially in a background process, while frames provide a stable output target that can be replaced, appended, cleared, or updated for a specific client. Layouts compose renderable values. Smart Cells persist JSON-friendly attributes and generate inspectable Elixir source. A Livebook App hides notebook code and presents outputs and controls. Single-session apps share one running session; multi-session apps create separate sessions for users or workflows.

Deep Dive

Interactivity changes a notebook from pull-based evaluation to an event-driven system. A normal cell runs because an author evaluates it. A control emits because a user clicks, submits, changes a value, or a timer ticks. That event must be received by a process, validated, associated with client identity when relevant, applied to owned state, and rendered. Every interactive project is therefore a small OTP system even when its surface resembles a form.

Choose the input primitive based on collaboration semantics. A regular Kino.Input is synchronized across users, useful for a shared notebook parameter during co-authoring. A Kino.Control form emits user-specific values and is better for a multi-session app or independent submissions. Assuming inputs are private is a data-leak and UX bug. The origin carried by events is opaque but useful for routing updates or grouping evidence.

Frames separate event handling from cell output position. Replacing frame content suits current state, appending suits a bounded log, and client-targeted rendering supports per-user feedback. Append without a bound creates a browser and runtime memory leak. Re-rendering an entire large table on every keypress produces latency and stale results. Debounce, submit explicitly, pre-aggregate, and cancel obsolete work.

Kino.listen starts a background listener and handles events sequentially. Sequential handling preserves a simple order but a slow callback creates backlog. Async handling is an opt-in trade-off, not a universal fix: results can complete out of order and contend for resources. A robust design assigns a request generation number and discards late results superseded by newer input.

Kino.Screen can provide LiveView-like per-user form behavior with a state transition and render callback. A custom Kino.JS component goes further, embedding bundled browser assets and exchanging events with a server process. The browser is an untrusted boundary: validate every payload on the Elixir side. Bundle assets rather than depend casually on external CDNs, which may be blocked and undermine offline reproducibility.

A Smart Cell is a source generator with a UI, not a secret alternative execution engine. Its attributes must serialize cleanly, restore after save/reopen, and convert deterministically into understandable source. Users should be able to convert it to a regular code cell and review what it does. If the cell uses a collaborative editor, synchronize pending edits before evaluation. Test events, exported source, attribute round-trips, and invalid configuration with Kino.Test.

Apps change presentation and lifecycle. Preview mode hides code and shows selected outputs. A single-session app is one shared live process graph; it fits a shared incident room or team control surface. A multi-session app gives each user a private session; it fits calculators, private uploads, and independent analysis. Neither mode automatically provides durable storage. A restarted session loses in-memory state unless it is reconstructed.

Collaboration in the notebook editor and collaboration in an app are also distinct. Real-time authoring shares document edits and regular inputs. App users interact with the deployed UI and may not see code. Durable team workflows need explicit identity, authorization, persistence, and conflict policy beyond shared process memory.

Accessibility belongs in the event contract too. Every control needs a clear label, keyboard path, validation message, and non-color status indicator. A fast visual demo that excludes keyboard or screen-reader users is not a finished internal tool.

How This Fits on Projects

Project 2 introduces inputs and forms. Project 4 adds frames and event streams. Project 7 builds a responsive dashboard. Project 9 constructs a Smart Cell. Projects 10 and 12 exercise multi-session user isolation and deployment.

Definitions and Key Terms

  • Renderable: a value Kino knows how to present.
  • Frame: a stable output container updated after cell evaluation.
  • Origin: an opaque client identifier attached to control events.
  • Listener: a background process consuming an event stream.
  • Smart Cell: a UI-backed cell that persists attributes and generates source.
  • Single-session app: one shared running notebook session for all users.
  • Multi-session app: a separate notebook session per user or workflow.

Mental Model Diagram

client A form ──event(origin A)──┐
client B form ──event(origin B)──┼──▶ listener / screen process
timer control ──interval event───┘              │
                                                v
                                      validate + state transition
                                                │
                       ┌────────────────────────┴─────────────────┐
                       v                                          v
               shared frame update                     client-specific frame

Smart Cell: JSON attributes -> deterministic source -> normal evaluator

How It Works

  1. Render an input or control.
  2. Subscribe or listen to its event stream.
  3. Validate the event and identify shared versus per-client behavior.
  4. Transition process-owned state.
  5. Replace or append to a bounded frame.
  6. Stop listeners or reconnect cleanly when the notebook lifecycle ends.
  7. In app mode, verify session semantics with two clients.

Invariants: shared and private state are intentional; event payloads are validated; history is bounded; late results cannot overwrite newer state; Smart Cell source is inspectable.

Failure modes: shared inputs leaking between users, duplicated listeners, slow sequential handlers, out-of-order async results, unbounded frame appends, CDN-only assets, non-serializable Smart Cell attributes, and assuming app memory is durable.

Minimal Concrete Example

EVENT STATE MACHINE

idle
  --submit valid form--> loading(request=42)
  --submit invalid-----> error_for_client(origin)

loading(request=42)
  --new submit---------> loading(request=43)
  --result 42----------> ignore as stale
  --result 43----------> ready(rendered dashboard)

Common Misconceptions

  • “Every input is private to the current browser.” Regular Kino inputs are shared by default.
  • “Kino.listen blocks the cell.” It runs a background listener.
  • “Async listeners preserve event order.” Concurrent completion may reorder results.
  • “Smart Cells hide implementation.” Their purpose includes generating inspectable source.

Check-Your-Understanding Questions

  1. Which primitive fits a private calculator form?
  2. How do you stop an old slow query from replacing a new result?
  3. When does a single-session app fit better than multi-session?

Check-Your-Understanding Answers

  1. A per-user form or screen, usually in a multi-session app.
  2. Tag requests with a generation or reference and render only the current one; cancel when possible.
  3. When all users intentionally share one live state, such as a collaborative incident room.

Real-World Applications

Internal tools, calculators, training labs, approval workbenches, shared control rooms, executable documentation, and domain-specific notebook extensions.

Where You Will Apply It

Project 2, Project 4, Project 7, Project 9, Project 10, and Project 12.

References

Key Insight

A Kino interface is a message-driven process system whose collaboration semantics must be designed explicitly.

Summary

Select shared or per-user controls deliberately, bound event work and frame history, generate transparent Smart Cell source, and test app session semantics with multiple clients.

Homework / Exercises

  1. Choose shared versus per-user state for four example widgets.
  2. Design a bounded frame policy for a telemetry stream.
  3. Define a Smart Cell attribute round-trip test.

Solutions

  1. Shared incident severity filter may be shared; private cost calculator inputs, file upload, and personal threshold should be per user unless collaboration requires otherwise.
  2. Keep the last N points or T minutes, aggregate older points, expose dropped count, and update at a fixed cadence.
  3. Initialize attributes, export source, persist JSON form, restore, export again, and assert equivalent source and validation behavior.

Theory Chapter 7: Numerical Elixir and Machine Learning

Fundamentals

Nx provides typed multidimensional tensors and numerical definitions for Elixir. A tensor has shape, element type, names when used, and a backend. Broadcasting aligns compatible dimensions without manually copying data. Nx.Defn builds numerical computation graphs in a restricted tensor-aware subset that can run in an evaluator or compile through a backend such as EXLA. Compilation cost and execution cost are different measurements. Bumblebee loads pretrained models and tokenizers or featurizers, while Nx.Serving wraps preprocessing, batching, execution, and postprocessing behind a serving contract. A machine-learning notebook must specify data and model provenance, deterministic choices, evaluation metrics, failure behavior, resource limits, and human override; a visually plausible prediction is not validation.

Deep Dive

Shape is the first contract in numerical work. A tensor described as batch by height by width by channels is different from channels by height by width, even if it contains the same number of elements. Broadcasting makes operations concise, but an accidentally compatible shape can compute the wrong result without raising. Write the intended shape next to every boundary and assert it before inference.

Element type controls range, precision, memory, and backend support. Integers do not represent normalized pixels; low-precision floats may change model behavior; automatic conversion can allocate a second large tensor. A notebook should display shape, type, backend, and approximate memory before running a heavy step. Named axes can make transformations easier to review where supported.

Numerical definitions build an expression graph rather than executing arbitrary Elixir step by step. The restricted model enables optimization and compilation. Ordinary I/O, arbitrary process operations, and dynamic structures do not belong inside a defn. Keep data loading and result presentation outside; keep tensor math inside. Separate the first-call compilation benchmark from steady-state execution or the results will mislead.

Backends and devices create resource lifecycles. Copying tensors between host and device costs time and memory. Recompiling for varying shapes destroys latency. A serving system often pads or batches inputs into compatible shapes and warms the model before user traffic. On a local CPU, a small model and bounded image size are educational. A huge model that swaps the machine teaches little about the intended contract.

Pretrained models add supply-chain and provenance questions. Record repository, revision or artifact digest, task, tokenizer or featurizer configuration, license, expected input range, label mapping, and local cache policy. “Latest” is not reproducible. Offline deployment requires the model artifact to exist in the image or managed storage before the app starts.

Inference is a pipeline: accept input, validate type and size, preprocess, construct tensors, run serving, decode output, calibrate or interpret scores, and present uncertainty. Each stage can fail independently. Reject unsupported images before allocating huge tensors. Redact or avoid retaining user content. Put a timeout and queue bound around serving.

Nx.Serving can batch requests to improve throughput, but batching adds wait time. A low-volume interactive app may prefer a short batch window; a throughput-oriented service may accept more latency. Device partitions can increase parallelism only when hardware and memory permit. Measure queue time, batch size, inference time, total latency, timeout count, and memory.

Evaluation must match the decision. Top prediction and confidence are not enough. Use a small labeled test set, confusion matrix, per-class precision and recall, abstention threshold, and error slices. For a triage tool, allow human correction and record it as feedback rather than silently treating model output as truth. State that confidence scores may not be calibrated probabilities.

Livebook is well suited to this work because it combines artifact provenance, tensor inspection, charts, examples, and an app preview. It is not a reason to skip normal packaging. When the model becomes a production dependency, move stable modules and tests into a Mix project, keep the notebook as evaluation and operations evidence, and deploy with explicit capacity.

How This Fits on Projects

Project 10 is the dedicated tensor and pretrained-model lab. Project 12 integrates a bounded model-assisted suggestion into an operations app with human override and provenance.

Definitions and Key Terms

  • Tensor: a typed multidimensional numerical array.
  • Shape: the ordered dimensions of a tensor.
  • Broadcasting: aligning compatible dimensions for elementwise operations.
  • defn: Nx’s tensor-aware numerical definition.
  • Backend: the implementation storing and executing tensor operations.
  • Warmup: preparing dependencies, model artifacts, and compiled paths before traffic.
  • Serving: a bounded interface around preprocessing, batching, inference, and postprocessing.

Mental Model Diagram

untrusted input
   │ validate format, dimensions, bytes
   v
preprocess ──▶ tensor[batch,height,width,channels] f32
   │                    │ shape/type evidence
   v                    v
Nx.Serving queue -> batch -> compiled model -> raw scores
                                            │
                                            v
                                labels + threshold + abstain
                                            │
                         ┌──────────────────┴─────────────────┐
                         v                                    v
                 human accepts                       human corrects

How It Works

  1. Select and pin a compatible model artifact.
  2. Validate and bound user input.
  3. Preprocess into a declared shape and type.
  4. Warm the serving path.
  5. Run bounded inference and record queue plus execution latency.
  6. Interpret scores with thresholds and an abstain state.
  7. Evaluate labeled examples and preserve human correction.

Invariants: model revision is known; input size is bounded; shapes are asserted; compilation and steady-state timings are separated; the UI supports failure and abstention; user content is not retained accidentally.

Failure modes: shape confusion, recompilation from variable shapes, model-cache misses in production, unbounded uploads, device memory exhaustion, blocking the UI process, interpreting score as certainty, data leakage, and no human override.

Minimal Concrete Example

INFERENCE CONTRACT
input: image/jpeg or image/png, maximum 5 MiB, maximum 2048 x 2048
preprocess: resize with documented crop policy
tensor: [1, 224, 224, 3], f32
model: repository + revision + artifact digest
output: ranked labels, score, latency, model revision
decision: suggest only; abstain below threshold; human owns final category

Common Misconceptions

  • “Compatible element count means correct shape.” Axis meaning and order still matter.
  • “The first inference time is normal latency.” It may include loading and compilation.
  • “A high score is calibrated confidence.” Scores require evaluation and calibration evidence.
  • “The model is the app.” Validation, serving, UX, monitoring, and human policy are equally important.

Check-Your-Understanding Questions

  1. Why benchmark compilation separately?
  2. What makes variable input shapes operationally expensive?
  3. Why should an internal triage model have an abstain state?

Check-Your-Understanding Answers

  1. Compilation is often a one-time or shape-specific cost and otherwise distorts steady-state latency.
  2. They can trigger new compilation, prevent batching, and increase unpredictable allocation.
  3. Some inputs are out of distribution or uncertain; forced labels turn uncertainty into confident error.

Real-World Applications

Image and text triage, forecasting, anomaly scoring, embeddings, search, quality inspection, and model evaluation workbenches.

Where You Will Apply It

Project 10 and Project 12.

References

Key Insight

Numerical code becomes dependable when shape, type, artifact, compilation, serving, and decision policy are one explicit contract.

Summary

Inspect tensors before trusting them, distinguish compile from run cost, bound serving resources, pin artifacts, evaluate errors, and keep a human decision boundary.

Homework / Exercises

  1. Calculate the approximate memory of a batch tensor.
  2. Design a test set for five image categories with an unknown class.
  3. Compare throughput and latency goals for batch size one and a larger batch.

Solutions

  1. Multiply all dimensions by bytes per element and include copies/intermediates separately.
  2. Include representative, ambiguous, corrupted, out-of-distribution, and class-imbalanced examples; measure per-class errors and abstention.
  3. Batch one minimizes queue delay but may underuse hardware; larger batches improve throughput but add wait time and memory, so choose from measured workload goals.

Theory Chapter 8: Production Delivery, Security, and Operations

Fundamentals

Livebook executes code with the authority of its runtime. Anyone with authoring access can potentially read files and execute commands available to that operating-system identity. Livebook binds locally and uses token authentication by default, but network exposure requires deliberate authentication, TLS, proxy, and identity controls. Standalone, remote-execution, and attached runtimes have different trust boundaries. Livebook Apps hide code from app users but do not sandbox it. Stamps remember approved resource access; they do not prove arbitrary code is safe. Docker app servers, Teams deployment groups, shared secrets and storage, authentication, authorization, audit logs, CLI deployment, warmup, capacity planning, rollback, and inactivity shutdown turn a notebook into an operated service.

Deep Dive

Start production design with authority, not packaging. List what the notebook runtime can reach: local files, environment variables, database networks, Erlang distribution, cloud metadata, model stores, and outbound internet. Then remove everything not required. A standalone runtime with a read-only API or database credential is the preferred default because it preserves a narrow interface. An attached runtime places evaluation inside the target node and should be reserved for controlled diagnostics on a disposable environment before any production consideration.

Remote Execution Smart Cells occupy a useful middle position. The notebook keeps its standalone runtime and packages, then performs a bounded call on a remote Elixir node. This supports charts and Kino dependencies without loading them into the target. It still relies on distributed Erlang authority and must use an allowlist, timeouts, bounded results, and protected cookies or network paths.

Authentication answers who may enter; authorization answers what they may do. An authoring instance is more privileged than an app surface. Keep authoring single-user unless a managed team model explicitly separates identities and configuration. App authentication may use Livebook Teams, email-domain login, OIDC, or other supported identity providers. Per-user identity should feed external authorization rather than merely decorate an audit log.

Secrets belong in workspace or deployment configuration. Shared secret systems synchronize encrypted values, but the runtime eventually receives plaintext to use them. Prevent display, log redaction failures, and overly broad reuse. Separate development, staging, and production credentials. A deployment group should override environment-specific secrets without changing notebook source.

Stamping protects a resource-approval workflow. A notebook signed in an expected workspace can retain permission to named secrets and file systems. A modified or foreign notebook loses that remembered trust. Teams adds organizational authority for deployment stamps. The stamp does not inspect business logic, constrain operating-system calls, or make third-party code harmless. Review source and dependencies.

An app has a lifecycle and capacity. Startup may compile Mix.install dependencies, load model artifacts, open pools, and warm numerical paths. Persist the Mix install directory or warm dependencies during image construction when appropriate. Size memory for the Livebook server plus every active app session and workload-specific data or models. Multi-session apps multiply session state. Configure inactivity shutdown where acceptable and test cold-start behavior.

Deployment should be repeatable and reversible. Preview locally, run clean replay, build an immutable artifact, deploy to staging, execute smoke checks, and promote the exact revision. Livebook CLI can deploy notebooks through CI when Teams configuration is present. Store tokens in CI secrets, never source. Record notebook revision, stamp status, dependency manifest, app-server image, deployment group, smoke-test result, and rollback target.

Observability must cover both application value and runtime health. Capture structured evaluation logs, request failures, session count, startup time, memory, queue length, inference latency, data freshness, model revision, and stale-source state. An operations dashboard that cannot say how old its data is encourages wrong decisions.

Mutation is intentionally excluded from this guide’s production Definition of Done. If an advanced learner adds it later, require a separate credential, allowlisted command, dry-run, explicit confirmation, idempotency key, two-person policy where appropriate, before/after evidence, and an audit record outside the notebook node. Convenience must not silently become control-plane authority.

How This Fits on Projects

Project 6 introduces least-privilege SQL. Project 8 applies shared storage and stamps. Project 11 compares runtime boundaries. Project 12 performs secured deployment, capacity checks, audit evidence, and rollback.

Definitions and Key Terms

  • Trust boundary: where authority, validation, or identity changes.
  • Authoring surface: the privileged interface used to edit and execute notebook code.
  • App server: a Livebook instance configured to run deployed notebook apps.
  • Deployment group: Teams configuration grouping app servers and deployments.
  • Warmup: preparing dependencies and resources before serving users.
  • Audit evidence: durable identity, time, action, version, and outcome records.
  • Rollback: restoring a known prior deployable revision.

Mental Model Diagram

authors ──strong auth──▶ authoring Livebook
                              │ reviewed + stamped notebook
                              v
CI deploy ───────────▶ deployment group ─────────▶ app servers
                                                    │
app users ──OIDC/auth──▶ code-hidden app surface    │
                                                    v
                                      standalone runtime sessions
                                         │        │        │
                                   read-only DB  API   model store

Audit: identity + notebook revision + action + time + outcome

How It Works

  1. Inventory runtime authority and remove unnecessary access.
  2. Separate authoring identity from app-user identity.
  3. Configure environment-specific secrets and storage.
  4. Review, stamp, and clean-replay the notebook.
  5. Warm dependencies and size the app server.
  6. Deploy immutable revision to staging and smoke test.
  7. Promote, observe, and retain a tested rollback path.

Invariants: no public unauthenticated authoring; production credentials are least privilege; app users cannot see source by default; every deploy names a revision; every dashboard names freshness; production exercises remain read-only.

Failure modes: exposing token URLs, treating app mode as a sandbox, attached runtime with broad authority, unpersisted dependency cache causing long restarts, underestimated multi-session memory, secrets printed in exceptions, invalid stamps, no rollback, and audit data stored only in volatile session memory.

Minimal Concrete Example

DEPLOYMENT GATE
notebook revision: git-sha
clean replay: PASS
source review: PASS
stamp authority: expected team workspace
runtime: standalone
database role: ops_readonly
data freshness alarm: configured
cold-start budget: PASS
two-client isolation test: PASS
rollback revision: previous-git-sha
mutation capability: NONE

Common Misconceptions

  • “Code-hidden means sandboxed.” It changes presentation, not runtime authority.
  • “Token authentication makes internet exposure safe by itself.” Network and identity controls still matter.
  • “A stamp proves the notebook is benign.” It retains resource approval and authority context, not semantic safety.
  • “Attached mode is the easiest production integration.” It is often the highest-coupling and highest-authority option.

Check-Your-Understanding Questions

  1. Why prefer a standalone runtime plus read-only API?
  2. What does Teams add beyond open-source real-time collaboration?
  3. Why must multi-session capacity be measured differently?

Check-Your-Understanding Answers

  1. It narrows authority, preserves package isolation, and makes the boundary explicit.
  2. Managed deployment, identity and authorization options, organizational stamping, shared secrets/storage, and audit capabilities.
  3. Each active session has process and data state; models or caches may also multiply unless deliberately shared.

Real-World Applications

Internal operations apps, production diagnostics, governed runbooks, data products, model-assisted support tools, and team knowledge systems.

Where You Will Apply It

Project 6, Project 8, Project 11, and Project 12.

References

Key Insight

A production notebook is an executable service whose authority, identity, capacity, evidence, and rollback must be designed as carefully as its code.

Summary

Prefer narrow standalone boundaries, separate identities and credentials, understand stamps, deploy immutable revisions, measure session capacity, observe freshness, and keep production workflows read-only.

Homework / Exercises

  1. Draw a trust map for a notebook that queries production data.
  2. Create a deployment evidence checklist.
  3. Design the controls required before adding one mutating operation.

Solutions

  1. Include author, app user, Livebook server, notebook runtime, database, secret store, network proxy, CI, audit store, and every credential boundary.
  2. Use the Minimal Concrete Example plus dependency versions, image digest, test dataset, session-mode test, memory budget, and approver.
  3. Separate credential, allowlist, dry-run, explicit confirmation, idempotency key, authorization, before/after evidence, external audit, rate limit, timeout, and rollback or compensation.

Glossary

  • App preview: a local code-hidden rendering of a notebook as an application.
  • Attached runtime: evaluation inside an existing external Elixir node.
  • Binding: an association between a name and value on an evaluation path.
  • Branching section: an independently evaluated section inheriting from a selected parent.
  • Clean replay: reconnecting a runtime and rebuilding the verified outcome from declared inputs.
  • Collection boundary: the deliberate point where deferred data work becomes materialized.
  • Control: a Kino component emitting events such as clicks, submits, or timer ticks.
  • Data provenance: evidence of a dataset’s source, retrieval, schema, and transformations.
  • DataFrame: an immutable named collection of typed same-length Series.
  • defn: an Nx numerical definition that builds tensor-aware computation graphs.
  • Evaluation path: the sequence of ancestor cells whose bindings are available to a cell.
  • Frame: a Kino output target that can be updated after its creating cell finishes.
  • Kino: Livebook’s library and protocol layer for rich output and interaction.
  • Live Markdown: Livebook’s readable .livemd notebook format.
  • Multi-session app: an app creating a separate notebook session per user or workflow.
  • Origin: an opaque client identifier included in Kino events.
  • Remote execution: bounded evaluation on another Elixir node while retaining the notebook’s own runtime.
  • Runtime: the node and evaluator processes that execute notebook cells.
  • Single-session app: an app in which users share one live notebook session.
  • Smart Cell: an interactive cell persisting JSON-friendly attributes and generating inspectable source.
  • Stamp: a signature that carries notebook authority context and approved Livebook resource access.
  • Standalone runtime: the default isolated Erlang VM started for one notebook.
  • Stale cell: a cell whose old result may not match changed upstream source.
  • Tensor: a typed multidimensional numerical value used by Nx.

Why Livebook + Elixir Matters

Modern engineering work often sits awkwardly between code, documentation, operations, and analysis. A script can execute but does not explain itself. A wiki page can explain but drifts away from the command actually run. A dashboard is polished but may hide its query and assumptions. Livebook lets one reviewed artifact combine narrative, executable transformations, rich evidence, and a controlled app surface.

This matters especially for Elixir. The language’s immutable values, pattern matching, isolated processes, and supervision are often taught abstractly. A notebook can make them observable: render a rejected value beside its reason, chart a mailbox backlog, show a producer PID before and after a supervised crash, or compare tensor shape and memory before inference.

As of July 15, 2026, the official GitHub pages list Livebook v0.19.8 as the latest stable release, with roughly 5.8 thousand stars, about 499 forks, and 71 releases. These are community and release-activity signals, not usage counts. The same official documentation describes Livebook as open source, real-time collaborative, versionable through .livemd, dependency-aware, and able to run notebooks as apps. Livebook repository and v0.19.8 release.

SCATTERED WORKFLOW                         LIVEBOOK WORKFLOW

wiki assumptions                          .livemd narrative
      │                                          │
      v                                          v
copied shell script                        declared setup + cells
      │                                          │
      v                                          v
spreadsheet / chart                       Kino table + VegaLite
      │                                          │
      v                                          v
manual screenshot                         manifest + clean replay
      │                                          │
      v                                          v
untracked internal tool                   preview -> reviewed app deploy

Drift between artifacts                   One reviewable executable artifact

Context and Evolution

Interactive programming predates modern notebooks, and literate programming long argued that programs should explain themselves. Livebook combines those traditions with the BEAM: each notebook can receive an isolated Erlang VM, collaborate in real time, connect to existing nodes, and turn event-driven Kino components into deployable apps. It should not replace a normal Mix project for every production system. Its strongest role is the boundary between exploration and engineering: executable learning, reviewable analysis, diagnostics, internal tools, and operational evidence.

Concept Summary Table

Concept Cluster What You Need to Internalize
Livebook Execution and Reproducibility Document state and runtime state differ; evaluation ancestry, stale tracking, setup, branches, declared inputs, and clean replay determine whether an outcome is trustworthy.
Functional Elixir and Data Modeling Immutable values, pattern matching, tagged results, explicit structures, collection boundaries, and named transformations turn cell history into a domain program.
BEAM Concurrency and OTP Processes own state, messages define protocols, links and monitors define failure relationships, and backpressure plus cleanup determine runtime health.
External Data, Contracts, and Persistence HTTP, SQL, files, users, and secrets are untrusted boundaries requiring budgets, schemas, redaction, provenance, and durable artifact policy.
Explorer Analytics and Visualization Types, lazy plans, collection, joins, quality invariants, and honest visual encodings must remain inspectable from source to chart.
Kino Interactivity, Apps, and Collaboration Inputs, forms, events, frames, Smart Cells, client origin, and single- versus multi-session semantics form a message-driven UI system.
Numerical Elixir and Machine Learning Shape, type, backend, model provenance, compile/run cost, serving bounds, evaluation, abstention, and human override form one inference contract.
Production Delivery, Security, and Operations Runtime authority, identity, secrets, stamps, app servers, capacity, deploy evidence, observability, and rollback are part of notebook correctness.

Project-to-Concept Map

Project Concepts Applied
Project 1: Executable Energy Field Report Execution and Reproducibility; Functional Elixir
Project 2: Interactive Capacity Planner Functional Elixir; Kino Interactivity
Project 3: Messy CSV Triage Clinic Execution and Reproducibility; Functional Elixir; External Contracts
Project 4: Fault-Tolerant Telemetry Lab BEAM Concurrency and OTP; Kino Interactivity
Project 5: City Mobility Data Pipeline Execution and Reproducibility; External Contracts; Explorer Analytics
Project 6: Safe SQL Incident Runbook External Contracts; Explorer Analytics; Production Security
Project 7: Interactive SLA Command Center BEAM Concurrency; Explorer Analytics; Kino Interactivity
Project 8: Portable Cloud Data Lake Lab Execution and Reproducibility; External Contracts; Explorer Analytics; Production Security
Project 9: Tested Domain Smart Cell Toolkit Functional Elixir; Kino Interactivity; Production Delivery
Project 10: Nx + Bumblebee Image Triage Lab Explorer Analytics; Kino Interactivity; Numerical Elixir and ML
Project 11: Safe Attached-Runtime Console BEAM Concurrency; External Contracts; Production Security
Project 12: Deployed Multi-Session Ops Decision App All eight concept chapters

Deep Dive Reading by Concept

Concept Book and Chapter Why This Matters
Livebook Execution and Reproducibility Official Livebook Learn notebooks and Mix.install docs The notebook format and evaluator are evolving product behavior, so primary documentation is authoritative.
Functional Elixir and Data Modeling Elixir in Action, 3rd ed. — Chapters 2–4 Develops immutable data, abstractions, and error modeling used in early projects.
BEAM Concurrency and OTP Elixir in Action, 3rd ed. — Chapters 5–9 Explains processes, servers, supervision, and recovery in a production-oriented sequence.
External Data, Contracts, and Persistence Designing Data-Intensive Applications, 2nd ed. — Chapters 1, 3, 11 Connects local data transformations with durable, trustworthy data systems.
Explorer Analytics and Visualization Data Science for Business — Chapters 2, 3, 7 Builds discipline around analytical framing, evidence, and evaluation.
Kino Interactivity, Apps, and Collaboration Official Kino Input, Control, Frame, SmartCell, and Test docs Public APIs and client semantics are best learned from version-matched primary sources.
Numerical Elixir and Machine Learning Machine Learning in Elixir and Livebook — Nx, data, and pretrained-model chapters Connects tensors and model workflows to the Livebook-native ecosystem.
Production Delivery, Security, and Operations Release It!, 2nd ed. — stability and production-operations chapters Frames deployment, capacity, recovery, and observability as design concerns.

Quick Start: Your First 48 Hours

Day 1

  1. Install Livebook Desktop or start the official container.
  2. Read Theory Chapters 1 and 2.
  3. Open the built-in Learn notebooks for Livebook and Elixir.
  4. Start Project 1 with three sections: assumptions, transformations, evidence.
  5. Deliberately change an upstream assumption and observe stale descendants.

Day 2

  1. Move calculation logic into a named module and add explicit invariant cases.
  2. Reconnect the runtime and evaluate from the top.
  3. Save a replay manifest with runtime versions and input checksum.
  4. Open the .livemd diff in Git and verify it is readable and secret-free.
  5. Read Project 2’s Core Question before creating its first input.

At 48 hours, success means one cleanly replayable notebook and a written explanation of document state versus runtime state.

Path 1: Elixir Beginner

Project 1 -> Project 2 -> Project 3 -> Project 4 -> Project 5 -> Project 7 -> Project 12

Path 2: Data and Analytics Engineer

Project 1 -> Project 3 -> Project 5 -> Project 6 -> Project 7 -> Project 8 -> Project 10 -> Project 12

Path 3: Elixir Application Engineer

Project 1 -> Project 4 -> Project 7 -> Project 9 -> Project 11 -> Project 12

Path 4: Internal Tools / Solution Architect

Project 2 -> Project 6 -> Project 7 -> Project 8 -> Project 11 -> Project 12

Path 5: Numerical and ML Explorer

Project 1 -> Project 3 -> Project 5 -> Project 7 -> Project 10 -> Project 12

Success Metrics

  • Reconnect any completed notebook and reproduce its verified outcome without relying on old bindings or processes.
  • Explain which cells become stale after a source change and why.
  • Model expected failures with tagged data while allowing genuine defects to fail visibly.
  • Inject a process crash and explain the observed supervision and recovery events.
  • Profile an Explorer source, preserve row lineage, and justify the collection boundary.
  • Build a Kino interaction that behaves correctly with two simultaneous clients.
  • Produce a Smart Cell whose attributes round-trip and whose generated source is inspectable.
  • State tensor shape, type, backend, artifact revision, and decision threshold before trusting inference.
  • Compare standalone, remote-execution, and attached runtime authority.
  • Deploy a code-hidden app with least-privilege credentials, capacity evidence, authentication, audit signals, and rollback.

Livebook Debugging and Verification Appendix

When a cell works only after manual steps

  1. Reconnect the runtime.
  2. Evaluate Setup only and inspect dependency errors.
  3. Evaluate one ancestry path at a time.
  4. Search the notebook for undeclared file paths and environment variables.
  5. Verify live versus snapshot mode.

When clicks happen twice

  1. Assume duplicate listeners from re-evaluation.
  2. Inspect stored PIDs or listener handles.
  3. Stop old listeners or reconnect.
  4. Make startup idempotent before continuing.

When an app works for one browser but not two

  1. Identify regular shared inputs versus per-user forms.
  2. Confirm single- or multi-session mode.
  3. Log origin without treating it as authenticated identity.
  4. Test reconnect, simultaneous submit, and late response.

When data memory grows

  1. Inspect eager versus lazy frames.
  2. Move filters and projection before collection.
  3. Bound SQL rows and frame history.
  4. Record actual source and materialized sizes.

When inference latency varies

  1. Separate dependency download, model load, compilation, queue, and execution.
  2. Fix input shapes.
  3. Warm the serving path.
  4. Record batch size and device memory.

Project Overview Table

# Project Difficulty Time Primary Focus Coolness Business
1 Executable Energy Field Report Level 1 5–8 hrs .livemd, Elixir values, replay Level 2 Level 1
2 Interactive Capacity Planner Level 1 6–10 hrs Kino inputs, forms, validation Level 3 Level 2
3 Messy CSV Triage Clinic Level 2 10–16 hrs boundary contracts, tagged errors Level 3 Level 2
4 Fault-Tolerant Telemetry Lab Level 3 16–24 hrs processes, supervision, frames Level 4 Level 2
5 City Mobility Data Pipeline Level 2 12–20 hrs Explorer, lazy plans, Parquet Level 3 Level 2
6 Safe SQL Incident Runbook Level 2 12–20 hrs KinoDB, secrets, read-only SQL Level 3 Level 3
7 Interactive SLA Command Center Level 3 18–28 hrs VegaLite, events, responsive state Level 4 Level 3
8 Portable Cloud Data Lake Lab Level 3 16–26 hrs file references, S3, portability Level 3 Level 3
9 Tested Domain Smart Cell Toolkit Level 3 20–30 hrs Kino.JS.Live, source generation Level 4 Level 2
10 Nx + Bumblebee Image Triage Lab Level 3 20–32 hrs tensors, serving, evaluation Level 4 Level 2
11 Safe Attached-Runtime Console Level 3 20–30 hrs runtimes, distributed Elixir, safety Level 4 Level 3
12 Deployed Multi-Session Ops Decision App Level 4 35–50 hrs apps, auth, deploy, operations Level 4 Level 4

Project List

The following projects progress from deterministic Elixir transformations to interactive, concurrent, data-rich, and operated Livebook applications. Each expanded guide is linked from its project title.

Project 1: Executable Energy Field Report

  • File: P01-reproducible-field-report.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Erlang cells; Python integration for comparison only
  • Coolness Level: Level 2 — Practical but Forgettable
  • Business Potential: Level 1 — Resume Gold
  • Difficulty: Level 1 — Beginner
  • Knowledge Area: Literate programming, functional transformations, reproducibility
  • Software or Tool: Livebook, ExUnit, Git
  • Main Book: Elixir in Action, Third Edition

What you will build: A .livemd household energy report that transforms explicit appliance readings and tariff assumptions into costs, scenario comparisons, and a replay manifest.

Why it teaches Livebook + Elixir: It uses cells to expose binding ancestry and stale state while pure Elixir functions turn raw values into auditable results.

Core challenges you will face

  • Rebinding without mutation -> Functional Elixir
  • Cell ordering and stale descendants -> Livebook execution
  • Deterministic inputs and invariant tests -> Reproducibility

Real World Outcome

Open the notebook and evaluate from the top. The first visible section explains units and tariff assumptions. The second renders a compact table with appliance, watts, daily hours, monthly kilowatt-hours, and monthly cost. A scenario selector compares baseline, reduced-use, and efficient-device assumptions. The evidence section reports the runtime versions, source checksum, calculation policy, and replay result.

ENERGY AUDIT — JULY 2026
Input rows: 6          Tariff: R$ 0.91/kWh
Baseline: 184.2 kWh    Estimated cost: R$ 167.62
Efficient scenario: 141.7 kWh   Savings: R$ 38.68 (23.1%)

Verification
[PASS] total = sum(appliance totals)
[PASS] no negative hours or watts
[PASS] baseline - efficient = reported savings
[PASS] clean-runtime replay

Change the tariff cell and observe later cells become stale. Re-evaluate only the correct ancestry, then reconnect the runtime and prove the final output returns without any manual hidden step.

The Core Question You Are Answering

“What must an executable document declare so another person can reconstruct both the reasoning and the result?”

The project shifts attention from syntax completion to evidence: values, assumptions, transformations, and replay.

Concepts You Must Understand First

  1. Immutable values and rebinding
    • What changes when a name is rebound?
    • Book Reference: Elixir in Action, 3rd ed. — Chapter 2
  2. Pattern matching and function clauses
    • Which invalid appliance shapes should be expected errors?
    • Book Reference: Programming Elixir 1.6 — Chapters 3–5
  3. Evaluation ancestry
    • Which cells depend on tariff and which depend only on appliance data?
    • Primary Reference: Livebook introductory Learn notebook

Questions to Guide Your Design

  1. Which assumptions belong in input cells and which belong in named functions?
  2. How will units prevent mixing watts, kilowatts, hours, and currency?
  3. What invariants can be checked without duplicating the implementation?
  4. What must the replay manifest record?

Thinking Exercise

Draw the cell dependency graph before creating the notebook. Change one input on paper and mark exactly which outputs become stale. Then list three hidden inputs that would make two runs disagree.

The Interview Questions They Will Ask

  1. “How is Elixir rebinding different from mutation?”
  2. “What makes a notebook reproducible?”
  3. “Why move calculation logic into a module?”
  4. “What does Livebook’s stale marker tell you?”
  5. “How would you test a .livemd tutorial in CI?”

Hints in Layers

Hint 1: Separate story from calculation

Use Markdown for assumptions, cells for orchestration, and named functions for transformations.

Hint 2: Make units part of names and contracts

Normalize to one energy unit before calculating money.

Hint 3: Preserve stages

Keep raw readings, validated readings, and scenario results separately inspectable.

Hint 4: Destroy comfortable state

Reconnect and evaluate from the top; if it fails, record the missing declaration before fixing it.

Books That Will Help

Topic Book Chapter
Elixir data and functions Elixir in Action, 3rd ed. Ch. 2–3
Functions and modules Programming Elixir 1.6 Ch. 4–7
Executable documentation habits The Pragmatic Programmer Topics on knowledge portfolios and automation

Common Pitfalls and Debugging

Problem 1: “The report works until the runtime reconnects”

  • Why: A binding or module was created in an undeclared manual cell.
  • Fix: Put required setup on the saved evaluation path.
  • Quick test: Reconnect and use Evaluate all.

Problem 2: “A tariff change leaves plausible old totals”

  • Why: Stale cells were not re-evaluated.
  • Fix: Follow the dependency ancestry and verify final invariants.
  • Quick test: Compare displayed tariff with manifest tariff.

Problem 3: “Totals differ by cents”

  • Why: Floating-point money or inconsistent rounding.
  • Fix: Use decimal semantics and one declared rounding boundary.
  • Quick test: Sum line items and compare with the declared tolerance.

Definition of Done

  • One saved .livemd file contains assumptions, narrative, transformations, and evidence
  • Invalid negative or missing inputs produce explicit domain errors
  • All totals and scenario differences satisfy independent invariants
  • Upstream edits produce understood stale descendants
  • Git diff is readable and contains no generated noise or secret
  • A fresh runtime evaluates top to bottom and marks clean replay PASS

Project 2: Interactive Capacity Planning Workbench

  • File: P02-capacity-planner.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Erlang; JavaScript only inside an optional custom Kino
  • Coolness Level: Level 3 — Genuinely Clever
  • Business Potential: Level 2 — Micro-SaaS / Pro Tool
  • Difficulty: Level 1 — Beginner
  • Knowledge Area: Forms, validation, scenario modeling, per-user state
  • Software or Tool: Livebook, Kino.Input, Kino.Control, Kino.Frame
  • Main Book: Elixir in Action, Third Edition

What you will build: An interactive planner that converts traffic, latency, utilization, redundancy, and growth assumptions into capacity scenarios with warnings and sensitivity evidence.

Why it teaches Livebook + Elixir: Kino makes functional validation and transformation observable while two clients expose shared-input versus per-user form semantics.

Core challenges you will face

  • User-specific versus shared values -> Kino collaboration model
  • Validation without crashing the listener -> Tagged results
  • Late or repeated submissions -> Event-driven state

Real World Outcome

The app-like notebook shows a form on the left and a result panel on the right. Inputs include peak requests per second, measured service time, target utilization, growth rate, redundancy factor, and safety margin. Submit renders required worker count, projected headroom, warnings, and a small sensitivity table. Invalid values appear beside the result without erasing the previous valid scenario.

Open two browser windows. Each window enters different assumptions and receives its own form result. A separate shared “team baseline” input intentionally synchronizes, demonstrating the difference rather than hiding it.

Scenario: Holiday peak
Required concurrency: 238
Capacity with redundancy: 476
Recommended provisioned capacity: 550
Projected utilization: 73%
Status: PASS — 12 percentage points below target ceiling
Warning: result is sensitive to service time above 180 ms

The Core Question You Are Answering

“How does an immutable calculation become an interactive multi-user workflow without hiding state or validation?”

Concepts You Must Understand First

  1. Tagged validation results
    • How can several field errors be displayed together?
    • Book Reference: Elixir in Action — Chapters 3–4
  2. Kino Input and Control semantics
    • Which values synchronize and which submissions are user specific?
    • Primary Reference: Kino.Input and Kino.Control docs
  3. Frame lifecycle
    • When should output replace versus append?
    • Primary Reference: Kino.Frame docs

Questions to Guide Your Design

  1. Is target utilization a fraction or percentage at the domain boundary?
  2. Which validation rules are field-level and which are cross-field?
  3. How will a second client prove isolation?
  4. What happens if the user submits rapidly?

Thinking Exercise

Trace three submissions: invalid, valid, then valid from another client. Draw the event, origin, validation result, state transition, and frame target for each.

The Interview Questions They Will Ask

  1. “What is the difference between Kino.Input and a Kino.Control form?”
  2. “How do you prevent stale asynchronous results?”
  3. “Why preserve the previous valid result after an invalid submission?”
  4. “Where should form state live?”
  5. “How do you test two-client behavior?”

Hints in Layers

Hint 1: Start synchronous

Build a pure scenario function before adding any control.

Hint 2: Accumulate validation

Return all actionable field errors rather than failing on the first.

Hint 3: Route by collaboration intent

Use a form for per-user submission and a regular input only for explicitly shared baseline data.

Hint 4: Test with two real clients

Do not infer isolation from one browser; exercise simultaneous values and reconnect.

Books That Will Help

Topic Book Chapter
Data modeling and failures Elixir in Action, 3rd ed. Ch. 3–4
Capacity thinking Release It!, 2nd ed. Stability and capacity chapters
Usable interactive feedback Design It! Experiments and design feedback chapters

Common Pitfalls and Debugging

Problem 1: “Another browser changes my field”

  • Why: A shared Kino.Input was used for private workflow state.
  • Fix: Use a per-user form or multi-session app semantics.
  • Quick test: Type different values in two clients.

Problem 2: “One invalid number crashes the listener”

  • Why: Boundary parsing raised instead of returning a validation result.
  • Fix: Convert boundary failures into tagged errors.
  • Quick test: Submit blank, negative, and extreme values.

Problem 3: “Old work overwrites a newer scenario”

  • Why: Results lack a request generation.
  • Fix: Tag and render only the latest request.
  • Quick test: Submit two deliberately different slow scenarios quickly.

Definition of Done

  • Pure calculation and validation are independently tested
  • Form reports all actionable errors without crashing
  • Shared baseline and per-user scenario behavior are visibly distinct
  • Results state units, assumptions, formula version, and sensitivity warning
  • Two-client simultaneous and reconnect tests pass
  • Clean-runtime replay reconstructs the workbench

Project 3: Messy CSV Triage Clinic

  • File: P03-csv-triage-clinic.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Erlang; Python integration for a comparison notebook
  • Coolness Level: Level 3 — Genuinely Clever
  • Business Potential: Level 2 — Micro-SaaS / Pro Tool
  • Difficulty: Level 2 — Intermediate
  • Knowledge Area: Parsing, validation, data quality, redaction
  • Software or Tool: Livebook, Kino file input, Explorer or NimbleCSV, Decimal
  • Main Book: Designing Data-Intensive Applications, Second Edition

What you will build: A billing-file intake clinic that uploads a deliberately dirty CSV, separates accepted and rejected rows, accumulates reasons, reconciles counts, and exports sanitized evidence.

Why it teaches Livebook + Elixir: Pattern matching and tagged results turn malformed boundary data into visible domain evidence instead of a runtime crash.

Core challenges you will face

  • Encoding, delimiter, and schema ambiguity -> Boundary contracts
  • Decimal money, dates, duplicates, and multi-error rows -> Domain modeling
  • Useful evidence without PII leakage -> Redaction and provenance

Real World Outcome

Choose a training CSV from the notebook file input. The screen first shows file name, bytes, checksum, detected headers, and contract version. After triage, tabs show accepted rows, rejected rows with multiple reasons, error counts by category, and a reconciliation card.

Subscription Billing Intake
Rows read: 1,250
Accepted: 1,173
Rejected: 77
Reconciliation: 1,250 = 1,173 + 77  [PASS]

Top rejection reasons
invalid_decimal_amount ........ 31
missing_customer_id ........... 22
end_before_start .............. 18
duplicate_invoice_id .......... 14

Export: sanitized-accepted.parquet
Evidence: triage-manifest.json
PII scan: PASS

The rejected table masks email and payment references. A row with three defects reports all three. A deliberately malformed encoding produces a file-level error panel instead of partial misleading totals.

The Core Question You Are Answering

“How can a bulk workflow remain useful in the presence of bad data without silently accepting it or failing at the first defect?”

Concepts You Must Understand First

  1. Boundary versus domain representation
    • When does a row become a trusted subscription?
    • Book Reference: Domain Modeling Made Functional — Chapters 4–6
  2. Decimal and date semantics
    • Where is rounding allowed and which timezone applies?
    • Book Reference: Designing Data-Intensive Applications — data encoding sections
  3. Error accumulation
    • Which errors are independent and useful together?
    • Book Reference: Elixir in Action — Chapter 4

Questions to Guide Your Design

  1. Which file defects invalidate the entire import?
  2. Which row defects can be quarantined?
  3. How are duplicates defined across normalized fields?
  4. Which fields must never appear in evidence?

Thinking Exercise

Create a ten-row matrix where each row has zero to three defects. Predict accepted and rejected counts, category counts, and redacted output before executing.

The Interview Questions They Will Ask

  1. “When should validation fail fast versus accumulate errors?”
  2. “Why should financial data avoid binary floats?”
  3. “How do you distinguish malformed encoding from a domain error?”
  4. “What invariants detect silent row loss?”
  5. “How do you make a rejection report safe to share?”

Hints in Layers

Hint 1: Validate in layers

File -> header -> row structure -> normalized fields -> domain policy.

Hint 2: Keep original row identity

Use a source line or stable record ID so evidence remains traceable.

Hint 3: Reconcile categories

Accepted plus rejected must equal input; category totals may exceed rejected when rows have multiple reasons.

Hint 4: Redact before rendering

Do not render sensitive raw values and try to scrub screenshots later.

Books That Will Help

Topic Book Chapter
Domain validation Domain Modeling Made Functional Ch. 4–7
Data encodings and evolution Designing Data-Intensive Applications, 2nd ed. Ch. 4
Elixir failure modeling Elixir in Action, 3rd ed. Ch. 4

Common Pitfalls and Debugging

Problem 1: “Category totals exceed rejected rows”

  • Why: One row can have several reasons.
  • Fix: Report unique rejected rows separately from reason frequency.
  • Quick test: Use the ten-row prediction matrix.

Problem 2: “Valid-looking amount changes by a cent”

  • Why: Float parsing and repeated rounding.
  • Fix: Use decimal representation and one declared rounding rule.
  • Quick test: Include values known to be awkward in binary floating point.

Problem 3: “Evidence export leaks customer data”

  • Why: Redaction happened only in the UI.
  • Fix: Build exports from sanitized domain evidence.
  • Quick test: Scan the artifact for seeded canary values.

Definition of Done

  • File, header, row, and domain validation layers are distinct
  • Multiple row errors are accumulated and traceable
  • Decimal, date, timezone, duplicate, and empty-field policies are documented
  • Input equals accepted plus rejected for every test fixture
  • UI and exports contain no seeded sensitive canary values
  • Manifest records checksum, schema, package versions, and clean replay

Project 4: Fault-Tolerant Telemetry Lab

  • File: P04-telemetry-supervision-lab.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Erlang; Python for telemetry-analysis comparison
  • Coolness Level: Level 4 — Hardcore Tech Flex
  • Business Potential: Level 2 — Micro-SaaS / Pro Tool
  • Difficulty: Level 3 — Advanced
  • Knowledge Area: BEAM processes, supervision, backpressure, interactive observability
  • Software or Tool: Livebook, Kino.Control, Kino.Frame, Supervisor
  • Main Book: Elixir in Action, Third Edition

What you will build: A simulated sensor system whose live dashboard survives an injected producer crash and visibly proves supervision, recovery, bounded history, and cleanup.

Why it teaches Livebook + Elixir: BEAM failure semantics become observable through process identities, event transcripts, and Kino frames instead of remaining an architecture diagram.

Core challenges you will face

  • Process ownership and message protocol -> OTP design
  • Crash recovery without pretending state persisted -> Supervision
  • Event rate greater than render rate -> Backpressure

Real World Outcome

The notebook renders four sensor cards, a bounded time-series frame, mailbox/lag indicators, and an event log. Select a sensor and click “Inject fault.” That card briefly shows recovering, the event log records the exit, and the new PID appears while other readings continue.

14:32:10 sensor-2 PID<0.241.0> temperature=21.8
14:32:12 action=inject_fault target=sensor-2
14:32:12 DOWN PID<0.241.0> reason=simulated_failure
14:32:12 supervisor restart sensor-2 -> PID<0.255.0>
14:32:13 sensor-2 PID<0.255.0> temperature=21.7

Stream continuity: PASS
Window: last 120 points
Dropped/coalesced events: 34 (reported)

An “Orderly stop” action terminates producers, listeners, and timers. Re-evaluating the start cell does not duplicate the system.

The Core Question You Are Answering

“What does fault tolerance actually preserve, restart, discard, and reveal?”

Concepts You Must Understand First

  1. Processes and mailboxes
    • Who owns the aggregate window?
    • Book Reference: Elixir in Action — Chapters 5–6
  2. Links, monitors, and supervisors
    • Which components should fail together?
    • Book Reference: Elixir in Action — Chapters 7–9
  3. Kino event handling
    • What happens when rendering is slower than production?
    • Primary Reference: Kino.Control and Kino.Frame docs

Questions to Guide Your Design

  1. What is the reading message contract?
  2. Which state is derived and safe to lose?
  3. How will delayed messages from an old PID be rejected?
  4. What queue or window bound proves backpressure is intentional?

Thinking Exercise

Draw the supervisor tree and failure matrix. For every process, predict what happens when it crashes, whether it restarts, and how the UI learns about the event.

The Interview Questions They Will Ask

  1. “How do links and monitors differ?”
  2. “Why does supervision not equal persistence?”
  3. “What causes mailbox growth?”
  4. “How do you avoid duplicated notebook listeners?”
  5. “What should happen after restart intensity is exceeded?”

Hints in Layers

Hint 1: One state owner

Make the aggregator the only owner of the current window.

Hint 2: Separate measurement and rendering cadence

Aggregate faster readings and render less often.

Hint 3: Version producers

Associate messages with sensor generation or current PID.

Hint 4: Preserve handles

Keep a system handle that can stop every child and listener idempotently.

Books That Will Help

Topic Book Chapter
Processes and servers Elixir in Action, 3rd ed. Ch. 5–6
Fault tolerance Elixir in Action, 3rd ed. Ch. 7–9
Supervision-oriented design Designing Elixir Systems with OTP Ch. 3–8

Common Pitfalls and Debugging

Problem 1: “Every reading appears twice”

  • Why: The start cell launched a second listener.
  • Fix: Stop the prior system or make startup return the existing instance.
  • Quick test: Start twice and verify one event per producer interval.

Problem 2: “Memory grows during the demo”

  • Why: Frames or mailboxes are unbounded.
  • Fix: Bound window, coalesce updates, and report drops.
  • Quick test: Run a high-rate fixture for ten minutes and inspect counts.

Problem 3: “The dashboard dies with a sensor”

  • Why: UI observer was linked unintentionally.
  • Fix: Use the intended supervision and monitoring relationships.
  • Quick test: Inject one producer fault and verify UI PID continuity.

Definition of Done

  • Supervisor tree and message contracts are documented
  • One injected crash shows old PID, reason, new PID, and recovery time
  • Other producers and aggregate output continue through the fault
  • Queue/history bounds and dropped/coalesced counts are visible
  • Repeated startup and orderly stop are idempotent
  • Reconnect leaves no reliance on prior processes and replay passes

Project 5: City Mobility Data Pipeline

  • File: P05-mobility-data-pipeline.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: SQL; Python integration for result comparison
  • Coolness Level: Level 3 — Genuinely Clever
  • Business Potential: Level 2 — Micro-SaaS / Pro Tool
  • Difficulty: Level 2 — Intermediate
  • Knowledge Area: DataFrames, lazy queries, data quality, columnar formats
  • Software or Tool: Livebook, Explorer, KinoExplorer, Parquet
  • Main Book: Data Science for Business

What you will build: A typed mobility pipeline that profiles raw trips, quarantines defects, enriches zones, computes service metrics lazily, writes Parquet, and emits a manifest.

Why it teaches Livebook + Elixir: Explorer makes immutable analytical transformations and collection boundaries visible inside a replayable notebook.

Core challenges you will face

  • Inferred versus declared data types -> Schema contracts
  • Join cardinality and row preservation -> Analytical correctness
  • Lazy optimization versus eager evidence -> Collection boundaries

Real World Outcome

The notebook begins with a source card showing file, bytes, checksum, row count, and inferred versus declared schema. A quality table reports nulls, duplicates, invalid durations, unknown zones, and quarantined rows. The pipeline-plan section explains which operations remain lazy. A final dashboard shows trips by day, median and p95 duration, and zone coverage.

CITY MOBILITY PIPELINE
Source rows: 2,400,000
Accepted rows: 2,372,814
Quarantined rows: 27,186
Row reconciliation: PASS
Zone join unmatched keys: 43
Collection boundary: daily_zone_summary (3,648 rows)
Output: mobility-clean.parquet
Manifest checksum: sha256:REDACTED_EXAMPLE

The output directory contains cleaned Parquet, quarantine Parquet, and a human-readable manifest. A second run against the same snapshot produces matching counts and checksums.

The Core Question You Are Answering

“Where does an analytical pipeline actually execute, and what evidence proves that no rows or meanings disappeared?”

Concepts You Must Understand First

  1. Series, DataFrames, and dtypes
    • Which apparent numbers are identifiers?
    • Primary Reference: Explorer.DataFrame docs
  2. Lazy execution and collection
    • Which operations should be pushed before materialization?
    • Primary Reference: Ten Minutes to Explorer
  3. Join cardinality
    • Can zone keys multiply trip rows?
    • Book Reference: Designing Data-Intensive Applications — Chapter 3

Questions to Guide Your Design

  1. What is the source timezone and interval convention?
  2. Which invalid rows are quarantined rather than repaired?
  3. What before/after evidence surrounds every join?
  4. Why is the chosen collection boundary bounded?

Thinking Exercise

Given a five-row trip table and a zone reference with one duplicate key, calculate the results of inner and left joins by hand. Identify the invariant that reveals multiplication.

The Interview Questions They Will Ask

  1. “What is the difference between eager and lazy DataFrames?”
  2. “What is predicate pushdown?”
  3. “How do you detect a many-to-many join accidentally?”
  4. “How should null and invalid values differ?”
  5. “Why choose Parquet for the output?”

Hints in Layers

Hint 1: Profile before cleaning

Save the first source profile as evidence before transformations.

Hint 2: Keep quality flags

Do not turn every invalid meaning into a null.

Hint 3: Push down early

Project columns and filter partitions before collection.

Hint 4: Manifest the boundary

Record source, plan revision, counts, date range, output bytes, and checksums.

Books That Will Help

Topic Book Chapter
Analytical problem framing Data Science for Business Ch. 2–3
Storage and encoding Designing Data-Intensive Applications, 2nd ed. Ch. 3–4
Data contracts Fundamentals of Software Architecture Architecture characteristics chapters

Common Pitfalls and Debugging

Problem 1: “Trip count increases after enrichment”

  • Why: Zone reference keys are not unique.
  • Fix: Validate cardinality before joining.
  • Quick test: Compare unique keys and before/after row counts.

Problem 2: “The lazy pipeline consumes all memory”

  • Why: An early collection or full unbounded operation materialized data.
  • Fix: Inspect stage types and move filtering/projection earlier.
  • Quick test: Record rows and bytes at the collection boundary.

Problem 3: “Durations become negative around time changes”

  • Why: Naive local timestamps or inconsistent timezone rules.
  • Fix: Normalize instants and document the source timezone.
  • Quick test: Include daylight-saving and midnight-crossing fixtures.

Definition of Done

  • Declared schema is compared with inferred schema
  • Source, accepted, and quarantined rows reconcile
  • Every join has key uniqueness and before/after evidence
  • Lazy plan and collection boundary are explained
  • Parquet outputs and manifest have stable checksums for the snapshot
  • Clean-runtime replay reproduces the same bounded summary

Project 6: Safe SQL Incident Investigation Runbook

  • File: P06-safe-sql-runbook.md
  • Main Programming Language: Elixir and SQL
  • Alternative Programming Languages: Erlang; Python for notebook comparison
  • Coolness Level: Level 3 — Genuinely Clever
  • Business Potential: Level 3 — Service & Support
  • Difficulty: Level 2 — Intermediate
  • Knowledge Area: Databases, operational diagnostics, security, evidence
  • Software or Tool: Livebook, KinoDB, PostgreSQL or SQLite training database
  • Main Book: Designing Data-Intensive Applications, Second Edition

What you will build: A read-only incident runbook that performs parameterized support queries, bounds and redacts results, visualizes anomalies, and exports an audit-style investigation transcript.

Why it teaches Livebook + Elixir: SQL Smart Cells accelerate exploration while Elixir modules enforce input, authorization, redaction, and evidence contracts around the query.

Core challenges you will face

  • Convenient UI versus safe query boundary -> Least privilege
  • PII-rich results versus shareable evidence -> Redaction
  • Fast diagnosis versus unbounded production load -> Time and row budgets

Real World Outcome

Use only the disposable training database. Choose an incident window and service, then submit. The runbook shows credential role, read-only confirmation, query timeout, row cap, data freshness, and redaction policy before results. Tabs present an anomaly summary, bounded sample, query timing, and executed-step transcript.

RUNBOOK: delayed_invoice_sync
Environment: TRAINING
Database role: support_readonly
Transaction mode: READ ONLY
Window: 2026-07-15 12:00Z..13:00Z
Rows scanned estimate: 82,410
Rows returned: 214 / limit 500
Query duration: 184 ms
PII redaction: PASS
Mutation capability: NONE

A seeded injection string is treated as a value and returns no unauthorized rows. A deliberately broad query stops at the budget and explains how to narrow it. The export includes query identifiers and parameters in redacted form, not secrets.

The Core Question You Are Answering

“How can an executable runbook accelerate diagnosis without becoming an unbounded production shell?”

Concepts You Must Understand First

  1. Parameterized SQL
    • Why must values remain separate from syntax?
    • Book Reference: Security in Computing — database security sections
  2. Least privilege and read-only transactions
    • What can the credential do outside the intended query?
    • Book Reference: Foundations of Information Security — access control chapters
  3. Operational evidence
    • Which facts must survive the app session?
    • Book Reference: Release It!, 2nd ed. — operations chapters

Questions to Guide Your Design

  1. Which queries are allowlisted?
  2. How are time window and row limit enforced twice?
  3. Which columns are redacted, aggregated, or forbidden?
  4. What does an empty result mean versus a failed query?

Thinking Exercise

Threat-model the runbook from four perspectives: malicious input, mistaken operator, compromised credential, and slow database. Give each risk a visible control and verification.

The Interview Questions They Will Ask

  1. “Why is a Smart Cell not a security boundary?”
  2. “How do parameterized queries prevent injection?”
  3. “What does read-only protect and not protect?”
  4. “How do you bound diagnostic database load?”
  5. “What belongs in an audit transcript?”

Hints in Layers

Hint 1: Begin with a query catalog

Do not accept arbitrary SQL in the completed runbook.

Hint 2: Enforce bounds at multiple layers

Validate parameters, add query limits, and set database timeouts.

Hint 3: Redact by construction

Select or derive safe columns rather than fetching everything and hiding later.

Hint 4: Separate UI identity and database authority

The app user is not automatically the database user; map permissions explicitly.

Books That Will Help

Topic Book Chapter
Data systems and indexes Designing Data-Intensive Applications, 2nd ed. Ch. 1, 3
Access control Foundations of Information Security Access-control chapters
Operational stability Release It!, 2nd ed. Stability and operations chapters

Common Pitfalls and Debugging

Problem 1: “The row limit still runs an expensive query”

  • Why: Limiting returned rows does not guarantee a cheap plan.
  • Fix: Narrow indexed predicates, inspect the plan, and set timeout.
  • Quick test: Run the deliberately broad training query.

Problem 2: “Redacted table is safe but export leaks PII”

  • Why: UI and export use different data paths.
  • Fix: Create one sanitized evidence representation.
  • Quick test: Seed canary values and scan every artifact.

Problem 3: “Attached notebook has more authority than expected”

  • Why: Runtime boundary was confused with a read-only database client.
  • Fix: Use standalone runtime and explicit read-only connection for this project.
  • Quick test: Confirm runtime type and database role in the header.

Definition of Done

  • Training environment and read-only role are displayed before queries
  • Query catalog uses parameters, allowlisted operations, timeouts, and row bounds
  • Injection, empty, timeout, stale-data, and partial-result cases are tested
  • UI and exports pass seeded PII canary scans
  • Investigation transcript records identity context, time, revision, parameters, and outcome
  • No production mutation path exists and clean replay passes

Project 7: Interactive SLA Command Center

  • File: P07-sla-command-center.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: SQL; JavaScript only for optional custom visualization
  • Coolness Level: Level 4 — Hardcore Tech Flex
  • Business Potential: Level 3 — Service & Support
  • Difficulty: Level 3 — Advanced
  • Knowledge Area: Operational analytics, event-driven UI, visualization
  • Software or Tool: Livebook, Kino, Explorer, VegaLite, KinoVegaLite
  • Main Book: Data Science for Business

What you will build: A responsive SLA dashboard whose filters update KPI cards, latency distributions, breach trends, and bounded evidence without manually re-evaluating cells.

Why it teaches Livebook + Elixir: It joins Explorer’s immutable summaries with Kino event processes and declarative chart specifications.

Core challenges you will face

  • Honest aggregation and encoding -> Analytical correctness
  • Rapid filters and slow queries -> Cancellation and stale results
  • Loading, empty, partial, and failed states -> Operational UX

Real World Outcome

The command center opens with a status strip showing source, last refresh, current request generation, and data freshness. Filters select period, service, severity, and region. KPI cards show request count, availability, breach rate, median, p95, and p99 latency. A histogram preserves distribution shape, a trend chart shows breaches over time, and a table drills into bounded incidents.

SERVICE: checkout-api       WINDOW: last 24 hours
Requests: 8,241,993         Availability: 99.94%
SLA breaches: 0.37%         p50 / p95 / p99: 42 / 211 / 804 ms
Data age: 47 seconds        Status: CURRENT
Filter generation: 18
Late result generation 17: discarded

Rapidly change the service three times. The loading state identifies the current generation, and an older slow response is discarded. Select a period with no data and see an intentional empty-state explanation rather than zero-filled metrics.

The Core Question You Are Answering

“How do you keep an interactive analytical view truthful while users and data change faster than computation?”

Concepts You Must Understand First

  1. Visual encodings and denominators
    • What population defines the breach rate?
    • Book Reference: Data Science for Business — Chapters 2–3
  2. Event streams and frame updates
    • Which process owns current filter generation?
    • Primary Reference: Kino.Control and Kino.Frame
  3. Percentiles and distributions
    • Why can an average hide an outage tail?
    • Book Reference: Designing Data-Intensive Applications — performance sections

Questions to Guide Your Design

  1. Which calculations can be pre-aggregated?
  2. How will obsolete work be canceled or ignored?
  3. Which chart encodings make comparison precise?
  4. How will partial source failure differ from a true zero?

Thinking Exercise

Sketch five UI states—initial, loading, ready, empty, and failed. For each, state what data is safe to display and what must be removed or labeled stale.

The Interview Questions They Will Ask

  1. “How do you prevent race conditions in dashboard filtering?”
  2. “Why is p95 insufficient without a population and window?”
  3. “How do you test an empty analytical result?”
  4. “When should a dashboard pre-aggregate?”
  5. “What is wrong with updating charts on every keystroke?”

Hints in Layers

Hint 1: One request generation

Increment a reference for every accepted filter change.

Hint 2: Render state, not fragments

Make loading, ready, empty, and failed explicit state variants.

Hint 3: Prefer bounded summaries

Do not send raw millions of points to the browser.

Hint 4: Test deliberate delay

Make one filter response slow and prove it cannot overwrite a newer result.

Books That Will Help

Topic Book Chapter
Analytical framing Data Science for Business Ch. 2–3
Performance distributions Designing Data-Intensive Applications, 2nd ed. Ch. 1
Resilient UI state Release It!, 2nd ed. Timeouts and stability patterns

Common Pitfalls and Debugging

Problem 1: “Old data flashes after a new filter”

  • Why: Completion order is confused with request order.
  • Fix: Render only the current generation.
  • Quick test: Inject variable delays into three rapid requests.

Problem 2: “Empty period displays perfect availability”

  • Why: Missing denominator was converted to zero incidents.
  • Fix: Render a no-population state.
  • Quick test: Select a known empty window.

Problem 3: “Browser becomes slow over time”

  • Why: Chart or log history is unbounded.
  • Fix: Replace bounded datasets and cap drill-down rows.
  • Quick test: Run repeated refreshes and observe point counts.

Definition of Done

  • Every KPI names unit, population, window, and freshness
  • Loading, ready, empty, partial, stale, and failed states are intentional
  • Late results cannot overwrite current filter state
  • Charts use bounded data and documented encodings
  • Two clients can filter independently where intended
  • Clean-runtime replay and a 30-minute interaction soak test pass

Project 8: Portable Cloud Data Lake Lab

  • File: P08-portable-cloud-data-lab.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: SQL; Python integration for interoperability checks
  • Coolness Level: Level 3 — Genuinely Clever
  • Business Potential: Level 3 — Service & Support
  • Difficulty: Level 3 — Advanced
  • Knowledge Area: File systems, object storage, data portability, security
  • Software or Tool: Livebook file references, Kino.FS, Explorer, S3-compatible storage
  • Main Book: Designing Data-Intensive Applications, Second Edition

What you will build: One notebook that runs against a local snapshot and an S3-compatible dataset, produces equivalent summaries, and records storage, cache, partition, and checksum evidence.

Why it teaches Livebook + Elixir: Livebook-managed file references and Explorer expose the difference between a portable resource contract and a hard-coded local path.

Core challenges you will face

  • Local path versus managed reference -> Portability
  • Large remote object versus bounded scan -> Resource planning
  • Shared storage credentials and stamps -> Security

Real World Outcome

The notebook’s source selector offers “local training snapshot” and “S3-compatible training bucket.” Both render the same schema and summary contract. The source panel shows logical reference, object version or checksum, local-cache behavior, selected partitions, bytes scanned, and whether the path is treated read-only.

PORTABILITY CHECK
Local source summary checksum:  8f0d...c21
S3 source summary checksum:     8f0d...c21
Schema equivalence: PASS
Row-count equivalence: PASS
Selected partitions: date=2026-07-01..2026-07-07
Rows collected: 4,218
Secret names used: [LB_TRAINING_S3_ACCESS] (values never displayed)

Disable the object store and rerun. The notebook reports remote unavailability, offers the declared snapshot fallback, and labels the result as cached rather than current.

The Core Question You Are Answering

“What must remain stable when the same analytical method moves between local disk, object storage, and another operator’s machine?”

Concepts You Must Understand First

  1. Logical reference versus resolved path
    • Who owns the cached local file?
    • Primary Reference: Kino.FS docs
  2. Columnar storage and partitions
    • How do selected columns and partitions reduce work?
    • Book Reference: Designing Data-Intensive Applications — Chapter 3
  3. Secrets and stamps
    • What approval does a stamp remember?
    • Primary Reference: Livebook stamping and shared storage docs

Questions to Guide Your Design

  1. Which metadata proves local and remote snapshots are equivalent?
  2. When may fallback occur, and how is freshness labeled?
  3. Can the selected format avoid reading the whole object?
  4. What happens when another user opens the notebook without storage access?

Thinking Exercise

Draw the lifecycle of an S3 file reference: notebook source, credential lookup, download or scan, resolved local path, transformation, and output. Mark trust and ownership changes.

The Interview Questions They Will Ask

  1. “Why should Kino.FS resolved paths be treated read-only?”
  2. “What is predicate or partition pruning?”
  3. “How do notebook stamps affect file access?”
  4. “How do you distinguish cached and current results?”
  5. “What makes an object-store workflow reproducible?”

Hints in Layers

Hint 1: Define one source adapter contract

Downstream transformations should not branch on arbitrary paths.

Hint 2: Compare manifests

Use schema, row counts, ranges, and checksums rather than visual similarity.

Hint 3: Fail closed on credentials

Never fall back to a broader credential or public bucket silently.

Hint 4: Simulate disconnection

Prove the UI labels fallback state and does not claim current data.

Books That Will Help

Topic Book Chapter
Storage and file layout Designing Data-Intensive Applications, 2nd ed. Ch. 3
Security boundaries Foundations of Information Security Access-control chapters
Cloud failure behavior Release It!, 2nd ed. Integration and timeout chapters

Common Pitfalls and Debugging

Problem 1: “Remote mode downloads far more data than expected”

  • Why: Format or operation cannot prune before download.
  • Fix: Select partitions/columns and measure actual bytes.
  • Quick test: Compare source object size with bytes read.

Problem 2: “Notebook works only on the author’s Mac”

  • Why: A hard-coded absolute path escaped the file-reference contract.
  • Fix: Keep source selection logical and resolve through Livebook.
  • Quick test: Run in a clean container or second machine profile.

Problem 3: “Modified notebook still accesses old resources”

  • Why: Stamp and permission expectations are misunderstood.
  • Fix: Inspect stamp status and require explicit reapproval where appropriate.
  • Quick test: Save a changed copy and observe resource authorization behavior.

Definition of Done

  • Local and S3-compatible modes use the same downstream contract
  • Equivalence is verified with schema, counts, ranges, and checksums
  • Resolved file paths are treated read-only
  • Credentials never appear in source, output, logs, or exports
  • Remote unavailable, unauthorized, stale cache, and partial read are tested
  • A fresh runtime in a second environment reproduces the declared snapshot result

Project 9: Tested Domain Smart Cell Toolkit

  • File: P09-smart-cell-toolkit.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: JavaScript; TypeScript for bundled UI assets; Erlang for generated-source comparison
  • Coolness Level: Level 4 — Hardcore Tech Flex
  • Business Potential: Level 2 — Micro-SaaS / Pro Tool
  • Difficulty: Level 3 — Advanced
  • Knowledge Area: Source generation, collaborative UI, packaging, testing
  • Software or Tool: Kino.JS, Kino.JS.Live, Kino.SmartCell, Kino.Test
  • Main Book: Domain Specific Languages

What you will build: A reusable Smart Cell package that configures a safe data-quality rule, persists its UI state, generates reviewable Elixir source, and tests attribute and event behavior.

Why it teaches Livebook + Elixir: It reveals Smart Cells as transparent domain-specific source generators backed by BEAM processes and browser events.

Core challenges you will face

  • JSON-friendly state versus Elixir domain values -> Serialization boundary
  • Friendly UI versus deterministic inspectable source -> DSL design
  • Collaborative edits and evaluation synchronization -> Smart Cell lifecycle

Real World Outcome

Install the local package into a training notebook and choose “Data quality rule” from the Smart menu. The cell UI lets the author select a variable, column, rule type, severity, message, and null behavior. An adjacent preview shows the source that evaluation will generate. Save, close, reopen, and confirm every attribute returns.

Convert the Smart Cell to a code cell. The generated source is formatted, readable, uses no hidden state, and produces the same result. The package test transcript reports attribute round-trip, invalid variable rejection, collaborative editor synchronization, source export, and event rendering.

SMART CELL TEST SUMMARY
attribute round-trip ........ PASS
invalid variable name ...... PASS
source deterministic ....... PASS
convert-to-code parity ..... PASS
collaborative sync ......... PASS
malformed attrs recovery ... PASS

The Core Question You Are Answering

“How do you give a domain workflow a friendly interface without replacing readable source with opaque magic?”

Concepts You Must Understand First

  1. Source generation and AST boundaries
    • Which values may be emitted safely?
    • Book Reference: Metaprogramming Elixir — AST and quoting chapters
  2. Kino.JS.Live state
    • Which data belongs in assigns, attrs, and browser state?
    • Primary Reference: Kino.SmartCell docs
  3. Package API design
    • What remains stable when the UI evolves?
    • Book Reference: Domain Specific Languages — representation chapters

Questions to Guide Your Design

  1. Are generated variable names validated as identifiers?
  2. Can equivalent attributes produce byte-stable source?
  3. What happens when older saved attrs lack a new field?
  4. How does the user recover from invalid configuration?

Thinking Exercise

Write the attribute schema and three migrations on paper: current, missing optional field, and unknown future field. Predict restored UI and generated source for each.

The Interview Questions They Will Ask

  1. “What is a Smart Cell under the hood?”
  2. “Why must attributes be JSON friendly?”
  3. “How do you prevent source injection?”
  4. “What does convert-to-code parity prove?”
  5. “How do collaborative editor changes reach evaluation?”

Hints in Layers

Hint 1: Start with deterministic source

Build and test attribute-to-source conversion before the custom UI.

Hint 2: Validate identifiers separately

Never interpolate arbitrary text as a variable or module name.

Hint 3: Version attributes

Use defaults and tolerant restoration for older cells.

Hint 4: Test through the Kino bridge

Exercise events and export with Kino.Test rather than only pure helpers.

Books That Will Help

Topic Book Chapter
DSL representation Domain Specific Languages Representation and semantic model chapters
Elixir AST Metaprogramming Elixir Ch. 1–4
Stable library design The Pragmatic Programmer Tracer bullets and orthogonality topics

Common Pitfalls and Debugging

Problem 1: “Cell reopens with missing values”

  • Why: Runtime assigns were never serialized through attrs.
  • Fix: Define and test complete attribute export/restoration.
  • Quick test: Save, close, reopen, and compare source.

Problem 2: “Generated code changes ordering randomly”

  • Why: Map iteration or formatting is nondeterministic.
  • Fix: Canonicalize ordering and formatting.
  • Quick test: Export the same attrs repeatedly and compare bytes.

Problem 3: “A crafted label changes generated source”

  • Why: Untrusted text was interpolated into syntax.
  • Fix: Keep labels data, validate identifiers, and construct syntax safely.
  • Quick test: Use quotes, newlines, interpolation markers, and invalid identifiers.

Definition of Done

  • Smart Cell persists and restores all JSON-friendly attributes
  • Attribute-to-source conversion is deterministic and readable
  • Invalid identifiers and malformed attrs fail safely
  • Convert-to-code produces equivalent observable behavior
  • Kino.Test covers events, synchronization, export, and restoration
  • Package documentation includes a clean-runtime example and no full solution shortcut

Project 10: Nx + Bumblebee Image Triage Lab

  • File: P10-nx-image-triage.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Python; Julia for external numerical comparison
  • Coolness Level: Level 4 — Hardcore Tech Flex
  • Business Potential: Level 2 — Micro-SaaS / Pro Tool
  • Difficulty: Level 3 — Advanced
  • Knowledge Area: Tensors, pretrained models, inference serving, evaluation
  • Software or Tool: Livebook, Kino.Input.image, Nx, Bumblebee, Nx.Serving
  • Main Book: Machine Learning in Elixir and Livebook

What you will build: A multi-session image triage lab that validates uploads, exposes tensor contracts, runs a pinned pretrained model, measures serving behavior, and requires human confirmation or correction.

Why it teaches Livebook + Elixir: Livebook makes model provenance, Nx shapes, warmup, inference latency, failure cases, and human feedback visible in one executable artifact.

Core challenges you will face

  • Image bytes to correct tensor shape and type -> Numerical contracts
  • Model artifact and cache reproducibility -> Provenance
  • Score versus trustworthy decision -> Evaluation and human override

Real World Outcome

Open the app preview in two clients. Each uploads a different training image and sees a private preview, validation summary, preprocessing description, tensor shape/type, ranked suggestions, threshold decision, queue time, inference time, and model revision. The user accepts, corrects, or marks “none of these.”

IMAGE TRIAGE
Input: 1280 x 853 JPEG, 412 KiB
Tensor: [1, 224, 224, 3] f32
Model: repository@revision (digest recorded)
Top suggestion: damaged_package   score=0.81
Decision: SUGGEST — human confirmation required
Queue / inference / total: 3 / 47 / 61 ms
Warm path: yes
Retention: input discarded after session

The evaluation section runs a labeled fixture set including ambiguous, corrupt, oversized, and out-of-distribution images. It reports per-class errors and abstention, not just successful demos.

The Core Question You Are Answering

“What evidence must surround a model prediction before it can safely influence a real workflow?”

Concepts You Must Understand First

  1. Tensor shape, type, and backend
    • Which axis order does the model expect?
    • Book Reference: Machine Learning in Elixir and Livebook — Nx chapters
  2. Pretrained artifact provenance
    • Can the exact model be loaded offline?
    • Primary Reference: Bumblebee docs
  3. Serving and evaluation
    • What latency is queueing, compilation, and inference?
    • Primary Reference: Nx.Serving docs

Questions to Guide Your Design

  1. What upload size and dimensions are accepted?
  2. Which resize/crop policy preserves relevant evidence?
  3. When does the app abstain?
  4. What data is retained after the session?

Thinking Exercise

Trace one image from browser bytes through validation, preprocessing, tensor creation, batching, model output, threshold, and human correction. Mark every allocation, trust boundary, and failure state.

The Interview Questions They Will Ask

  1. “What is broadcasting and how can it hide a shape bug?”
  2. “Why separate cold-start and warm inference latency?”
  3. “What does Nx.Serving add around a model?”
  4. “Why is a score not necessarily probability?”
  5. “How do you deploy a pretrained model offline?”

Hints in Layers

Hint 1: Make the contract visible before inference

Render input bounds, target shape, type, and model revision.

Hint 2: Warm once, measure twice

Report cold preparation separately from steady-state requests.

Hint 3: Include abstention

Low or ambiguous results should not be forced into a label.

Hint 4: Test concurrent clients

Measure queue and batch behavior without mixing private uploads or outputs.

Books That Will Help

Topic Book Chapter
Nx and model workflows Machine Learning in Elixir and Livebook Nx and pretrained-model chapters
Production inference AI Engineering Inference and evaluation chapters
Analytical evaluation Data Science for Business Ch. 7–8

Common Pitfalls and Debugging

Problem 1: “First request takes seconds; later ones take milliseconds”

  • Why: Download, model load, and compilation are mixed with inference.
  • Fix: Warm deliberately and report phases separately.
  • Quick test: Compare cold and five warm requests.

Problem 2: “Concurrent users see each other’s image”

  • Why: Shared input or frame semantics were used.
  • Fix: Use multi-session or client-targeted per-user state.
  • Quick test: Upload distinct canary images from two clients.

Problem 3: “Model deploy fails without internet”

  • Why: Artifact was fetched lazily at runtime.
  • Fix: Pin, prefetch, verify digest, and package/cache for offline use.
  • Quick test: Start with network disabled.

Definition of Done

  • Upload byte, format, and dimension bounds are enforced before heavy allocation
  • Tensor shape, type, backend, and memory estimate are displayed
  • Model repository, revision, digest, label mapping, and license are recorded
  • Cold, queue, inference, and total latency are measured separately
  • Labeled error set includes corrupt, ambiguous, and out-of-distribution inputs
  • Two-client privacy, abstention, human correction, and clean replay pass

Project 11: Safe Attached-Runtime Triage Console

  • File: P11-attached-runtime-console.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Erlang; Gleam on a disposable BEAM target
  • Coolness Level: Level 4 — Hardcore Tech Flex
  • Business Potential: Level 3 — Service & Support
  • Difficulty: Level 3 — Advanced
  • Knowledge Area: Distributed Elixir, live diagnostics, trust boundaries
  • Software or Tool: Livebook runtimes, Remote Execution Smart Cell, disposable Phoenix node
  • Main Book: Elixir in Action, Third Edition

What you will build: A disposable-node diagnostic console that compares standalone, remote-execution, and attached modes while enforcing allowlisted read-only observations, timeouts, and bounded output.

Why it teaches Livebook + Elixir: The learner sees exactly where evaluation runs, which packages are available, and how runtime choice changes authority and failure coupling.

Core challenges you will face

  • Notebook dependency needs versus target-node purity -> Runtime selection
  • Distributed-node access versus least privilege -> Security
  • Useful inspection versus target impact -> Bounded diagnostics

Real World Outcome

Start a disposable local Phoenix or OTP application with an explicit node and training cookie. The notebook displays a three-column runtime comparison. In standalone mode it proves package installation and isolation. Through Remote Execution it runs one allowlisted diagnostic on the target while retaining local Kino output. In attached mode it proves the evaluator PID belongs to the target and shows that new Mix.install dependencies are unavailable.

TRIAGE CONSOLE — DISPOSABLE NODE ONLY
Notebook runtime: standalone@127.0.0.1
Target node: training_app@127.0.0.1
Connection: healthy
Allowed diagnostics: process_summary, queue_depths, app_versions
Result limit: 100 rows / 256 KiB
Timeout: 2,000 ms
Mutation capability: NONE
Audit transcript: enabled

Disconnect the target and observe bounded failure. Run an intentionally slow diagnostic and see timeout without freezing the notebook. The guide explicitly stops before production connection.

The Core Question You Are Answering

“Where does notebook code execute, and how does that location change dependencies, authority, risk, and failure?”

Concepts You Must Understand First

  1. Erlang distribution
    • How do node name and cookie participate in connection?
    • Book Reference: Elixir in Action — distributed systems chapter
  2. Runtime types
    • Why can standalone install packages while attached cannot?
    • Primary Reference: Livebook runtimes docs
  3. Read-only diagnostic design
    • What makes observation bounded?
    • Book Reference: Release It!, 2nd ed. — operational stability chapters

Questions to Guide Your Design

  1. Which diagnostics are allowlisted and why?
  2. How is output size bounded before rendering?
  3. What can the target credential or cookie reach?
  4. Which mode best supports charts without loading packages remotely?

Thinking Exercise

Draw three trust maps for standalone API access, Remote Execution, and attached runtime. Mark evaluator location, dependency location, credential, target authority, and failure propagation.

The Interview Questions They Will Ask

  1. “How does attached runtime differ from remote execution?”
  2. “Why is an Erlang cookie sensitive?”
  3. “Why can attached mode not use Mix.install normally?”
  4. “How do you bound live process inspection?”
  5. “When should production diagnostics use an API instead?”

Hints in Layers

Hint 1: Start with a disposable target

Never make first experiments against production.

Hint 2: Prove evaluator location

Render node identity and selected process evidence in every mode.

Hint 3: Keep the query catalog narrow

Allow named diagnostics, not arbitrary strings.

Hint 4: Prefer Remote Execution for rich local dependencies

Keep Kino and analytical packages on the notebook node when possible.

Books That Will Help

Topic Book Chapter
Distributed Elixir Elixir in Action, 3rd ed. Distribution chapter
Operational safety Release It!, 2nd ed. Stability patterns
Security boundaries Foundations of Information Security Authentication and access-control chapters

Common Pitfalls and Debugging

Problem 1: “Attached notebook cannot install chart dependency”

  • Why: It uses the target node’s existing code and dependencies.
  • Fix: Use Remote Execution from a standalone runtime when appropriate.
  • Quick test: Display runtime type before package setup.

Problem 2: “Process list freezes the browser”

  • Why: Inspection produced unbounded results.
  • Fix: Filter and summarize on the target before transfer.
  • Quick test: Seed many training processes and verify output cap.

Problem 3: “Connection fails intermittently”

  • Why: Node naming, cookie, distribution network, or DNS differs.
  • Fix: Verify each layer explicitly without printing the cookie.
  • Quick test: Run the connection checklist and bounded ping transcript.

Definition of Done

  • Standalone, Remote Execution, and attached modes are demonstrated and compared
  • Evaluator and dependency location are visible in each mode
  • Only allowlisted read-only diagnostics exist
  • Timeout, output bound, disconnect, and target crash cases are tested
  • Cookie and target details are redacted from saved evidence
  • All work uses a disposable node; no production connection is required

Project 12: Deployed Multi-Session Operations Decision App

  • File: P12-deployed-ops-app.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: SQL; JavaScript for packaged custom UI only
  • Coolness Level: Level 4 — Hardcore Tech Flex
  • Business Potential: Level 4 — Open Core Infrastructure
  • Difficulty: Level 4 — Expert
  • Knowledge Area: Internal tools, deployment, identity, operations, data and ML governance
  • Software or Tool: Livebook Apps, Kino, Explorer, VegaLite, optional Nx.Serving, Docker, Livebook Teams
  • Main Book: Release It!, Second Edition

What you will build: A read-only operations decision app combining safe data queries, SLA evidence, runbook guidance, optional model suggestions, per-user notes, durable exports, authentication, deployment, capacity checks, audit signals, and rollback.

Why it teaches Livebook + Elixir: It integrates every concept while forcing a separation between notebook code, runtime authority, app-user state, operational data, and deployment control.

Core challenges you will face

  • Multi-session privacy with shared reference data -> App architecture
  • Fresh operational evidence with bounded access -> Data and runtime contracts
  • Notebook preview to operated service -> Security, capacity, CI, rollback

Real World Outcome

After authentication, each user gets a private app session. The header shows environment, app revision, data age, model revision if enabled, and health. The left column contains a bounded incident queue and filters. The center contains SLA and service charts. The right contains a diagnostic guide and optional model suggestion with accept/correct/abstain. The bottom contains per-user notes and an exportable handoff report.

┌─────────────────────────────────────────────────────────────────────┐
│ STAGING | Data age 41s | App 9b73c2a | Model r17 | Health READY    │
├────────────────┬────────────────────────────┬───────────────────────┤
│ Incident queue │ SLA and service evidence   │ Decision assistant    │
│ severity/owner │ latency/errors/saturation  │ suggest/correct/skip  │
│ max 100 rows   │ bounded comparison window  │ human owns decision   │
├────────────────┴────────────────────────────┴───────────────────────┤
│ Private notes | read-only runbook | provenance | export handoff    │
└─────────────────────────────────────────────────────────────────────┘

Two authenticated test users receive isolated forms, notes, uploads, and outputs. Shared reference-data caching is proven not to contain user content. The deployment evidence page shows clean replay, stamp authority, secret names, image digest, memory budget, warmup result, smoke tests, and rollback revision.

The Core Question You Are Answering

“What must change when an executable notebook becomes a multi-user service that influences operational decisions?”

Concepts You Must Understand First

  1. App session semantics
    • What is shared by process, user, app server, or external store?
    • Primary Reference: Livebook Apps and Kino.Workspace docs
  2. Runtime and credential authority
    • Why is the completed app standalone and read-only?
    • Primary Reference: Livebook security, runtime, and Teams docs
  3. Deployment and recovery
    • How will the exact revision be promoted and rolled back?
    • Book Reference: Release It!, 2nd ed. — production chapters
  4. Decision support
    • What evidence and override accompany any model suggestion?
    • Book Reference: AI Engineering — evaluation chapters

Questions to Guide Your Design

  1. Which data is private session state, shared cache, and durable evidence?
  2. Which identity provider and authorization groups gate the app?
  3. How are cold start, session memory, and inactivity shutdown budgeted?
  4. What smoke test proves a deploy and what event triggers rollback?

Thinking Exercise

Create a full trust-boundary and state-ownership matrix. Rows include app user, author, CI, app server, runtime session, database, object store, model store, and audit sink. Columns include identity, credential, readable data, writable data, lifetime, and evidence.

The Interview Questions They Will Ask

  1. “How do Livebook single-session and multi-session apps differ?”
  2. “Why does hiding notebook code not sandbox execution?”
  3. “How do you avoid multiplying model memory per session?”
  4. “What does a deployment stamp guarantee?”
  5. “How do you deploy and roll back a Livebook App?”
  6. “What operational metrics prove the app is safe to keep serving?”

Hints in Layers

Hint 1: Write the authority map first

Do not build UI before selecting runtime, credentials, and data boundaries.

Hint 2: Keep user content session-local or explicitly durable

Never place private inputs in a shared cache.

Hint 3: Gate deployment with evidence

Clean replay, two-client isolation, cold start, memory, and redaction must pass before promotion.

Hint 4: Make rollback boring

Retain the previous immutable revision and one documented revert action.

Books That Will Help

Topic Book Chapter
Production stability Release It!, 2nd ed. Stability, capacity, and operations chapters
Architecture trade-offs Fundamentals of Software Architecture Characteristics and risk chapters
Data reliability Designing Data-Intensive Applications, 2nd ed. Ch. 1, 3, 11
ML decision systems AI Engineering Evaluation and inference chapters

Common Pitfalls and Debugging

Problem 1: “Two users see the same private note”

  • Why: Shared input, shared frame, or single-session state was used.
  • Fix: Use multi-session and audit shared caches for user data.
  • Quick test: Seed distinct canaries in two authenticated sessions.

Problem 2: “Restart takes too long”

  • Why: Dependencies or model artifacts compile/download on every boot.
  • Fix: Persist or prewarm approved artifacts and measure cold start.
  • Quick test: Restart with network disabled and time readiness.

Problem 3: “Dashboard is healthy but data is old”

  • Why: Process health and source freshness are conflated.
  • Fix: Make freshness a separate health dimension and alarm.
  • Quick test: Freeze the source while app processes remain alive.

Problem 4: “Rollback deploys but secrets or storage fail”

  • Why: Revision compatibility with current environment was not retained.
  • Fix: Version deployment contract and smoke-test rollback in staging.
  • Quick test: Revert to the previous revision and run the same gate.

Definition of Done

  • App runs in multi-session mode with two-client privacy tests
  • Runtime is standalone and all operational access is read-only and bounded
  • Auth, authorization, secret, storage, stamp, and audit policies are documented
  • UI shows app revision, data freshness, source mode, and model provenance
  • Cold start, warmup, memory per session, inactivity shutdown, and load are measured
  • CI or a documented repeatable deployment promotes an immutable revision
  • Staging smoke tests and previous-revision rollback both pass
  • No production mutation is present; all eight concept chapters are evidenced

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor
1. Executable Energy Field Report Level 1 5–8 hrs Foundation ★★★☆☆
2. Interactive Capacity Planner Level 1 6–10 hrs Foundation ★★★★☆
3. Messy CSV Triage Clinic Level 2 10–16 hrs Medium ★★★★☆
4. Fault-Tolerant Telemetry Lab Level 3 16–24 hrs High ★★★★★
5. City Mobility Data Pipeline Level 2 12–20 hrs High ★★★★☆
6. Safe SQL Incident Runbook Level 2 12–20 hrs High ★★★★☆
7. Interactive SLA Command Center Level 3 18–28 hrs High ★★★★★
8. Portable Cloud Data Lake Lab Level 3 16–26 hrs High ★★★★☆
9. Tested Domain Smart Cell Toolkit Level 3 20–30 hrs Very High ★★★★★
10. Nx + Bumblebee Image Triage Lab Level 3 20–32 hrs Very High ★★★★★
11. Safe Attached-Runtime Console Level 3 20–30 hrs Very High ★★★★★
12. Deployed Multi-Session Ops Decision App Level 4 35–50 hrs Capstone ★★★★★

Recommendation

If you are new to Elixir: Start with Project 1, then follow Projects 2, 3, and 4. They connect values, validation, messages, and recovery without assuming prior OTP fluency.

If you already build Elixir systems: Start with Project 4 or Project 9. They expose Livebook-specific process lifecycle and extension APIs rather than repeating language syntax.

If you work with data: Start with Project 3, then do Projects 5, 7, and 8 before adding Project 10.

If you want internal tools quickly: Build Project 2, then Project 6, and only then attempt Project 12. The security boundary is part of the product.

If you operate Phoenix systems: Complete Project 11 in a disposable environment, but prefer the standalone read-only architecture used by Project 12 for production.

Final Overall Project: Livebook Operations Studio

The Goal: Combine the strongest artifacts from Projects 3–12 into a governed portfolio repository rather than one monolithic notebook.

  1. Publish a shared data-contract package used by the CSV, Explorer, SQL, and app notebooks.
  2. Package the Project 9 Smart Cell and renderer as a versioned Kino extension.
  3. Keep read-only diagnostic modules in a normal Mix project with ExUnit.
  4. Maintain separate analysis, model-evaluation, and deployed-app .livemd files.
  5. Add a manifest generator that records notebook revision, runtime, packages, sources, and output checksums.
  6. Add a staging app server, deployment gate, capacity run, and rollback exercise.
  7. Publish an operator handbook that states authority, freshness, failure, and escalation boundaries.

Success Criteria: A new operator can clone the repository, replay training notebooks, review generated source, deploy the app to staging, verify two-client isolation and read-only access, observe a controlled fault, and roll back without private instructions.

From Learning to Production: What Is Next

Your Project Production Equivalent Gap to Fill
Field report Executable package documentation CI replay, supported versions, maintained fixtures
Capacity planner Internal decision tool Identity, persistence, accessibility, product ownership
CSV clinic Data-quality intake service Queueing, durable quarantine, schema registry, alerting
Telemetry lab Supervised ingestion service Durable telemetry, load testing, observability, release
Mobility pipeline Scheduled analytical pipeline Orchestrator, incremental processing, lineage catalog
SQL runbook Governed support console Central authorization, immutable audit, query catalog review
SLA command center Operational dashboard Data SLAs, caching, paging, availability ownership
Cloud data lab Data-lake workflow Storage policy, lifecycle, encryption, catalog, cost controls
Smart Cell toolkit Published Kino package Semantic versioning, compatibility matrix, Hex release
Image triage lab Model inference service Evaluation ownership, monitoring, artifact registry, privacy review
Runtime console Production diagnostics API Narrow service API, RBAC, rate limits, independent audit
Ops decision app Managed internal platform SLOs, on-call, change management, disaster recovery

Summary

This learning path covers Livebook and Elixir through twelve hands-on projects, beginning with a replayable executable document and ending with a secured multi-session internal application.

# Project Name Main Language Difficulty Time Estimate
1 Executable Energy Field Report Elixir Level 1 5–8 hrs
2 Interactive Capacity Planner Elixir Level 1 6–10 hrs
3 Messy CSV Triage Clinic Elixir Level 2 10–16 hrs
4 Fault-Tolerant Telemetry Lab Elixir Level 3 16–24 hrs
5 City Mobility Data Pipeline Elixir Level 2 12–20 hrs
6 Safe SQL Incident Runbook Elixir + SQL Level 2 12–20 hrs
7 Interactive SLA Command Center Elixir Level 3 18–28 hrs
8 Portable Cloud Data Lake Lab Elixir Level 3 16–26 hrs
9 Tested Domain Smart Cell Toolkit Elixir + JS Level 3 20–30 hrs
10 Nx + Bumblebee Image Triage Lab Elixir Level 3 20–32 hrs
11 Safe Attached-Runtime Console Elixir Level 3 20–30 hrs
12 Deployed Multi-Session Ops Decision App Elixir Level 4 35–50 hrs

Expected Outcomes

  • Explain Livebook’s evaluator, runtime, collaboration, and app models from first principles
  • Use functional Elixir and OTP to build visible, fault-aware workflows
  • Produce typed, bounded, provenance-rich data and numerical results
  • Extend Livebook with tested transparent Smart Cells
  • Choose safe runtime boundaries for diagnostics
  • Deploy and operate a multi-user Livebook App with evidence and rollback

Additional Resources and References

Official Livebook and Kino

Official Elixir Data and Numerical Ecosystem

Books

  • Elixir in Action, Third Edition by Saša Jurić — language, processes, supervision, and production thinking
  • Programming Elixir 1.6 by Dave Thomas — functions, modules, protocols, and OTP foundations
  • Designing Elixir Systems with OTP by James Edward Gray II and Bruce A. Tate — supervision-oriented design
  • Machine Learning in Elixir and Livebook by Sean Moriarity, Bruce Tate, and Sophie DeBenedetto — Nx-native numerical and ML workflow
  • Designing Data-Intensive Applications, Second Edition by Martin Kleppmann — data contracts, storage, and reliability
  • Release It!, Second Edition by Michael T. Nygard — stability, capacity, and production operations
  • AI Engineering by Chip Huyen — evaluation, serving, and inference-system boundaries