Project 9: Tested Domain Smart Cell Toolkit

Build a reusable Livebook Smart Cell that turns a small data-quality rule editor into deterministic, inspectable Elixir source without hiding state or accepting source injection.

Quick Reference

Attribute	Value
Difficulty	Level 3 — Advanced
Time Estimate	20–30 hours
Language	Elixir; bundled JavaScript and CSS for the editor surface
Prerequisites	Projects 1–4 or equivalent Elixir, process, Kino, and testing experience
Key Topics	Smart Cell lifecycle, JSON attributes, Kino.JS.Live, deterministic source generation, compatibility, Kino.Test

1. Learning Objectives

By completing this project, you will:

Explain why a Smart Cell is a persisted editor plus a source generator, not an opaque execution engine.
Design a versioned, JSON-compatible attribute schema with defaults and recovery rules.
Separate browser state, server assigns, persisted attributes, and generated program text.
Generate byte-stable, formatted, reviewable source from validated domain attributes.
Prevent identifier, string, and expression injection across the browser-to-source boundary.
Use Kino.Test to exercise connection, events, source export, editor synchronization, and malformed state.
Package the cell so another notebook can install it, convert it to ordinary code, and understand the result.

2. Theoretical Foundation

2.1 Core Concepts

A Smart Cell is a transparent DSL adapter. The editor captures a narrow domain intent—such as “column amount must be non-null and violations are errors”—and translates it into ordinary Elixir source. Evaluation still happens through Livebook’s normal evaluator. The generated source is part of the public contract and must remain understandable after conversion to a regular code cell.

State exists in four places. Browser controls hold transient interaction state. A Kino.JS.Live process owns server-side assigns and handles events. Smart Cell attributes are the JSON-friendly state saved in .livemd. Generated source is the executable representation. Confusing these layers causes lost settings, stale previews, and files that reopen differently from how they were saved.

Determinism is a product feature. Equivalent normalized attributes must generate equivalent source. Canonical field ordering, stable defaults, explicit formatting, and deterministic list order keep Git diffs reviewable and make source-export tests meaningful.

Source generation is a security boundary. Free-form labels may be safely emitted as quoted data, but variable names, aliases, function names, and expressions affect syntax. Validate identifiers against a narrow grammar, map rule types to an allowlist, and construct quoted data rather than concatenating arbitrary text into source.

Compatibility is schema evolution. Old notebooks will outlive the first package version. Restoration must tolerate missing optional keys, reject impossible values with a recoverable UI state, preserve or intentionally ignore unknown fields, and never silently change rule meaning.

2.2 Why This Matters

Smart Cells let teams encode a repeated workflow at the right level: less error-prone than copying snippets, but more transparent than sending users to a closed service. A safe implementation can standardize database queries, charts, validation rules, or model configuration while retaining code review, version control, and the escape hatch of conversion to source. The same lessons apply to configuration UIs, low-code tools, compiler front ends, and domain-specific languages.

2.3 Historical Context / Background

Livebook originally centered on Markdown and code cells, then added Smart Cells so higher-level editors could generate normal notebook code. Kino provides the live-process and browser bridge that powers those editors. This design deliberately preserves literate programming: UI convenience enriches source rather than replacing it. Current official APIs document Smart Cell registration, persisted attributes, source generation, and test support in Kino.SmartCell, Kino.JS.Live, and Kino.Test.

2.4 Common Misconceptions

“Assigns are saved automatically.” Only exported attributes become notebook state.
“JSON-friendly means semantically valid.” Serialization proves transport compatibility, not domain correctness.
“Escaping a string makes every interpolation safe.” Syntax-bearing values need allowlisting or validated AST construction.
“The preview is the source.” It may lag pending editor state unless synchronization is designed and tested.
“Conversion to code is a downgrade.” It is a critical transparency and portability feature.
“A pure helper test is sufficient.” The browser/server bridge, event protocol, and export callback need integration tests.

3. Project Specification

3.1 What You Will Build

Create a small package exposing a “Data quality rule” Smart Cell. Its editor lets an author choose:

the DataFrame variable to inspect;
the target column;
one allowlisted rule: non-null, numeric range, accepted category set, or uniqueness;
severity: information, warning, or error;
null behavior where the rule supports it;
a human-readable message stored strictly as data.

The cell shows a generated-source preview and a compact description of the rule. Evaluation produces a structured validation result that another notebook section may render. The package includes tests, package documentation, one minimal .livemd example, and a compatibility policy for saved attributes.

3.2 Functional Requirements

Registration: the cell appears under a recognizable Smart Cell name after package setup.
Restoration: valid saved attributes restore the same visible editor and exported source.
Rule editing: changing rule type reveals only relevant fields and preserves valid shared fields.
Validation: invalid identifiers, incomplete ranges, duplicate categories, and unsupported rule types show actionable errors.
Source preview: the preview corresponds to the last synchronized valid attributes.
Deterministic export: normalized equivalent attributes generate byte-identical formatted source.
Conversion parity: converting to an ordinary code cell preserves observable evaluation behavior.
Compatibility: at least two older attribute shapes restore through documented defaults or migrations.
Recovery: malformed attributes open into a safe repair state rather than crashing the notebook.

3.3 Non-Functional Requirements

Security: no arbitrary expression, module, or function entry; labels remain quoted data.
Reliability: repeat event delivery and reconnect do not duplicate state transitions.
Usability: validation is local, specific, and does not erase the previous valid source.
Maintainability: domain normalization and source generation are pure modules independent of Kino.
Compatibility: attribute schema version and supported migrations are documented.
Performance: ordinary edits update within a human-imperceptible interval and do not regenerate large unrelated assets.

3.4 Example Usage / Output

SMART CELL: Data quality rule

DataFrame variable: trips
Column: duration_minutes
Rule: numeric range
Minimum: 1
Maximum: 360
Null behavior: reject
Severity: error
Message: Duration must be within the service contract

Source preview: VALID
Rule identifier: quality.duration_minutes.range.v1

After evaluation, a consumer sees a structured result transcript:

Rule quality.duration_minutes.range.v1
Rows evaluated: 24,000
Violations: 37
Severity: error
Sample row identifiers: [redacted bounded sample]
Status: FAIL

3.5 Real World Outcome

In a clean notebook, install the local package through the normal dependency setup, open the Smart menu, and insert the cell. Configure a rule, save the notebook, close the session, reopen it, and observe every field restored. Convert a copy to a normal code cell; the source is formatted, short enough to review, and produces an equivalent structured result. Run the package test suite and obtain:

SMART CELL VERIFICATION
attribute round-trip ............... PASS
legacy attrs v0 -> current ......... PASS
legacy attrs v1 -> current ......... PASS
invalid identifier rejection ....... PASS
malformed attrs recovery ........... PASS
source byte stability .............. PASS
convert-to-code parity ............. PASS
pending editor synchronization ..... PASS
duplicate event tolerance .......... PASS

The outcome is visible in three places: the editor works, the saved .livemd diff contains stable attributes, and the generated source remains useful without the custom UI.

4. Solution Architecture

4.1 High-Level Design

┌──────────────────── Browser editor ────────────────────┐
│ allowlisted controls + validation hints + source view │
└──────────────────────────┬─────────────────────────────┘
                           │ typed events
                           v
┌──────────────── Kino.JS.Live process ──────────────────┐
│ normalize event -> update assigns -> export attrs     │
│ send canonical editor state / validation feedback     │
└───────────────┬──────────────────────┬─────────────────┘
                │                      │
                v                      v
       persisted JSON attrs      pure domain compiler
       in .livemd                 attrs -> source model
                │                      │
                └──────────┬───────────┘
                           v
                   formatted Elixir source
                           │
                           v
                 normal Livebook evaluator

4.2 Key Components

Component	Responsibility	Key Decisions
Attribute schema	Define persisted keys, defaults, version, and accepted values	Store only JSON primitives, lists, and maps
Attribute normalizer	Migrate, validate, and canonicalize saved/editor state	Pure function returning valid state or repair diagnostics
Rule model	Represent validated domain meaning	No browser-specific fields or arbitrary expressions
Source generator	Turn rule model into deterministic source	Allowlisted templates, canonical ordering, standard formatter
Smart Cell server	Own assigns, handle events, export attrs/source	Keep event handlers thin and idempotent
Browser editor	Render controls and send typed events	Bundle assets; server remains authority
Kino tests	Exercise bridge, restoration, source, and events	Compare observable source and state, not private callback order

4.3 Data Structures

Use a persisted schema similar to this conceptual shape:

attrs = {
  schema_version: integer,
  dataframe_variable: string,
  column: string,
  rule: {
    type: allowlisted string,
    options: JSON-friendly map
  },
  severity: allowlisted string,
  message: string
}

Normalize this into an internal rule model with validated identifiers, canonical categories, explicit optional values, and no unknown syntax-bearing data. Keep UI-only properties such as focus, open panels, and transient draft text out of persisted domain attributes unless restoring them has user value.

4.4 Algorithm Overview

Key Algorithm: Attributes to Deterministic Source

Decode attributes and select a schema migration by version.
Apply defaults for fields that were optional in the historical version.
Validate the DataFrame identifier and column as data.
Map rule type and severity through fixed allowlists.
Normalize unordered values, such as category sets, into a canonical order.
Build a source model using fixed templates and safely quoted literals.
Format the complete expression with the standard formatter.
Export only if the rule model is valid; otherwise export a safe diagnostic expression or retain the last valid source according to the documented policy.

Complexity Analysis:

Time: O(A + C log C), where A is attribute size and C is category count requiring canonical ordering.
Space: O(A + S), where S is generated source length.

5. Implementation Guide

5.1 Development Environment Setup

Use a normal Mix library project for stable modules and tests, and a separate training notebook that installs the local package. Record compatible Elixir, Livebook, and Kino versions. The verification workflow should support these commands:

$ mix test
Expected: pure domain and Kino integration tests pass

$ mix format --check-formatted
Expected: package source and tests are formatted

Livebook -> New notebook -> Add dependency from local path
Expected: “Data quality rule” appears in the Smart menu

Do not place a full implementation in the guide. The learner should derive callbacks from the current official APIs.

5.2 Project Structure

smart-cell-toolkit/
├── lib/
│   ├── quality_rule.ex              # validated domain model
│   ├── quality_rule_attrs.ex        # migrations and normalization
│   ├── quality_rule_source.ex       # deterministic source generation
│   └── quality_rule_cell.ex         # Kino Smart Cell bridge
├── assets/
│   ├── editor.js                    # bundled browser editor
│   └── editor.css
├── test/
│   ├── quality_rule_attrs_test.exs
│   ├── quality_rule_source_test.exs
│   └── quality_rule_cell_test.exs
├── examples/
│   └── data-quality-rule.livemd
├── mix.exs
└── README.md

5.3 The Core Question You’re Answering

“How do you give a domain workflow a friendly interface without replacing readable source with opaque magic?”

Before implementation, decide what must remain understandable after the package disappears. The answer should be: domain meaning and generated source remain; only the editing convenience is lost.

5.4 Concepts You Must Understand First

Smart Cell lifecycle
- When are attributes initialized, restored, edited, synchronized, and exported?
- Primary reference: Kino.SmartCell
Live process state
- Which assigns are authoritative and which browser values are only drafts?
- Primary reference: Kino.JS.Live
Safe syntax construction
- Which fields are code identifiers and which are quoted runtime data?
- Book reference: Metaprogramming Elixir — quoting and AST chapters
Schema evolution
- How will a one-year-old notebook restore after attributes change?
- Book reference: Designing Data-Intensive Applications — evolvability and encoding chapters
Integration testing
- Which public events and outputs prove the cell works?
- Primary reference: Kino.Test

5.5 Questions to Guide Your Design

What is the smallest useful rule vocabulary?
Can every persisted value be represented as ordinary JSON without losing meaning?
What is the canonical order of keys, category values, and generated options?
How does an invalid draft differ from the last valid exported source?
Which old attribute forms are migrated, rejected, or preserved?
Can free-form user text ever change syntax?
What does a collaborator see while another author edits?

5.6 Thinking Exercise

Create a state table for four layers: browser draft, server assigns, persisted attributes, generated source. Trace these events by hand:

restore a valid old notebook;
enter an invalid variable name;
change rule type while a now-irrelevant option exists;
save before a collaborative editor blur event;
reconnect after a server process restart.

For each event, state which layer changes, what the user sees, and whether evaluation is allowed.

5.7 The Interview Questions They’ll Ask

“What is a Livebook Smart Cell under the hood?”
“Why must persisted attributes be JSON-compatible?”
“How do you prevent source injection from labels and identifiers?”
“What does deterministic source generation buy you?”
“How do you evolve saved attributes without breaking old notebooks?”
“What does convert-to-code parity prove?”

5.8 Hints in Layers

Hint 1: Build the compiler first

Start with pure attribute normalization and source generation. A static map should produce a stable result before a browser exists.

Hint 2: Separate syntax from data

Use an allowlist for syntax-bearing choices. Treat column names and messages as safely quoted data wherever possible.

Hint 3: Version at the boundary

Migrate persisted attributes into one current normalized shape. Do not spread historical cases throughout the UI and generator.

Hint 4: Test the bridge

Use Kino.Test to connect, send events, request source, restore saved data, and observe replies. Pure tests cannot reveal synchronization bugs.

5.9 Books That Will Help

Topic	Book	Chapter
Domain-specific language design	Domain Specific Languages by Martin Fowler	Semantic model and representation chapters
Elixir syntax representation	Metaprogramming Elixir by Chris McCord	Chapters 1–4
Evolvable persisted formats	Designing Data-Intensive Applications, 2nd ed.	Encoding and evolvability chapters
Stable library design	The Pragmatic Programmer	Orthogonality, tracer bullets, and contracts topics

5.10 Implementation Phases

Phase 1: Domain Compiler (5–7 hours)

Goals: define the current attribute schema, two historical fixtures, normalization, validation, and source-generation contract.

Tasks:

Write valid, invalid, legacy, and malicious attribute fixtures.
Define allowlists and identifier policy.
Specify canonical ordering and formatting.
Test repeated generation and migration idempotency.

Checkpoint: pure tests prove stable source and safe rejection without Kino.

Phase 2: Smart Cell Bridge (7–10 hours)

Goals: register the cell, restore attributes, handle typed events, and export source.

Tasks:

Map normalized attributes into server assigns.
Design the browser/server event protocol.
Implement validation feedback and source preview.
Exercise connection and event handling with Kino.Test.

Checkpoint: a minimal editor round-trips one rule and converts to equivalent code.

Phase 3: Compatibility, Collaboration, and Packaging (8–13 hours)

Goals: harden restoration, pending-edit synchronization, documentation, and package behavior.

Tasks:

Restore historical fixtures and malformed notebooks.
Test two-client editing and evaluation boundaries.
Bundle assets and remove external runtime dependencies.
Create the example .livemd and compatibility matrix.

Checkpoint: save/reopen, convert-to-code, clean-runtime install, and full test transcript all pass.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Rule vocabulary	Free expression; fixed rules; plugin callbacks	Fixed rules first	Makes validation and source review tractable
Invalid draft behavior	Export broken source; clear source; retain last valid source with warning	Retain last valid source and visibly block evaluation of draft	Avoids silent broken notebooks while preserving recoverability
Attribute migration	Handle everywhere; normalize once	Normalize once at initialization	Keeps current logic simple
Browser assets	CDN; bundled assets	Bundled	Supports offline and reproducible use
Category order	Preserve incidental UI order; canonical sort	Canonical sort unless order has domain meaning	Produces stable source
Source construction	String concatenation; constrained template/AST	Constrained construction plus formatter	Reduces injection and formatting drift

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Pure unit	Prove normalization, migration, validation, and source stability	equivalent attrs, invalid ranges, malicious identifiers
Smart Cell integration	Prove bridge behavior	connect, event, export, restore, editor sync
Compatibility	Protect saved notebooks	v0, v1, current, unknown/malformed fields
Conversion	Preserve transparency	Smart Cell result versus converted source result
Collaboration	Detect pending and conflicting edit problems	two clients, save during edit, reconnect
Packaging	Prove fresh installation	clean Mix cache and fresh Livebook runtime

6.2 Critical Test Cases

Round-trip: save valid attributes, restore them, and export byte-identical source.
Migration idempotency: migrating historical attributes twice produces the same current form.
Identifier attack: quotes, newlines, interpolation markers, aliases, and invalid Unicode identifiers cannot change source structure.
Free-text safety: messages containing source-like characters remain literal data.
Malformed recovery: invalid persisted state yields a repairable editor and no server crash.
Pending edit: evaluation sees the latest synchronized value or is visibly blocked until synchronization.
Duplicate event: repeated delivery does not append duplicate options or change meaning.
Conversion parity: the generated code reports the same rule identity, severity, and result.

6.3 Test Data

Fixture current_valid:
  variable=trips, column=duration_minutes, range=1..360, severity=error

Fixture legacy_v0:
  no schema version, old “level” field, missing null policy

Fixture hostile_identifier:
  includes quote, newline, interpolation marker, and expression-like text

Fixture malformed:
  unsupported rule, reversed range, severity as nested object

Expected:
  valid -> stable source
  legacy -> documented normalized source
  hostile -> actionable rejection
  malformed -> safe repair state

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Assigns never exported	Cell reopens with defaults	Define complete attribute serialization and round-trip test
Map iteration leaks into source	Git diff changes without semantic edit	Canonicalize every unordered collection
Browser treated as authority	Crafted event bypasses UI restriction	Revalidate all events server-side
Label inserted into syntax	Source breaks or changes behavior	Quote labels strictly as data
Migration mixed into callbacks	Old notebooks fail in surprising paths	Normalize once before current-state logic
Listener duplication	Each UI event is handled more than once	Ensure one Smart Cell process and idempotent event transitions

7.2 Debugging Strategies

Inspect four states separately: browser draft, server assigns, exported attrs, and source.
Reduce to one event: reproduce with a single typed event through Kino.Test.
Compare normalized forms: diff canonical attributes before diffing generated text.
Reopen the saved notebook: a live session can hide serialization omissions.
Convert to code: if the generated result is mysterious, the DSL model is too opaque.
Disable network: verify bundled assets and local package behavior.

7.3 Performance Traps

Avoid regenerating large source on every keystroke, shipping oversized option lists to the browser, recompiling browser assets at runtime, or retaining full collaboration event histories. Debounce cosmetic preview updates, but synchronize authoritative state before export or evaluation.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a “starts with” string rule while preserving strict data quoting.
Add a compact human-readable rule summary distinct from source preview.
Export a compatibility report showing the persisted schema version.

8.2 Intermediate Extensions

Add column suggestions from a known DataFrame while treating suggestions as hints, not authority.
Add rule composition with explicit all/any semantics and stable ordering.
Publish a custom renderer for the structured validation result.

8.3 Advanced Extensions

Package and publish the toolkit with a Livebook/Kino compatibility matrix.
Add property-based tests for arbitrary JSON attributes and source safety.
Design a migration deprecation policy with automated fixture generation from prior releases.

9. Real-World Connections

9.1 Industry Applications

Governed query builders: guide analysts toward reviewed query templates.
Data-quality authoring: let domain experts create checks without copying syntax.
Model configuration: expose safe task and threshold choices while retaining code review.
Training environments: turn complex setup into inspectable generated examples.
Operational runbooks: encode allowlisted diagnostics rather than arbitrary commands.

Livebook: notebook host and Smart Cell integration surface.
Kino: rich output, JavaScript bridge, Smart Cell, and test APIs.
KinoDB: a production example of database-oriented Smart Cells.
KinoVegaLite: an example of packaging a domain renderer for Livebook.

9.3 Interview Relevance

This project demonstrates DSL design, source safety, serialization, backward compatibility, event-driven state, browser/server trust boundaries, integration testing, and API ergonomics. Be ready to explain why UI convenience does not justify hidden execution.

10. Resources

10.1 Essential Reading

Kino.SmartCell — current Smart Cell behavior and callbacks.
Kino.JS.Live — server process and browser event bridge.
Kino.Test — integration-testing utilities.
Domain Specific Languages by Martin Fowler — semantic model and representation strategies.
Metaprogramming Elixir by Chris McCord — quoted syntax and AST foundations.

10.2 Video Resources

Use the current Livebook “Learn” notebooks shipped with the installed version for Smart Cell and Kino demonstrations.
Prefer official Livebook release demonstrations linked from Livebook News when API behavior has changed.

10.3 Tools & Documentation

Kino documentation: version-current API index.
Livebook documentation: notebook, runtime, and security context.
Elixir formatter: canonical source formatting and syntax verification.
Git diff: practical detector for nondeterministic generated text.

Project 7: Interactive SLA Command Center: prepares event-state and frame reasoning.
Project 10: Nx + Bumblebee Image Triage Lab: can later consume a model-configuration Smart Cell.
Project 12: Deployed Multi-Session Operations Decision App: may consume packaged validation rules without granting arbitrary source entry.

11. Self-Assessment Checklist

11.1 Understanding

I can explain the difference between assigns, attributes, browser state, and generated source.
I can explain why JSON compatibility and domain validity are separate checks.
I can identify every field capable of affecting syntax.
I can explain the restoration policy for each historical fixture.

11.2 Implementation

All functional requirements are met.
Pure and Kino integration tests pass.
Generated source is deterministic, formatted, and inspectable.
Save/reopen and convert-to-code parity pass.
No external CDN or hidden runtime state is required.

11.3 Growth

I documented one design choice I would change in a second version.
I can explain the package in terms of compiler front end, state machine, and compatibility contract.
I can review a new Smart Cell for injection and persistence risks.

12. Submission / Completion Criteria

Minimum Viable Completion:

One allowlisted rule type edits, persists, restores, and generates readable source.
Invalid identifiers are rejected server-side.
Pure source-generation and one Kino bridge test pass.

Full Completion:

All four rule types, versioned attributes, two historical migrations, malformed recovery, deterministic source, convert-to-code parity, bundled assets, and clean-runtime installation work.
Test evidence includes events, restoration, source export, duplicate delivery, and pending-editor synchronization.

Excellence (Going Above & Beyond):

Publish a versioned package with compatibility policy, property-based safety tests, accessible editor behavior, and a second independent notebook consuming the generated result protocol.

This guide was generated from LEARN_LIVEBOOK_ELIXIR_DEEP_DIVE.md. For the complete learning path, see the parent guide.

Project 9: Tested Domain Smart Cell Toolkit

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Historical Context / Background

2.4 Common Misconceptions

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Structures

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: Domain Compiler (5–7 hours)

Phase 2: Smart Cell Bridge (7–10 hours)

Phase 3: Compatibility, Collaboration, and Packaging (8–13 hours)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

7.3 Performance Traps

8. Extensions & Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools & Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria