Project 10: Nx + Bumblebee Image Triage Lab

Build a multi-session image-triage lab that makes upload validation, tensor contracts, model provenance, serving latency, abstention, and human correction visible.

Quick Reference

Attribute Value
Difficulty Level 3 — Advanced
Time Estimate 20–32 hours
Language Elixir; optional Python only for an external comparison
Prerequisites Explorer basics, Kino interaction, process lifecycle, elementary classification metrics
Key Topics Nx tensors, image preprocessing, Bumblebee, Nx.Serving, artifact provenance, multi-session privacy, evaluation

1. Learning Objectives

By completing this project, you will:

  1. Describe every image-processing stage from browser upload to model decision.
  2. Validate byte size, format, dimensions, and decoded shape before expensive allocation.
  3. State tensor shape, element type, axis meaning, backend, and approximate memory.
  4. Pin and verify a compatible pretrained model, featurizer, labels, revision, digest, and license.
  5. Separate artifact download, model load, compilation, queue, inference, and total latency.
  6. Explain how Nx.Serving batches work and why latency and throughput goals conflict.
  7. Evaluate a labeled fixture set with per-class metrics, ambiguous cases, corrupt images, and out-of-distribution inputs.
  8. Implement abstention and a human-owned accept/correct/none decision.
  9. Prove two-client privacy and document data-retention behavior.

2. Theoretical Foundation

2.1 Core Concepts

Tensor contracts are semantic, not merely dimensional. A tensor with shape [1, 224, 224, 3] communicates batch, height, width, and channels only if those axes are documented. Another arrangement can have the same element count and still be wrong. Type determines precision, range, memory, and backend compatibility. Render shape, type, names when available, backend, and memory estimate before trusting inference.

Preprocessing is part of the model. Decode, color space, alpha handling, orientation metadata, crop or letterbox policy, resize algorithm, normalization, and channel order change predictions. A model revision without its preprocessing contract is incomplete provenance.

defn and compilation change the cost model. Nx numerical definitions build computation graphs suitable for evaluators and compilers. The first compatible call may include compilation; later calls reuse compiled work. Variable shapes can trigger additional compilation or prevent efficient batching. Compare cold preparation and warm steady state separately.

A model artifact is a supply-chain dependency. Record the source repository, exact revision or immutable artifact, digest, task, label mapping, license, and cache location. Production-like offline verification must prove the approved model is already available rather than fetched implicitly after startup.

Serving is more than prediction. Nx.Serving can encapsulate preprocessing, batching, execution, and postprocessing. A serving process accumulates requests up to a batch size or timeout. Larger batches may improve throughput but add queue delay and memory. Queue time, execution time, and end-to-end time must be measured separately.

Scores are evidence, not authority. A classifier score is not automatically a calibrated probability. The lab needs labeled evaluation, an abstention threshold, out-of-distribution examples, and human correction. The model suggests; the human owns the triage decision.

2.2 Why This Matters

Many machine-learning demonstrations show one impressive prediction and omit the engineering system around it. Real workflows fail through malformed input, hidden preprocessing, artifact drift, cold starts, memory exhaustion, privacy mistakes, or confident misclassification. This lab teaches numerical Elixir while treating the model as one bounded component in an observable decision process.

2.3 Historical Context / Background

Nx introduced typed tensor computation and compilable numerical definitions to the Elixir ecosystem. Bumblebee builds on Nx and Axon to load compatible pretrained models and preprocessing artifacts, including repositories hosted on Hugging Face. Nx.Serving turns one-off inference into a supervised, batch-aware serving contract. Livebook and Kino make this stack inspectable and interactive without requiring the learner to build an entire web application first. Current primary references are Nx, Nx numerical definitions, Nx.Serving, Bumblebee, and Bumblebee examples.

2.4 Common Misconceptions

  • “Matching element count means the shape is right.” Axis meaning and ordering still determine correctness.
  • “Resize is cosmetic.” Resize and crop policy can discard the evidence the classifier needs.
  • “The first request is normal latency.” It may include download, load, allocation, and compilation.
  • “A high score is a probability.” Calibration must be measured for the specific data distribution.
  • “Multi-session means the model is copied efficiently.” Session isolation can multiply model memory unless architecture deliberately shares approved serving resources.
  • “Deleting the visible preview deletes the upload.” Browser, runtime, cache, log, and exported artifact retention must each be considered.

3. Project Specification

3.1 What You Will Build

Build a Livebook lab and app preview for triaging training images into a small, documented label set such as intact package, damaged package, unreadable label, contamination, and unknown. Select a model that is supported by the current Bumblebee release and whose license permits the exercise. The system must:

  • accept a bounded JPEG or PNG through a per-user image input;
  • reject malformed, oversized, or unsupported inputs before inference;
  • display preprocessing and tensor evidence;
  • run a pinned model through Nx.Serving;
  • show ranked suggestions, score, latency phases, and model revision;
  • abstain when the evidence is weak or outside the supported label policy;
  • require accept, correct, or none-of-these input from the human;
  • evaluate a fixed labeled fixture set;
  • discard or explicitly persist user data according to a written retention policy.

3.2 Functional Requirements

  1. Input bounds: enforce MIME signature, byte limit, decoded dimension limit, and accepted channel policy.
  2. Orientation and color: normalize orientation and document RGB/alpha handling.
  3. Tensor evidence: display expected and actual shape, type, backend, axes, and estimated bytes.
  4. Artifact evidence: display model source, revision, digest, task, labels, license, and cache/offline status.
  5. Warmup evidence: separate download, load, compile, and warmup from request latency.
  6. Serving metrics: report queue, batch, inference, postprocessing, and total duration.
  7. Decision policy: distinguish suggest, abstain, and failure.
  8. Human result: record accept, corrected label, or none, without overwriting raw model output.
  9. Evaluation: report confusion matrix, per-class precision/recall, abstention, corrupt-input handling, and out-of-distribution behavior.
  10. Isolation: two simultaneous clients never see each other’s image, preview, result, or correction.

3.3 Non-Functional Requirements

  • Privacy: training images and corrections remain session-local unless the user explicitly exports a redacted evaluation artifact.
  • Resource safety: queue, concurrency, upload size, image dimensions, batch size, and timeout are bounded.
  • Reproducibility: approved artifact is immutable or digest-verified; fixture set has checksums.
  • Usability: failures explain whether input, artifact, capacity, or inference failed.
  • Honesty: score wording never implies guaranteed correctness or calibrated probability without evidence.
  • Portability: a documented CPU path works; optional accelerator measurements are additional evidence.

3.4 Example Usage / Output

IMAGE TRIAGE
Input: training-fixture-014.jpg
Decoded: 1280 x 853 RGB, 412 KiB
Preprocess: orientation normalized -> center crop -> 224 x 224 -> model normalization
Tensor: [1, 224, 224, 3] f32, backend=EXLA host, approx=0.57 MiB

Model: approved-repository@immutable-revision
Artifact digest: sha256:EXAMPLE_REDACTED
Top suggestions:
  damaged_package .... 0.81
  contamination ...... 0.11
  intact_package ..... 0.05

Decision: SUGGEST — human confirmation required
Queue / inference / total: 3 / 47 / 61 ms
Warm path: yes

3.5 Real World Outcome

Open the app preview from two browser profiles. Upload distinct canary images and verify private previews and results. In each session, the user sees an input-contract card, tensor card, provenance card, ranked result, and decision controls. A corrupt or oversized image produces a bounded error before the serving process is called. A deliberately unfamiliar image results in abstention under the documented policy.

The evaluation section produces an evidence report such as:

EVALUATION FIXTURE r3
Images: 120 labeled + 20 out-of-distribution + 8 corrupt
Macro precision / recall: 0.78 / 0.74
Abstention rate: 16.7%
Correct among non-abstained: 84.0%
Corrupt rejected before inference: 8 / 8
Cross-client canary leak scan: PASS
Artifact offline startup: PASS
Retention scan after session close: PASS

The learner can point to the exact input, artifact, preprocessing, serving, evaluation, privacy, and human-decision contracts rather than merely displaying a prediction.

4. Solution Architecture

4.1 High-Level Design

┌────────────── Browser session A ──────────────┐
│ private image input + preview + correction   │
└──────────────────────┬────────────────────────┘
                       │ bounded image payload
                       v
┌──────────── Session validation process ─────────────┐
│ signature -> bytes -> dimensions -> orientation    │
│ retention policy -> request id -> tensor contract  │
└──────────────────────┬──────────────────────────────┘
                       │ validated tensor/input
                       v
             ┌───────────────────────┐
             │ Nx.Serving boundary   │
             │ queue -> batch -> ML  │
             └───────────┬───────────┘
                         │ ranked scores + timings
                         v
┌──────────── Session decision process ──────────────┐
│ threshold -> suggest/abstain -> human correction  │
│ render private evidence -> optional safe export   │
└────────────────────────────────────────────────────┘

Shared only when proven safe: immutable model metadata and serving resources.
Never shared: uploaded bytes, preview, user correction, private notes.

4.2 Key Components

Component Responsibility Key Decisions
Input validator Bound format, bytes, dimensions, and decode policy Reject before tensor allocation
Preprocessor Produce exact model-compatible input One documented resize/crop/normalize path
Tensor inspector Render shape/type/backend/memory evidence Inspect before serving
Artifact loader Verify model, preprocessor, labels, revision, digest, license Fail closed when artifact differs
Nx.Serving Bound queue, batch, compilation, and execution Measure queue separately from inference
Decision policy Convert ranked scores into suggest/abstain/fail Human remains final authority
Evaluation harness Run fixed labeled and adversarial fixtures Separate fixture data from interactive uploads
Retention controller Delete or export according to explicit policy No implicit user-content cache

4.3 Data Structures

input_evidence = {
  request_id, origin/session_id, source_name,
  bytes, format, width, height, channels,
  validation_status, preprocessing_revision
}

model_evidence = {
  repository, revision, digest, license,
  task, labels_revision, backend, compile_shape
}

decision_evidence = {
  raw_ranked_scores, policy_revision,
  threshold, state: suggest|abstain|failure,
  human_action: accept|correct|none,
  timings, retention_status
}

Use opaque request identifiers in user-facing evidence. Never use uploaded file names as trusted paths.

4.4 Algorithm Overview

Key Algorithm: Bounded Triage Request

  1. Receive a per-session image reference and allocate a request generation.
  2. Verify file signature and byte limit before decoding.
  3. Decode under dimension and format bounds; normalize orientation and color policy.
  4. Apply the pinned preprocessing contract and assert target tensor shape/type.
  5. Submit a bounded batch request to Nx.Serving with a timeout.
  6. Record queue, inference, postprocessing, and total time.
  7. Rank labels, apply abstention policy, and preserve raw output.
  8. Render the result only if the request generation is current.
  9. Accept human correction as separate evidence.
  10. Dispose of user content according to retention policy.

Complexity Analysis:

  • Decode/preprocess time: O(W × H) in source pixel count, constrained by input dimensions.
  • Model execution: architecture-dependent; treat as measured bounded cost.
  • Output ranking: O(L log K) or O(L log L), depending on top-K strategy and label count L.
  • Session memory: bounded input bytes + decoded pixels + tensors + result metadata; document copies separately.

5. Implementation Guide

5.1 Development Environment Setup

Choose versions through Livebook’s current package search and record the resolved dependency set. Use a small supported model and CPU-capable backend first. The setup verification should look like this:

Livebook runtime: Standalone
Nx backend: explicit and displayed
Bumblebee model support check: PASS
Approved artifact available: PASS
Fixture manifest checksum: PASS

$ network disabled during offline verification
Expected: approved artifact still loads; no fallback download occurs

Do not paste a complete dependency or inference implementation into the guide. Build it from the official version-matched examples and document every choice.

5.2 Project Structure

image-triage-lab/
├── notebooks/
│   └── image-triage.livemd
├── lib/
│   ├── image_contract.ex
│   ├── image_preprocessor.ex
│   ├── triage_serving.ex
│   ├── decision_policy.ex
│   └── evaluation_report.ex
├── fixtures/
│   ├── manifest.json
│   ├── labeled/
│   ├── out-of-distribution/
│   └── corrupt/
├── test/
│   ├── image_contract_test.exs
│   ├── decision_policy_test.exs
│   └── evaluation_report_test.exs
├── artifacts/
│   └── approved-model-manifest.json
└── README.md

5.3 The Core Question You’re Answering

“What evidence must surround a model prediction before it can safely influence a real workflow?”

If the answer is only “the label and score,” the project is not ready. The required answer includes input validity, preprocessing, shape/type, artifact identity, latency phases, evaluation, uncertainty, privacy, and human authority.

5.4 Concepts You Must Understand First

  1. Tensor shape, type, axes, and backend
    • Can you explain the meaning of every dimension and copy?
    • Primary reference: Nx
  2. Numerical definitions and compilation
  3. Serving lifecycle
    • How do batch size and timeout affect latency and throughput?
    • Primary reference: Nx.Serving
  4. Bumblebee artifact contracts
    • Which repository files define model, featurizer, tokenizer, and labels?
    • Primary reference: Bumblebee
  5. Classification evaluation
    • Why do per-class errors and abstention matter?
    • Book reference: AI Engineering — evaluation and inference chapters
  6. Kino per-user interaction

5.5 Questions to Guide Your Design

  1. What exact file signatures, byte limits, and dimensions are accepted?
  2. What happens to EXIF orientation, alpha, grayscale, and animated images?
  3. Does crop or letterbox preserve the defect the model should see?
  4. Which immutable model revision and license are approved?
  5. How is label mapping verified against the model output?
  6. What score or margin causes abstention, and how was it chosen?
  7. Where can private content exist, and when is each copy deleted?
  8. Is the serving process shared, and if so, what data can cross that boundary safely?

5.6 Thinking Exercise

Trace one 12-megapixel phone image on paper. Mark:

  • compressed bytes in the browser and runtime;
  • decoded pixel allocation;
  • resized allocation;
  • tensor allocation and any backend transfer;
  • batch padding or copy;
  • model intermediates;
  • result and preview retention.

Then decide which allocations are rejected, bounded, reused, or deleted. Repeat for a corrupt image and a second concurrent client.

5.7 The Interview Questions They’ll Ask

  1. “How can broadcasting or reshaping hide a semantic tensor bug?”
  2. “Why separate cold-start, queue, inference, and total latency?”
  3. “What does Nx.Serving provide beyond calling a model function?”
  4. “How do fixed shapes affect compilation and batching?”
  5. “Why is a model score not necessarily a calibrated probability?”
  6. “How do you make a pretrained model deployment reproducible and offline-capable?”

5.8 Hints in Layers

Hint 1: Prove the input contract first

Reject malicious and oversized fixtures before loading any model. The validation pipeline should be testable independently.

Hint 2: Make shape visible

Render expected versus actual axes and type immediately before serving.

Hint 3: Warm once and measure phases

Do not combine download, loading, compilation, queue, and execution into one latency number.

Hint 4: Evaluate decisions, not screenshots

Use a labeled manifest, confusion matrix, per-class metrics, abstention, and human correction. One successful picture proves almost nothing.

5.9 Books That Will Help

Topic Book Chapter
Nx-native ML workflow Machine Learning in Elixir and Livebook Nx, transfer learning, and pretrained-model chapters
Inference systems AI Engineering by Chip Huyen Evaluation and inference chapters
Decision metrics Data Science for Business Chapters 7–8
Reliable service boundaries Release It!, 2nd ed. Timeouts, capacity, and stability patterns

5.10 Implementation Phases

Phase 1: Input and Artifact Contracts (6–8 hours)

Goals: validate inputs without model execution and select an approved artifact.

Tasks:

  1. Build valid, corrupt, oversized, unusual-color, and out-of-distribution fixtures.
  2. Write input and retention contracts.
  3. Verify model support, task, labels, revision, digest, and license.
  4. Define exact preprocessing and compile shapes.

Checkpoint: every invalid fixture fails at the intended bounded stage, and offline artifact verification passes.

Phase 2: Serving and Interactive Decision (7–11 hours)

Goals: produce visible tensor evidence, warm serving, and private per-session results.

Tasks:

  1. Connect preprocessing to Nx tensors with shape assertions.
  2. Create a bounded serving path and latency instrumentation.
  3. Apply suggest/abstain policy.
  4. Add accept/correct/none controls without mutating raw output.
  5. Test two simultaneous clients with distinct canaries.

Checkpoint: warm requests are bounded, instrumented, isolated, and human-confirmed.

Phase 3: Evaluation and Failure Hardening (7–13 hours)

Goals: evaluate the workflow, not just the model call.

Tasks:

  1. Run the fixed labeled fixture manifest.
  2. Report per-class metrics, confusion, abstention, and failures.
  3. Stress queue and memory under bounded concurrency.
  4. Test offline restart, corrupt cache, serving timeout, and session cleanup.

Checkpoint: evaluation report, privacy scan, retention scan, and clean replay pass.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Model size Largest available; small supported model Small supported model first Keeps learning focused on contracts and serving
Resize policy Stretch; center crop; letterbox Choose from domain evidence and document it Each changes visible information
Inference result Forced top-1; suggestion with abstention Suggestion with abstention Preserves uncertainty and human authority
Artifact loading Runtime network fetch; preverified cache Preverified immutable cache for offline test Makes startup reproducible
Serving location Per-session copy; deliberately shared serving Measure both, then share only immutable serving resources if isolation is proven Prevents accidental model-memory multiplication
User content Retain by default; discard by default Discard by default Minimizes privacy risk

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Input contract Reject unsafe data early wrong magic bytes, huge dimensions, alpha/grayscale policy
Tensor contract Prove numerical boundary axis order, dtype, backend, normalized range
Artifact Preserve reproducibility digest mismatch, missing cache, offline startup
Decision Validate suggest/abstain behavior threshold boundary, tie, unknown class
Evaluation Measure real task quality confusion matrix, per-class recall, OOD abstention
Concurrency/privacy Protect multiple users simultaneous canary uploads, queue saturation, cleanup
Failure recovery Keep UI truthful timeout, serving crash, corrupt image, memory budget

6.2 Critical Test Cases

  1. Magic-byte mismatch: a renamed non-image is rejected before decode.
  2. Decompression bomb dimensions: dimensions exceed policy and no tensor is allocated.
  3. Axis sentinel: a synthetic color pattern proves channel and axis order.
  4. Artifact mismatch: wrong digest fails closed and does not silently fetch another revision.
  5. Cold versus warm: preparation and five warm requests are reported separately.
  6. Threshold edge: just-below and just-above scores enter different documented states.
  7. OOD image: unsupported content produces abstention or an explicit limitation finding.
  8. Cross-client canaries: image A, result A, and correction A never appear in session B.
  9. Queue saturation: bounded overload returns a visible busy/timeout state without unbounded memory.
  10. Session close: uploaded bytes and private correction are absent from retained runtime evidence.

6.3 Test Data

Fixture manifest fields:
  id, relative_path, sha256, expected_label_or_ood,
  width, height, format, known_edge_case, license/source

Required groups:
  representative labeled images
  ambiguous in-domain images
  class-imbalanced slice
  out-of-distribution images
  corrupt/truncated files
  oversized-dimension headers
  two private canary images

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Input bounds checked after decode Memory spike before rejection Inspect bytes/header and enforce dimensions first
Axis order assumed Plausible but wrong predictions Use a synthetic channel/shape sentinel
First request reported as inference Misleading multi-second latency Separate download, load, compile, queue, execute
Artifact fetched during startup Offline deployment fails Prefetch, pin, digest, and verify cache
Score labeled “confidence” Users over-trust result Use score wording and measured decision policy
Shared Kino input Users see another image Use multi-session/per-user controls and canary test
Correction overwrites model output Evaluation evidence disappears Store raw output and human action separately

7.2 Debugging Strategies

  • Render the contract: shape, type, axes, backend, and ranges before model execution.
  • Time each stage: identify whether delay is queue, compile, artifact, or inference.
  • Use sentinel images: controlled colors and geometry reveal orientation/channel errors.
  • Disable network: expose implicit model fetches.
  • Run one client, then two: isolate privacy problems from model problems.
  • Inspect process memory between requests: detect retained uploads or repeated model copies.

7.3 Performance Traps

Variable shapes can cause recompilation; repeated device copies waste bandwidth; decoding full-resolution uploads before bounds wastes memory; large batch windows increase interactive latency; one model per session can exhaust memory; unbounded correction logs retain private data. Measure peak memory and queue time under the expected concurrent-session budget.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a visual crop preview showing exactly what the model receives.
  • Add a model/provenance card export without user image content.
  • Add an explicit “why abstained” explanation based on decision policy.

8.2 Intermediate Extensions

  • Compare two resize policies against the same labeled fixture set.
  • Compare evaluator and EXLA host paths with compilation separated.
  • Add an active-learning export containing only explicit user-approved, licensed fixtures.

8.3 Advanced Extensions

  • Add a shared supervised serving process and prove session isolation under batching.
  • Evaluate score calibration and choose a threshold from a held-out validation set.
  • Package the model and cache into a container and verify immutable offline startup.

9. Real-World Connections

9.1 Industry Applications

  • Warehouse damage triage: prioritize human inspection while retaining final authority.
  • Manufacturing quality review: classify visible defects under a documented camera and lighting contract.
  • Content moderation assistance: rank cases for review without automatic enforcement.
  • Medical/research pre-screening: only as a carefully governed assistive workflow with domain validation far beyond this project.
  • Support intake: attach structured suggestions and uncertainty to uploaded evidence.
  • Nx: tensors, numerical definitions, backends, and serving.
  • Bumblebee: supported pretrained model and preprocessing integrations.
  • Axon: neural-network construction and execution underlying many Bumblebee models.
  • Kino: image inputs and interactive presentation.

9.3 Interview Relevance

This project supports discussion of tensor semantics, compilation, batching, inference architecture, artifact supply chains, cold starts, resource limits, privacy, evaluation metrics, abstention, and human-in-the-loop design.

10. Resources

10.1 Essential Reading

  • Nx API — tensor creation, shape, type, axes, and backends.
  • Nx numerical definitions — graph-building and compilation constraints.
  • Nx.Serving — run, batched serving, batch size, timeout, partitions, and distribution.
  • Bumblebee — supported artifact loading and high-level tasks.
  • Bumblebee examples — current image and serving patterns.
  • AI Engineering by Chip Huyen — evaluation, inference, and operating model-backed systems.

10.2 Video Resources

  • Use official Numerical Elixir and Livebook presentations linked from the current project documentation.
  • Prefer current Livebook Launch Week or release demonstrations at Livebook News for app and ML integration behavior.

10.3 Tools & Documentation

  • Kino.Input: image-input contract.
  • Kino.Workspace: app/session context.
  • Livebook system resources panel: observe runtime memory and process behavior.
  • Fixture manifest and checksum tool: preserve labeled-set provenance.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain every tensor axis, type conversion, and backend copy.
  • I can distinguish artifact load, compilation, queue, inference, and postprocessing.
  • I can explain why the score is not automatically a probability.
  • I can defend the abstention and retention policies.

11.2 Implementation

  • All input, tensor, artifact, serving, and decision requirements are met.
  • Labeled, ambiguous, OOD, corrupt, and oversized fixtures are tested.
  • Two-client privacy and session cleanup pass.
  • Offline startup uses the approved digest-verified artifact.
  • Clean-runtime replay reproduces the evaluation report.

11.3 Growth

  • I documented the largest model limitation revealed by evaluation.
  • I can estimate capacity per request and per concurrent session.
  • I can explain which components belong in a normal Mix application before production.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • One bounded image reaches a visible, asserted tensor contract and pinned model.
  • The result can suggest or abstain and requires a human action.
  • Corrupt and oversized fixtures fail safely.

Full Completion:

  • Complete artifact provenance, fixed preprocessing, offline startup, phase-specific latency, Nx.Serving bounds, labeled evaluation, per-class metrics, OOD tests, two-client isolation, retention scan, and clean replay all pass.

Excellence (Going Above & Beyond):

  • Demonstrate a supervised shared serving architecture with measured batching, capacity model, calibrated decision threshold, immutable container artifact, and privacy review—without changing the human-owned decision boundary.

This guide was generated from LEARN_LIVEBOOK_ELIXIR_DEEP_DIVE.md. For the complete learning path, see the parent guide.