Project 29: Nx Demand Forecasting Engine

Build a reproducible demand-forecasting engine that turns ordinary Elixir data into well-shaped tensors, trains a small numerical model, beats a declared baseline, and explains exactly where compilation and evaluation choices can mislead you.

Quick Reference

Attribute Value
Difficulty Level 3: Advanced
Suggested Seniority Senior Elixir engineer
Time Estimate 20-32 hours
Main Programming Language Elixir
Alternative Programming Languages Python for comparison; Erlang for data-loading comparison
Coolness Level Level 4: Hardcore Tech Flex
Business Potential Level 2: Micro-SaaS / Pro Tool
Prerequisites Elixir collections, structs, pipelines, basic statistics, linear algebra vocabulary
Key Topics Nx tensors, shapes and dtypes, defn, JIT, time-series windows, leakage, baselines, reproducibility

1. Learning Objectives

By completing this project, you will:

  1. Convert validated tabular observations into correctly shaped Nx tensors.
  2. Reason about axes, broadcasting, dtypes, containers, and backend boundaries.
  3. Express a trainable numerical function inside the restricted defn environment.
  4. Build chronological training, validation, and test splits without future-data leakage.
  5. Compare a learned model against naive and seasonal baselines using explicit metrics.
  6. Measure eager, first-JIT, and warmed-JIT behavior without hiding compilation cost.
  7. Produce a reproducible forecast artifact with data, configuration, and model fingerprints.

2. All Theory Needed (Per-Concept Breakdown)

Tensor Semantics: Shapes, Axes, Dtypes, Broadcasting, and Containers

Fundamentals

An Nx tensor is a typed, multidimensional numerical value with a fixed shape. A scalar has no axes, a vector has one, and a batch of fixed-width observations commonly has shape {batch, features}. Shape is not cosmetic: it determines whether an operation represents feature-wise arithmetic, a matrix product, a reduction, or an accidental broadcast. Dtype determines precision, range, storage, and sometimes which operations are valid. Nx operations return new tensor values rather than mutating data. A backend performs the actual numerical work, while Nx.Container allows nested Elixir data structures whose leaves are tensors to participate in traversal and numerical-function input handling. Learning Nx begins by treating shape, dtype, and axis meaning as part of every function’s contract.

Deep Dive

Most early numerical bugs are valid computations with the wrong meaning. Suppose historical demand windows are intended to have shape {examples, lookback} and targets {examples, 1}. If targets are accidentally {examples}, subtracting predictions may broadcast one dimension across another and yield a square error matrix instead of one error per example. No exception is guaranteed; the arithmetic may be perfectly legal. The defense is to document symbolic shapes at every boundary and assert actual shapes before training.

Axes need domain names in your mental model even when the runtime represents them by position. For this project, useful meanings include example, time-within-window, feature, and forecast-horizon. A reduction across the example axis computes an aggregate across observations; a reduction across features changes each observation. Write the intended input and output shape next to every pipeline stage. Prefer a consistent convention such as batch-first, then lookback, then feature. Avoid casual reshaping that preserves element count while destroying semantics.

Dtype choices affect results. Raw counts may arrive as integers, but gradient-based optimization requires floating-point tensors. Converting too late can cause integer division or unsupported differentiation. Using unnecessarily wide dtypes increases memory and may reduce accelerator throughput; using a narrow dtype may magnify rounding or overflow. This project should establish an explicit dtype policy: validate counts as non-negative integers in ordinary Elixir, transform normalized model inputs to a chosen float dtype, and report metrics in a stable display precision. Missing values are not automatically represented by a special Elixir nil inside a numeric tensor. Handle them before tensor construction through rejection, imputation, or an explicit mask tensor.

Broadcasting aligns shapes according to documented rules and is useful for subtracting a feature mean or applying a scale vector across a batch. It is dangerous when shape assumptions remain implicit. Test a batch of one as well as normal batches; singleton dimensions often hide an accidental squeeze. Likewise, the final incomplete batch can expose code that assumed a fixed compile-time batch size. Decide whether to pad with a mask, drop incomplete training examples, or compile for variable leading dimensions where the backend supports it. Evaluation must never silently drop test rows merely to satisfy a convenient shape.

Nx containers connect numerical leaves to ordinary Elixir structure. A forecasting batch could be represented conceptually as a map containing input tensor, target tensor, and mask tensor. Container-aware transformations and defn inputs can preserve this organization, but static metadata such as product identifiers or timestamps should not be smuggled into numerical definitions as arbitrary dynamic values. Keep an index outside the tensor pipeline that maps example positions back to business keys and dates.

Backend choice is a boundary, not a business-domain decision. Begin with the default binary backend for correctness and small fixtures. Add an optimizing compiler backend only after the shape and metric tests are trustworthy. Transfers between backends and host conversions can be expensive and may force synchronization. Batch reporting rather than converting every prediction to an Elixir number inside a loop.

Name each boundary conversion in the design so an accidental host transfer is visible during review and benchmarking.

How this fits on the project

Tensor semantics define the dataset builder, batch contract, normalization stage, model input/output, metric calculations, and backend comparison.

Definitions and key terms

  • Shape: The ordered size of each tensor dimension.
  • Axis: One dimension with a domain meaning such as example or feature.
  • Dtype: The numerical representation, precision, and range of tensor elements.
  • Broadcasting: Applying operations across compatible dimensions without explicit copying.
  • Container: A supported nested data structure whose leaves can be traversed as tensors.
  • Backend: The implementation that stores tensors and executes operations.

Mental model diagram

validated rows
  {date, item, demand, features}
          |
          v
[chronological window builder]
          |
          +--> X shape={examples, lookback, features} dtype=float
          +--> y shape={examples, horizon}            dtype=float
          +--> index={example -> item/date}            ordinary Elixir
          |
          v
[batch container: %{inputs: X, targets: y, mask: M}]
          |
          v
       Nx backend

How it works

  1. Validate dates, counts, and feature availability outside Nx.
  2. Sort observations chronologically within each item series.
  3. Build lookback windows and future targets without crossing split boundaries.
  4. Construct tensors with an explicit shape and dtype contract.
  5. Normalize using statistics computed only from the training split.
  6. Batch examples while preserving masks and business-key indexes.
  7. Invariant: every prediction position maps to exactly one known item and forecast date.
  8. Failure modes include accidental broadcasting, target leakage, empty tensors, singleton dimension loss, overflow, and backend transfer churn.

Minimal concrete example

TENSOR CONTRACT, NOT RUNNABLE CODE
inputs:  float32 {batch, 28 days, 4 features}
targets: float32 {batch, 7 forecast days}
mask:    u8      {batch, 7 forecast days}

assert axis 0 means examples
assert axis 1 means chronological position
assert normalization parameters came from training dates only

Common misconceptions

  • If Nx accepts two shapes, the calculation must represent the intended domain operation.
  • Missing numerical values can be placed in a tensor as ordinary nil.
  • Changing backends cannot affect memory behavior or compilation constraints.

Check-your-understanding questions

  1. Why can {batch} targets produce a plausible but wrong loss against {batch, 1} predictions?
  2. Why must normalization statistics be computed only on the training interval?
  3. What data belongs outside the tensor pipeline?

Check-your-understanding answers

  1. Broadcasting may expand both operands and compute pairwise errors rather than aligned errors.
  2. Test-period statistics leak future information and make evaluation optimistic.
  3. Business identifiers, source timestamps, rejection details, and other non-numerical indexing metadata.

Real-world applications

  • Inventory replenishment forecasts
  • Workforce and call-volume planning
  • Energy-load estimation
  • Capacity reservation and anomaly baselines

Where you will apply it

  • Project 29 dataset builder, model contract, metrics, and backend benchmark

References

Key insight

In numerical Elixir, shape and axis meaning are as important as ordinary function types.

Summary

Correct tensor work requires explicit shapes, deliberate dtypes, leakage-free preprocessing, restrained host/backend transfers, and a durable mapping from numerical positions back to business observations.

Homework/Exercises

  1. Write shape contracts for univariate and multivariate seven-day forecasts.
  2. Explain how to handle a final batch with three real examples in a nominal batch of eight.

Solutions

  1. Univariate may use {batch, lookback, 1} to {batch, 7}; multivariate changes the final input axis to the feature count while preserving target meaning.
  2. Use variable leading dimensions or pad to eight with an explicit mask; never score padded targets as real observations.

defn, Compilation Boundaries, and Honest Forecast Evaluation

Fundamentals

Nx.Defn defines numerical functions that Nx can transform, differentiate, and compile. The defn environment is deliberately restricted: tensor operations and supported control-flow constructs form a numerical program, while arbitrary Elixir side effects, dynamic data structures, and ordinary process interactions do not belong inside it. JIT compilation specializes work based on input structure and shapes, so the first call may include compilation cost while subsequent compatible calls reuse compiled work. A forecasting engine also needs evaluation discipline outside defn: chronological splits, declared baselines, stable metrics, deterministic seeds, and an artifact recording exactly which data and configuration produced a model. Fast tensor execution cannot rescue an invalid experimental design.

Deep Dive

The numerical function boundary should be narrow and explicit. Ordinary Elixir is excellent for reading files, validating rows, attaching business identifiers, selecting date ranges, and reporting errors. defn is appropriate for prediction, loss, gradient computation, parameter updates, and tensor metrics. Trying to hide data ingestion or logging inside numerical definitions fights the model and makes compilation opaque. Design one pure numerical core with tensors in and tensors out, then orchestrate it from normal Elixir.

Compilation changes how control flow is interpreted. A branch dependent on a tensor value needs supported numerical control flow rather than an ordinary Elixir if that expects a host boolean. Loops need transformable constructs and stable carried state. Shapes often influence compilation cache keys, so highly variable shapes can trigger repeated compilation. For training, use a small set of planned batch shapes. For forecasts, either define accepted shapes or precompile representative templates and report when a new shape incurs compilation.

Different timing categories answer different questions. Eager execution helps debug correctness. The first JIT call includes tracing and compilation, representing cold-start behavior. Warmed JIT measures repeated compatible requests. Report all three rather than advertising only the smallest number. Force completion before stopping a timer when the backend can execute asynchronously. Include batch size, shape, dtype, backend, Elixir/OTP versions, and hardware in benchmark output.

The model can remain deliberately small: a linear autoregressive predictor or a compact multilayer perceptron is enough to teach parameters, loss, gradients, and updates. Complexity should be earned by baseline performance. First implement a last-value baseline and a seasonal-naive baseline that predicts demand from the corresponding prior period. The learned model must be compared against both on an untouched chronological test interval. If it does not win, the project still succeeds when the report explains why; hiding a losing model would teach the wrong lesson.

Time-series splits differ from random tabular splits. Training dates precede validation dates, which precede test dates. Windows must not use target observations from beyond their split. Hyperparameter choices use validation performance only. Test metrics are computed once for the final selected configuration. Useful metrics include MAE for interpretability, RMSE for stronger penalty on large misses, and a scaled or percentage metric only when its zero-demand behavior is explicit. Also report error by horizon and by high-volume versus low-volume item groups so a single average cannot hide operational failure.

Reproducibility requires more than a random seed. Fingerprint the normalized input data, feature specification, date split, lookback and horizon, dtype, backend, initialization seed, model structure, training steps, optimizer settings, and source revision. Persist the best parameter artifact only alongside this manifest. A loader must reject incompatible feature order or model version instead of producing a numerically valid but meaningless forecast.

Numerical stability should be visible. Feature scales can make gradient descent diverge, too high a learning rate can produce non-finite values, and percentage metrics can explode around zero. Add finite-value checks at defined checkpoints and classify failures with step and metric context. Do not convert every tensor element to host data merely to check it; reduce to compact health indicators within the numerical core.

How this fits on the project

This concept governs the prediction/loss/update core, compilation plan, benchmark report, chronological evaluation, and model manifest.

Definitions and key terms

  • defn: An Nx numerical definition that can be transformed and compiled.
  • JIT: Just-in-time compilation specialized to input structure and shape.
  • Baseline: A simple forecast that a learned model must be compared against.
  • Leakage: Information from validation/test or the future influencing training.
  • Horizon: The number of future periods predicted.
  • Artifact manifest: Metadata required to reproduce and safely load a model.

Mental model diagram

ordinary Elixir orchestration                Nx.Defn numerical core
----------------------------------           ------------------------------
read -> validate -> split -> batch  -------> predict(parameters, inputs)
manifest + seed                    -------> loss(predictions, targets, mask)
training loop control              <------- gradients + compact metrics
report + artifact                  <------- updated parameters

evaluation: seasonal baseline ----- compare on same untouched test interval

How it works

  1. Freeze chronological splits and preprocessing parameters.
  2. Initialize parameters from a recorded seed.
  3. Run the numerical prediction and loss in eager mode until correct.
  4. Differentiate loss and update parameters for planned batch shapes.
  5. Select configuration using validation data.
  6. Evaluate learned and baseline forecasts on the same test dates.
  7. Benchmark eager, cold JIT, and warm JIT separately.
  8. Invariant: test observations never influence parameters or configuration selection.
  9. Failure modes include recompilation storms, non-finite loss, shape-specific cache churn, leakage, and misleading aggregate metrics.

Minimal concrete example

PSEUDOCODE
numerical_step(parameters, batch):
  predictions = model(parameters, batch.inputs)
  loss = masked_mean_absolute_error(predictions, batch.targets, batch.mask)
  gradients = gradient(loss with respect to parameters)
  return {updated_parameters, compact_training_metrics}

Common misconceptions

  • defn is ordinary Elixir that simply runs faster.
  • Warm JIT time represents cold request latency.
  • A lower training loss proves a useful forecasting model.

Check-your-understanding questions

  1. Why should file parsing stay outside defn?
  2. What does a seasonal-naive baseline protect against?
  3. Why can variable batch shapes cause latency spikes?

Check-your-understanding answers

  1. It is side-effecting, dynamic orchestration rather than transformable numerical computation.
  2. It prevents claiming value when the model merely learns an obvious repeating pattern less effectively.
  3. A new shape may require a new compiled specialization rather than cache reuse.

Real-world applications

  • Batch replenishment planning
  • Embedded forecasts inside Phoenix/Elixir systems
  • Numerical preprocessing in Livebook workflows
  • Low-latency compiled inference services

Where you will apply it

  • Project 29 numerical model, training, evaluation, benchmarks, and artifacts

References

Key insight

Compilation accelerates a numerical program; it does not validate the experiment that produced it.

Summary

Keep defn pure and shape-aware, measure cold and warm execution honestly, use chronological evaluation, and require a learned forecast to justify itself against simple baselines.

Homework/Exercises

  1. Design a three-way chronological split for two years of daily observations.
  2. Define a benchmark table that cannot hide compilation overhead.

Solutions

  1. Use the earliest interval for training, the next contiguous interval for validation, and the final untouched interval for testing; ensure windows do not cross boundaries improperly.
  2. Include eager, cold JIT, warm JIT, batch shape, dtype, backend, iterations, and environment, with synchronization before timing ends.

3. Project Specification

3.1 What You Will Build

Build demand_forecast, a CLI-oriented forecasting engine for daily item demand. It imports a documented CSV schema, validates and groups series, creates leakage-free windows, trains a small Nx model, compares it with last-value and weekly seasonal baselines, writes a forecast CSV, and stores a reproducibility manifest with the selected parameters.

3.2 Functional Requirements

  1. Validate dates, item IDs, demand, optional price, promotion, and availability fields.
  2. Reject duplicate item/date rows and report gaps according to policy.
  3. Build configurable lookback and forecast-horizon windows.
  4. Split chronologically and compute preprocessing statistics from training only.
  5. Train a deterministic small model in Nx.
  6. Evaluate naive, seasonal, and learned forecasts on identical rows.
  7. Report MAE, RMSE, horizon error, and excluded/masked counts.
  8. Export forecasts with item/date provenance and an artifact manifest.
  9. Benchmark eager, cold JIT, and warmed JIT modes.

3.3 Non-Functional Requirements

  • Identical data, manifest, seed, and environment produce equivalent metrics within a declared tolerance.
  • Shape and dtype contracts are asserted at subsystem boundaries.
  • Training never loads all raw source text twice merely for convenience.
  • Metric reports state denominators and zero-demand behavior.
  • The engine refuses incompatible model artifacts instead of guessing feature order.

3.4 Example Usage / Output

$ mix forecast.train --input data/daily_demand.csv --lookback 28 --horizon 7 --seed 1042
dataset items=120 rows=175200 rejected=3
split train=2023-01-01..2024-06-30 validation=2024-07-01..2024-09-30 test=2024-10-01..2024-12-31
tensor inputs={18340,28,4} targets={18340,7} dtype=f32
baseline last_value       test_mae=8.41
baseline weekly_seasonal  test_mae=6.92
model linear_window       test_mae=6.31 test_rmse=10.84
artifact=artifacts/linear-window-7d manifest_sha256=8ce1...

3.5 Data Formats / Schemas / Protocols

INPUT CSV
date,item_id,demand,price,promotion,available
2024-01-01,SKU-0042,18,12.50,false,true

FORECAST CSV
item_id,forecast_date,horizon_day,predicted_demand,model_version
SKU-0042,2025-01-01,1,19.4,linear-window-v1

ARTIFACT MANIFEST
{schema_version, data_digest, feature_order, split_dates, lookback,
 horizon, dtype, backend, seed, model, optimizer, training_steps,
 parameter_digest, source_revision, metrics}

3.6 Edge Cases

  • New item with less history than the lookback
  • Zero demand throughout a metric interval
  • Missing dates caused by store closure versus data loss
  • Out-of-stock days where observed demand is censored
  • Negative demand representing returns
  • Final incomplete batch
  • One-item dataset producing singleton dimensions
  • Non-finite loss after a parameter update
  • Artifact built with a different feature order or dtype

3.7 Real World Outcome

3.7.1 How to Run

$ mix deps.get
$ mix test
$ mix forecast.train --input test/fixtures/demand.csv --seed 1042
$ mix forecast.predict --artifact artifacts/reference --from 2025-01-01

3.7.2 Golden Path Demo

The deterministic fixture contains weekly seasonality, promotions, missing days, and one invalid row. The engine rejects the invalid row, derives training-only normalization, trains the reference model, beats the last-value baseline, and explains whether it beats the stronger seasonal baseline.

3.7.3 Exact Terminal Transcript

$ mix forecast.benchmark --artifact artifacts/reference --batch 64
backend=EXLA.Client dtype=f32 input={64,28,4}
eager_median_ms=2.84
jit_first_call_ms=118.72
jit_warm_median_ms=0.43
compiled_variants=1 synchronized=true

3.8 Scope Boundary Versus Projects 1-13

Projects 1-13 focus on BEAM processes, supervision, distribution, real-time flow, ETS, telemetry, and releases. This project teaches Elixir as a numerical-computing environment. It does not add another event pipeline or live dashboard; its core difficulty is tensor semantics, compilation, experimental validity, and reproducibility.

4. Solution Architecture

4.1 High-Level Design

CSV --> [validator] --> [series index] --> [chronological splitter]
                                               |
                                               v
                                     [window + tensor builder]
                                      /                  \
                              [baselines]           [Nx model]
                                                       |
                                             eager -> JIT -> metrics
                                      \                  /
                                       [evaluation report]
                                               |
                                  forecast CSV + artifact manifest

4.2 Key Components

Component Responsibility Key decision
Row Validator Parse and classify input Preserve rejected rows and reasons
Series Index Group and order observations Define duplicate and missing-date policy
Splitter Freeze chronological intervals Prevent windows crossing future boundaries
Tensor Builder Create inputs, targets, masks Assert axis and dtype contracts
Baseline Suite Produce naive forecasts Evaluate on exactly the same rows
Numerical Core Predict, calculate loss, update parameters Keep side effects outside defn
Evaluator Aggregate metrics by horizon/item group Publish denominator and masks
Artifact Store Persist parameters and manifest Reject incompatible loads

4.3 Data Structures (No Full Code)

  • Observation: item, date, demand, feature values, availability classification.
  • Series: ordered observations plus gaps and rejection summary.
  • SplitPlan: fixed train, validation, and test date intervals.
  • TensorBatch: input, target, and mask tensors with a separate row index.
  • ModelManifest: complete compatibility and reproducibility contract.
  • MetricReport: aggregate plus horizon and cohort breakdowns.

4.4 Algorithm Overview

Within each item, sort validated observations, apply the documented gap policy, and create windows whose targets remain in their assigned split. Fit preprocessing statistics on training windows. Train the model through deterministic batches, select configuration on validation MAE, and evaluate once on the test interval alongside baselines.

Window construction is O(N * L) if each of N examples copies a lookback of length L; a view-oriented or chunked builder can reduce allocation pressure. Model cost depends on tensor shapes and architecture. Host-side series indexes are O(N), while batch tensor memory is bounded by configured batch size.

5. Implementation Guide

5.1 Development Environment Setup

Use current Nx with the default backend first. Add EXLA only after correctness tests pass. Record Elixir, OTP, Nx, backend, and native runtime versions in benchmark output.

5.2 Project Structure

demand_forecast/
├── lib/
│   ├── mix/tasks/forecast.train.ex
│   ├── mix/tasks/forecast.predict.ex
│   └── demand_forecast/
│       ├── row_validator.ex
│       ├── series_index.ex
│       ├── split_plan.ex
│       ├── windows.ex
│       ├── tensor_builder.ex
│       ├── baselines.ex
│       ├── model.ex
│       ├── trainer.ex
│       ├── evaluator.ex
│       └── artifact.ex
├── test/fixtures/
└── test/

5.3 The Core Question You Are Answering

“How do I make a numerical forecast fast on the BEAM without confusing tensor validity, compiled speed, and genuine predictive value?”

5.4 Concepts You Must Understand First

  1. Matrix/vector shape and element-wise versus matrix operations.
  2. Time-series chronology, windows, horizon, and leakage.
  3. Nx tensor, backend, and container contracts.
  4. Nx.Defn restrictions, JIT specialization, and differentiation.
  5. MAE, RMSE, baseline comparison, and reproducibility.

5.5 Questions to Guide Your Design

  1. What domain meaning does every axis carry?
  2. How are missing and out-of-stock observations represented?
  3. Can any training transform see validation or test dates?
  4. Which batch shapes compile, and how is recompilation observed?
  5. What evidence proves the learned model adds value over seasonality?

5.6 Thinking Exercise

Draw the lineage of one test prediction backward through its target, input window, preprocessing statistics, model parameters, validation selection, and training rows. If any arrow reaches a later test observation, the design leaks future information.

5.7 The Interview Questions They Will Ask

  1. “How do Nx broadcasting bugs produce valid but incorrect results?”
  2. “What code belongs outside a defn?”
  3. “Why is a random train/test split usually invalid for forecasting?”
  4. “How do you benchmark a JIT-compiled function honestly?”
  5. “What makes a model artifact safe to reload?”
  6. “What would you do when the learned model loses to seasonal naive?”

5.8 Hints in Layers

Hint 1: Start with baselines — Build evaluation and two naive predictors before any trainable model.

Hint 2: Annotate shapes — Put symbolic input/output shapes in module documentation and assert them at boundaries.

Hint 3: Separate worlds — Keep CSV, dates, IDs, and reporting in ordinary Elixir; keep tensor math in a small numerical core.

Hint 4: Record the experiment — Treat the manifest as required output, not cleanup after training.

5.9 Books That Will Help

Topic Book Precise reading
Nx fundamentals Machine Learning in Elixir and Livebook — Moriarity, Tate, DeBenedetto Chapter “The Nx Numerical Computing Library”
Data preparation Machine Learning in Elixir and Livebook Chapter “Data Preprocessing with Explorer,” adapting the concepts to the project loader
Neural/numerical models Machine Learning in Elixir and Livebook Chapter “Neural Networks with Axon” for parameter/loss context, even if the core model remains manual
Forecast evaluation Forecasting: Principles and Practice, 3rd ed. — Hyndman and Athanasopoulos Chapters 3 “The Forecaster’s Toolbox” and 5 “The Time Series Regression Model”

5.10 Implementation Phases

Phase 1: Data Contract and Baselines (5-7 hours)

  • Validate rows and build chronological series.
  • Freeze splits and generate windows without tensors.
  • Implement last-value and weekly seasonal baselines plus metrics.

Phase 2: Tensor Pipeline (5-8 hours)

  • Define axis/dtype contracts and build masked batches.
  • Test singleton and incomplete batches.
  • Verify normalization uses only training data.

Phase 3: Numerical Model (6-9 hours)

  • Implement prediction, loss, gradients, and updates in defn pseudocode design.
  • Add finite-value checks and deterministic initialization.
  • Select a configuration using validation results.

Phase 4: Artifact and Performance (4-8 hours)

  • Evaluate the untouched test interval against baselines.
  • Persist parameters and complete manifest.
  • Benchmark eager, cold JIT, and warmed JIT modes.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
First model complex neural network / linear window Linear window Makes shape, loss, and baseline reasoning visible
Split random / chronological Chronological Prevents future leakage
Missing days silently zero / reject / explicit policy + mask Explicit policy + mask Separates no demand from no observation
Backend optimized immediately / default then optimized Default then optimized Establishes correctness before performance
Benchmark warm only / eager+cold+warm All three Reports operational reality

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Validate parsing and series rules duplicates, gaps, returns, closures
Shape Protect tensor contracts singleton batch, horizon seven, masks
Numerical Check model math hand-calculated tiny loss and update
Leakage Protect chronology future sentinel values never affect training
Property Stress window generation arbitrary series and split boundaries
Reproducibility Compare repeated runs manifests, metrics, parameter digest
Performance Characterize execution eager, cold JIT, warm JIT, shape variants

6.2 Critical Test Cases

  1. Duplicate item/date input is rejected with both source line numbers.
  2. A window never includes a target outside its split.
  3. Validation/test outliers do not alter normalization parameters.
  4. {batch} versus {batch, 1} target mismatch fails an assertion.
  5. Padded targets do not contribute to loss or metrics.
  6. All-zero demand yields a defined metric report without division surprises.
  7. A deliberately high learning rate triggers a non-finite training diagnostic.
  8. Two seeded reference runs produce equivalent metrics and manifests.
  9. Loading an artifact with reordered features is rejected.
  10. Baselines and model score exactly the same item/date rows.

6.3 Test Data

Use a tiny hand-computable series, a deterministic seasonal series, a multi-item fixture with gaps, a zero-demand series, a one-item singleton case, and a larger generated fixture for performance. Keep split dates and expected windows visible in fixture documentation.

6.4 Definition of Done

  • Input, rejection, forecast, and artifact schemas are documented.
  • Every tensor boundary asserts shape, dtype, and axis meaning.
  • Chronological leakage tests pass with future sentinel values.
  • Last-value and seasonal baselines are scored on identical test rows.
  • The learned model’s result is reported honestly whether it wins or loses.
  • Repeated seeded runs meet the declared reproducibility tolerance.
  • Cold JIT time is separated from warm execution time.
  • Artifact loads reject incompatible feature order, dtype, or model version.
  • Edge cases have explicit policy rather than silent row deletion.
  • Tests pass on the default backend before optional optimized-backend tests.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem Why it happens Fix Quick test
Suspiciously low loss Target broadcast across examples Assert exact prediction/target shapes Use a two-row hand-computed batch
Great test result, poor production result Random split or preprocessing leakage Freeze chronological lineage Add future sentinel values
Frequent latency spikes New shapes trigger compilation Plan and count compiled variants Alternate batch sizes in benchmark
Percentage metric explodes Demand can be zero Choose explicit zero-safe metrics Test all-zero series
Artifact predicts nonsense Feature order differs at load Validate full manifest compatibility Swap two feature columns
Training becomes non-finite Scale or learning rate is unstable Normalize, reduce step size, add finite checks Use an intentionally unstable fixture

7.2 Debugging Strategies

  • Print shape, dtype, axis labels, and finite summaries at subsystem boundaries.
  • Reproduce a loss calculation on a two-example tensor by hand.
  • Compare model predictions against both baselines for a single item and horizon.
  • Inspect the exact date lineage for one suspicious test prediction.
  • Count compilation variants and backend transfers during benchmarks.

7.3 Performance Traps

  • Converting each prediction separately from tensor to host data
  • Recompiling for many small shape variations
  • Building every possible window eagerly in memory
  • Timing asynchronous execution without synchronization
  • Using oversized dtypes or retaining gradients during evaluation

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add moving-average and median baselines.
  • Render horizon-by-horizon error as an ASCII table.
  • Add explicit holiday flags to the feature contract.

8.2 Intermediate Extensions

  • Use Explorer for columnar preprocessing while preserving the same lineage tests.
  • Add rolling-origin backtesting with multiple chronological folds.
  • Compare a manual linear model with an Axon model under the same evaluator.

8.3 Advanced Extensions

  • Serve versioned artifacts with Nx.Serving and bounded batches.
  • Add probabilistic intervals and verify empirical coverage.
  • Detect drift between training and forecast feature distributions.
  • Run CPU versus accelerator benchmarks including transfer and cold-start costs.

9. Real-World Connections

9.1 Industry Applications

Retail, logistics, staffing, energy, and SaaS capacity planning all depend on forecasts whose lineage matters as much as their raw error. An Elixir-native engine can keep ingestion, orchestration, APIs, and numerical inference within one operational stack while using Nx for compiled tensor work.

  • Nx for tensor operations and numerical definitions
  • EXLA for XLA-backed compilation
  • Explorer for dataframe-oriented preprocessing
  • Axon for neural-network model construction
  • Scholar for traditional machine-learning algorithms

9.3 Interview Relevance

The project reveals whether a candidate can cross functional Elixir, numerical contracts, experimental design, and production performance. Senior-level answers should discuss shape bugs, leakage, cold starts, model manifests, and baseline honesty rather than only model architecture.

10. Resources

10.1 Essential Official Reading

10.2 Books and Deeper Study

  • Machine Learning in Elixir and Livebook by Sean Moriarity, Bruce Tate, and Sophie DeBenedetto — Nx, preprocessing, and Axon chapters.
  • Forecasting: Principles and Practice, 3rd edition, by Rob J. Hyndman and George Athanasopoulos — Chapters 3 and 5.

10.3 Verification Checklist Before Sharing the Project

  • Show tensor shapes and split dates in the golden transcript.
  • Include at least one model that loses to a baseline during development.
  • Publish eager, cold-JIT, and warm-JIT measurements separately.
  • Verify every metric denominator and mask rule.
  • Keep all implementation material at the level of pseudocode, tensor contracts, schemas, and CLI transcripts.