Project 41: Complete ML Pipeline and House-Price Predictor

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 5: Master
  • Time: 4 weeks
  • Language: Python (NumPy; Pandas may be used only for data loading)
  • Prerequisites: Projects 14, 30, 34, 36, and at least two of Projects 13 and 38-39
  • Merged from: ML-Math P20; ML-Found P19

What You Will Build

Build a reproducible command-line pipeline that ingests raw housing data, validates its schema, profiles missing values and outliers, creates leakage-safe train/validation/test splits, fits preprocessing only on training folds, and compares several models implemented in earlier projects. The final artifact is not merely a predictor: it is a versioned decision report containing the data contract, transformations, cross-validation results, error slices, selected model, test metrics, configuration, seed, and serialized inference bundle. Add a small prediction API only after the offline evaluation is trustworthy.

Real World Outcome

Running one command on a housing CSV produces a validation report, saved split indices, feature pipeline, cross-validation comparison, final RMSE/R² with uncertainty, residual plots, model file, and a sample API response. Repeating the run with the same data and seed reproduces the metrics within a documented tolerance.

Core Question

How do you turn messy raw data into a prediction that can be reproduced, audited, and safely used for a decision?

Concepts You Must Understand First

  • Data contracts, missingness, categorical encoding, scaling, and feature engineering — Designing Machine Learning Systems by Chip Huyen.
  • Leakage boundaries: preprocessing, feature selection, and imputation must be fitted inside each training fold.
  • Cross-validation and model selection — An Introduction to Statistical Learning, Chapter 5.
  • Regression diagnostics: RMSE, MAE, R², residual structure, and subgroup error.
  • Reproducibility: seeds, immutable split indices, configuration snapshots, and dataset fingerprints.

Build Milestones

  1. Implement schema checks, data profiling, deterministic splitting, and a baseline mean predictor.
  2. Build fit/transform preprocessing components and prove that validation/test statistics never influence training.
  3. Compare linear/ridge regression, a decision tree, and one additional eligible model with identical folds.
  4. Retrain the selected configuration, evaluate once on the held-out test set, and generate error-slice plots.
  5. Serialize preprocessing plus model together and expose a validated prediction command or HTTP endpoint.

Hints in Layers

  1. Start with one numeric-only end-to-end slice before adding categoricals and engineered features.
  2. Give every transformer fit and transform phases; log which row IDs each phase sees.
  3. Hash the raw file, store fold indices, and rerun from the saved manifest to test reproducibility.

Common Pitfalls and Debugging

  • Validation looks excellent but test error jumps: preprocessing or selection saw validation/test data. Fit every learned transformation within training folds and audit row IDs.
  • Metrics change between identical runs: random state or row ordering is uncontrolled. Seed all generators and persist shuffled indices and configuration.
  • API predictions differ from notebook predictions: serving uses a different transformation path. Serialize one composite preprocessing-model artifact and test it against offline examples.

Definition of Done

  • Schema violations and malformed requests fail with actionable diagnostics.
  • A naive baseline and at least three model configurations use identical cross-validation folds.
  • Leakage tests prove that held-out rows never influence fitted transformations.
  • The final report includes uncertainty, residual analysis, and at least two meaningful error slices.
  • A clean process can reproduce the selected model and metrics from the saved manifest.
  • Serialized inference returns the same predictions as the evaluation pipeline.

Previous: Project 40 · Complete learning path · Next: Project 42