Project 34: Linear Regression and Curve-Fitting Engine

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Advanced
  • Time: 3-4 weeks
  • Language: Python with NumPy, Pandas, and Matplotlib
  • Prerequisites: Projects 5, 12, and 33
  • Merged from: HS P09; ML-Math P17; NN P03; ML-Found P07, ML-Found P13

What You Will Build

Build a from-scratch regression system that fits a line with closed-form least squares and gradient descent, then extends to multiple features and polynomial, exponential, or sinusoidal bases. Implement preprocessing, an explicit intercept, feature standardization fitted only on training data, train/validation/test separation, and regularized fitting. Report MSE, RMSE, MAE, R², parameter uncertainty intuition, and residual diagnostics. Compare the normal equation or least-squares solve with gradient descent, emphasizing conditioning and avoiding an explicit inverse where possible. Add a model-selection runner that compares basis families and degrees without selecting on the final test set. The final report must translate coefficients back into the problem’s units and state where extrapolation is unsafe.

Real World Outcome

Given housing or temperature data, the tool saves fitted curves, residual-versus-prediction and residual-distribution plots, learning curves, metrics by split, learned coefficients, and a plain-language interpretation. It can demonstrate underfitting with a line, overfitting with a high-degree polynomial, and improvement from regularization or a suitable sinusoidal basis.

Core Question

How does “best fit” become a precise optimization problem, and what evidence distinguishes a useful relationship from leakage, overfitting, or a misleadingly low training error?

Concepts You Must Understand First

  1. Least squares: minimize the sum or mean of squared residuals. See Géron, Hands-On Machine Learning, ch. 4.
  2. Design matrix: nonlinear basis functions can still be linear in learned coefficients.
  3. Normal equation and conditioning: solve XᵀXw=Xᵀy, preferably with stable least-squares methods.
  4. Gradient of MSE: derive parameter updates using matrix shapes and the chain rule.
  5. Generalization diagnostics: residual patterns, held-out metrics, and learning curves expose model mismatch. See James et al., An Introduction to Statistical Learning.

Build Milestones

  1. Fit simple linear regression by covariance formulas and verify against hand data.
  2. Implement matrix-based multivariate fitting and gradient descent, including standardization and intercept handling.
  3. Add polynomial and other basis transforms plus L2 regularization.
  4. Create leakage-safe splits, metrics, residual plots, and learning curves.
  5. Compare candidate families on validation data and produce one final untouched test report.

Hints in Layers

  1. Use a noiseless line first; both solvers should recover parameters almost exactly.
  2. Fit all preprocessing statistics on training data and apply them unchanged to later splits.
  3. Watch the condition number as polynomial degree grows; scaling and QR/SVD solves are safer than explicit inversion.

Common Pitfalls and Debugging

  • Symptom: gradient descent diverges while closed form works. Cause: unscaled features or excessive learning rate. Fix: standardize training features and sweep rates.
  • Symptom: validation performance is implausibly strong. Cause: preprocessing or selection saw validation/test targets. Fix: audit the pipeline boundary and preserve an untouched test set.
  • Symptom: R² looks good but errors curve systematically. Cause: wrong functional form. Fix: inspect residuals and compare justified basis families.

Definition of Done

  • Closed-form and gradient solutions agree on well-conditioned fixtures.
  • Multiple features, intercept, basis transforms, and L2 regularization are supported.
  • Preprocessing and model selection are leakage-safe.
  • Reports include MSE, RMSE, MAE, R², residuals, and learning curves.
  • Underfitting, overfitting, and conditioning are demonstrated deliberately.
  • Coefficients and limitations are interpreted in real-world terms.

Previous: Project 33 · Complete learning path · Next: Project 35