Project 36: Logistic Regression Classifier

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Advanced
  • Time: 2-3 weeks
  • Language: Python with NumPy, Pandas, and Matplotlib
  • Prerequisites: Projects 12 and 33-34
  • Merged from: ML-Math P18; NN P04; ML-Found P14

What You Will Build

Build binary logistic regression from raw arrays through calibrated probability output. Implement standardized features, logits, a numerically stable sigmoid, binary cross-entropy from logits, analytic gradients, L2 regularization, and mini-batch or full-batch gradient descent. Plot the decision boundary for 2D data and provide a feature-contribution explanation for each prediction. Evaluate accuracy, precision, recall, F1, confusion matrix, ROC-AUC, log loss, and calibration; make classification threshold configurable rather than embedding 0.5 as truth. Use a real dataset such as spam, fraud, or medical-style synthetic data, with stratified splits and imbalanced-class analysis. Compare against a trusted library only after the from-scratch model passes gradient checks.

Real World Outcome

The program trains with a visible loss curve, reports held-out discrimination and calibration metrics, and lets a user move the decision threshold to inspect false-positive/false-negative tradeoffs. Each prediction displays probability, logit, and the largest signed feature contributions. A reliability diagram shows whether “0.8 probability” corresponds to roughly 80% positives.

Core Question

How can a linear score become a probability-like classification model, and why must probability quality be judged separately from thresholded accuracy?

Concepts You Must Understand First

  1. Log odds and sigmoid: a linear logit maps through sigmoid into (0,1).
  2. Bernoulli likelihood and cross-entropy: maximum likelihood yields log loss. See Géron, Hands-On Machine Learning, classification chapters.
  3. Stable computation: use log-domain identities or logits to avoid log(0) and exponential overflow.
  4. Regularization: L2 penalizes large weights and changes the optimization objective.
  5. Metrics and calibration: ranking, threshold decisions, and probability reliability answer different questions.

Build Milestones

  1. Implement stable sigmoid and cross-entropy-with-logits with extreme-value tests.
  2. Derive vectorized weight/intercept gradients and verify them with central differences.
  3. Train on synthetic separable and overlapping 2D data; visualize loss and boundary.
  4. Add scaling, L2 regularization, stratified splits, and real-data ingestion.
  5. Build threshold, ROC, confusion, contribution, and calibration reports.

Hints in Layers

  1. Keep raw logits for loss; do not clip probabilities until you understand what the clipping hides.
  2. Do not regularize the intercept unless you deliberately choose and document that convention.
  3. Choose thresholds on validation data using problem costs, then report once on untouched test data.

Common Pitfalls and Debugging

  • Symptom: loss becomes NaN on confident mistakes. Cause: direct log(sigmoid(z)). Fix: use a stable cross-entropy-from-logits identity.
  • Symptom: model predicts only the majority class. Cause: imbalance and threshold-insensitive accuracy. Fix: inspect precision/recall, class distribution, and threshold curves.
  • Symptom: gradient check fails only for intercept. Cause: shape/broadcasting or accidental regularization. Fix: test intercept separately on a tiny fixture.

Definition of Done

  • Stable loss remains finite for very large positive and negative logits.
  • Analytic gradients pass finite-difference checks.
  • Training is reproducible and loss decreases on controlled data.
  • Evaluation includes discrimination, threshold, and calibration views.
  • Feature scaling and regularization are fitted without leakage.
  • From-scratch predictions agree closely with a trusted reference model.

Previous: Project 35 · Complete learning path · Next: Project 37