Project 45: MLP Classifier from First Principles

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 4: Expert
  • Time: 3-4 weeks
  • Language: Python with NumPy only
  • Prerequisites: Projects 30, 36-37, and 42-44
  • Merged from: ML-Math P19; Prog Final Neural Network; NN P06, NN P08; ML-Found P17

What You Will Build

Build a vectorized multilayer perceptron without TensorFlow or PyTorch. Progress from a hidden layer that solves XOR to a configurable stack of dense layers trained by mini-batch gradient descent. Implement ReLU/sigmoid/tanh, stable softmax and cross-entropy, initialization, forward caches, matrix-form backward passes, SGD, regularization, class weighting, and checkpointing. Validate first on a small imbalanced fraud-style binary dataset, where precision/recall matter, then train a 784→128→64→10 network on MNIST and analyze both correct and mistaken digits.

Real World Outcome

One training command prints architecture and parameter counts, learning curves, confusion matrix, class-sensitive metrics, and gradient-check status. The binary model beats a majority baseline without ignoring the rare class; the MNIST model reaches a defensible target such as 95%+ test accuracy and saves reproducible weights and misclassification plots.

Core Question

How do linear layers, nonlinear activations, a loss function, and backpropagation combine into a network that learns nonlinear decisions?

Concepts You Must Understand First

  • Dense-layer matrix multiplication and batched matrix calculus — Project 43.
  • Reverse-mode backpropagation and parameter ownership — Project 44.
  • ReLU, sigmoid, tanh, softmax, and cross-entropy — Deep Learning, Chapter 6.
  • Xavier/He initialization, saturation, and symmetry breaking.
  • Imbalanced classification metrics, class weighting, and threshold selection.

Build Milestones

  1. Implement dense layers and activations; prove a one-hidden-layer network learns XOR.
  2. Add stable softmax/cross-entropy, vectorized mini-batches, initialization, and per-layer gradient checks.
  3. Train a binary MLP with class weighting; report precision, recall, PR curve, and confusion matrix.
  4. Train the configurable MLP on MNIST; save checkpoints and inspect misclassified digits.
  5. Compare activations, depths, initialization schemes, and seeded runs in one experiment report.

Hints in Layers

  1. Overfit a tiny batch before attempting the full dataset; failure here indicates implementation, not generalization.
  2. Subtract each row’s maximum logit before exponentiating and combine softmax with cross-entropy backward.
  3. Track activation and gradient statistics per layer to detect saturation or dead ReLUs.

Common Pitfalls and Debugging

  • Loss is nan: softmax overflow or log(0). Use log-sum-exp stabilization and clip only at the final logarithm boundary.
  • Accuracy is high but fraud recall is zero: class imbalance rewards predicting the majority. Use class weights and report PR metrics.
  • Every hidden unit behaves identically: weights were initialized equally. Use seeded random initialization scaled by fan-in.

Definition of Done

  • XOR is learned consistently by a hidden-layer network.
  • Every layer’s gradients pass finite-difference checks on a tiny batch.
  • Training is vectorized, seeded, checkpointed, and recoverable.
  • Binary evaluation includes minority-class recall, precision, and threshold analysis.
  • MNIST performance meets a declared target and includes error inspection.
  • An experiment report explains activation, initialization, and architecture tradeoffs.

Previous: Project 44 · Complete learning path · Next: Project 46