Project 33: Gradient Descent Visualizer and Optimizer

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Advanced
  • Time: 2-3 weeks
  • Language: Python with NumPy and Matplotlib
  • Prerequisites: Projects 26 and 30
  • Merged from: ML-Math P09; NN P02; ML-Found P06

What You Will Build

Build an optimization workbench for scalar and two-parameter functions. Implement gradients from analytic callbacks and central differences, then implement batch gradient descent, momentum, and Adam without optimizer libraries. Animate parameter trajectories over curves, contours, and 3D loss surfaces while plotting loss, gradient norm, step norm, and effective learning rate. Include convex bowls, ill-conditioned ellipses, Rosenbrock’s valley, saddle points, and multimodal functions. Add stopping rules and divergence diagnostics rather than relying on a fixed iteration count. A comparison runner should launch algorithms from identical starts and seeds, making oscillation, slow progress, overshooting, and local-minimum dependence observable.

Real World Outcome

Selecting a function and starting point produces an animated path plus a convergence report. On an elongated quadratic, the tool visibly compares zig-zagging vanilla descent with momentum; on Rosenbrock, it shows sensitivity to learning rate. Failed runs are labeled with exploding loss, non-finite gradient, or stalled progress rather than drawn as successes.

Core Question

Why does stepping opposite the gradient usually reduce a function locally, and how do scale, curvature, and optimizer state determine whether those steps converge?

Concepts You Must Understand First

  1. Gradient: the vector of partial derivatives points toward steepest local increase. See Orland, Math for Programmers, optimization chapters.
  2. Local linear approximation: sufficiently small negative-gradient steps should reduce smooth objectives.
  3. Curvature and conditioning: unequal curvature across directions causes oscillation and slow convergence.
  4. Momentum and Adam: moving averages alter direction and per-coordinate scale. See Goodfellow et al., Deep Learning, ch. 8.
  5. Stopping evidence: loss change, gradient norm, step norm, and iteration budget answer different questions.

Build Milestones

  1. Implement finite-difference and analytic gradient adapters with numerical cross-checks.
  2. Build vanilla descent with structured history, multiple stop rules, and non-finite guards.
  3. Add contour/surface animation and convergence plots for standard test functions.
  4. Implement momentum and Adam with correctly initialized state and bias correction.
  5. Create reproducible hyperparameter sweeps and a comparative failure report.

Hints in Layers

  1. Start with f(x)=x²; print every update before attempting 2D animation.
  2. Keep optimizer state separate from parameters and objective; test the first two updates by hand.
  3. Normalize visualization axes carefully: an apparently short path may hide huge loss changes.

Common Pitfalls and Debugging

  • Symptom: loss alternates or grows. Cause: learning rate is too large for local curvature. Fix: sweep rates logarithmically or add backtracking.
  • Symptom: Adam moves far on its first step. Cause: moment bias correction or epsilon placement is wrong. Fix: compare initial updates with the published equations.
  • Symptom: optimizer claims convergence on a plateau. Cause: only loss change was checked. Fix: also report gradient and step norms.

Definition of Done

  • Numerical and analytic gradients agree on smooth test functions.
  • Vanilla, momentum, and Adam share one reproducible optimizer interface.
  • Histories contain parameters, loss, gradients, steps, and stop reason.
  • At least four landscapes demonstrate distinct optimizer behavior.
  • Divergence, stagnation, and maximum-iteration exits are distinguished.
  • A report compares algorithms under identical starts and budgets.

Previous: Project 32 · Complete learning path · Next: Project 34