Project 46: Backpropagation Flow Visualizer

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 3: Advanced
  • Time: 2 weeks
  • Language: Python with Plotly or Matplotlib
  • Prerequisites: Project 45
  • Source: ML-Found P18

What You Will Build

Instrument the MLP from Project 45 to capture per-layer activations, pre-activations, parameters, incoming gradients, parameter gradients, and update magnitudes during training. Present an interactive network view in which node/edge color encodes gradient magnitude, plots show activation distributions and layer-wise norms, and a step control reveals forward and backward flow. Add controlled experiments for sigmoid saturation, dead ReLUs, poor initialization, excessive learning rate, and increasing depth so the dashboard teaches why training succeeds or fails.

Real World Outcome

An interactive session can pause on any mini-batch, animate one optimization step, inspect a neuron’s activation and gradient, and compare healthy ReLU/He training with a saturated sigmoid or exploding-gradient configuration. The dashboard flags suspect layers and exports a diagnostic snapshot alongside the loss curve.

Core Question

What happens to signal and gradient magnitude as information travels forward and error travels backward through a deep network?

Concepts You Must Understand First

  • Forward caches, computational graphs, and matrix backpropagation — Projects 43-45.
  • Activation saturation and dead neurons — Deep Learning, Chapters 6 and 8.
  • Vanishing/exploding gradients and initialization scaling.
  • Gradient norms, update-to-weight ratios, and distribution summaries.
  • Skip connections as alternate gradient paths — He et al., Deep Residual Learning.

Build Milestones

  1. Add non-invasive hooks that record layer statistics without changing computed gradients.
  2. Draw architecture, weight magnitude, activations, and gradients for a single training step.
  3. Add time-series views for loss, gradient norms, saturation, dead-unit fraction, and update ratios.
  4. Create reproducible pathology presets and compare activations, depth, and initialization.
  5. Implement warnings with evidence, not opaque labels, and export a diagnostic report.

Hints in Layers

  1. Store summaries by default; retaining every tensor for every step will exhaust memory.
  2. Use log-scaled gradient magnitude because layers may differ by many orders of magnitude.
  3. Verify instrumentation by training with hooks enabled and disabled under the same seed.

Common Pitfalls and Debugging

  • Training changes when visualization is enabled: hooks mutate arrays or retain graph state. Record copies or read-only summaries and test trajectory equivalence.
  • All gradients look equally dark: linear color scaling hides small values. Use a log scale with an explicit legend and zero handling.
  • Dashboard freezes on long runs: full tensors are retained. Sample batches/steps and aggregate histograms online.

Definition of Done

  • Instrumented and uninstrumented runs match under an identical seed.
  • Users can inspect activation, gradient, parameter, and update statistics per layer.
  • At least four reproducible pathologies have visibly distinct signatures.
  • The view supports pause, single-step, comparison, and exported snapshots.
  • Warning rules cite the measured statistic and threshold that triggered them.

Previous: Project 45 · Complete learning path · Next: Project 47