Project 43: Matrix Calculus Backpropagation Workbench

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 4: Expert
  • Time: 2-3 weeks
  • Language: Python with NumPy
  • Prerequisites: Projects 20, 26, and 42
  • Source: ML-Math P22

What You Will Build

Create a shape-aware notebook or CLI laboratory that turns scalar backpropagation into batched matrix calculus. Begin with an affine layer, then compose activation and stable softmax-cross-entropy operations. Every operation publishes input/output shapes, its local derivative contract, and a vector-Jacobian product rather than materializing enormous Jacobians unnecessarily. Derive gradients for weights, bias, inputs, and batch reductions; compare each parameter block with centered finite differences; and emit diagnostics that identify transpose, broadcasting, and reduction mistakes by layer and tensor index.

Real World Outcome

For a two-layer MLP and a fixed mini-batch, the workbench prints the forward loss, shape trace, analytical/numerical relative errors for W1, b1, W2, b2, and inputs, plus a pass/fail chain-rule report. Deliberately enabling a transpose or mean/sum bug points to the offending block.

Core Question

How does backpropagation work when inputs, outputs, and derivatives are vectors and matrices rather than scalars?

Concepts You Must Understand First

  • Jacobians, gradients, differentials, and a consistent numerator/denominator layout convention.
  • Matrix chain rule and vector-Jacobian products — Parr and Howard, The Matrix Calculus You Need for Deep Learning.
  • Broadcasting and the need to reduce gradients over expanded axes.
  • Stable softmax-cross-entropy and batch mean versus batch sum conventions.
  • Centered finite differences and robust relative-error denominators.

Build Milestones

  1. Derive and implement affine forward/backward operations with exhaustive shape assertions.
  2. Add ReLU or sigmoid and stable softmax-cross-entropy, documenting each local derivative.
  3. Compose a two-layer batched network using reverse-mode vector-Jacobian products.
  4. Build per-tensor and sampled per-element numerical gradient checks.
  5. Add fault injection for transpose, broadcast-reduction, and batch-scaling errors.

Hints in Layers

  1. Write the shape beside every symbol before writing algebra or code.
  2. Seed the upstream gradient with the loss derivative and work backward one operator at a time.
  3. Test non-square dimensions and batch size greater than one; symmetric shapes conceal transpose bugs.

Common Pitfalls and Debugging

  • Gradients have plausible values but wrong shapes: broadcasting was reversed incorrectly. Sum over axes that were added or had size one in the forward pass.
  • Errors differ by exactly the batch size: one path averages while the other sums. Choose and document one loss-reduction convention.
  • Checks fail only near zero: raw relative error is unstable. Combine absolute and relative tolerances and sample away from activation kinks.

Definition of Done

  • Every operator documents and asserts its forward/backward shape contract.
  • All parameter and input gradients pass numerical checks on non-square batches.
  • The derivation clearly connects local Jacobians to efficient vector-Jacobian products.
  • Stable softmax-cross-entropy agrees with a trusted reference on extreme logits.
  • Fault injection produces a localized, useful mismatch report.

Previous: Project 42 · Complete learning path · Next: Project 44