Project 48: CNN from Scratch

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 5: Master
  • Time: 3 weeks
  • Language: Python with NumPy only
  • Prerequisites: Projects 43-47
  • Source: NN P09

What You Will Build

Implement and train a convolutional neural network without a deep-learning framework. Build Conv2D, ReLU, max-pooling, flatten, dense, and softmax-cross-entropy layers with complete forward and backward passes. Start with a literal loop implementation whose indices are easy to audit, then optionally add im2col for speed. Train a compact LeNet-style model on MNIST, visualize learned kernels and feature maps, and compare it fairly with the dense MLP from Project 45 to explain how local connectivity and parameter sharing change image learning.

Real World Outcome

A command trains the CNN, reports architecture/output shapes and parameter count, plots train/validation loss and accuracy, saves checkpoints, and produces a gallery of input images, intermediate feature maps, predictions, and errors. Every convolution and pooling gradient passes numerical checks on tiny tensors, and the final model reaches a declared MNIST target.

Core Question

How do local receptive fields and shared kernels learn spatial features while using fewer parameters than dense image models?

Concepts You Must Understand First

  • Convolution, channels, padding, stride, receptive fields, and feature maps — Project 47.
  • Parameter sharing and translation tolerance — Deep Learning, Chapter 9.
  • Max-pooling forward routing and backward argmax masks.
  • Tensor shape transformations between convolutional and dense stages.
  • Matrix backpropagation, stable loss, and gradient checking — Projects 43-45.

Build Milestones

  1. Implement loop-based Conv2D forward and validate shapes/values on hand-computed tensors.
  2. Derive and implement gradients for inputs, kernels, and biases; numerically check each one.
  3. Add max-pooling backward, ReLU, flatten, dense, and stable classification loss.
  4. Overfit a tiny MNIST subset, then train a seeded LeNet-style network on full data.
  5. Visualize feature maps/kernels and compare parameters, speed, and errors with the dense MLP.

Hints in Layers

  1. Use tiny non-square inputs and kernels; square symmetric cases hide axis mistakes.
  2. Cache pooling argmax positions and convolution input patches needed by backward.
  3. Optimize only after the loop version passes checks; use it as an oracle for im2col.

Common Pitfalls and Debugging

  • Backward shapes match but gradients fail: kernel/input axes were transposed. Write every tensor shape beside the summation and test one element manually.
  • Pooling gradient appears in several cells: ties or masks are mishandled. Define a deterministic tie policy and route each upstream value accordingly.
  • Training is impossibly slow: loops dominate. First reduce the test dataset, then add a verified im2col path.

Definition of Done

  • Convolution and pooling forward outputs pass hand-computed tests.
  • Input, weight, bias, and pooling gradients pass finite-difference checks.
  • A tiny batch can be deliberately overfit before full training.
  • The full model meets a declared MNIST accuracy target reproducibly.
  • Feature-map and kernel visualizations support a written interpretation.
  • Dense-versus-CNN comparison includes parameters, runtime, accuracy, and error examples.

Previous: Project 47 · Complete learning path · Next: Project 49