Project 42: Manual Neuron and Single-Neuron Backpropagation

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 3: Advanced
  • Time: 1-2 weeks
  • Language: Python without ML frameworks
  • Prerequisites: Projects 26, 33, and 36
  • Merged from: ML-Math P11; NN P01

What You Will Build

Implement one inspectable neuron from scalar arithmetic: weighted sum, bias, activation, loss, local derivatives, parameter gradients, and gradient-descent updates. Train it first with a threshold/perceptron rule on linearly separable Boolean gates, then with sigmoid and differentiable loss so every chain-rule factor can be printed. The program must expose the forward cache and backward calculation for an individual example, compare analytical gradients with centered finite differences, learn AND/OR/NAND/NOR, and demonstrate—rather than hide—why a single neuron cannot solve XOR.

Real World Outcome

A CLI training run prints the equation for one forward/backward step, a decreasing loss curve, learned decision boundary, final truth-table predictions, and a gradient-check report. A separate XOR run terminates with a clear “not linearly separable” diagnosis instead of pretending that more epochs will fix the model.

Core Question

How does changing a weight change the prediction and loss, and how does that local sensitivity become learning?

Concepts You Must Understand First

  • Linear decision boundaries and weighted sums — Grokking Deep Learning by Andrew Trask.
  • Sigmoid, step, and identity activations; saturation and derivative behavior.
  • Mean-squared or binary cross-entropy loss and the chain rule — Neural Networks and Deep Learning by Michael Nielsen.
  • Gradient descent, learning rate, and finite-difference derivative checks.
  • Linear separability and the geometric reason XOR needs a hidden layer.

Build Milestones

  1. Implement deterministic forward prediction and truth-table tests for a hand-configured threshold neuron.
  2. Add loss and explicitly derive gradients for weights, bias, and pre-activation.
  3. Train differentiable and perceptron variants on AND/OR; plot points and the learned boundary.
  4. Add centered finite-difference checks for every parameter and test gradient accumulation over a batch.
  5. Run XOR, visualize the failed boundary, and explain the representational limitation.

Hints in Layers

  1. Keep all variables scalar for the first version and name each intermediate quantity from the derivation.
  2. Check gradients on one example before summing a batch; use relative error when values are near zero.
  3. If learning stalls, print activation and derivative together—sigmoid may be saturated.

Common Pitfalls and Debugging

  • Loss increases or oscillates: update sign is reversed or learning rate is too large. Verify parameter -= rate × gradient on a one-parameter quadratic.
  • Gradient check fails: the backward pass uses recomputed or overwritten intermediates. Cache exactly the values from the corresponding forward pass.
  • XOR never converges: this is a capacity limit, not necessarily a bug. Plot the four points and prove no single line separates the labels.

Definition of Done

  • Forward output matches hand calculations for at least three parameter/input cases.
  • Analytical gradients pass centered finite-difference checks under a stated tolerance.
  • The neuron learns at least four linearly separable Boolean gates from more than one seed.
  • A trace explains every factor in one backward update.
  • Decision-boundary and loss plots make learning observable.
  • The XOR experiment documents the limitation and motivates an MLP.

Previous: Project 41 · Complete learning path · Next: Project 43