Project 61: GPT from Scratch

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 5: Master
  • Time: 3+ months
  • Language: Python using the Project 50 framework; NumPy backend first
  • Prerequisites: Projects 37, 44, and 49-50; Project 60 is a recommended final readiness check
  • Source: ML-Found Final GPT from Scratch

What You Will Build

Build a small decoder-only transformer language model from first principles. Begin with a character tokenizer for correctness, then optionally implement byte-pair encoding. Add token/positional embeddings, scaled dot-product causal self-attention, multi-head attention, feed-forward blocks, residual connections, layer normalization, dropout if supported, final vocabulary logits, stable cross-entropy, autoregressive batching, Adam training, checkpoints, and temperature/top-k generation. Implement the missing tensor operations in your Project 50 framework and numerically verify each new backward rule before composing the full model.

Real World Outcome

A training command prints vocabulary, context length, architecture, parameter count, train/validation loss, perplexity, throughput, gradient norms, and periodic samples. A generation command loads a checkpoint and accepts prompt, length, temperature, top-k, and seed. Attention-mask tests prove tokens cannot see the future, and small-corpus output becomes measurably better than unigram/Markov baselines.

Core Question

How can causal self-attention, repeated transformer blocks, and next-token likelihood produce a model that learns and generates text?

Concepts You Must Understand First

  • Tokenization, embeddings, conditional language modeling, cross-entropy, and perplexity.
  • Query-key-value scaled dot-product attention — Vaswani et al., Attention Is All You Need.
  • Causal masks, multi-head attention, residual streams, and layer normalization.
  • Reverse-mode autograd/framework design — Projects 44 and 50.
  • Autoregressive sequence training and sampling — Project 49.

Build Milestones

  1. Implement tokenizer, shifted next-token batches, and unigram/Markov baselines.
  2. Implement embeddings, masked single-head attention, and hand-calculated causal-mask tests.
  3. Add multi-head attention, feed-forward block, residuals, layer norm, and gradient checks.
  4. Stack decoder blocks, overfit a tiny batch, then train a small model with checkpoint/resume.
  5. Add deterministic generation controls, attention visualization, evaluation, and scaling benchmarks.

Hints in Layers

  1. Use tiny dimensions and a sequence of three tokens to calculate attention scores and masks by hand.
  2. Before full training, force the model to memorize one small batch; inability means an implementation or optimization bug.
  3. Subtract row maxima in softmax, mask future logits before softmax, and test that forbidden probabilities are exactly zero within tolerance.

Common Pitfalls and Debugging

  • Validation loss is implausibly low: targets leaked through an incorrect causal mask or batch shift. Perturb future tokens and prove earlier logits are unchanged.
  • Training produces nan: unstable softmax/layer norm or exploding gradients. Inspect per-block norms, use epsilon correctly, and clip gradients if necessary.
  • Generation repeats or becomes nonsense: decoding settings or undertraining dominate. Compare temperatures/top-k and report validation loss before judging samples.

Definition of Done

  • Tokenizer round-trips text and batching aligns every input with its next-token target.
  • Causal-mask tests prove no output depends on future tokens.
  • New attention/layer-normalization autograd operations pass numerical checks.
  • The complete model deliberately overfits a tiny batch before corpus training.
  • Checkpoint/resume preserves optimizer state and reproduces the loss trajectory.
  • Generation is seeded/configurable and evaluation compares against simpler language baselines.

Previous: Project 60 · Complete learning path