Project 49: Recurrent Character Generator

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Level 5: Master
  • Time: 3 weeks
  • Language: Python with NumPy only
  • Prerequisites: Projects 6, 40, and 43-45
  • Source: NN P10

What You Will Build

Build a character-level recurrent neural network that reads text one character at a time, maintains hidden state, predicts the next-character distribution, and generates seeded samples. Implement vocabulary encoding, sequence batching, recurrent forward equations, stable softmax loss, backpropagation through time (BPTT), gradient clipping, hidden-state handling, checkpointing, and temperature-controlled sampling. Use a small corpus such as Shakespeare, names, or source code and make the temporal mechanics inspectable rather than delegating them to a framework.

Real World Outcome

Training reports loss, perplexity, gradient norms, and periodic generated samples. A generation command loads a checkpoint and accepts seed text, length, temperature, and random seed. Diagnostic mode can show hidden-state norms and per-timestep gradients, making vanishing or exploding behavior visible.

Core Question

How can a network carry information through time, learn sequence dependencies, and assign probabilities to the next symbol?

Concepts You Must Understand First

  • Markov sequence models and conditional next-token distributions — Project 40.
  • Recurrent state equation, unrolling through time, and shared parameters.
  • BPTT and gradient accumulation across timesteps — Deep Learning, Chapter 10.
  • Cross-entropy, perplexity, categorical sampling, and temperature.
  • Vanishing/exploding gradients, clipping, and truncated BPTT.

Build Milestones

  1. Build deterministic vocabulary maps and a baseline frequency/Markov generator.
  2. Implement one recurrent step and unrolled forward loss on very short sequences.
  3. Derive BPTT for all parameters; check gradients with sequence length one, then several steps.
  4. Add clipping, mini-batching or stream chunks, state-reset policy, checkpoints, and sampling.
  5. Train on a real corpus and analyze how temperature and context length affect output.

Hints in Layers

  1. With sequence length one, BPTT reduces to ordinary backpropagation and is much easier to verify.
  2. Remember that recurrent parameters are reused, so their gradients sum contributions from every timestep.
  3. Sample from probabilities rather than taking argmax; temperature below one sharpens and above one diversifies.

Common Pitfalls and Debugging

  • Gradient check passes for one step but fails for sequences: shared-parameter contributions are overwritten. Accumulate gradients across all timesteps.
  • Loss becomes nan: exploding gradients or unstable softmax. Track norms, clip by global norm, and use log-sum-exp.
  • Generated text repeats forever: argmax decoding or overly low temperature collapses diversity. Use categorical sampling and inspect learned probabilities.

Definition of Done

  • Vocabulary encoding/decoding round-trips arbitrary supported text.
  • Multi-timestep recurrent gradients pass numerical checks on a tiny problem.
  • Training logs loss, perplexity, gradient norms, and deterministic checkpoints.
  • Generation supports seed text, temperature, length, and reproducible random sampling.
  • Samples improve visibly over a Markov/frequency baseline.
  • A diagnostic explains at least one vanishing/exploding-gradient experiment.

Previous: Project 48 · Complete learning path · Next: Project 50