Project 13: Naive Bayes Spam Filter

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Intermediate
  • Time: 1-2 weeks
  • Language: Python (alternatives: Go, Rust, JavaScript, C++)
  • Prerequisites: Project 12
  • Merged from: ML-Math P14; Prog P05; ML-Found P10

What You Will Build

Build a multinomial Naive Bayes email classifier from scratch, without a machine-learning library. Create a deterministic tokenizer, split labeled messages before fitting, count class and token frequencies, estimate priors and Laplace-smoothed likelihoods, and combine evidence in log space. For each prediction, report spam/ham scores, posterior-like normalized confidence, and the tokens contributing most strongly in each direction. Evaluate on held-out data with a confusion matrix, accuracy, precision, recall, F1, calibration bins, and examples of costly false positives.

Real World Outcome

Training prints class priors and interpretable token statistics. Entering a new message returns its label, score margin, and strongest evidence rather than a bare answer. An evaluation report compares a majority baseline with the model, displays confusion counts and threshold curves, inspects misclassifications, and saves the vocabulary/model so the same test message receives the same result after reload.

Core Question

How can Bayes’ rule combine many uncertain word clues, and why can a deliberately false conditional-independence assumption still produce a useful classifier?

Concepts You Must Understand First

  1. Conditional probability and Bayes’ rule: posterior is proportional to likelihood times prior.
  2. Naive conditional independence: tokens are assumed independent given class, simplifying a joint likelihood.
  3. Maximum-likelihood counts and Laplace smoothing: unseen words must not zero the entire message probability. See Géron, Hands-On Machine Learning, classification chapters.
  4. Log probabilities: products of small values become sums and avoid underflow. See Jurafsky and Martin, Speech and Language Processing, Chapter 4.
  5. Classification metrics and calibration: accuracy can hide false-positive/false-negative tradeoffs. See Manning et al., Introduction to Information Retrieval, Chapter 13.

Build Milestones

  1. Load a labeled corpus, normalize/tokenize deterministically, and create stratified train/test splits.
  2. Fit class priors and per-class token counts using training data only.
  3. Add smoothing, unknown-token behavior, log-score prediction, and model serialization.
  4. Produce confusion-matrix metrics, threshold analysis, calibration bins, and baseline comparison.
  5. Explain individual predictions and review representative errors for data or assumption failures.

Hints in Layers

  1. Start with tiny messages where you can calculate every count and posterior by hand.
  2. Compute one log score per class: log prior + sum(token_count * log likelihood).
  3. Normalize two log scores safely by subtracting their maximum before exponentiating.

Common Pitfalls and Debugging

  • Symptom: any unseen token forces probability zero. Cause: likelihoods were unsmoothed. Fix: apply consistent additive smoothing including vocabulary size.
  • Symptom: test accuracy is suspiciously perfect. Cause: vocabulary/counts were fitted before splitting. Fix: make the split first and audit all fitted state.
  • Symptom: long messages produce zero or NaN scores. Cause: raw probabilities underflow. Fix: accumulate log probabilities.

Definition of Done

  • Tokenization, split, and training are deterministic and leakage-free.
  • Priors, likelihoods, smoothing, and log scoring match hand-calculated fixtures.
  • Predictions include interpretable positive and negative token evidence.
  • Evaluation includes confusion metrics, threshold tradeoffs, calibration, and a baseline.
  • Saved and reloaded models produce identical predictions.
  • Documentation discusses independence assumptions and false-positive risk.

Previous: Project 12 · Complete learning path · Next: Project 14