Project 37: Information-Theory and Loss Engineering Lab

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Advanced
  • Time: 2-3 weeks
  • Language: Python with NumPy and Matplotlib
  • Prerequisites: Projects 12, 30, and 36
  • Source: ML-Math P23

What You Will Build

Build a workbench for entropy, cross-entropy, KL divergence, mutual information, coding length, and classification losses. Implement discrete calculations from normalized distributions with explicit zero-probability conventions and stable log operations. Let users compare true and predicted distributions, then visualize how confidence changes log loss, Brier score, gradients, and calibration. Simulate communication or compression to connect average surprise with expected code length. Add mutual-information experiments on independent, noisy-correlated, and deterministically related variables. Finally, compare loss landscapes for squared error, binary/categorical cross-entropy, and label smoothing, recording how each treats confident mistakes and class imbalance. The goal is not a formula catalog but an experimental system where identities and inequalities are testable.

Real World Outcome

The dashboard accepts probability vectors and produces entropy, cross-entropy, KL, per-outcome surprise, and a decomposition check H(p,q)=H(p)+KL(p||q). Interactive plots show loss and gradient versus predicted probability. A second panel estimates mutual information from samples and reveals estimator sensitivity to binning and sample size.

Core Question

Why does logarithmic surprise provide a natural measure of information, and how do information-theoretic quantities explain the behavior of ML loss functions?

Concepts You Must Understand First

  1. Self-information and entropy: rare events carry more surprise; entropy averages it. See Cover and Thomas, Elements of Information Theory.
  2. Cross-entropy and KL: cross-entropy measures coding under the wrong distribution; KL is the excess over true entropy.
  3. Zero conventions: 0 log 0 contributes zero, but assigning zero probability to a possible event creates infinite loss.
  4. Mutual information: dependence equals divergence between joint and product-of-marginals distributions.
  5. Proper scoring and calibration: log loss and Brier score reward honest probabilities in different ways.

Build Milestones

  1. Implement validated distributions, stable logs, entropy, cross-entropy, and KL with identity tests.
  2. Visualize surprise and coding-length intuition for controlled discrete sources.
  3. Implement joint/marginal tables and mutual-information experiments.
  4. Compare classification losses, gradients, confidence, label smoothing, and calibration.
  5. Add finite-sample estimator studies and a reproducible experiment report.

Hints in Layers

  1. Validate nonnegativity and normalization before calculating; report whether you renormalize.
  2. Use natural logs for nats or base-2 logs for bits, and label the unit everywhere.
  3. Estimate mutual information first from an exact joint table, then confront sampling and binning bias.

Common Pitfalls and Debugging

  • Symptom: KL is negative. Cause: distributions are unnormalized, axes are swapped, or rounding is severe. Fix: validate inputs and compare with the decomposition identity.
  • Symptom: entropy becomes NaN with zero entries. Cause: evaluating 0*log(0) literally. Fix: mask zero-mass terms using the limiting convention.
  • Symptom: sampled MI grows when bins increase. Cause: finite-sample estimator bias. Fix: sweep bins/sample size and disclose the estimator.

Definition of Done

  • Entropy, cross-entropy, and KL pass known distributions and identity checks.
  • Bits/nats and zero-probability policies are explicit.
  • Mutual information distinguishes independent and dependent examples.
  • Loss plots include values and gradients for confident correct/incorrect predictions.
  • Calibration and scoring-rule experiments are reproducible.
  • The report discusses finite-sample and discretization bias.

Previous: Project 36 · Complete learning path · Next: Project 38