Project 25: PCA Image Compressor

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Advanced
  • Time: 2-3 weeks
  • Language: Python with NumPy, Pillow, and Matplotlib
  • Prerequisites: Projects 12, 20, and 24
  • Merged from: ML-Math P07; ML-Found P04

What You Will Build

Build Principal Component Analysis from the centering step through image reconstruction. First prove the pipeline on a small tabular dataset: center features, construct the covariance matrix, find principal directions, sort them by eigenvalue, project, and reconstruct. Then treat image rows, patches, or a collection of similarly sized images as observations and compress them by retaining k components. Provide controls for component count or target explained variance, and report reconstruction error and a realistic storage estimate. Implement power iteration plus deflation or a symmetric eigensolver exercise, then compare with a trusted SVD-based reference. The final artifact should teach what information is preserved, not merely call a library PCA class.

Real World Outcome

Given an image and a target such as 95% retained variance, the tool displays the original, reconstructions at several k values, a scree/cumulative-variance plot, mean-squared error, and approximate compression ratio. A second demo projects a labeled dataset into two dimensions and reveals which structure survives.

Core Question

How can a change of basis preserve most variation with fewer coordinates, and what exactly is lost when low-variance directions are discarded?

Concepts You Must Understand First

  1. Centering: PCA describes variation around the mean; failing to center changes the first direction.
  2. Covariance matrix: it encodes pairwise feature variation and is symmetric positive semidefinite. See Deisenroth et al., Mathematics for Machine Learning, ch. 6.
  3. Eigenvectors as axes: covariance eigenvectors are orthogonal principal directions; eigenvalues measure captured variance.
  4. Projection/reconstruction: scores are coordinates in the new basis; multiplying back and restoring the mean approximates the input.
  5. PCA versus SVD: SVD is usually the numerically safer route. See Géron, Hands-On Machine Learning, dimensionality-reduction chapter.

Build Milestones

  1. Implement centering, covariance, sorted eigenpairs, projection, and reconstruction on 2D synthetic data.
  2. Add explained-variance ratios and verify orthogonality and total-variance identities.
  3. Implement power iteration/deflation for learning value and compare with eigh or SVD.
  4. Compress grayscale images, then support color by a documented channel or patch strategy.
  5. Build interactive k/variance controls and export a quantitative quality report.

Hints in Layers

  1. Preserve the training mean and reuse it during projection and reconstruction.
  2. For covariance, define whether observations are rows and use a consistent (n-1) denominator.
  3. Clip tiny negative eigenvalues caused by rounding only after recording them; large negatives indicate a bug.

Common Pitfalls and Debugging

  • Symptom: first component mostly reproduces image brightness. Cause: data was not centered. Fix: subtract and later restore the mean.
  • Symptom: reconstruction dimensions are transposed. Cause: observations/features conventions changed. Fix: annotate every matrix shape and test a tiny fixture.
  • Symptom: “compression ratio” is misleading. Cause: basis storage was ignored. Fix: count retained scores, components, mean, and numeric precision.

Definition of Done

  • PCA is implemented without a high-level PCA API.
  • Principal directions are ordered, orthonormal, and numerically verified.
  • Explained-variance ratios sum to approximately one.
  • Increasing k never materially worsens reconstruction error.
  • The UI compares quality, retained variance, and honest storage cost.
  • A reference SVD/eigendecomposition confirms results within tolerance.

Previous: Project 24 · Complete learning path · Next: Project 26