Sprint: Math from Foundations to Machine Learning Mastery - Real World Projects

Goal: Build mathematical intuition by turning ideas into working software whose answers can be inspected, tested, and explained. This guide starts with arithmetic and high-school mathematics, then moves through calculus, probability, linear algebra, classical machine learning, neural networks, and research-level ML mathematics. Every one of the 61 projects includes its actual scope, observable outcome, milestones, hints, failure modes, and completion checks, so the five older guides are provenance rather than required reading.

Introduction

This is one self-contained, prerequisite-ordered curriculum. The original five guides contained 93 named projects and capstones; projects that taught substantially the same mathematics through the same kind of artifact were merged, while projects with different learning angles stayed separate. A numerical derivative explorer and a symbolic differentiator remain distinct, for example, because approximation and symbolic rules produce different understanding.

You will build calculators, visualizers, simulators, statistical tools, machine-learning models, neural-network infrastructure, and finally a small GPT-style transformer. The point is not merely to obtain a numeric answer. Each project makes the reasoning observable through transcripts, plots, residuals, invariants, reference comparisons, or reproducible experiments.

Arithmetic and logic
        |
        v
Algebra and functions ---> discrete mathematics
        |                         |
        v                         v
Geometry and vectors ---> matrices and graphs
        |                         |
        +------------+------------+
                     v
          calculus and probability
                     |
                     v
       optimization and classical ML
                     |
                     v
       neural networks from first principles
                     |
                     v
 analysis, kernels, graphs, random matrices, GPT

In scope: the mathematics, implementation decisions, validation strategy, and conceptual checkpoints needed to complete every project. Out of scope: production-ready security, large-scale distributed training, and full runnable solutions. The guide gives layered hints and pseudocode-level direction without removing the learning challenge.

How to Use This Guide

Follow the numbered order unless a project explicitly lists only prerequisites you have already demonstrated.
Before coding, answer the project’s core question and complete its short thinking work mentally or on paper.
Build the smallest milestone that creates observable output, then add correctness checks before features.
Treat the Definition of Done as the completion gate. A screenshot alone is not proof; retain test cases, seeds, tolerances, plots, and a short explanation of what failed during development.
Use Python first for the default path. Reimplement selected projects in another language only after the mathematics is correct and measured.

The Source and Merged from fields record provenance. They are not prerequisites and you do not need to open the old guides to finish a project.

Prerequisites & Background Knowledge

Essential prerequisites

Basic Python: variables, functions, loops, collections, modules, files, and exceptions.
Ability to run commands in a terminal and create an isolated virtual environment.
Willingness to write tests and compare output with hand-computed examples.

Helpful, but taught during the path

NumPy arrays and plotting with Matplotlib.
Basic data handling with CSV files and pandas.
Familiarity with Git and notebooks.

Self-assessment

Can you explain why 0.1 + 0.2 may not equal the decimal value 0.3 exactly in binary floating point?
Can you write a function, call it from a test, and interpret a failed assertion?
Can you distinguish a mathematical assumption from an implementation detail?
Can you preserve a fixed random seed and rerun an experiment?

Recommended environment

Python 3.12 or newer.
pytest, NumPy, Matplotlib, pandas, and scikit-learn for reference comparisons.
Jupyter or a plain CLI; every project should still have a deterministic testable core outside the notebook UI.

Time investment

Foundations: 4-10 hours per project.
Intermediate numerical and ML projects: 10-20 hours.
Neural-network and advanced projects: 20-40 hours.
Full path: roughly 12-24 months at a sustainable part-time pace.

Big Picture / Mental Model

Every project uses the same learning loop:

mathematical claim
       |
       v
representation contract ---> domain and shape checks
       |                              |
       v                              v
algorithm or simulation ------> observable artifact
       |                              |
       +----------> validation <------+
                         |
                  explanation in words

The representation changes—expression tree, vector, matrix, probability distribution, computational graph—but the discipline does not. State the domain, preserve invariants, make error visible, and compare against a case whose answer you understand.

Theory Primer

This primer is intentionally compact. Each project repeats the concepts it requires and turns them into concrete design and verification work.

Numbers, algebra, and functions

Arithmetic software must distinguish a mathematical value from its finite machine representation. Operator precedence, domains, equality-preserving transformations, and function behavior become explicit contracts. Algebraic solving preserves a solution set while reshaping an equation; function analysis studies how inputs map to outputs, where that map is undefined, and how transformations change its graph. Floating-point results require tolerances and residual checks rather than blind equality. These ideas power Projects 1-7 and reappear in every numerical project.

Logic, discrete structures, and exact algorithms

Logic turns conditions into propositions that can be evaluated. Sets describe allowed values, constraints rule out invalid states, graphs encode relationships, and modular arithmetic creates finite algebraic systems. Search algorithms are correct because they maintain invariants about explored and unexplored states—not because they happen to find an answer. Projects 8-10 make this visible through Sudoku explanations, traversal frontiers, shortest paths, and RSA round trips.

Probability and statistics

Probability models uncertain outcomes; statistics connects those models to observed samples. A random variable maps outcomes to numbers, a distribution assigns probability, and an estimator summarizes or infers something from finite data. Simulation approximates expectations, but reproducibility requires explicit generators and seeds. Inference requires assumptions, uncertainty intervals, and an effect-size decision—not only a p-value. Projects 11-14 establish the foundation used by regression, classification, evaluation, generalization, and random-matrix experiments.

Geometry and linear algebra

Vectors encode magnitude, direction, features, or state. Matrices encode linear maps, systems of equations, transformations, covariance, and graph structure. Coordinates change, but the underlying geometric object should remain coherent. Rank and conditioning reveal whether a result is unique or numerically trustworthy; eigenvectors reveal directions preserved by a transformation. Projects 15-25 move from visible geometry to matrix computation, eigendecomposition, and PCA.

Calculus and numerical computation

Derivatives measure local change and integrals measure accumulation. Symbolic methods manipulate rules exactly within an expression language; numerical methods approximate using finite resolution. Approximation error, truncation error, floating-point error, and conditioning are different phenomena and should be measured separately. Projects 26-32 make these distinctions visible through tangent estimates, symbolic checks, root-finding traces, integration convergence, stability stress tests, dynamics, and epidemic models.

Optimization and classical machine learning

Machine learning chooses parameters that reduce an objective on data while preserving performance on unseen cases. Optimization describes how parameters move; statistics describes uncertainty and evaluation; information theory explains several common losses and split criteria. Data leakage, poor scaling, class imbalance, and metric choice can invalidate an otherwise correct algorithm. Projects 33-41 build the algorithms and then combine them in a reproducible pipeline.

Neural networks and automatic differentiation

A neural network composes affine transformations and nonlinearities. Backpropagation is repeated application of the chain rule through a computational graph; autograd records that graph and accumulates derivatives in reverse order. Shape contracts, initialization, saturation, gradient checks, and reproducible training loops matter as much as the layer equations. Projects 42-50 progress from one neuron to an MLP, diagnostic visualizer, CNN, RNN, and reusable framework.

Advanced mathematical ML

Convexity supplies global optimality conditions; real analysis supplies rigorous limits and convergence; measure theory supplies the language of modern probability; functional analysis explains kernels and infinite-dimensional feature spaces. Graph spectra reveal global structure from local connectivity, while random-matrix experiments distinguish high-dimensional signal from noise. Projects 51-60 form an advanced assurance track. Project 61 integrates the earlier representation, optimization, probability, linear algebra, autograd, and systems work in a small transformer.

Glossary

Domain: The set of inputs for which an expression or function is defined.
Invariant: A property that must remain true throughout an algorithm or transformation.
Residual: The difference left when a proposed answer is substituted back into the original problem.
Conditioning: How sensitive a problem’s answer is to small input changes.
Convergence: Movement of an approximation or sequence toward a limiting value.
Estimator: A rule that maps observed data to an estimated quantity.
Loss function: A scalar measure of how poorly parameters explain the target data.
Gradient: The vector of partial derivatives pointing toward fastest local increase.
Generalization: Performance on data not used to fit the model.
Computational graph: A graph of operations and values used to evaluate a function and its derivatives.
Eigenvector: A nonzero vector whose direction is preserved by a linear transformation.
Reproducibility: The ability to regenerate a result from recorded code, inputs, configuration, and randomness.

Why Mathematical Software Matters

Mathematical software turns invisible reasoning into something inspectable. A plot can expose a discontinuity, a residual can reject a false root, a gradient check can locate a broken derivative, and a held-out metric can expose data leakage. The path also teaches how modern tools should be used: Python documents why binary floating point is approximate, NumPy recommends explicit random generators for reproducible simulation, scikit-learn separates model selection from final evaluation, and PyTorch describes autograd as a dynamically recorded operation graph. You will recreate the core ideas before using those libraries as trusted references.

Concept Summary Table

Concept cluster	What you must internalize	Projects
Numbers, algebra, functions	Domains, invariants, representations, residuals	1-7
Discrete mathematics	Logic, constraints, graphs, modular arithmetic	8-10
Probability and statistics	Sampling, distributions, Bayes, inference	11-14
Geometry and linear algebra	Vectors, matrices, transformations, spectra	15-25
Calculus and numerics	Change, accumulation, approximation, stability	26-32
Classical ML	Optimization, regression, classification, evaluation	33-41
Neural networks	Chain rule, autograd, architectures, training systems	42-50
Advanced ML mathematics	Analysis, measure, kernels, graph spectra, transformers	51-61

Project-to-Concept Map

Project range	Primary concepts	Observable evidence
1-10	Algebra and discrete mathematics	Transcripts, proof tables, search traces, round trips
11-14	Probability and inference	Seeded simulations, distributions, calibrated decisions
15-25	Geometry and linear algebra	Interactive plots, transformations, reconstructions
26-32	Calculus and numerical methods	Error curves, convergence traces, stable simulations
33-41	Classical machine learning	Loss histories, held-out metrics, decision reports
42-50	Neural networks	Gradient checks, training curves, generated samples
51-60	Advanced ML mathematics	Counterexamples, certificates, bounds, spectra
61	Transformer integration	Reproducible training and autoregressive generation

Deep Dive Reading by Concept

Concept	Suggested reading	Use it for
Algebra and functions	Precalculus by OpenStax, functions and algebra chapters	Projects 1-7, 15-19
Linear algebra	Introduction to Linear Algebra by Gilbert Strang, Chapters 1-6	Projects 16, 20-25, 42-61
Calculus	Calculus by James Stewart, limits through integration	Projects 26-33
Probability	Introduction to Probability by Blitzstein and Hwang, Chapters 1-8	Projects 11-14, 40, 54-55
Statistical learning	An Introduction to Statistical Learning, Chapters 2-8	Projects 34-41
Deep learning	Deep Learning by Goodfellow, Bengio, and Courville, Chapters 6-10	Projects 42-50, 61
Numerical methods	Numerical Analysis by Burden and Faires, roots, differentiation, integration	Projects 28-31
Analysis	Understanding Analysis by Stephen Abbott, sequences and continuity	Projects 52-55

Quick Start: Your First 48 Hours

Day 1

Install the recommended Python environment and run one pytest smoke test.
Read the Numbers, Algebra, and Functions primer.
Build Project 1’s tokenizer and display one deterministic parse trace.

Day 2

Add domain errors and floating-point tolerance checks.
Complete Project 1’s reference cases and Definition of Done.
Write a short note explaining one failure and why the final invariant catches it.

Recommended Learning Paths

Complete first-principles path: Projects 1-61 in order.
Experienced programmer refreshing mathematics: validate Projects 1-10 quickly, then complete 11-41 and 42-50 in order.
Machine-learning practitioner: Projects 5-7, 11-14, 16, 20-30, then 33-61.
Graphics and simulation path: Projects 5-6, 15-21, 24-31.
Rigorous mathematics path: Projects 1-30, then 51-60 before returning to neural systems.

Success Metrics

Every completed project has deterministic reference tests or a documented statistical tolerance.
You can explain each algorithm’s invariant, assumption, and primary failure mode without opening the implementation.
Plots and metrics are generated from recorded inputs and seeds rather than hand-edited screenshots.
Numerical methods include an error or convergence study, not only a final answer.
ML projects keep training, validation, and final test decisions separate.
Neural-network gradients are checked numerically before model accuracy is trusted.

Source Guides and Deduplication

Code	Original guide	Source entries
`HS`	High School Math with Python	22
`ML-Math`	Math for Machine Learning Projects	32
`Prog`	Learn Math for Programming	7
`NN`	AI Prediction and Neural Networks Deep Dive	11
`ML-Found`	Machine Learning Foundations Projects	21
Total	Five guides	93

The 93 source entries become 61 canonical projects. The source codes remain in each project only to make the merge auditable.

Project Overview Table

Phase	Projects	Focus
1	1-7	Numeracy, algebra, functions, sequences, and counting
2	8-10	Logic applications, graph theory, and number theory
3	11-14	Probability, distributions, Bayesian inference, and experimentation
4	15-25	Geometry, vectors, matrices, simulation, and spectral foundations
5	26-32	Calculus and reliable numerical computation
6	33-41	Optimization, statistics, and classical machine learning
7	42-50	Backpropagation and neural-network systems
8	51-61	Advanced ML mathematics and final capstones

Project List

The projects below contain the actual learning and build specification. Nothing in the older guides is required to understand or complete them.

Phase 1 — Mathematical Foundations (Projects 1-7)

Project 1: Scientific Calculator and Numeric Foundations

Difficulty: Beginner
Time: 8-12 hours
Language: Python (alternatives: C, JavaScript, Rust)
Prerequisites: Basic Python syntax
Source: ML-Math P01

What You Will Build

Build a command-line scientific calculator that accepts expressions such as (3 + 4) * 2^3, sqrt(16) + log(exp(5)), and sin(pi / 2). Instead of delegating expression evaluation to eval, create a tokenizer, a precedence-aware parser, and an evaluator with explicit domain checks. The calculator should distinguish unary minus from subtraction, support right-associative exponentiation, report syntax errors at a useful position, and print results with a documented floating-point tolerance. Include a verification suite that compares ordinary operations with hand-computed cases and selected transcendental results with Python’s math module.

Real World Outcome

Running the tool produces a stable transcript: valid expressions return a number and evaluation trace; malformed expressions return a deterministic syntax diagnostic; sqrt(-1) and log(0) return domain errors rather than crashes. A test report demonstrates correct precedence, nested parentheses, functions, constants, scientific notation, and near-equality behavior such as 0.1 + 0.2 versus 0.3.

Core Question

How do you turn human mathematical notation, with its implicit ordering and domain rules, into an unambiguous sequence of computer operations?

Concepts You Must Understand First

Operator precedence and associativity: why 2^3^2 groups differently from subtraction. See King, C Programming: A Modern Approach, Chapters 4 and 7.
Tokens, stacks, and expression trees: represent structure before computing values. See Aho et al., Compilers, Chapter 2.
Shunting-yard or recursive-descent parsing: convert infix notation into a form that can be evaluated safely. See Sedgewick and Wayne, Algorithms, §4.3.
Function domains and inverse relationships: log, exp, roots, and trigonometric functions. See Stewart, Calculus: Early Transcendentals, Chapter 1.
Floating-point representation: numerical equality needs tolerances, not string or bit equality. See Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, §2.4.

Build Milestones

Tokenize integers, decimals, parentheses, constants, and + - * / ^, rejecting unknown characters.
Parse arithmetic with correct precedence, associativity, nested parentheses, and unary signs.
Add sqrt, sin, cos, log, and exp with arity and domain validation.
Add deterministic formatting, error categories, and a trace or postfix/AST inspection mode.
Test normal, boundary, malformed, and numerically awkward expressions against trusted results.

Hints in Layers

Start by converting infix tokens to postfix with an operator stack; postfix evaluation needs only a value stack.
Store precedence, associativity, and arity as data rather than spreading special cases through the parser.
Separate the unrounded computed value from display formatting, and centralize approximate comparison in one helper.

Common Pitfalls and Debugging

Symptom: -2^2 returns the wrong convention. Cause: unary minus was assigned an accidental precedence. Fix: specify and test the grammar explicitly, including (-2)^2.
Symptom: nested functions leave extra values on the stack. Cause: argument boundaries or function arity are not represented. Fix: validate the final stack size and trace tokens one at a time.
Symptom: tests around decimals fail intermittently. Cause: exact floating-point equality. Fix: compare absolute/relative error with a justified tolerance.

Definition of Done

Arithmetic precedence, associativity, unary signs, and nested parentheses are correct.
At least five scientific functions enforce their domains and argument counts.
Invalid syntax produces stable, specific errors without executing arbitrary code.
Tests cover hand-computed examples, edge cases, and trusted-library comparisons.
The README documents grammar, numeric tolerance, limitations, and one failure case.

Project 2: Logic, Sets, and Proof Lab

Difficulty: Intermediate
Time: 8-16 hours
Language: Python (alternatives: JavaScript, Julia, Haskell)
Prerequisites: Project 1
Source: HS P12

What You Will Build

Build a CLI reasoning lab with three connected modes. truth parses propositions containing not, and, or, and implication, generates every assignment, and classifies the formula as a tautology, contradiction, or contingency. quantify evaluates bounded forall and exists statements over explicit finite sets, preserving quantifier order and reporting witnesses or the smallest counterexample. proof-check reads a structured proof note and checks that assumptions, proof method, legal step annotations, and conclusion are present. It does not pretend to prove arbitrary mathematics; it makes logical structure and missing obligations observable.

Real World Outcome

The command (p and q) -> p generates a four-row CSV truth table and a tautology verdict. A nested finite-domain statement prints the witness chosen for each existential obligation or the first assignment that refutes it. An induction proof note missing its base case fails with a focused diagnostic, while a structurally complete proof produces a checklist suitable for review.

Core Question

What distinguishes a convincing example from a valid argument, and which parts of logical validity can a finite program check deterministically?

Concepts You Must Understand First

Sets and membership: union, intersection, difference, Cartesian products, and finite domains. See Orland, Math for Programmers, logic and set chapters.
Propositional logic: truth values, connective precedence, implication, equivalence, and inference rules.
Quantifiers: forall versus exists, variable binding, and why swapping quantifier order changes meaning.
Proof methods: direct proof, contraposition, contradiction, counterexample, and mathematical induction. See Graham, Knuth, and Patashnik, Concrete Mathematics, Chapter 1.
Sound tool scope: structural validation can reveal omissions but cannot certify unrestricted natural-language reasoning.

Build Milestones

Parse proposition expressions into an AST and generate deterministic truth tables for two to four variables.
Add equivalence, tautology, contradiction, and contingency classification.
Parse finite set declarations and nested quantifiers, returning explicit witness/counterexample traces.
Define a small proof-note schema for assumptions, method, steps, base/induction cases, and conclusion.
Create valid and deliberately broken examples for each reasoning mode.

Hints in Layers

Reuse the parsing discipline from Project 1, but make every AST node return a Boolean.
Evaluate quantifiers recursively: universal nodes short-circuit on a counterexample; existential nodes short-circuit on a witness.
Label proof-check results as structural diagnostics, never as proof of semantic truth.

Common Pitfalls and Debugging

Symptom: p -> q behaves like p and q. Cause: implication was encoded incorrectly. Fix: verify it against not p or q for all four assignments.
Symptom: forall x exists y and exists y forall x give identical traces. Cause: quantifiers were flattened. Fix: preserve nesting in the AST and bind variables per recursive frame.
Symptom: the checker approves an invalid argument. Cause: “all sections present” was reported as semantic validity. Fix: separate completeness warnings from verified finite truth claims.

Definition of Done

Truth tables and formula classifications match hand-worked cases.
Bounded nested quantifiers preserve order and report concrete witnesses/counterexamples.
Set operations are deterministic and covered by tests.
Proof notes receive method-specific structural checks, including induction base cases.
Output clearly states what the tool has and has not proved.

Project 3: Algebraic Equation Solver

Difficulty: Beginner
Time: 6-10 hours
Language: Python (alternatives: JavaScript, C++)
Prerequisites: Projects 1-2
Source: HS P01

What You Will Build

Build a deterministic CLI solver for one-variable linear and quadratic equations. Parse both sides, normalize them to ax^2 + bx + c = 0, combine like terms, classify the equation, and emit a step-by-step transcript in which every transformation preserves the solution set. Handle unique linear solutions, two real roots, a repeated root, no real roots, identities with infinitely many solutions, and contradictions with no solutions. Verify every numeric root by substituting the unrounded value into the original equation and reporting the residual.

Real World Outcome

Given 3x - 7 = 2x + 5, the program explains subtracting the right side, combining terms, and obtaining x = 12, then prints a zero residual. Given x^2 - 5x + 6 = 0, it reports discriminant 1, sorted roots 2 and 3, and two substitution checks. Unsupported expressions, division-by-zero transformations, and malformed input return stable error classes and exit codes.

Core Question

Why does applying the same legal operation to both sides preserve an equation’s truth, and how can software make that invariant auditable?

Concepts You Must Understand First

Equality-preserving transformations: adding, subtracting, multiplying, or dividing both sides, with nonzero constraints.
Terms, coefficients, degree, and canonical form: normalize diverse syntax into a small coefficient representation.
Linear equation classification: bx + c = 0, including b = 0 identity and contradiction cases.
Quadratic formula and discriminant: root count follows from D = b^2 - 4ac. See any high-school algebra text, quadratic equations chapter.
Residual validation: computation and display rounding must remain separate.

Build Milestones

Parse signed constant, x, and x^2 terms on both sides, including implicit coefficients such as -x.
Normalize to coefficients (a, b, c) and log explicit equality-preserving steps.
Implement complete linear classification, then quadratic discriminant branching and sorted roots.
Substitute solutions into the original parsed expressions and enforce a documented tolerance.
Add transcript/golden tests for every solution class and invalid-input path.

Hints in Layers

Represent a polynomial as a map from degree to coefficient; moving a term means subtracting its coefficient.
Dispatch after normalization: quadratic when a != 0, linear when a == 0 and b != 0, otherwise classify c.
Generate explanations from the same transformation objects that modify coefficients so text cannot drift from computation.

Common Pitfalls and Debugging

Symptom: 0x = 0 causes division by zero. Cause: the solver assumed every normalized equation has a unique solution. Fix: classify zero coefficients before dividing.
Symptom: the printed steps yield a different equation from the solver. Cause: explanation strings were handwritten separately. Fix: store operation, operand, before, and after states together.
Symptom: a displayed rounded root fails substitution. Cause: the rounded display value was reused internally. Fix: retain full precision for validation.

Definition of Done

Linear and quadratic inputs normalize to a documented canonical form.
All real solution-set classifications are implemented explicitly.
Every algebra step preserves equality and can be replayed from the transcript.
Every reported root passes substitution within tolerance.
Tests include malformed syntax, reducible quadratics, identities, and contradictions.

Project 4: Inequality and Absolute-Value Region Analyzer

Difficulty: Intermediate
Time: 6-12 hours
Language: Python (alternatives: R, Julia, JavaScript)
Prerequisites: Project 3
Source: HS P13

What You Will Build

Build a solver and visualizer for one-dimensional linear, compound, and absolute-value inequalities plus two-dimensional systems of linear constraints. The one-dimensional engine should emit interval notation, preserve open versus closed endpoints, reverse the comparison when multiplying or dividing by a negative value, and support intersection and union. Translate |x-a| < r into a distance interval and |x-a| > r into two rays. The 2D mode should sample or clip a plotting window, shade the intersection of half-planes, and verify candidate points against every original constraint.

Real World Outcome

For -3 <= 2x + 1 < 7, the tool reports [-2, 3), renders the correct closed/open endpoints, and tests boundary membership. For constraints such as x + y <= 6, x >= 0, and y >= 0, it creates a labeled feasible-region plot and exports representative inside, boundary, and outside points with pass/fail details for each inequality.

Core Question

How does solving an inequality describe a set of valid points rather than a single answer, and how do algebraic boundary rules become visible geometry?

Concepts You Must Understand First

Order properties: adding preserves order; multiplying by a negative reverses it.
Intervals as sets: open, closed, half-open, bounded, and unbounded intervals; unions and intersections.
Absolute value as distance: |x-a| measures distance from a, producing inside or outside regions.
Linear inequalities in two variables: each line divides the plane into two half-planes.
Boundary semantics: strict constraints exclude the line; inclusive constraints include it.

Build Milestones

Parse and normalize one-variable linear inequalities, including negative coefficients and compound chains.
Represent solution sets as interval objects supporting union, intersection, membership, and stable notation.
Add absolute-value patterns for <, <=, >, and >=, including impossible/whole-line cases.
Render number lines and 2D feasible regions with distinct strict/inclusive boundaries.
Validate results using generated point-membership tests against the original expressions.

Hints in Layers

Keep endpoints and inclusion flags as structured data; do not infer them back from formatted strings.
For 2D constraints, test a known point such as the origin to select which side of the boundary is valid.
Use the original constraint evaluator as an oracle for plotted sample points and boundary checks.

Common Pitfalls and Debugging

Symptom: solving -2x < 6 gives x < -3. Cause: the sign was not reversed on division by -2. Fix: centralize inequality transformations and test negative multipliers.
Symptom: the report says (1, 4] but the plot fills both endpoints. Cause: inclusion flags were lost during rendering. Fix: pass boundary style directly from the interval object.
Symptom: shading shows the union of constraints. Cause: masks were combined with OR. Fix: a feasible system requires every constraint, so combine with AND.

Definition of Done

Linear, compound, and absolute-value inequalities produce correct interval sets.
Negative-coefficient transformations reverse order correctly.
Number-line plots preserve open/closed and infinite endpoints.
Multiple 2D constraints produce their intersection with labeled boundaries.
Boundary and random membership tests agree with original constraints.

Project 5: Function Grapher and Analysis Workbench

Difficulty: Intermediate-Advanced
Time: 16-28 hours
Language: Python (alternatives: JavaScript, Julia, R)
Prerequisites: Projects 3-4
Merged from: HS P02, HS P19; ML-Math P02

What You Will Build

Build a safe multi-function graphing and analysis workbench. Parse expressions without arbitrary execution, establish a requested plotting window, determine valid sample points, and draw functions with axes and labels. Then add structural diagnostics: approximate roots and intercepts, increasing/decreasing sample intervals, piecewise boundaries, composition, candidate inverse windows, asymptotic trends, singularities, holes, and jump discontinuities. Parameter sweeps should show how transformations such as a*f(b(x-h))+k affect shape. Every feature report must state whether it is exact from structure or estimated from sampling.

Real World Outcome

A command can compare 1/x, (x^2-1)/(x-1), and a piecewise function on the same fixed grid. The output saves a plot without drawing false lines across singularities and prints domain exclusions, approximate zeros, suspicious discontinuities, and confidence/step-size notes. A parameter animation or set of plots demonstrates horizontal/vertical scale, reflection, and translation with reproducible colors and bounds.

Core Question

What can a graph reveal about a function’s structure, and which apparent features are artifacts of finite sampling rather than mathematical facts?

Concepts You Must Understand First

Function, domain, and range: a rule plus permitted inputs, not merely a formula.
Composition and inverses: inverses require one-to-one behavior on the chosen domain.
Transformations and piecewise definitions: shifts, scales, reflections, and boundary rules. See Orland, Math for Programmers.
Continuity and asymptotes: holes, jumps, vertical singularities, and end behavior.
Sampling and numerical feature detection: a sign change suggests a root but can miss tangencies or cross a pole.

Build Milestones

Reuse or adapt the safe expression parser, evaluate over a deterministic grid, and mask domain errors.
Plot multiple functions while splitting line segments around invalid values or implausibly large jumps.
Detect approximate intercepts, sign changes, local direction changes, and suspicious discontinuities.
Add piecewise definitions, composition, domain diagnostics, and interval-based inverse checks.
Implement parameter sweeps and generate a structured analysis report with accuracy caveats.

Hints in Layers

Return (value, status) for every sample so an invalid domain is data, not an exception that ends the run.
Detect features with brackets first, then refine them; never call a coarse-grid estimate exact.
Keep symbolic domain rules for common functions (sqrt, denominator, log) alongside numerical probes.

Common Pitfalls and Debugging

Symptom: a vertical line appears across x=0 for 1/x. Cause: the plot connected samples on opposite sides of a pole. Fix: break segments at invalid points and large jumps.
Symptom: (x-1)^2 has no reported root. Cause: sign-change detection misses tangent roots. Fix: also search local minima of |f(x)| and label them candidates.
Symptom: an inverse is claimed for x^2 over all reals. Cause: global one-to-one behavior was not checked. Fix: require a monotonic restricted interval.

Definition of Done

Expressions are evaluated safely on a deterministic grid with domain failures isolated.
Plots avoid connecting across known or detected discontinuities.
Reports cover roots/intercepts, transformations, domain, piecewise boundaries, and asymptotic clues.
Composition and restricted inverse behavior have tested examples.
Every numerical inference includes resolution or tolerance limitations.
Parameter sweeps are reproducible and visibly demonstrate transformations.

Project 6: Sequences, Series, and Recurrence Lab

Difficulty: Intermediate
Time: 8-16 hours
Language: Python (alternatives: Julia, R, JavaScript)
Prerequisites: Project 5
Source: HS P14

What You Will Build

Build a sequence engine that accepts arithmetic and geometric parameters, explicit formulas, and a constrained recurrence definition. Produce term tables, partial-sum tables, and plots of values and errors. For arithmetic and geometric families, compare iterative generation and finite sums with their closed forms. For recurrences, record state transitions and explore fixed-point behavior, oscillation, growth, and sensitivity to the starting value. Add convergence diagnostics that report finite evidence and a tolerance window without falsely claiming a proof from a limited number of terms.

Real World Outcome

The tool can generate the first 100 terms of an arithmetic progression, verify its partial-sum formula, show a geometric series approaching its finite limit when |r| < 1, and plot a recurrence such as x[n+1] = cos(x[n]) approaching a fixed point. A report distinguishes convergent-looking, divergent, oscillatory, and inconclusive runs and includes the parameters and stopping rule needed to reproduce them.

Core Question

How do local rules for producing the next value create global patterns, and what can finite computation legitimately say about infinite behavior?

Concepts You Must Understand First

Sequence notation: index, term, explicit formula, and recurrence relation.
Arithmetic and geometric progressions: common difference/ratio and finite sum formulas.
Series and partial sums: a sequence of accumulated totals is distinct from the terms being summed.
Limits and convergence intuition: approaching a value differs from reaching it; oscillation can remain bounded.
Fixed points: a recurrence approaches L only if its update is compatible with L = f(L) and locally stable.

Build Milestones

Implement structured arithmetic/geometric generators and compare terms with explicit formulas.
Compute finite partial sums and compare them against trusted closed-form results.
Add safe recurrence definitions, initial state, trace output, and termination limits.
Plot terms, partial sums, and successive differences on suitable scales.
Add convergence heuristics, parameter sweeps, and adversarial examples that fool simplistic rules.

Hints in Layers

Keep terms[n] and partial_sums[n] separate; many conceptual bugs come from mixing them.
A useful heuristic requires several consecutive small changes, not one lucky near-equality.
Compare different initial values and plot |x[n]-L| on a log scale when a reference limit is known.

Common Pitfalls and Debugging

Symptom: a geometric sum is wrong by one term. Cause: inconsistent zero-based versus one-based indexing. Fix: document the first term and test n=1 before larger cases.
Symptom: 1, 0, 1, 0... is declared convergent. Cause: boundedness was confused with convergence. Fix: inspect a trailing window and detect periodic behavior.
Symptom: recurrence evaluation never stops. Cause: no iteration budget or non-finite check. Fix: require maximum steps and stop on overflow/NaN.

Definition of Done

Arithmetic and geometric terms and finite sums match hand calculations.
Recurrences produce reproducible state traces with explicit initial conditions.
Terms, partial sums, and error/difference trends are plotted separately.
Convergence output is labeled heuristic and includes tolerance/window settings.
Tests include convergent, divergent, oscillating, and inconclusive examples.

Project 7: Counting, Combinatorics, and Modeling Studio

Difficulty: Advanced
Time: 14-28 hours
Language: Python (alternatives: R, Julia, JavaScript)
Prerequisites: Projects 2 and 6
Source: HS P20

What You Will Build

Build a scenario-driven modeling studio that turns a small real-world question into explicit variables, assumptions, units, counting rules, uncertainty ranges, and a decision-ready report. Support sum/product rules, factorials, permutations, combinations, and probability derived from favorable versus possible outcomes. Scenarios can be loaded from JSON/YAML or CLI parameters—for example staffing schedules, password spaces, tournament outcomes, inventory bundles, or sampling plans. The engine should reject unit-inconsistent equations, compare alternative assumptions, and produce sensitivity tables showing which inputs most affect the result.

Real World Outcome

Given a scenario for selecting a five-person committee with role constraints, the tool prints the valid counting formula, evaluates it, verifies small cases by enumeration, and explains whether order matters. Given an operational estimate, it checks dimensions, runs low/base/high assumptions, ranks influential inputs, and exports a Markdown report containing result, uncertainty, limitations, and a warning when independence or uniformity was assumed without evidence.

Core Question

How do you translate an ambiguous situation into a mathematical model whose assumptions, units, and counting choices can be inspected and challenged?

Concepts You Must Understand First

Sum and product rules: mutually exclusive alternatives add; sequential independent choices multiply.
Permutations versus combinations: decide whether order matters and whether repetition is allowed. See Graham et al., Concrete Mathematics, counting chapters.
Sample spaces and probability: specify what counts as an outcome before declaring outcomes equally likely.
Dimensional analysis: valid addition requires compatible units; multiplication/division creates derived units.
Sensitivity analysis: vary one or several assumptions to expose fragile conclusions.

Build Milestones

Define a scenario schema with quantities, units, assumptions, constraints, and requested outputs.
Implement counting primitives and a trace that explains why each rule applies.
Add brute-force enumeration for small instances as a verification oracle.
Build a unit checker and low/base/high plus one-at-a-time sensitivity analysis.
Export a report that separates facts, assumptions, computed results, and limitations.

Hints in Layers

Ask four questions before choosing a formula: order, repetition, distinct objects, and constraints.
Represent units symbolically as exponent maps so km / hour survives calculations.
Keep the pure model function separate from scenario iteration; then sensitivity analysis is repeated evaluation, not duplicated logic.

Common Pitfalls and Debugging

Symptom: committee counts are too large by a factorial factor. Cause: permutations were used where order is irrelevant. Fix: verify a tiny case by listing outcomes.
Symptom: an equation adds hours to people. Cause: units were stripped before computation. Fix: carry unit metadata through every operation.
Symptom: sensitivity results never change. Cause: scenarios mutated display values but not model inputs. Fix: log the actual parameter map passed to each run.

Definition of Done

Counting primitives handle order/repetition cases and explain their selection.
Small cases agree with explicit enumeration.
Scenario inputs retain units and inconsistent operations fail clearly.
Low/base/high and sensitivity runs are reproducible.
Reports distinguish observations, assumptions, results, uncertainty, and limitations.
At least three meaningfully different scenarios exercise the engine.

Phase 2 — Discrete Mathematics and Cryptography (Projects 8-10)

Project 8: Sudoku Constraint Solver

Difficulty: Beginner-Intermediate
Time: 8-14 hours (about a weekend)
Language: Python (alternatives: JavaScript, C++)
Prerequisites: Projects 2 and 7
Source: Prog P01

What You Will Build

Build a Sudoku solver that treats each empty cell as a set of candidates constrained by its row, column, and 3×3 box. Read a standard 81-character or formatted grid, validate the givens, repeatedly apply constraint propagation, and explain every forced placement. When deduction stalls, choose a constrained cell, branch, and backtrack on contradiction. The final report should distinguish deductions from guesses, show search depth and explored branches, and reject malformed, contradictory, or non-unique puzzles according to a documented mode.

Real World Outcome

For a valid puzzle, the CLI prints the solved grid plus a replayable sequence such as “r4c7 = 9 because 9 is its only remaining candidate.” It reports propagation rounds and search statistics. A contradictory puzzle points to the conflicting row/column/box; an unsatisfiable puzzle returns no solution; an optional uniqueness check searches for a second solution rather than assuming the first is unique.

Core Question

How can a set of local constraints shrink a huge search space, and when should deterministic deduction give way to systematic backtracking?

Concepts You Must Understand First

Sets and predicates: candidates are values satisfying all active constraints.
Constraint satisfaction problems: variables, domains, constraints, assignments, and contradictions.
Propagation: every assignment removes possibilities from peers and may force new assignments.
Backtracking search: branch, propagate, detect failure, undo, and try the next value.
Search heuristics: choosing the cell with the fewest candidates usually exposes contradictions sooner. See Skiena and Revilla, Programming Challenges.

Build Milestones

Parse and validate grids; construct peer sets for every cell.
Compute candidates and implement naked singles with an explanation log.
Add hidden singles in each row, column, and box; propagate until no changes occur.
Add recursive backtracking with complete state isolation and minimum-remaining-values selection.
Detect invalid, unsatisfiable, solved, and optionally non-unique outcomes with tests.

Hints in Layers

Precompute each cell’s peers once; repeated row/column/box scans are simpler when relationships are explicit.
Make propagation return one of three states: contradiction, stable unsolved, or solved.
Copy or immutably derive candidate state at each branch; debugging shared mutable state is far harder than spending modest memory.

Common Pitfalls and Debugging

Symptom: a backtracked branch contaminates later attempts. Cause: candidate sets were mutated in place across recursion. Fix: clone state or use an explicit reversible change log.
Symptom: the solver loops without progress. Cause: a propagation pass reports change even when no candidate changed. Fix: compare before/after state and assert monotonic candidate removal.
Symptom: a full grid is accepted though invalid. Cause: completion was checked without constraints. Fix: validate every unit before declaring success.

Definition of Done

Input validation identifies malformed and contradictory givens.
Constraint propagation explains forced placements before search.
Backtracking solves puzzles that deduction alone cannot.
Branch state is isolated and search statistics are reproducible.
Tests cover easy, hard, invalid, unsatisfiable, and multiple-solution cases.

Project 9: Graph Traversal Visualizer

Difficulty: Intermediate
Time: 8-14 hours (about a weekend)
Language: Python (alternatives: JavaScript, C++)
Prerequisites: Projects 2 and 8
Source: Prog P06

What You Will Build

Build an interactive or frame-by-frame visualizer for breadth-first search (BFS) and depth-first search (DFS) on user-defined directed or undirected graphs. Support adjacency-list input, validate missing/duplicate nodes, and display the active node, frontier data structure, visited set, traversal tree, and discovery order after every operation. BFS should reconstruct shortest unweighted paths; DFS should reveal branches, backtracking, and connected/reachable components. Deterministic neighbor ordering and a seeded graph generator make runs testable and comparable.

Real World Outcome

Selecting a start and goal animates BFS expanding in layers and returns a highlighted shortest path with distance. Switching to DFS shows a different exploration tree and explicit backtracking. Disconnected nodes remain visibly unvisited, and directed edges are respected. A textual trace makes the algorithm understandable even without the animation and can be compared against expected queue/stack states.

Core Question

How does the choice of frontier discipline—first-in-first-out versus last-in-first-out—change exploration order and the guarantees you get?

Concepts You Must Understand First

Graph vocabulary: vertices, edges, direction, paths, cycles, components, and reachability.
Adjacency representations: lists suit sparse graphs; matrices make edge lookup direct but consume more space.
Queue and stack behavior: BFS uses FIFO; iterative DFS uses LIFO, while recursive DFS uses the call stack.
Visited invariants: mark discovery early enough to avoid repeated insertion and cycle loops.
Shortest unweighted paths: BFS layer number equals edge distance. See Cormen et al., Introduction to Algorithms, graph traversal chapters.

Build Milestones

Parse directed/undirected graphs and render nodes, edges, labels, and disconnected components.
Implement BFS as an event generator emitting enqueue, dequeue, discover, and finish events.
Implement iterative or recursive DFS with discover, edge, backtrack, and finish events.
Add parent maps, traversal trees, reachability, and BFS shortest-path reconstruction.
Build controls for play/pause/step/reset and tests for exact deterministic traces.

Hints in Layers

Keep algorithms UI-independent by yielding immutable events; the renderer only consumes them.
Sort neighbors before traversal so insertion order does not silently change teaching output.
For a shortest path, follow parent[goal] backward to the start and reverse the result.

Common Pitfalls and Debugging

Symptom: cyclic graphs enqueue the same node many times. Cause: nodes are marked visited only when removed. Fix: mark them when discovered/enqueued.
Symptom: BFS returns a non-shortest path. Cause: a stack or weighted interpretation slipped in. Fix: inspect frontier events and assert nondecreasing discovery distance.
Symptom: animation state differs from algorithm state. Cause: mutable containers were passed to old frames. Fix: snapshot event payloads.

Definition of Done

Directed and undirected graph inputs are validated and rendered.
BFS and DFS produce deterministic, inspectable event traces.
The frontier, visited set, and traversal tree are visible at each step.
BFS reconstructs correct shortest paths in unweighted graphs.
Cycles, self-loops, disconnected graphs, and unreachable goals are tested.

Project 10: RSA Cryptosystem from Scratch

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python (alternatives: C++, Rust)
Prerequisites: Projects 1-2
Source: Prog P03

What You Will Build

Build a deliberately small educational RSA system without using a cryptography library for the number-theory operations. Generate candidate primes, test primality at an appropriate toy scale, compute n and Euler’s totient, choose a public exponent coprime to the totient, and find the private exponent with the extended Euclidean algorithm. Implement square-and-multiply modular exponentiation, encode short byte messages into integers smaller than n, and demonstrate key generation, encryption, and decryption. Clearly label the result unsafe for real secrets because it lacks secure key sizes, vetted randomness, padding, and side-channel defenses.

Real World Outcome

The CLI creates a public/private toy keypair, prints the mathematical invariants, encrypts a short message into integer blocks, and recovers the exact original bytes. A property-test run checks many messages and prime pairs. Invalid public exponents, oversized blocks, non-primes, and corrupt ciphertext produce useful diagnostics rather than silent nonsense.

Core Question

How do modular arithmetic and multiplicative inverses create a transformation that is easy to apply publicly but reversible only with a private exponent?

Concepts You Must Understand First

Divisibility, primes, and greatest common divisors: Euclid’s algorithm exposes coprimality.
Congruences: numbers can be equivalent modulo n; arithmetic wraps within residue classes.
Euler’s totient and theorem: for coprime values, exponentiation cycles modulo n.
Modular inverses: extended Euclid finds d such that ed ≡ 1 (mod φ(n)).
Fast modular exponentiation: square-and-multiply avoids constructing enormous intermediate powers. See Hardy and Wright, An Introduction to the Theory of Numbers.

Build Milestones

Implement and test gcd, extended gcd, modular inverse, and modular exponentiation.
Add toy prime generation/testing with deterministic test fixtures and optional seeded randomness.
Generate keys and assert p != q, gcd(e, φ)=1, and (e*d) mod φ = 1.
Encode messages into safe-size blocks; encrypt, decrypt, and reconstruct bytes exactly.
Add property tests, tampering/error cases, serialization, and an explicit security disclaimer.

Hints in Layers

Build the arithmetic primitives first and compare each with Python’s three-argument pow or hand-worked examples.
Keep byte encoding separate from RSA math; assert every message integer is in [0, n).
Use tiny known primes for readable tests, then larger toy primes only after all invariants pass.

Common Pitfalls and Debugging

Symptom: decryption returns unrelated integers. Cause: d was computed modulo the wrong value or e is not coprime to the totient. Fix: print and assert the key invariants.
Symptom: part of a message cannot be recovered. Cause: its encoded integer is at least n. Fix: reduce block size based on modulus bit length.
Symptom: implementation seems secure because examples work. Cause: mathematical correctness was confused with production cryptographic safety. Fix: document missing OAEP padding, CSPRNG, key size, and side-channel protections.

Definition of Done

Core number-theory functions pass hand-calculated and randomized property tests.
Key generation enforces prime, distinctness, and coprimality invariants.
Short messages round-trip exactly through block encoding, encryption, and decryption.
Invalid keys, blocks, and ciphertext paths fail deterministically.
Documentation prominently states that the system is educational and unsafe for real secrets.

Phase 3 — Probability and Statistics (Projects 11-14)

Project 11: Monte Carlo Probability Lab

Difficulty: Beginner-Intermediate
Time: 12-20 hours
Language: Python (alternatives: R, Julia, JavaScript, Rust)
Prerequisites: Project 7
Merged from: HS P05; ML-Math P12; ML-Found P09

What You Will Build

Build a seeded simulation suite that begins with descriptive statistics and grows into reusable Monte Carlo experiments. Load or generate a small dataset and compute frequency tables, mean, median, variance, and standard deviation. Then simulate coins, dice, cards, a roulette-style casino game, and geometric estimation of π. For every experiment, define the sample space and random variable, compute theoretical probabilities or expected value when available, track running estimates, and display confidence intervals and error against trial count. Include a bankroll simulation so negative expected value becomes observable over many repeated plays.

Real World Outcome

A single command runs reproducible experiments and generates a report containing summary statistics, empirical versus theoretical frequencies, an EV table, and convergence plots. The π view shows inside/outside points and the running estimate 4 * inside / total. Casino runs display bankroll paths from multiple seeds plus average drift and uncertainty, making clear that short-term wins can coexist with a negative long-run expectation.

Core Question

How can repeated random experiments estimate quantities and decisions reliably while every individual run remains uncertain?

Concepts You Must Understand First

Descriptive statistics: center, spread, frequency tables, outliers, and why mean and median answer different questions.
Sample spaces and events: probability is defined over explicit possible outcomes, not vague “randomness.”
Random variables, expectation, and variance: EV is a probability-weighted average, not a guaranteed payoff.
Law of large numbers: sample averages stabilize with more independent trials. See Blitzstein and Hwang, Introduction to Probability, Chapter 1.
Standard error and confidence intervals: Monte Carlo precision generally improves on the order of 1/sqrt(n). See Bhargava, Grokking Algorithms, Monte Carlo discussion.

Build Milestones

Implement tested descriptive-statistics functions and frequency reports for a small dataset.
Define a common experiment interface with seed, trial count, outcome, and payoff.
Add coin, dice, card, and casino experiments with theoretical probability/EV comparisons.
Add π estimation, point visualization, running estimates, and interval/error plots.
Run batches across seeds and sample sizes; export assumptions, uncertainty, and convergence evidence.

Hints in Layers

Keep the random generator object explicit and pass it into experiments so seeds truly control every draw.
Record cumulative sums and squared sums online; you need not retain millions of outcomes for convergence statistics.
Validate games by enumerating one-round outcomes before trusting a large simulation.

Common Pitfalls and Debugging

Symptom: the same seed produces different reports. Cause: multiple implicit RNGs or changed draw order. Fix: use one injected generator and log configuration.
Symptom: roulette appears profitable. Cause: payouts omit the lost stake or probabilities do not sum to one. Fix: enumerate every outcome and compute theoretical EV first.
Symptom: confidence intervals are implausibly narrow. Cause: variance or standard error used the wrong denominator. Fix: test against a Bernoulli case with known variance.

Definition of Done

Descriptive statistics match hand-computed fixtures.
Every random experiment is seeded and reproducible.
Empirical frequencies and EV are compared with theoretical values where known.
π and casino experiments include convergence and uncertainty visualizations.
Batch runs across seeds demonstrate variation rather than hiding it.
The report states assumptions, trial count, seed, and limitations.

Project 12: Distribution and Random-Variable Visualizer

Difficulty: Beginner-Intermediate
Time: 10-16 hours
Language: Python (alternatives: Julia, R, JavaScript)
Prerequisites: Project 11
Merged from: ML-Math P13; ML-Found P12

What You Will Build

Build an interactive, seeded explorer for uniform, Bernoulli/binomial, Poisson, exponential, and normal distributions. For each family, accept valid parameters, generate samples, compute empirical moments, and overlay a histogram or mass plot with the theoretical PMF/PDF. Plot the CDF separately and make its probability interpretation explicit. Add a Central Limit Theorem experiment that repeatedly sums or averages samples from a clearly non-normal parent distribution and shows the standardized averages becoming approximately normal as sample size grows.

Real World Outcome

Commands such as normal --mean 0 --std 1 --samples 10000 and binomial --n 20 --p .3 produce reproducible plots and reports comparing theoretical and empirical mean/variance. The CDF view answers interval or threshold questions. A CLT panel compares average distributions for n=1, 2, 10, 30, including standardized axes and a metric or visual reference showing convergence toward a bell curve.

Core Question

How do a random variable’s distribution and parameters determine the shapes, averages, variability, and tail behavior observed in data?

Concepts You Must Understand First

Random variables: functions mapping outcomes to numeric values; discrete and continuous cases differ.
PMF, PDF, and CDF: PMFs assign mass, PDFs assign density, and CDFs give cumulative probability. See DasGupta, Probability for Statistics and Machine Learning, Chapters 2-3.
Expectation and variance: theoretical moments can validate a sampler.
Distribution families: Bernoulli/binomial for successes, Poisson for counts, exponential for waiting, normal for additive noise.
Central Limit Theorem: standardized averages often approach a normal distribution under suitable conditions. See Blitzstein and Hwang, Chapters 3-7.

Build Milestones

Define validated distribution objects exposing sampling, PMF/PDF, CDF, mean, and variance where practical.
Implement deterministic sampling and compare sample moments across growing sample sizes.
Overlay normalized histograms/mass bars with theoretical curves and plot CDFs.
Add interval-probability queries and parameter controls with invalid-range diagnostics.
Build the CLT experiment using several parent distributions and sample-size settings.

Hints in Layers

For continuous distributions, histogram height must use density normalization before comparison with a PDF.
Validate samplers first with mean/variance and quantiles; a pretty histogram can hide systematic errors.
Standardize CLT averages with their theoretical mean and standard error so panels are comparable.

Common Pitfalls and Debugging

Symptom: PDF values are interpreted as point probabilities. Cause: density and mass were conflated. Fix: demonstrate probability as area over an interval.
Symptom: empirical overlays are consistently shifted. Cause: distribution parameters use a different convention, such as rate versus scale. Fix: name conventions and test theoretical moments.
Symptom: CLT plots narrow but do not align. Cause: raw averages were not standardized. Fix: center by the parent mean and scale by sigma/sqrt(n).

Definition of Done

At least five common distributions support validated parameters and seeded samples.
Empirical means/variances converge toward theoretical references.
PMF/PDF and CDF plots are correctly labeled and normalized.
Threshold/interval probability queries agree with numerical or known cases.
A reproducible CLT experiment compares several sample sizes and parent distributions.

Project 13: Naive Bayes Spam Filter

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python (alternatives: Go, Rust, JavaScript, C++)
Prerequisites: Project 12
Merged from: ML-Math P14; Prog P05; ML-Found P10

What You Will Build

Build a multinomial Naive Bayes email classifier from scratch, without a machine-learning library. Create a deterministic tokenizer, split labeled messages before fitting, count class and token frequencies, estimate priors and Laplace-smoothed likelihoods, and combine evidence in log space. For each prediction, report spam/ham scores, posterior-like normalized confidence, and the tokens contributing most strongly in each direction. Evaluate on held-out data with a confusion matrix, accuracy, precision, recall, F1, calibration bins, and examples of costly false positives.

Real World Outcome

Training prints class priors and interpretable token statistics. Entering a new message returns its label, score margin, and strongest evidence rather than a bare answer. An evaluation report compares a majority baseline with the model, displays confusion counts and threshold curves, inspects misclassifications, and saves the vocabulary/model so the same test message receives the same result after reload.

Core Question

How can Bayes’ rule combine many uncertain word clues, and why can a deliberately false conditional-independence assumption still produce a useful classifier?

Concepts You Must Understand First

Conditional probability and Bayes’ rule: posterior is proportional to likelihood times prior.
Naive conditional independence: tokens are assumed independent given class, simplifying a joint likelihood.
Maximum-likelihood counts and Laplace smoothing: unseen words must not zero the entire message probability. See Géron, Hands-On Machine Learning, classification chapters.
Log probabilities: products of small values become sums and avoid underflow. See Jurafsky and Martin, Speech and Language Processing, Chapter 4.
Classification metrics and calibration: accuracy can hide false-positive/false-negative tradeoffs. See Manning et al., Introduction to Information Retrieval, Chapter 13.

Build Milestones

Load a labeled corpus, normalize/tokenize deterministically, and create stratified train/test splits.
Fit class priors and per-class token counts using training data only.
Add smoothing, unknown-token behavior, log-score prediction, and model serialization.
Produce confusion-matrix metrics, threshold analysis, calibration bins, and baseline comparison.
Explain individual predictions and review representative errors for data or assumption failures.

Hints in Layers

Start with tiny messages where you can calculate every count and posterior by hand.
Compute one log score per class: log prior + sum(token_count * log likelihood).
Normalize two log scores safely by subtracting their maximum before exponentiating.

Common Pitfalls and Debugging

Symptom: any unseen token forces probability zero. Cause: likelihoods were unsmoothed. Fix: apply consistent additive smoothing including vocabulary size.
Symptom: test accuracy is suspiciously perfect. Cause: vocabulary/counts were fitted before splitting. Fix: make the split first and audit all fitted state.
Symptom: long messages produce zero or NaN scores. Cause: raw probabilities underflow. Fix: accumulate log probabilities.

Definition of Done

Tokenization, split, and training are deterministic and leakage-free.
Priors, likelihoods, smoothing, and log scoring match hand-calculated fixtures.
Predictions include interpretable positive and negative token evidence.
Evaluation includes confusion metrics, threshold tradeoffs, calibration, and a baseline.
Saved and reloaded models produce identical predictions.
Documentation discusses independence assumptions and false-positive risk.

Project 14: A/B Testing and Uncertainty Dashboard

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python (alternatives: R, JavaScript, Julia, Go)
Prerequisites: Project 12
Merged from: ML-Math P15; ML-Found P11

What You Will Build

Build a reproducible analyzer for two-variant conversion experiments. Accept visitor and conversion counts or row-level Bernoulli data, validate assignment and missingness assumptions, compute rates, absolute/relative lift, pooled standard error, a two-sided proportion z-test, p-value, and confidence interval for the rate difference. Add power and minimum-detectable-effect calculations, warnings for small samples or peeking, and a decision panel that separates statistical significance from practical significance. A simulation mode should reveal false positives, low power, and interval coverage across repeated experiments.

Real World Outcome

Given control 203/4532 and treatment 248/4621, the dashboard shows rates, lift, test statistic, p-value, and a confidence interval, then states what the result does and does not justify. Error-bar and sampling-distribution plots show uncertainty. The recommendation incorporates a user-provided minimum worthwhile effect, sample-size adequacy, and data-quality warnings rather than reducing the decision to p < .05.

Core Question

When observed conversion rates differ, how much evidence is there for a real effect, how large might it be, and is it useful enough to act on?

Concepts You Must Understand First

Sampling variability and Bernoulli/binomial models: observed proportions fluctuate even with no effect.
Null/alternative hypotheses and z statistics: standardize a difference under the null model.
P-values: probability of equally or more extreme data under the null, not probability the null is true. See Reinhart, Statistics Done Wrong, Chapter 1.
Confidence intervals and effect size: quantify plausible magnitude, not only threshold crossing. See James et al., An Introduction to Statistical Learning, Chapter 2.
Power and experimental validity: randomization, independence, predeclared analysis, and adequate sample size. See Wheelan, Naked Statistics, Chapter 10.

Build Milestones

Validate counts/data and compute conversion rates, absolute lift, relative lift, and standard errors.
Implement a two-proportion z-test and confidence interval, checked against trusted software.
Add practical-effect thresholds, power, sample-size, and minimum-detectable-effect calculations.
Build plots and warnings for imbalance, low expected counts, repeated peeking, and missing data.
Simulate null and alternative experiments to verify Type I error, power, and interval coverage.

Hints in Layers

Use the pooled proportion for the null-test variance but an unpooled variance for the ordinary difference interval; document the choice.
Create exact boundary fixtures: identical rates, zero conversions, total conversion, and tiny samples.
Keep “statistically detectable” and “worth shipping” as independent dashboard fields.

Common Pitfalls and Debugging

Symptom: p-value is reported as the chance the treatment is ineffective. Cause: null-conditional probability was reversed. Fix: include a fixed plain-language interpretation template.
Symptom: relative lift explodes. Cause: control rate is near zero. Fix: always report absolute difference and flag unstable relative measures.
Symptom: simulation false positives exceed the selected alpha. Cause: repeated looks or incorrect standard error. Fix: freeze one analysis point and compare formulas with a reference package.

Definition of Done

Input validation catches impossible counts and documents analysis assumptions.
Test statistics, p-values, and intervals match trusted reference cases.
Reports include absolute/relative effect, uncertainty, power, and practical threshold.
Visuals and language avoid common p-value and confidence-interval misinterpretations.
Simulations demonstrate Type I error, power, and coverage behavior.
Decision output includes warnings rather than an unconditional ship/no-ship command.

Phase 4 — Geometry and Linear Algebra (Projects 15-25)

Project 15: Conic Sections Mapper

Difficulty: Intermediate-Advanced
Time: 10-18 hours
Language: Python (alternatives: Julia, JavaScript, C#)
Prerequisites: Projects 3 and 5
Source: HS P15

What You Will Build

Build a conic-analysis tool for axis-aligned general quadratic equations Ax^2 + Cy^2 + Dx + Ey + F = 0. Parse and normalize coefficients, classify circles, ellipses, parabolas, hyperbolas, and degenerate/unsupported cases, then complete the square to produce standard form. Extract center or vertex, axes/radius, orientation, foci, directrix where appropriate, and asymptotes for hyperbolas. Plot the curve from the derived geometry and verify the transformation by sampling points and checking residuals in both original and standard equations.

Real World Outcome

For 4x^2 + 9y^2 - 8x + 36y + 4 = 0, the report shows each completing-the-square step, identifies an ellipse, prints its center and semi-axes, and saves a correctly scaled plot with foci. Invalid, degenerate, imaginary, or rotated Bxy cases are classified explicitly rather than forced into a misleading picture.

Core Question

How does the algebraic pattern of a quadratic equation encode the geometry of a circle, ellipse, parabola, or hyperbola?

Concepts You Must Understand First

Completing the square: rewrite ax^2 + bx as a shifted square plus a constant.
Standard conic forms: signs and denominators determine shape and scale.
Geometric parameters: center/vertex, axes, eccentricity, foci, directrices, and asymptotes.
Quadratic-form viewpoint: coefficients summarize curvature; an xy term indicates rotation.
Residual checks: derived points should satisfy the original equation within tolerance.

Build Milestones

Parse two-variable quadratic expressions into stable coefficient records.
Classify supported axis-aligned forms and detect degenerate, empty, or rotated cases.
Generate completing-the-square steps and normalized standard form.
Extract geometric parameters and render equal-aspect plots with meaningful bounds.
Sample the plotted parameterization and verify original-equation residuals.

Hints in Layers

Group x terms, y terms, and constants before completing either square.
Normalize the final right-hand side to 1, then read denominators and signs structurally.
Derive plotting bounds from center/axes rather than using an arbitrary window.

Common Pitfalls and Debugging

Symptom: ellipses look circular. Cause: unequal plot-axis scaling. Fix: enforce equal aspect ratio.
Symptom: a negative-radius circle is plotted. Cause: empty/imaginary cases were not classified after normalization. Fix: validate squared-length parameters before plotting.
Symptom: completing-square steps change the curve. Cause: constants added inside groups were not balanced outside. Fix: track symbolic before/after coefficients and residual-test samples.

Definition of Done

Supported quadratic inputs classify correctly, including degenerate/empty cases.
Standard-form transformations are shown and algebraically equivalent.
Geometry tables include the appropriate parameters for each conic.
Plots use equal scaling and derived, reproducible bounds.
Sampled plotted points satisfy the original equation within tolerance.

Project 16: Vector Geometry and 2D Graphics Toolkit

Difficulty: Intermediate-Advanced
Time: 12-20 hours
Language: Python (alternatives: C++, Julia, JavaScript)
Prerequisites: Projects 5 and 15
Merged from: HS P16; ML-Found P01

What You Will Build

Build a dimension-safe vector library and interactive 2D graphics playground. Begin with a unit-circle checkpoint that converts degrees/radians and visualizes sine/cosine as coordinates. Implement vector addition, subtraction, scalar multiplication, magnitude, normalization, dot product, angle, projection, and parallel/orthogonal decomposition. Then represent polygon vertices as vectors and let the user translate, rotate, scale, and steer shapes. Draw original and transformed vectors, projection components, angles, basis axes, and verification diagnostics so every operation has geometric meaning.

Real World Outcome

The CLI can explain the angle and projection between two vectors and show their decomposition. The graphics window displays a triangle or spaceship that rotates around a chosen pivot, moves along its heading, and scales predictably from keyboard controls. A side panel reports norms, dot products, radians/degrees, and invariant checks such as orthogonal residual near zero and length preservation under rotation.

Core Question

How do coordinates, trigonometry, and dot products turn abstract arrows into measurable geometry and controllable on-screen motion?

Concepts You Must Understand First

Radians and unit circle: (cos θ, sin θ) defines direction; radians connect angle to arc length.
Vectors versus points: both use coordinates, but vectors represent displacement and obey vector operations.
Norm and normalization: magnitude measures length; unit vectors preserve direction only for nonzero inputs.
Dot product: u·v = |u||v|cos θ, connecting algebra to angle and orthogonality. See Orland, Math for Programmers, Chapters 2 and 5.
Projection and rotation: projection extracts the component along a direction; 2D rotation combines sine and cosine. See 3Blue1Brown, Essence of Linear Algebra, Episode 3.

Build Milestones

Build degree/radian conversion and an interactive unit-circle visualization with quadrant tests.
Implement vector arithmetic, norms, normalization, dot products, angles, and dimension checks.
Add projection/decomposition and verify orthogonality and reconstruction invariants.
Render shapes from vertex vectors and support translation, rotation, scaling, and pivot choice.
Add interactive controls, numeric diagnostics, and property tests for geometric identities.

Hints in Layers

Clamp the computed cosine to [-1, 1] before acos to absorb tiny floating-point drift.
Projection of u onto v is (u·v / v·v)v; reject a zero v explicitly.
Apply transformations to immutable model coordinates, then render; repeatedly transforming already-rounded screen points accumulates distortion.

Common Pitfalls and Debugging

Symptom: rotations go the wrong way or by the wrong amount. Cause: degrees were passed to radian-based trig or screen y-axis orientation was ignored. Fix: centralize conversion and document coordinates.
Symptom: normalization produces NaN. Cause: a zero vector was divided by its magnitude. Fix: return a clear domain error.
Symptom: projections do not reconstruct the original vector. Cause: denominator or vector direction is wrong. Fix: assert parallel + orthogonal ≈ original and orthogonal·v ≈ 0.

Definition of Done

Unit-circle controls correctly connect angles, radians, sine, and cosine.
Vector operations enforce dimensions and handle zero-vector domains.
Projection/decomposition pass reconstruction and orthogonality checks.
Shapes translate, rotate, and scale around documented pivots.
Rotation preserves length within tolerance and deterministic tests cover quadrants.
Visuals and numeric diagnostics tell the same geometric story.

Project 17: Ballistics and Trigonometry Simulator

Difficulty: Beginner-Intermediate
Time: 8-14 hours
Language: Python (alternatives: JavaScript, C++)
Prerequisites: Projects 5 and 16
Source: HS P03

What You Will Build

Build a deterministic projectile-motion simulator for a constant-gravity, no-air-resistance world. Accept launch speed, angle, initial height, gravitational acceleration, and time step with explicit units. Decompose the launch velocity with sine and cosine, compute analytic position, flight time, horizontal range, and maximum height, then independently advance the state with fixed discrete time steps. Plot the trajectory and velocity components, interpolate the landing event between samples, and report analytic-versus-simulated error as the step size changes.

Real World Outcome

A command such as --speed 30 --angle 45 --height 2 --dt .02 prints initial velocity components, predicted apex, flight time, and range; saves a labeled trajectory; and tabulates simulation errors. A step-size sweep visibly converges toward the analytic solution. Impossible or ambiguous inputs—negative speed, nonpositive gravity, unsupported units—fail with deterministic messages.

Core Question

How do trigonometric components and constant-acceleration equations predict a curved trajectory, and what error appears when continuous motion is sampled in discrete time?

Concepts You Must Understand First

Unit-circle trigonometry: horizontal and vertical components are v cos θ and v sin θ.
Position, velocity, and acceleration: gravity changes vertical velocity at a constant rate.
Projectile equations: x=x0+vxt and y=y0+vyt-(g/2)t^2 form a parabola.
Quadratic root selection: landing time is the meaningful future root of y(t)=0.
Time stepping and interpolation: a simulated point may pass below ground between samples.

Build Milestones

Validate parameters/units and derive horizontal and vertical velocity components.
Implement analytic position, apex, landing time, and range with edge-case classification.
Implement fixed-step simulation and stop/interpolate at ground crossing.
Plot trajectory and component/time graphs; annotate launch, apex, and landing.
Run step-size sweeps and compare metrics against analytic references.

Hints in Layers

Solve the vertical quadratic for time and select the largest nonnegative root.
Keep analytic and simulated implementations separate so one can validate the other.
When consecutive y values straddle zero, linearly interpolate the final time/x instead of accepting a below-ground sample.

Common Pitfalls and Debugging

Symptom: a 45-degree shot travels almost nowhere. Cause: degrees were passed directly to radian trig functions. Fix: convert once at input and test cardinal angles.
Symptom: the simulated range jumps as dt changes. Cause: landing was rounded to the first below-ground point. Fix: interpolate the crossing and report residual error.
Symptom: initial height produces the wrong flight time. Cause: a ground-launch shortcut was used. Fix: solve the full vertical equation.

Definition of Done

Inputs use explicit units and enforce valid ranges.
Analytic metrics match hand-computed baseline cases.
Simulation tracks position/velocity and interpolates landing.
Plots label launch, apex, landing, axes, and units.
Step-size experiments demonstrate and quantify convergence.
Documentation states the no-drag/flat-ground assumptions.

Project 18: Complex Plane Explorer

Difficulty: Intermediate-Advanced
Time: 8-16 hours
Language: Python (alternatives: Julia, MATLAB/Octave, JavaScript)
Prerequisites: Projects 5 and 16
Source: HS P18

What You Will Build

Build a complex-number toolkit that parses rectangular values, converts between rectangular and polar forms, performs arithmetic, powers, and roots, and plots every result geometrically. Implement core operations from real/imaginary pairs rather than relying entirely on Python’s complex type; use the standard library only as a validation oracle. Show multiplication as scaling by modulus and rotation by argument, division as inverse scaling/rotation, and the n roots of a value as evenly spaced points. Include Euler-form demonstrations and careful branch conventions for principal arguments.

Real World Outcome

Entering 3+4i reports modulus 5, principal argument in radians/degrees, and a plotted vector. Multiplying by i visibly rotates it 90 degrees. Asking for fifth roots displays five points on the correct circle and verifies that raising each root to the fifth power reconstructs the input within tolerance. Round-trip rectangular→polar→rectangular errors are printed rather than hidden.

Core Question

How does adding one imaginary dimension turn multiplication into a geometric operation involving both rotation and scale?

Concepts You Must Understand First

Imaginary unit and rectangular form: i^2=-1, with real and imaginary coordinates.
Modulus and argument: distance from origin and directed angle, computed robustly with atan2.
Polar multiplication/division: moduli multiply/divide while arguments add/subtract.
Euler’s identity: e^(iθ)=cos θ+i sin θ links exponentials and rotations.
Powers, roots, and branches: arguments differ by multiples of 2π, creating multiple roots.

Build Milestones

Define a rectangular complex value and tested addition, subtraction, multiplication, and division.
Add modulus/argument and rectangular↔polar conversion with a documented principal-angle interval.
Implement polar multiplication/division and integer powers; visualize before and after.
Generate all nth roots, plot them, and verify power round-trips.
Add parser, Euler-form display, tolerance reports, and comparisons with trusted complex arithmetic.

Hints in Layers

Use atan2(imag, real), not atan(imag/real), to preserve quadrants and vertical directions.
Normalize displayed angles consistently, but retain unwrapped angles when explaining repeated rotations.
Roots have arguments (θ + 2πk)/n for k=0..n-1; stopping at the principal root loses valid answers.

Common Pitfalls and Debugging

Symptom: arguments are wrong in quadrants II and III. Cause: one-argument arctangent lost sign/quadrant information. Fix: use atan2 with tests on all axes/quadrants.
Symptom: only one square root appears. Cause: only the principal branch was generated. Fix: iterate all root indices.
Symptom: conversions fail near the negative real axis. Cause: equivalent angles π and -π were compared directly. Fix: compare reconstructed values or wrapped angular distance.

Definition of Done

Rectangular arithmetic agrees with hand and trusted-library cases.
Polar conversions handle every quadrant and round-trip within tolerance.
Multiplication/division visuals correctly show scale and rotation.
Powers and all nth roots are implemented and verified.
Principal-angle conventions and zero-value limitations are documented.

Project 19: Chaos and Mandelbrot Explorer

Difficulty: Intermediate
Time: 10-18 hours
Language: Python (alternatives: JavaScript, Julia, C++)
Prerequisites: Projects 6 and 18
Source: HS P04

What You Will Build

Build a reproducible fractal explorer centered on complex iteration. For each pixel, map screen coordinates to a complex c, iterate z[n+1]=z[n]^2+c from zero, and record escape time when magnitude exceeds a justified threshold. Render a Mandelbrot image with configurable viewport, resolution, maximum iterations, and color mapping; support zooming and selected-point orbit diagnostics. Add a Julia mode by fixing c and varying the initial z. To preserve the source project’s broader chaos lesson, include an optional seeded triangle chaos-game view demonstrating a geometric iterated-function system and burn-in behavior.

Real World Outcome

The tool renders a deterministic Mandelbrot set, allows the user to zoom into a selected rectangle, and prints the orbit/escape iteration for clicked points. Changing maximum iterations reveals boundary detail without moving stable exterior points. Julia mode shows how one chosen parameter reshapes the set. Fixed seeds make the optional Sierpiński chaos-game image byte-reproducible and let probability changes alter point density.

Core Question

How can repeatedly applying an extremely simple rule create stable global structure, intricate boundaries, and sensitivity to tiny initial changes?

Concepts You Must Understand First

Iteration and recurrence: current state is repeatedly transformed into next state.
Complex multiplication: squaring doubles angle and squares modulus, driving the geometry.
Escape criteria: for the quadratic Mandelbrot iteration, once |z|>2, the orbit will diverge.
Coordinate mapping: pixels must map consistently to a complex viewport without distortion.
Convergence, divergence, and finite evidence: “did not escape by N” is not a proof of membership.

Build Milestones

Implement and unit-test complex orbit generation and escape-time classification.
Map an equal-aspect complex viewport to pixels and render a baseline Mandelbrot image.
Add color maps, maximum-iteration controls, zoom history, and reproducible image metadata.
Add Julia-set rendering plus orbit plots for selected points.
Optionally add the seeded triangle chaos game with burn-in and transform/probability controls.

Hints in Layers

Test known points: c=0 remains bounded, while c=2 escapes quickly.
Separate mathematical coordinates from raster coordinates and test corner mappings.
Store escape iterations as a numeric array before coloring; then color changes do not require recomputation.

Common Pitfalls and Debugging

Symptom: the set appears stretched or mirrored. Cause: viewport aspect or pixel-y direction is wrong. Fix: test corners and enforce equal coordinate scale.
Symptom: every pixel is classified identically. Cause: z was not reset per pixel or c was not updated. Fix: log the first few orbits for known points.
Symptom: boundary points are declared definitely inside. Cause: finite iteration was treated as proof. Fix: label them “not escaped within limit.”

Definition of Done

Escape-time iteration passes known bounded/escaping fixtures.
Viewport-to-pixel mapping is tested and preserves aspect ratio.
Mandelbrot images are deterministic and include viewport/iteration metadata.
Zoom, Julia mode, and selected-orbit diagnostics work without stale state.
Finite-iteration limitations are explicit in the UI/report.
If included, chaos-game results reproduce exactly from seed and parameters.

Project 20: Matrix Systems and Visualization Studio

Difficulty: Advanced
Time: 16-26 hours
Language: Python (alternatives: Julia, MATLAB/Octave, C++, Rust)
Prerequisites: Projects 3 and 16
Merged from: HS P17; ML-Math P04

What You Will Build

Build a matrix calculator and linear-system studio from first principles. Support shape-checked addition, scalar multiplication, matrix multiplication, transpose, determinant for small matrices, inverse when appropriate, and Gaussian/Gauss-Jordan elimination with recorded row operations. Solve Ax=b and classify unique, infinitely many, or inconsistent systems using rank/pivot structure rather than determinant alone. Visualize two-variable equations as lines and three-variable systems where practical, and show row operations step by step. Report residuals and a conditioning warning when small input perturbations cause large solution changes.

Real World Outcome

The CLI accepts matrices in a stable text/JSON format, prints dimensions, and shows each row-reduction step. A unique system returns its solution and ||Ax-b||; an underdetermined system reports free variables and a parameterized solution; an inconsistent one identifies a contradictory row. A 2D plot shows line intersection, coincidence, or parallelism, while a stress example demonstrates why an almost singular system can be numerically unreliable despite having a formal solution.

Core Question

How do matrix operations encode simultaneous equations, and what do pivots, rank, and conditioning reveal that a computed answer alone cannot?

Concepts You Must Understand First

Matrix shape and operations: multiplication composes row-column dot products and requires compatible dimensions. See Orland, Math for Programmers.
Elementary row operations: swap, scale by nonzero value, and add a multiple preserve the solution set.
Echelon form, pivots, and rank: pivot structure controls constraints and free variables.
Determinant and inverse: determinant detects invertibility for square matrices but is not the universal solving method.
Residual versus conditioning: a small residual can coexist with a solution highly sensitive to input error.

Build Milestones

Implement a validated matrix type and fundamental operations with explicit shape errors.
Add elimination with partial pivoting and a replayable row-operation transcript.
Classify augmented systems by ranks/pivots and return unique or parameterized solutions.
Implement determinant/inverse as derived tools and verify identities/residuals.
Add 2D visualizations, perturbation experiments, and comparisons with NumPy as an oracle.

Hints in Layers

During elimination, choose the largest available absolute pivot in the column to reduce avoidable numerical error.
Determine inconsistency from a row [0 ... 0 | nonzero]; free non-pivot columns imply infinitely many solutions.
Test the implementation against exact integer/rational examples before ill-conditioned floating-point systems.

Common Pitfalls and Debugging

Symptom: matrix multiplication gives plausible wrong shapes/values. Cause: elementwise multiplication or swapped indices. Fix: assert dimensions and test each output as a row-column dot product.
Symptom: elimination divides by a tiny or zero pivot. Cause: no pivoting/tolerance policy. Fix: search lower rows, swap, and classify near-zero using documented scale-aware tolerance.
Symptom: every singular system is called inconsistent. Cause: singularity was equated with no solution. Fix: compare coefficient and augmented rank and expose free variables.

Definition of Done

Fundamental matrix operations enforce shape contracts and pass reference tests.
Elimination uses pivoting and produces a replayable row-operation trace.
Systems are correctly classified as unique, infinite, or inconsistent.
Solutions include residual checks; parameterized solutions include free variables.
Visual examples match algebraic classifications.
A perturbation case demonstrates conditioning/sensitivity and is documented.

Project 21: Matrix Transformations and 3D Rendering Engine

Difficulty: Advanced
Time: 3-5 weeks
Language: Python with NumPy and Pygame or Matplotlib
Prerequisites: Projects 16 and 20
Merged from: HS P10; Prog P02; ML-Math P05; ML-Found P02

What You Will Build

Build a staged graphics engine that makes every matrix operation visible. Begin with a 2D image or polygon editor supporting translation, scale, rotation, reflection, shear, and a four-point perspective warp computed as a homography. Add homogeneous coordinates so affine and projective transformations compose into matrices, then graduate to a 3D wireframe pipeline: model-space vertices, world and camera transforms, perspective projection, clipping, and screen coordinates. The interface must display the current matrix and transformation order beside the rendered result. Include both nearest-neighbor and bilinear image resampling, plus a rotatable cube or loaded mesh. This combines the sources’ image-transformation lab, matrix visualizer, and renderer into one coherent artifact rather than several nearly identical demos.

Real World Outcome

A learner can load an image, drag its four corners onto a photographed sign or screen, and see the source rectified through a computed 3x3 homography. The same interface can load a cube, issue rotate, scale, translate, and camera commands, and show before/after views with the composed 3x3 or 4x4 matrix. A saved diagnostic frame shows correspondence points, axes, transformed vertices, clipped edges, projection parameters, and the exact pipeline stage where each coordinate lives.

Core Question

How can one matrix pipeline turn mathematical coordinates into a correctly transformed image on a flat screen, and why does changing multiplication order change the picture?

Concepts You Must Understand First

Linear maps and basis vectors: a matrix’s columns show where basis directions move. See Paul Orland, Math for Programmers, chs. 4-6.
Homogeneous coordinates: translation becomes matrix multiplication by appending a coordinate. See Gabriel Gambetta, Computer Graphics from Scratch, transformation chapters.
Composition order: column-vector pipelines apply the rightmost transform first; matrix multiplication is not commutative.
Perspective projection: divide camera-space x and y by depth, while guarding the near plane. See Dunn and Parberry, 3D Math Primer for Graphics and Game Development.
Image resampling: inverse-map destination pixels to source pixels; bilinear interpolation reduces holes and jaggedness.
Projective homographies: four non-collinear point correspondences determine a planar perspective warp up to scale; degenerate point layouts must be rejected.

Build Milestones

Draw axes and transform 2D points with tested rotation, scale, reflection, and shear matrices.
Add 3x3 homogeneous translation, composition, inverse mapping, and image interpolation.
Solve a four-point homography, inverse-map every destination pixel through it, and rectify a perspective-distorted planar image.
Represent a 3D mesh as vertices and edges; implement 4x4 model, view, and projection matrices.
Clip unsafe geometry, perform the perspective divide, map normalized coordinates to the viewport, animate a cube, and expose every pipeline stage in an inspector.

Hints in Layers

Start with a triangle whose coordinates you can calculate by hand; draw its local and transformed axes.
Keep model, view, and projection separate until display time, and label vector conventions explicitly.
For affine and homography image warps, iterate destination pixels and apply the inverse transform; reject nearly collinear four-point inputs. For 3D, clip vertices at or behind the near plane before dividing by depth.

Common Pitfalls and Debugging

Symptom: rotate-then-translate behaves like translate-then-rotate. Cause: matrices were composed in the wrong order. Fix: test two deliberately noncommuting transforms on one point and document the convention.
Symptom: transformed images have gaps. Cause: forward pixel mapping. Fix: inverse-map every destination pixel and interpolate.
Symptom: geometry explodes or flips near the camera. Cause: perspective division by zero or negative depth. Fix: inspect camera-space z and clip against a positive near plane.

Definition of Done

Unit tests verify identity, inverse, composition order, and known 90-degree rotations.
The 2D tool transforms and saves both polygons and raster images.
Bilinear interpolation visibly improves a rotated checkerboard over nearest-neighbor sampling.
A four-point homography rectifies a perspective-distorted test image, rejects degenerate correspondences, and matches its corner targets within tolerance.
A cube can be translated, rotated, camera-transformed, clipped, and perspective-projected.
The UI exposes matrices and intermediate coordinates, not only the final picture.
A short report explains coordinate conventions and three diagnosed failure cases.

Project 22: Matrix Cipher Encoder

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python
Prerequisites: Projects 10 and 20
Source: HS P06

What You Will Build

Build a Hill-cipher-style educational encoder that turns text blocks into numeric vectors, multiplies them by a key matrix modulo an alphabet size, and reverses the process with a modular matrix inverse. The program must generate or accept keys, reject keys that cannot be inverted in the chosen modulus, preserve a documented policy for spaces and punctuation, pad incomplete blocks, and show the numeric work for at least one block. Include a key-analysis command reporting determinant, greatest common divisor, modular inverse, and round-trip status. This is not presented as secure modern cryptography; its purpose is to make matrix invertibility, modular arithmetic, and reversible transformations concrete.

Real World Outcome

Running the CLI on MEET AT NINE produces ciphertext, a block-by-block transformation trace, and a successful decrypted message. Supplying a bad key produces a precise diagnostic such as “determinant 6 shares factor 2 with modulus 26; no inverse exists” instead of corrupt output.

Core Question

What exact mathematical property makes a matrix transformation reversible when arithmetic wraps around a finite alphabet?

Concepts You Must Understand First

Modular arithmetic: addition and multiplication occur in residue classes. Revisit Project 10 and modular inverse via the extended Euclidean algorithm.
Determinants and invertibility: over modulus m, nonzero determinant is insufficient; gcd(det(K), m) must equal 1.
Matrix-vector multiplication: text blocks must follow one consistent row- or column-vector convention. See Lay et al., Linear Algebra and Its Applications.
Adjugate inverse in 2D: for a small matrix, multiply the adjugate by the modular inverse of its determinant.
Encoding contracts: alphabet, case folding, padding, and unsupported characters must be deterministic.

Build Milestones

Define a text-to-number mapping and prove its round trip independently of encryption.
Encrypt fixed-size blocks with a 2x2 key and display every intermediate vector.
Implement determinant and modular inverse checks, then derive the inverse key matrix.
Decode arbitrary-length messages with deterministic padding removal and punctuation policy.
Add seeded key generation plus property tests over many valid messages and keys.

Hints in Layers

Begin with alphabet size 26 and a known invertible key; verify one block by hand.
Normalize every intermediate matrix entry modulo m, including negative values in the inverse.
Generate candidates until the determinant is coprime to m; never assume a random matrix is usable.

Common Pitfalls and Debugging

Symptom: encryption works but decryption returns nonsense. Cause: key determinant has no modular inverse or vector orientation changed. Fix: print K_inv @ K mod m and require identity before accepting the key.
Symptom: only some messages round-trip. Cause: padding or punctuation handling is ambiguous. Fix: specify an encoded length or reversible escape policy and test boundary-sized messages.
Symptom: inverse entries are negative or too large. Cause: ordinary inverse arithmetic leaked into modular arithmetic. Fix: reduce every result modulo the alphabet size.

Definition of Done

Valid keys satisfy K_inv @ K ≡ I (mod m) in tests.
Invalid keys are rejected with determinant and gcd diagnostics.
Empty, short, exact-block, and multi-block messages behave deterministically.
The CLI shows one transparent numerical encryption/decryption trace.
At least 100 seeded property cases round-trip correctly.
Documentation states clearly why this historical cipher is not secure today.

Project 23: Dot-Product Recommendation Engine

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python with NumPy and Pandas
Prerequisites: Projects 12, 16, and 20
Source: ML-Found P03

What You Will Build

Build a movie recommender from ratings or explicit movie-feature vectors. Represent users and items in the same vector space, implement dot product, vector norm, cosine similarity, and top-k ranking yourself, and then compare user-user and item-item recommendations. Handle missing ratings deliberately rather than silently treating every absence as dislike. The tool should explain each recommendation by showing the strongest aligned features or the most similar contributing users. Use a small hand-checkable fixture first, then a MovieLens-style dataset. Include popularity and random baselines so a plausible-looking list is not mistaken for proof that the mathematics works.

Real World Outcome

After a user rates a few titles, the CLI returns five unseen movies with similarity scores and explanations such as “recommended because your science-fiction/action vector aligns with users 17 and 42.” An evaluation report compares precision@k or hit rate against baselines and identifies cold-start and sparse-data cases.

Core Question

When does vector alignment represent meaningful preference similarity, and when do magnitude, missing values, or popularity distort the dot product?

Concepts You Must Understand First

Dot product: multiply aligned components and sum; geometrically it combines magnitudes and angle. See Orland, Math for Programmers, vector chapters.
Cosine similarity: normalization isolates direction from magnitude. See Manning, Raghavan, and Schütze, Introduction to Information Retrieval, ch. 6.
Feature vectors: every dimension needs a defined scale and meaning. See Géron, Hands-On Machine Learning, ch. 2.
Sparse matrices: unobserved ratings are not necessarily zeros; choose overlap and imputation rules explicitly.
Ranking evaluation: assess only unseen held-out preferences and compare with simple baselines.

Build Milestones

Implement and test dot product, norm, cosine similarity, and stable top-k selection without library similarity helpers.
Create a tiny ratings matrix whose expected nearest neighbors and recommendations can be computed by hand.
Load a real ratings dataset, filter already-seen items, and build user-user and item-item recommenders.
Add explanations and policies for zero vectors, low overlap, cold-start users, and ties.
Hold out known ratings and compare recommendation quality with popularity and random baselines.

Hints in Layers

Validate the math on three users and five movies before loading a large dataset.
Compute similarity only on co-rated dimensions or document a centered/imputed alternative; require a minimum overlap.
To explain a score, retain per-neighbor or per-feature contributions before summing them.

Common Pitfalls and Debugging

Symptom: prolific raters appear similar to everyone. Cause: raw dot products reward magnitude. Fix: compare cosine similarity and mean-centered ratings.
Symptom: already watched movies dominate results. Cause: seen items were not masked. Fix: exclude them before top-k selection.
Symptom: offline accuracy looks perfect. Cause: evaluation used training ratings. Fix: create a user-stratified holdout before computing recommendations.

Definition of Done

Core vector operations match hand calculations and NumPy references.
Recommendations exclude rated items and are deterministic under a fixed seed.
Missing values, low overlap, ties, and zero norms have explicit policies.
Every result includes a human-readable contribution explanation.
Held-out ranking quality beats at least one baseline.
The report names limitations including sparsity, popularity bias, and cold start.

Project 24: Eigenvalue and Eigenvector Explorer

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Project 20
Source: ML-Math P06

What You Will Build

Build an interactive explorer that applies a 2x2 matrix repeatedly to points, vectors, and a grid while highlighting invariant directions. Users can enter a matrix, drag candidate vectors, and watch whether direction is preserved while length and orientation change. Implement the characteristic polynomial for 2x2 matrices and power iteration for a dominant eigenpair, then compare results with a trusted numerical library. Show real, repeated, complex, and defective cases without pretending every matrix has a full real eigenbasis. Residuals and iteration history must be visible so eigenpairs are validated rather than accepted because a plotted arrow “looks right.”

Real World Outcome

For a selected matrix, the tool displays eigenvalues, normalized eigenvectors, residual norms ||Av - λv||, and an animation of repeated transformation. A convergence panel graphs power-iteration estimates, while matrices with rotations or repeated eigenvalues produce an honest explanation of why real directions or convergence may be absent.

Core Question

Which directions survive a linear transformation without turning, and how can repeated multiplication reveal them?

Concepts You Must Understand First

Eigenpair equation: Av = λv, with nonzero v; λ scales or reverses an invariant direction. See Axler, Linear Algebra Done Right.
Characteristic polynomial: det(A - λI) = 0 identifies eigenvalue candidates for small matrices.
Geometric multiplicity: repeated eigenvalues do not guarantee enough independent eigenvectors.
Power iteration: repeated normalized multiplication tends toward a uniquely dominant eigenvector when its assumptions hold.
Residual verification: a small ||Av - λv|| is stronger evidence than matching rounded output.

Build Milestones

Transform a grid and arbitrary vectors while preserving original and transformed views.
Solve 2x2 characteristic equations and compute corresponding real eigenvectors.
Animate eigenvectors and ordinary vectors through repeated applications of A.
Implement power iteration with normalization, sign-insensitive convergence, and iteration diagnostics.
Catalog symmetric, diagonal, shear, rotation, repeated-eigenvalue, and defective examples.

Hints in Layers

Begin with diagonal matrices, where eigenvectors are visible coordinate axes.
Compare directions using absolute dot product because v and -v represent the same eigendirection.
Separate “no real eigenvector,” “nonunique dominant magnitude,” and “defective matrix” diagnostics; they are different phenomena.

Common Pitfalls and Debugging

Symptom: power iteration alternates signs forever. Cause: eigenvector sign ambiguity or a negative dominant eigenvalue. Fix: compare |dot(v_new, v_old)| or residuals.
Symptom: normalization yields NaN. Cause: the transformed vector became zero. Fix: detect near-zero norm and restart or report the null-space case.
Symptom: rotation matrix produces bogus real arrows. Cause: complex eigenvalues were forced into real arithmetic. Fix: detect negative discriminant and explain the complex pair.

Definition of Done

Known diagonal and symmetric matrices produce correct eigenpairs.
Every reported pair includes a residual norm and normalized vector.
The animation distinguishes invariant from turning directions.
Power iteration exposes estimates, convergence tolerance, and failure reason.
Complex, repeated, and defective examples are handled explicitly.
Results are cross-checked against NumPy within documented tolerance.

Project 25: PCA Image Compressor

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with NumPy, Pillow, and Matplotlib
Prerequisites: Projects 12, 20, and 24
Merged from: ML-Math P07; ML-Found P04

What You Will Build

Build Principal Component Analysis from the centering step through image reconstruction. First prove the pipeline on a small tabular dataset: center features, construct the covariance matrix, find principal directions, sort them by eigenvalue, project, and reconstruct. Then treat image rows, patches, or a collection of similarly sized images as observations and compress them by retaining k components. Provide controls for component count or target explained variance, and report reconstruction error and a realistic storage estimate. Implement power iteration plus deflation or a symmetric eigensolver exercise, then compare with a trusted SVD-based reference. The final artifact should teach what information is preserved, not merely call a library PCA class.

Real World Outcome

Given an image and a target such as 95% retained variance, the tool displays the original, reconstructions at several k values, a scree/cumulative-variance plot, mean-squared error, and approximate compression ratio. A second demo projects a labeled dataset into two dimensions and reveals which structure survives.

Core Question

How can a change of basis preserve most variation with fewer coordinates, and what exactly is lost when low-variance directions are discarded?

Concepts You Must Understand First

Centering: PCA describes variation around the mean; failing to center changes the first direction.
Covariance matrix: it encodes pairwise feature variation and is symmetric positive semidefinite. See Deisenroth et al., Mathematics for Machine Learning, ch. 6.
Eigenvectors as axes: covariance eigenvectors are orthogonal principal directions; eigenvalues measure captured variance.
Projection/reconstruction: scores are coordinates in the new basis; multiplying back and restoring the mean approximates the input.
PCA versus SVD: SVD is usually the numerically safer route. See Géron, Hands-On Machine Learning, dimensionality-reduction chapter.

Build Milestones

Implement centering, covariance, sorted eigenpairs, projection, and reconstruction on 2D synthetic data.
Add explained-variance ratios and verify orthogonality and total-variance identities.
Implement power iteration/deflation for learning value and compare with eigh or SVD.
Compress grayscale images, then support color by a documented channel or patch strategy.
Build interactive k/variance controls and export a quantitative quality report.

Hints in Layers

Preserve the training mean and reuse it during projection and reconstruction.
For covariance, define whether observations are rows and use a consistent (n-1) denominator.
Clip tiny negative eigenvalues caused by rounding only after recording them; large negatives indicate a bug.

Common Pitfalls and Debugging

Symptom: first component mostly reproduces image brightness. Cause: data was not centered. Fix: subtract and later restore the mean.
Symptom: reconstruction dimensions are transposed. Cause: observations/features conventions changed. Fix: annotate every matrix shape and test a tiny fixture.
Symptom: “compression ratio” is misleading. Cause: basis storage was ignored. Fix: count retained scores, components, mean, and numeric precision.

Definition of Done

PCA is implemented without a high-level PCA API.
Principal directions are ordered, orthonormal, and numerically verified.
Explained-variance ratios sum to approximately one.
Increasing k never materially worsens reconstruction error.
The UI compares quality, retained variance, and honest storage cost.
A reference SVD/eigendecomposition confirms results within tolerance.

Phase 5 — Calculus and Numerical Computing (Projects 26-32)

Project 26: Numerical Derivative and Tangent Explorer

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Projects 5-6
Merged from: HS P07; ML-Found P05

What You Will Build

Build an interactive calculus laboratory that evaluates a safe set of functions, estimates derivatives using forward, backward, and central differences, and draws the tangent line at a movable point. Plot the original function and derivative together; identify numerical critical-point candidates and classify them using nearby sign changes rather than labels alone. Add an error experiment that sweeps step size h across many orders of magnitude and compares estimates with known analytic derivatives. This reveals both truncation error at large h and floating-point cancellation at tiny h. Include continuous, nonsmooth, and discontinuous examples so the program learns to report when “a derivative value” is not trustworthy.

Real World Outcome

For f(x)=x^3-3x+1, clicking the graph displays the point, secant sample points, central-difference slope, tangent equation, and critical points. A log-scale chart shows derivative error versus h and identifies a useful range. At abs(x) near zero, the tool warns that left and right estimates disagree.

Core Question

How does a limit become a computable slope, and why can making the finite-difference step smaller eventually make the estimate worse?

Concepts You Must Understand First

Difference quotient and limit: derivatives are limiting local rates, not merely symbolic rules. See Stewart, Calculus, derivative chapters.
Tangent as local linear model: f(a)+f'(a)(x-a) approximates nearby values.
Forward versus central difference: central difference cancels leading error and is typically more accurate for smooth functions.
Continuity and differentiability: corners, jumps, and domain boundaries need one-sided or failure diagnostics.
Floating-point cancellation: subtracting nearly equal function values loses significant digits. See Orland, Math for Programmers, ch. 8.

Build Milestones

Implement three finite-difference estimators with explicit function-domain errors.
Plot f, estimated f’, secant points, and a tangent line controlled by mouse or CLI input.
Sweep h and compare each method against analytic references on polynomials, sine, exponential, and logarithm.
Locate derivative sign changes and classify candidate minima/maxima with local evidence.
Add diagnostics for boundaries, discontinuities, corners, overflow, and unstable estimates.

Hints in Layers

Test x^2 at x=3, where the exact slope is 6 and hand calculations are easy.
Compare central estimates at h and h/2; large disagreement is a practical warning signal.
Estimate left and right slopes separately before declaring a derivative near suspicious points.

Common Pitfalls and Debugging

Symptom: error grows as h approaches zero. Cause: subtractive cancellation. Fix: graph error over h and choose a scale-aware default.
Symptom: a corner is reported as a smooth critical point. Cause: only a symmetric estimate was checked. Fix: compare one-sided slopes.
Symptom: tangent line is vertically displaced. Cause: the line formula omitted f(a) or used x instead of x-a. Fix: require it to pass exactly through (a,f(a)).

Definition of Done

Forward, backward, and central estimates pass known-function tests.
The tangent visualization updates and passes through the selected point.
An error-versus-h experiment demonstrates both error regimes.
Nonsmooth and domain-boundary cases produce meaningful warnings.
Critical points are supported by derivative sign evidence.
Outputs are reproducible and compared with analytic references.

Project 27: Symbolic Differentiation Engine

Difficulty: Intermediate
Time: 2-3 weeks
Language: Python
Prerequisites: Projects 5 and 26
Source: ML-Math P08

What You Will Build

Build a small computer-algebra engine that tokenizes and parses expressions into an abstract syntax tree, differentiates the tree recursively, simplifies the result, prints it with correct precedence, and verifies it numerically. Support constants, variables, sums, products, quotients, integer powers, and common unary functions such as sine, cosine, exponential, and logarithm. Preserve a distinction between parsing, transformation, simplification, evaluation, and display so bugs can be localized. The engine must show an optional derivation trace naming each rule applied. Verification samples legal domain points and compares the symbolic derivative with Project 26’s central differences, turning symbolic manipulation into a testable software contract.

Real World Outcome

Entering sin(x*x) + x^3/log(x) returns a readable derivative, a tree/step trace, simplification before-and-after, and a verification table over valid positive x values. Malformed input and illegal domains produce deterministic parser or evaluator diagnostics rather than Python exceptions.

Core Question

How can differentiation rules be represented as local tree transformations that compose correctly for arbitrarily nested expressions?

Concepts You Must Understand First

Expression trees: operators are internal nodes and literals/variables are leaves. See Abelson and Sussman, Structure and Interpretation of Computer Programs, symbolic-data sections.
Grammar and precedence: parsing must distinguish unary minus, grouping, exponentiation, and function calls.
Derivative rules: linearity, product, quotient, power, and chain rules. See Stewart, Calculus.
Structural recursion: differentiate children and combine them according to node type.
Safe simplification: identities like x+0=x are local; aggressive algebra can change domains or divide by zero.

Build Milestones

Define immutable AST node types, a tokenizer, and a precedence-aware parser with round-trip tests.
Implement evaluation with variable environments and clear domain errors.
Add recursive derivative rules and a trace recording which rule produced each subtree.
Implement conservative simplifications such as constant folding, neutral elements, and zero products.
Numerically validate generated derivatives over seeded legal samples and report counterexamples.

Hints in Layers

Begin with prefix-constructed ASTs before writing the parser; prove transformation logic independently.
Treat each node as owning evaluate, differentiate, and format, or use exhaustive visitors.
For verification, scale tolerance with function magnitude and reject samples too close to domain boundaries.

Common Pitfalls and Debugging

Symptom: -x^2 differentiates as 2x. Cause: unary-minus precedence was parsed incorrectly. Fix: add explicit grammar tests and print the AST.
Symptom: nested functions miss factors. Cause: chain rule omitted the inner derivative. Fix: test sin(x^2) and trace each recursive result.
Symptom: simplification changes valid domains. Cause: cancellation such as x/x -> 1 ignored x=0. Fix: keep simplification conservative or track assumptions.

Definition of Done

Parsing and pretty-printing preserve structure for a representative expression suite.
Every supported node differentiates recursively with a named rule trace.
Simplification reduces obvious clutter without unsafe cancellations.
Numerical checks agree at many seeded valid points within tolerance.
Syntax, unknown-function, and domain errors are deterministic.
The implementation does not delegate differentiation to SymPy or another CAS.

Project 28: Numerical Polynomial Root Finder

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python
Prerequisites: Projects 3, 5, and 26-27
Source: ML-Math P03

What You Will Build

Build a root-finding workbench for polynomial and user-selected smooth functions. Implement bisection as the dependable bracketed baseline and Newton’s method as the fast derivative-based method, with an optional secant fallback. Every solver returns a structured history containing iterates, function values, interval widths or step sizes, residual, stop reason, and iteration count. Plot the function, current bracket, tangent steps, and convergence curve. Detect invalid brackets, flat derivatives, cycles, divergence, non-finite values, and roots of even multiplicity that do not change sign. Compare methods on simple roots, repeated roots, closely spaced roots, and functions whose starting points send Newton in the wrong direction.

Real World Outcome

For a polynomial such as x^3-2x-5, the CLI prints side-by-side iteration tables and highlights the converged root on a plot. For a poor Newton starting point, it reports why the method failed or switches to a maintained bracket. Final output includes both |f(x)| and an uncertainty/step measure rather than only rounded x.

Core Question

How can an algorithm know that an approximate root is trustworthy, and what tradeoff separates guaranteed bracketing from fast local iteration?

Concepts You Must Understand First

Continuity and the intermediate value theorem: opposite endpoint signs bracket at least one root.
Bisection: halving a valid bracket gives predictable error reduction.
Newton iteration: tangent-line intersection gives x_next=x-f/f'; convergence is local, not guaranteed.
Multiplicity: repeated roots flatten the curve and may not create a sign change.
Residual versus location error: small |f(x)| does not always imply small x error for ill-conditioned roots. See Sedgewick and Wayne, Algorithms, numerical methods discussions.

Build Milestones

Represent polynomials robustly and evaluate values and derivatives, including edge cases.
Implement bisection with bracket invariants and a formal interval-width bound.
Implement Newton and secant iterations with derivative and non-finite guards.
Add a hybrid mode that uses Newton only when its proposal stays safely inside a bracket.
Visualize histories and build a failure-case test catalog.

Hints in Layers

Make solver output a record rather than a bare float; termination evidence is part of the result.
Use both residual and step/bracket tolerance, plus a maximum iteration count.
In hybrid mode, preserve the sign-changing bracket after every accepted step.

Common Pitfalls and Debugging

Symptom: bisection “converges” without a root. Cause: endpoints never bracketed a sign change. Fix: validate before iteration and report even-multiplicity limitations.
Symptom: Newton jumps to infinity. Cause: derivative is nearly zero or start is outside the attraction basin. Fix: guard slope size and fall back to bisection.
Symptom: solver loops despite stable printed digits. Cause: exact equality was used. Fix: define residual, step, and iteration tolerances.

Definition of Done

Bisection preserves its bracket invariant and reports an error bound.
Newton converges rapidly on well-behaved simple roots.
Hybrid mode recovers from unsafe Newton proposals.
Repeated-root, no-real-root, invalid-bracket, and flat-derivative cases are documented.
Iteration plots and tables expose convergence behavior.
Final answers include residual, uncertainty measure, and stop reason.

Project 29: Numerical Integration and Area Estimator

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Project 26
Merged from: HS P08; ML-Math P10

What You Will Build

Build an integration visualizer that estimates definite integrals using left/right Riemann sums, midpoint rectangles, trapezoids, and Simpson’s rule, then compares their convergence as the partition count grows. Shade signed area above and below the axis and keep signed integral separate from geometric area. Allow piecewise interval splitting around discontinuities and add a simple adaptive refinement mode that subdivides regions whose coarse and fine estimates disagree. Test against functions with known antiderivatives, then use the engine on distance-from-velocity and probability-density examples. Every result includes method, interval, evaluation count, estimated value, reference error when available, and a convergence table.

Real World Outcome

For a selected function and bounds, the app draws the curve and chosen rectangles/trapezoids, animates refinement, and plots error versus evaluation count for each method. It can demonstrate cancellation across the x-axis, reject an improper singular interval unless explicitly handled, and export a reproducible comparison report.

Core Question

How does accumulated local contribution become area or total change, and how can refinement provide evidence that a numerical integral is reliable?

Concepts You Must Understand First

Definite integral: the limit of signed sums represents accumulation. See Stewart, Calculus, integration chapters.
Quadrature rules: midpoint, trapezoid, and Simpson methods approximate local shape differently.
Signed versus geometric area: negative contributions cancel in an integral but not in total geometric area.
Convergence and error: comparing n and 2n is evidence, not a proof, but exposes instability.
Adaptive subdivision: spend evaluations where curvature or irregularity makes local error larger. See Press et al., Numerical Recipes.

Build Milestones

Implement equal-width Riemann, midpoint, and trapezoid sums with shared interval validation.
Add Simpson’s rule, enforcing its even-subinterval requirement.
Visualize partitions and maintain distinct signed-integral and geometric-area modes.
Generate convergence/error plots on polynomial, trigonometric, and exponential references.
Implement recursive or iterative adaptive refinement with evaluation and depth limits.

Hints in Layers

Verify constants and linear functions first; trapezoids should integrate linear functions exactly up to rounding.
Cache function evaluations so adaptive refinement does not repeatedly sample the same points.
Split at known discontinuities and domain boundaries rather than allowing one panel to cross them blindly.

Common Pitfalls and Debugging

Symptom: area becomes unexpectedly small. Cause: positive and negative regions canceled. Fix: label signed integral and optionally integrate |f| for geometric area.
Symptom: Simpson’s output is wrong only for some n. Cause: an odd subinterval count. Fix: reject or normalize n explicitly.
Symptom: refinement never stabilizes near a singularity. Cause: the method assumes a bounded regular integrand. Fix: detect non-finite samples, split, transform, or report unsupported improper integration.

Definition of Done

Four quadrature methods match known test integrals within expected tolerances.
Visual partitions correspond exactly to the computed samples and weights.
Signed and geometric area are clearly distinguished.
Convergence plots compare error against evaluation cost.
Adaptive mode has tolerance, evaluation, and recursion limits.
Discontinuous and non-finite functions fail with actionable diagnostics.

Project 30: Numerical Stability and Conditioning Lab

Difficulty: Expert
Time: 2-3 weeks
Language: Python with NumPy
Prerequisites: Projects 1, 20, 26, and 28-29
Source: ML-Math P24

What You Will Build

Build a stress laboratory that runs mathematically equivalent formulas across benign and adversarial inputs, measures their disagreement, and explains the failure mechanism. Include catastrophic cancellation in the quadratic formula and finite differences, overflow/underflow in exponentials and probabilities, unstable summation, variance formulas, nearly singular linear systems, and integration/root-finding tolerances. Implement stable alternatives such as a cancellation-resistant quadratic formula, logsumexp, Welford variance, pairwise or compensated summation, scaling, and residual-based checks. Generate condition estimates by perturbing inputs and measuring relative output change. Each experiment should distinguish problem conditioning from algorithmic stability: sometimes no implementation can recover information the input has already lost.

Real World Outcome

The lab produces an HTML/Markdown report with tables of naive versus stable answers, relative/absolute error, perturbation amplification, runtime, and a diagnosis. A heatmap for a nearly singular matrix or polynomial shows regions where small input changes cause large output changes, while regression tests preserve every discovered counterexample.

Core Question

When a computation returns a wrong-looking answer, is the mathematical problem inherently sensitive, or did the chosen algorithm unnecessarily amplify ordinary floating-point error?

Concepts You Must Understand First

Floating-point model: numbers have finite precision, range, rounding, and special values; machine arithmetic is not real arithmetic.
Forward and backward error: ask whether the answer is close, or whether it exactly solves a nearby problem.
Conditioning: relative input perturbations may be amplified by the problem itself. See Trefethen and Bau, Numerical Linear Algebra.
Cancellation and scale: subtracting close values destroys significant digits; overflow can occur before a ratio becomes small.
Residuals: a small residual helps validate equations but must be interpreted with scale and condition.

Build Milestones

Build a reproducible experiment harness using float32/float64 and high-precision or symbolic references.
Add cancellation, summation, variance, exponential/probability, derivative, root, and integration cases.
Add ill-conditioned matrix systems and empirical perturbation-based condition estimates.
Implement stable counterparts and explain why each rewrite helps.
Export a ranked report and preserve minimal adversarial inputs as regression tests.

Hints in Layers

Use decimal or mpmath only as a reference oracle, not as the “fix” for every algorithm.
Sweep input magnitude logarithmically and report both relative and absolute error.
To separate conditioning from stability, compare several algorithms against the same high-precision problem and perturb the exact inputs independently.

Common Pitfalls and Debugging

Symptom: relative error is infinite near zero. Cause: the denominator is meaningless at that scale. Fix: report absolute error or a mixed tolerance.
Symptom: softmax becomes NaN. Cause: exponentials overflow. Fix: subtract the maximum before exponentiation and use log-domain computations.
Symptom: solver residual is tiny but solution varies wildly. Cause: the system is ill-conditioned. Fix: report condition estimate and perturbation sensitivity alongside residual.

Definition of Done

The suite contains at least six distinct numerical failure families.
Every experiment has a trusted reference and deterministic input generation.
Stable alternatives materially improve error on documented adversarial cases.
Reports distinguish conditioning, algorithmic instability, and tolerance mistakes.
Float precision and input scale can be swept reproducibly.
Counterexamples become regression tests with clear diagnostic assertions.

Project 31: Physics Engine and Numerical Dynamics Lab

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with Pygame or Matplotlib
Prerequisites: Projects 16, 21, 26, and 29-30
Merged from: Prog P04; ML-Found P08

What You Will Build

Build a 2D particle physics engine where forces produce acceleration, numerical integration updates velocity and position, and collisions and springs make stability visible. Implement fixed-step explicit Euler, semi-implicit Euler, and Verlet integration behind the same interface. Support gravity, drag, Hooke-law springs, circles colliding with boundaries and one another, and a pause/step inspector. Track kinetic, gravitational, and spring potential energy, momentum, constraint error, and simulation time. Provide controlled experiments that vary time step, spring stiffness, and integrator so the learner can see Euler’s energy drift and Verlet’s different behavior. Keep rendering decoupled from state updates so frame-rate changes do not silently alter physics.

Real World Outcome

An interactive window lets the user add balls and springs, toggle gravity, choose an integrator, and slow or single-step the simulation. Live plots compare total energy and momentum. A benchmark scene exports a table showing when each integrator remains stable or explodes as time step and stiffness change.

Core Question

How can continuous laws of motion be approximated by discrete updates without allowing numerical error to overwhelm the physical behavior?

Concepts You Must Understand First

Newton’s second law: net force gives acceleration a=F/m; force accumulation must reset every step. See Serway and Jewett, Physics for Scientists and Engineers.
State derivatives: velocity is position’s derivative and acceleration is velocity’s derivative.
Numerical integration: Euler and Verlet make different assumptions about how state changes during a step. See Millington, Game Physics Engine Development, ch. 3.
Hooke’s law and damping: spring force depends on displacement; damping opposes relative motion.
Impulse collision response: project relative velocity onto the collision normal and apply restitution while correcting overlap carefully.

Build Milestones

Separate vector math, bodies, force generators, integrators, collision detection, and rendering.
Simulate free fall with fixed-step Euler variants and compare against the analytic trajectory.
Add springs and Verlet integration; plot energy drift across integrators and time steps.
Detect and resolve circle-wall and circle-circle collisions with restitution and overlap correction.
Build parameter-sweep scenarios and an inspector for forces, velocities, contacts, and energy.

Hints in Layers

Validate one falling particle with no rendering; its position should approach y0+v0t+½at².
Accumulate forces first, integrate all bodies second, and resolve contacts in a clearly documented order.
Use an accumulator for a fixed physics step independent of display FPS; cap catch-up work to avoid a spiral of death.

Common Pitfalls and Debugging

Symptom: motion changes with monitor frame rate. Cause: variable render delta drives integration. Fix: use a fixed simulation step and interpolate rendering if needed.
Symptom: springs gain energy and explode. Cause: explicit Euler plus a stiff spring or large dt. Fix: compare semi-implicit/Verlet and reduce step size.
Symptom: objects stick or jitter at boundaries. Cause: repeated penetration and impulse application. Fix: separate positional correction from velocity impulse and inspect contact normals.

Definition of Done

At least three integrators share one tested state-update interface.
Free-fall error is measured against an analytic reference.
Gravity, drag, springs, boundaries, and circle collisions work interactively.
Energy and momentum diagnostics expose drift and collision behavior.
Simulation results are independent of rendering frame rate.
A parameter sweep documents stable and unstable time-step regimes.

Project 32: Epidemic Growth Simulator

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Projects 6, 12, and 26
Source: HS P11

What You Will Build

Build a discrete SIR-style epidemic simulator with susceptible, infected, and recovered compartments. Start with exponential growth to reveal why a constant percentage produces nonlinear curves, then add finite population, contact/infection rate, recovery rate, and optional vaccination or intervention schedules. Update the coupled recurrences from the same previous state so population is conserved. Plot all compartments, daily new infections, cumulative cases, and effective reproduction intuition. Provide seeded stochastic and deterministic modes, parameter sweeps, and scenarios such as delayed distancing or increased recovery. Frame outputs as model behavior under assumptions, not forecasts or medical advice; every run must display limitations and units.

Real World Outcome

The dashboard compares baseline and intervention scenarios, reports peak infected count and day, final attack fraction, and verifies S+I+R=N at every step. A heatmap shows how peak size or timing changes across transmission and recovery rates, with parameters and random seed exported alongside plots.

Core Question

How can simple coupled rate equations create rapid growth, a peak, and decline, and which assumptions control whether those curves resemble reality?

Concepts You Must Understand First

Exponential versus logistic effects: unrestricted proportional growth differs from growth limited by a finite susceptible pool.
Compartment recurrence: flows leaving one compartment enter another; use a consistent time index.
Rates and units: transmission and recovery parameters depend on the chosen time step.
Equilibria and thresholds: infection grows initially when effective transmission exceeds recovery.
Model uncertainty: parameter sensitivity and structural assumptions matter more than precise-looking curves. See Guttag, Introduction to Computation and Programming Using Python, modeling chapters.

Build Milestones

Implement and graph pure exponential growth with doubling-time checks.
Add deterministic SIR recurrences and assert nonnegative compartments and population conservation.
Compute observable summaries: incidence, prevalence, peak timing, and final sizes.
Add interventions, vaccination, and a seeded stochastic transition option.
Build one- and two-parameter sweeps with side-by-side scenario reports.

Hints in Layers

Calculate all flows from copies of the old S, I, and R values before updating any compartment.
Clamp only tiny floating-point negatives; large negative populations indicate a step-size or formula bug.
Express intervention schedules as functions of time so scenarios remain reproducible and testable.

Common Pitfalls and Debugging

Symptom: total population drifts. Cause: flows were added or removed inconsistently. Fix: model each flow once and assert conservation every step.
Symptom: results change drastically when switching days to hours. Cause: rates were not rescaled with dt. Fix: attach units and convert rates explicitly.
Symptom: compartments become negative. Cause: one step moves more people than available. Fix: reduce dt or bound discrete transition counts.

Definition of Done

Exponential growth matches a hand-computed doubling example.
SIR runs preserve population and nonnegative state within tolerance.
Plots distinguish incidence, prevalence, and cumulative quantities.
At least three interventions can be compared reproducibly.
Parameter sweeps expose sensitivity rather than one “correct” curve.
Every report states assumptions, units, seed, and non-forecast limitations.

Phase 6 — Optimization and Classical Machine Learning (Projects 33-41)

Project 33: Gradient Descent Visualizer and Optimizer

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Projects 26 and 30
Merged from: ML-Math P09; NN P02; ML-Found P06

What You Will Build

Build an optimization workbench for scalar and two-parameter functions. Implement gradients from analytic callbacks and central differences, then implement batch gradient descent, momentum, and Adam without optimizer libraries. Animate parameter trajectories over curves, contours, and 3D loss surfaces while plotting loss, gradient norm, step norm, and effective learning rate. Include convex bowls, ill-conditioned ellipses, Rosenbrock’s valley, saddle points, and multimodal functions. Add stopping rules and divergence diagnostics rather than relying on a fixed iteration count. A comparison runner should launch algorithms from identical starts and seeds, making oscillation, slow progress, overshooting, and local-minimum dependence observable.

Real World Outcome

Selecting a function and starting point produces an animated path plus a convergence report. On an elongated quadratic, the tool visibly compares zig-zagging vanilla descent with momentum; on Rosenbrock, it shows sensitivity to learning rate. Failed runs are labeled with exploding loss, non-finite gradient, or stalled progress rather than drawn as successes.

Core Question

Why does stepping opposite the gradient usually reduce a function locally, and how do scale, curvature, and optimizer state determine whether those steps converge?

Concepts You Must Understand First

Gradient: the vector of partial derivatives points toward steepest local increase. See Orland, Math for Programmers, optimization chapters.
Local linear approximation: sufficiently small negative-gradient steps should reduce smooth objectives.
Curvature and conditioning: unequal curvature across directions causes oscillation and slow convergence.
Momentum and Adam: moving averages alter direction and per-coordinate scale. See Goodfellow et al., Deep Learning, ch. 8.
Stopping evidence: loss change, gradient norm, step norm, and iteration budget answer different questions.

Build Milestones

Implement finite-difference and analytic gradient adapters with numerical cross-checks.
Build vanilla descent with structured history, multiple stop rules, and non-finite guards.
Add contour/surface animation and convergence plots for standard test functions.
Implement momentum and Adam with correctly initialized state and bias correction.
Create reproducible hyperparameter sweeps and a comparative failure report.

Hints in Layers

Start with f(x)=x²; print every update before attempting 2D animation.
Keep optimizer state separate from parameters and objective; test the first two updates by hand.
Normalize visualization axes carefully: an apparently short path may hide huge loss changes.

Common Pitfalls and Debugging

Symptom: loss alternates or grows. Cause: learning rate is too large for local curvature. Fix: sweep rates logarithmically or add backtracking.
Symptom: Adam moves far on its first step. Cause: moment bias correction or epsilon placement is wrong. Fix: compare initial updates with the published equations.
Symptom: optimizer claims convergence on a plateau. Cause: only loss change was checked. Fix: also report gradient and step norms.

Definition of Done

Numerical and analytic gradients agree on smooth test functions.
Vanilla, momentum, and Adam share one reproducible optimizer interface.
Histories contain parameters, loss, gradients, steps, and stop reason.
At least four landscapes demonstrate distinct optimizer behavior.
Divergence, stagnation, and maximum-iteration exits are distinguished.
A report compares algorithms under identical starts and budgets.

Project 34: Linear Regression and Curve-Fitting Engine

Difficulty: Advanced
Time: 3-4 weeks
Language: Python with NumPy, Pandas, and Matplotlib
Prerequisites: Projects 5, 12, and 33
Merged from: HS P09; ML-Math P17; NN P03; ML-Found P07, ML-Found P13

What You Will Build

Build a from-scratch regression system that fits a line with closed-form least squares and gradient descent, then extends to multiple features and polynomial, exponential, or sinusoidal bases. Implement preprocessing, an explicit intercept, feature standardization fitted only on training data, train/validation/test separation, and regularized fitting. Report MSE, RMSE, MAE, R², parameter uncertainty intuition, and residual diagnostics. Compare the normal equation or least-squares solve with gradient descent, emphasizing conditioning and avoiding an explicit inverse where possible. Add a model-selection runner that compares basis families and degrees without selecting on the final test set. The final report must translate coefficients back into the problem’s units and state where extrapolation is unsafe.

Real World Outcome

Given housing or temperature data, the tool saves fitted curves, residual-versus-prediction and residual-distribution plots, learning curves, metrics by split, learned coefficients, and a plain-language interpretation. It can demonstrate underfitting with a line, overfitting with a high-degree polynomial, and improvement from regularization or a suitable sinusoidal basis.

Core Question

How does “best fit” become a precise optimization problem, and what evidence distinguishes a useful relationship from leakage, overfitting, or a misleadingly low training error?

Concepts You Must Understand First

Least squares: minimize the sum or mean of squared residuals. See Géron, Hands-On Machine Learning, ch. 4.
Design matrix: nonlinear basis functions can still be linear in learned coefficients.
Normal equation and conditioning: solve XᵀXw=Xᵀy, preferably with stable least-squares methods.
Gradient of MSE: derive parameter updates using matrix shapes and the chain rule.
Generalization diagnostics: residual patterns, held-out metrics, and learning curves expose model mismatch. See James et al., An Introduction to Statistical Learning.

Build Milestones

Fit simple linear regression by covariance formulas and verify against hand data.
Implement matrix-based multivariate fitting and gradient descent, including standardization and intercept handling.
Add polynomial and other basis transforms plus L2 regularization.
Create leakage-safe splits, metrics, residual plots, and learning curves.
Compare candidate families on validation data and produce one final untouched test report.

Hints in Layers

Use a noiseless line first; both solvers should recover parameters almost exactly.
Fit all preprocessing statistics on training data and apply them unchanged to later splits.
Watch the condition number as polynomial degree grows; scaling and QR/SVD solves are safer than explicit inversion.

Common Pitfalls and Debugging

Symptom: gradient descent diverges while closed form works. Cause: unscaled features or excessive learning rate. Fix: standardize training features and sweep rates.
Symptom: validation performance is implausibly strong. Cause: preprocessing or selection saw validation/test targets. Fix: audit the pipeline boundary and preserve an untouched test set.
Symptom: R² looks good but errors curve systematically. Cause: wrong functional form. Fix: inspect residuals and compare justified basis families.

Definition of Done

Closed-form and gradient solutions agree on well-conditioned fixtures.
Multiple features, intercept, basis transforms, and L2 regularization are supported.
Preprocessing and model selection are leakage-safe.
Reports include MSE, RMSE, MAE, R², residuals, and learning curves.
Underfitting, overfitting, and conditioning are demonstrated deliberately.
Coefficients and limitations are interpreted in real-world terms.

Project 35: PyMath Core Engine

Difficulty: Expert
Time: 4-6 weeks
Language: Python with NumPy, Matplotlib, and optional SymPy only as a verifier
Prerequisites: Projects 3, 5, 11, 20, 26-29, and 34
Merged from: HS PyMath Engine; HS High School Math Core Engine

What You Will Build

Build one modular command-line and notebook system integrating the strongest earlier high-school mathematics tools behind stable contracts. Its expression pipeline parses, normalizes, evaluates, and verifies inputs. Modules solve equations and inequalities, analyze functions, classify linear systems, inspect sequences and conics, compute numerical derivatives and integrals, fit regression models, and run seeded probability/modeling scenarios. A shared report layer renders formulas, tables, plots, assumptions, warnings, and verification traces. Do not paste earlier scripts into one file: define reusable expression, result, diagnostic, plotting, and scenario interfaces so modules compose. Include a scenario command that takes raw input through parsing, analysis, visualization, and an auditable report in one run.

Real World Outcome

The CLI supports commands such as solve, analyze-function, differentiate, integrate, fit-line, and simulate. A demo notebook processes a problem bundle and emits a single report containing exact/numeric answers, plots, residuals, convergence evidence, assumptions, and deterministic reproduction metadata. Invalid or unsupported problems return typed diagnostics rather than stack traces.

Core Question

How can many mathematical techniques become a coherent, trustworthy software system whose answers carry enough evidence to be checked independently?

Concepts You Must Understand First

Representation contracts: expressions, matrices, intervals, datasets, and scenarios require explicit shapes, domains, and units.
Separation of concerns: parsing, mathematics, verification, presentation, and persistence should not depend circularly on one another.
Exact versus approximate computation: preserve exact forms where possible and attach tolerances/error evidence to numeric results.
Verification strategies: substitution, residuals, invariants, convergence studies, and seeded replication validate different modules.
Model assumptions: probabilistic and regression outputs need units, data provenance, and limitation statements. References include Saha, Doing Math with Python; Downey, Think Stats; and Orland, Math for Programmers.

Build Milestones

Design common input, result, diagnostic, and verification records plus a plugin-like module boundary.
Integrate equation/function and matrix/geometry capabilities with regression tests from their original projects.
Integrate calculus operations with shared domain checks, convergence controls, and plotting.
Add seeded probability, regression, and scenario modules with consistent report output.
Build an end-to-end notebook/CLI demo, documentation, and a cross-module test suite.

Hints in Layers

Choose one vertical slice—parse a function, evaluate it, plot it, verify a claim, export a report—before adding commands.
Return structured results; presentation code should never scrape printed strings.
Let every module register its assumptions and validation evidence so the report layer remains generic.

Common Pitfalls and Debugging

Symptom: adding one command breaks unrelated modules. Cause: shared mutable state or tight coupling. Fix: isolate pure domain services behind typed interfaces.
Symptom: numeric answers disagree without explanation. Cause: modules use inconsistent tolerances/domains. Fix: centralize numeric policy while allowing explicit overrides.
Symptom: the engine is only a menu of scripts. Cause: no shared contracts or cross-module workflow. Fix: require a scenario to compose at least three capabilities through structured data.

Definition of Done

One documented CLI exposes algebra, functions, matrices, calculus, statistics, and modeling modules.
Modules share structured results and diagnostics rather than print-only integration.
Every operation supplies verification evidence or a stated limitation.
Existing project fixtures remain regression tests after integration.
A deterministic end-to-end report composes at least three modules.
Architecture and extension documentation let another developer add a command safely.

Project 36: Logistic Regression Classifier

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with NumPy, Pandas, and Matplotlib
Prerequisites: Projects 12 and 33-34
Merged from: ML-Math P18; NN P04; ML-Found P14

What You Will Build

Build binary logistic regression from raw arrays through calibrated probability output. Implement standardized features, logits, a numerically stable sigmoid, binary cross-entropy from logits, analytic gradients, L2 regularization, and mini-batch or full-batch gradient descent. Plot the decision boundary for 2D data and provide a feature-contribution explanation for each prediction. Evaluate accuracy, precision, recall, F1, confusion matrix, ROC-AUC, log loss, and calibration; make classification threshold configurable rather than embedding 0.5 as truth. Use a real dataset such as spam, fraud, or medical-style synthetic data, with stratified splits and imbalanced-class analysis. Compare against a trusted library only after the from-scratch model passes gradient checks.

Real World Outcome

The program trains with a visible loss curve, reports held-out discrimination and calibration metrics, and lets a user move the decision threshold to inspect false-positive/false-negative tradeoffs. Each prediction displays probability, logit, and the largest signed feature contributions. A reliability diagram shows whether “0.8 probability” corresponds to roughly 80% positives.

Core Question

How can a linear score become a probability-like classification model, and why must probability quality be judged separately from thresholded accuracy?

Concepts You Must Understand First

Log odds and sigmoid: a linear logit maps through sigmoid into (0,1).
Bernoulli likelihood and cross-entropy: maximum likelihood yields log loss. See Géron, Hands-On Machine Learning, classification chapters.
Stable computation: use log-domain identities or logits to avoid log(0) and exponential overflow.
Regularization: L2 penalizes large weights and changes the optimization objective.
Metrics and calibration: ranking, threshold decisions, and probability reliability answer different questions.

Build Milestones

Implement stable sigmoid and cross-entropy-with-logits with extreme-value tests.
Derive vectorized weight/intercept gradients and verify them with central differences.
Train on synthetic separable and overlapping 2D data; visualize loss and boundary.
Add scaling, L2 regularization, stratified splits, and real-data ingestion.
Build threshold, ROC, confusion, contribution, and calibration reports.

Hints in Layers

Keep raw logits for loss; do not clip probabilities until you understand what the clipping hides.
Do not regularize the intercept unless you deliberately choose and document that convention.
Choose thresholds on validation data using problem costs, then report once on untouched test data.

Common Pitfalls and Debugging

Symptom: loss becomes NaN on confident mistakes. Cause: direct log(sigmoid(z)). Fix: use a stable cross-entropy-from-logits identity.
Symptom: model predicts only the majority class. Cause: imbalance and threshold-insensitive accuracy. Fix: inspect precision/recall, class distribution, and threshold curves.
Symptom: gradient check fails only for intercept. Cause: shape/broadcasting or accidental regularization. Fix: test intercept separately on a tiny fixture.

Definition of Done

Stable loss remains finite for very large positive and negative logits.
Analytic gradients pass finite-difference checks.
Training is reproducible and loss decreases on controlled data.
Evaluation includes discrimination, threshold, and calibration views.
Feature scaling and regularization are fitted without leakage.
From-scratch predictions agree closely with a trusted reference model.

Project 37: Information-Theory and Loss Engineering Lab

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Projects 12, 30, and 36
Source: ML-Math P23

What You Will Build

Build a workbench for entropy, cross-entropy, KL divergence, mutual information, coding length, and classification losses. Implement discrete calculations from normalized distributions with explicit zero-probability conventions and stable log operations. Let users compare true and predicted distributions, then visualize how confidence changes log loss, Brier score, gradients, and calibration. Simulate communication or compression to connect average surprise with expected code length. Add mutual-information experiments on independent, noisy-correlated, and deterministically related variables. Finally, compare loss landscapes for squared error, binary/categorical cross-entropy, and label smoothing, recording how each treats confident mistakes and class imbalance. The goal is not a formula catalog but an experimental system where identities and inequalities are testable.

Real World Outcome

The dashboard accepts probability vectors and produces entropy, cross-entropy, KL, per-outcome surprise, and a decomposition check H(p,q)=H(p)+KL(p||q). Interactive plots show loss and gradient versus predicted probability. A second panel estimates mutual information from samples and reveals estimator sensitivity to binning and sample size.

Core Question

Why does logarithmic surprise provide a natural measure of information, and how do information-theoretic quantities explain the behavior of ML loss functions?

Concepts You Must Understand First

Self-information and entropy: rare events carry more surprise; entropy averages it. See Cover and Thomas, Elements of Information Theory.
Cross-entropy and KL: cross-entropy measures coding under the wrong distribution; KL is the excess over true entropy.
Zero conventions: 0 log 0 contributes zero, but assigning zero probability to a possible event creates infinite loss.
Mutual information: dependence equals divergence between joint and product-of-marginals distributions.
Proper scoring and calibration: log loss and Brier score reward honest probabilities in different ways.

Build Milestones

Implement validated distributions, stable logs, entropy, cross-entropy, and KL with identity tests.
Visualize surprise and coding-length intuition for controlled discrete sources.
Implement joint/marginal tables and mutual-information experiments.
Compare classification losses, gradients, confidence, label smoothing, and calibration.
Add finite-sample estimator studies and a reproducible experiment report.

Hints in Layers

Validate nonnegativity and normalization before calculating; report whether you renormalize.
Use natural logs for nats or base-2 logs for bits, and label the unit everywhere.
Estimate mutual information first from an exact joint table, then confront sampling and binning bias.

Common Pitfalls and Debugging

Symptom: KL is negative. Cause: distributions are unnormalized, axes are swapped, or rounding is severe. Fix: validate inputs and compare with the decomposition identity.
Symptom: entropy becomes NaN with zero entries. Cause: evaluating 0*log(0) literally. Fix: mask zero-mass terms using the limiting convention.
Symptom: sampled MI grows when bins increase. Cause: finite-sample estimator bias. Fix: sweep bins/sample size and disclose the estimator.

Definition of Done

Entropy, cross-entropy, and KL pass known distributions and identity checks.
Bits/nats and zero-probability policies are explicit.
Mutual information distinguishes independent and dependent examples.
Loss plots include values and gradients for confident correct/incorrect predictions.
Calibration and scoring-rule experiments are reproducible.
The report discusses finite-sample and discretization bias.

Project 38: K-Means Clustering from Scratch

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python with NumPy and Matplotlib
Prerequisites: Projects 16, 20, and 33
Source: ML-Found P15

What You Will Build

Build k-means clustering without a clustering library. Implement vectorized distances, assignment, centroid recomputation, within-cluster sum of squares (inertia), convergence detection, and multiple random restarts. Add k-means++ initialization and explicit handling for empty clusters. Animate assignments and centroid movement on 2D data, then apply the same engine to standardized customer features or image colors. Provide elbow and silhouette-style diagnostics while stressing that neither automatically discovers a uniquely correct k. Include deliberately difficult datasets—unequal density, non-spherical moons, outliers, and differently scaled features—to expose the geometric assumptions behind Euclidean centroid clustering.

Real World Outcome

For a dataset and k, the CLI logs inertia and centroid shift per iteration, saves an animation, and reports the best of several seeded restarts. A model-selection view plots inertia and silhouette summaries across k. Failure-gallery plots explain why k-means splits elongated or nonconvex clusters poorly.

Core Question

How can alternating between nearest-centroid assignments and mean updates reduce within-cluster variance, and what data geometry makes that objective meaningful or misleading?

Concepts You Must Understand First

Euclidean distance and scale: large-scale features dominate squared distance unless standardized.
Centroid as mean: the arithmetic mean minimizes squared distance within a fixed cluster.
Alternating optimization: assignment and update steps each do not increase the k-means objective.
Initialization: the objective is nonconvex, so different seeds reach different local solutions. See Arthur and Vassilvitskii, “k-means++.”
Cluster validation: elbow and silhouette summarize geometry but do not supply ground truth. See Géron, Hands-On Machine Learning, ch. 9.

Build Milestones

Implement and test distance, assignment, mean-update, and inertia calculations on tiny data.
Add deterministic stopping by assignment stability or centroid shift plus an iteration cap.
Implement random and k-means++ starts, empty-cluster repair, and best-of-n restarts.
Animate 2D convergence and add elbow/silhouette comparisons across k.
Apply the engine to real features and a documented failure gallery.

Hints in Layers

Assert that inertia never materially increases after a full assignment/update cycle.
For k-means++, sample new centers in proportion to squared distance from the nearest selected center.
Repair empty clusters by reseeding to a far-away point or splitting a high-variance cluster, and log the event.

Common Pitfalls and Debugging

Symptom: one feature determines every cluster. Cause: inconsistent scales. Fix: fit standardization on the analysis data and explain transformed units.
Symptom: output changes wildly by run. Cause: single random initialization. Fix: seed, use k-means++, and compare multiple restarts.
Symptom: centroid contains NaN. Cause: empty cluster mean. Fix: detect and reseed before division.

Definition of Done

Core calculations match hand-computed fixtures.
Inertia history is nonincreasing within numerical tolerance.
Random and k-means++ initialization plus multiple restarts are supported.
Empty clusters, duplicates, and ties have deterministic policies.
Animation and k-selection diagnostics are exported.
Failure cases document scale, shape, density, and outlier limitations.

Project 39: Decision Tree Classifier from Scratch

Difficulty: Advanced
Time: 2-3 weeks
Language: Python with NumPy and Pandas
Prerequisites: Projects 2, 12, and 37
Source: ML-Found P16

What You Will Build

Build an interpretable classification tree using recursive binary partitioning. Implement class counts, Gini impurity and entropy, weighted impurity reduction/information gain, candidate thresholds, recursive node construction, prediction traversal, and probability estimates at leaves. Add stopping controls for depth, sample count, impurity, and minimum gain, plus either reduced-error or cost-complexity-style pruning. Print and visualize the tree, and return a decision path explaining every prediction. Evaluate on synthetic and real tabular data with train/validation/test separation. Compare unrestricted and constrained trees to reveal overfitting, and compute feature importance from accumulated impurity reduction while documenting its biases.

Real World Outcome

Training on a Titanic-style dataset prints the root split, node/leaf counts, depth, train/test metrics, and a readable tree. Querying one row returns its probability and path, such as sex <= ... -> age > ... -> leaf. Learning curves compare deep, shallow, and pruned versions.

Core Question

How can greedy local questions partition a feature space into useful prediction regions, and why does a perfectly pure training tree often generalize poorly?

Concepts You Must Understand First

Entropy and Gini impurity: both quantify class mixture, with zero at a pure node. See Géron, Hands-On Machine Learning, ch. 6.
Weighted information gain: a split is judged by child impurity weighted by child size.
Recursive partitioning: each internal node owns a rule and child subproblems. See James et al., An Introduction to Statistical Learning, tree chapter.
Stopping and pruning: limiting structure regularizes a high-variance learner.
Evaluation and explanation: leaf frequencies estimate probabilities; a path provides a local reason, not a causal explanation.

Build Milestones

Implement impurity/count helpers and verify them on pure, balanced, and skewed labels.
Enumerate valid numeric thresholds and select the split with highest weighted gain.
Recursively build nodes, leaves, prediction traversal, and probability output.
Add depth/sample/gain constraints and validation-based pruning.
Export text/diagram views, decision paths, feature importance, and held-out comparisons.

Hints in Layers

Candidate thresholds can be midpoints between sorted distinct feature values where labels may change.
Keep node data minimal: feature, threshold, children, counts, impurity, and sample count.
Use a validation set for pruning decisions; the test set remains untouched until the tree is final.

Common Pitfalls and Debugging

Symptom: gain is negative or children look purer but score worse. Cause: child impurities were not weighted by size. Fix: test the formula on a hand split.
Symptom: recursion never terminates. Cause: a split sends all rows to one child. Fix: reject empty/no-progress splits and require positive gain.
Symptom: training accuracy is perfect, test accuracy poor. Cause: unrestricted depth. Fix: tune stopping/pruning on validation data.

Definition of Done

Gini, entropy, and weighted gain match hand calculations.
Every accepted split creates two nonempty smaller child problems.
Prediction probabilities and decision paths are reproducible.
Multiple pre-pruning controls and one post-pruning strategy work.
Held-out metrics compare unrestricted, constrained, and pruned trees.
Feature importance is reported with a warning about interpretation bias.

Project 40: Markov Chain Text Generator

Difficulty: Intermediate
Time: 1-2 weeks
Language: Python
Prerequisites: Projects 6, 12, and 20
Source: ML-Math P16

What You Will Build

Build a character- or word-level n-gram text generator as a finite Markov chain. Tokenize a corpus, count transitions from states of order n to next tokens, normalize counts into probability distributions, and sample seeded continuations. Provide order-1 through higher-order models, start/end tokens, unknown-state fallback, and optional additive smoothing. Visualize a small transition matrix or graph and inspect the highest-probability successors of any state. Measure held-out negative log-likelihood/perplexity where possible, and compare diversity, coherence, memory, and sparsity as order changes. Add a synthetic transition-matrix mode that simulates arbitrary chains and estimates occupancy/stationary behavior, connecting text generation to general Markov processes.

Real World Outcome

After training on a text file, the CLI reports vocabulary, state count, transition sparsity, and held-out score, then emits reproducible samples from a prompt. An inspector displays successor probabilities and observed counts. A comparison report shows that low order is repetitive but robust, while high order copies longer fragments and suffers unseen states.

Core Question

How much sequence structure can be captured by assuming the next token depends only on a finite recent state, and what breaks as that memory grows?

Concepts You Must Understand First

Markov property: the modeled future depends on the current state, not the full history.
Conditional probability: transition rows are normalized distributions over next states.
Transition matrices: repeated multiplication evolves state distributions and may approach stationary behavior.
N-gram state design: increasing order stores more context but causes sparse, unseen states. See Jurafsky and Martin, Speech and Language Processing, n-gram chapters.
Sampling and likelihood: seeded categorical draws generate sequences; log probabilities avoid underflow when scoring them.

Build Milestones

Build deterministic tokenization, start/end markers, and transition counts for a tiny hand corpus.
Normalize transitions and implement seeded categorical sampling with probability checks.
Generalize to configurable n-gram order and prompt-conditioned generation.
Add smoothing/fallback, held-out log-likelihood, and order-comparison experiments.
Visualize small chains and estimate occupancy/stationary distributions through simulation.

Hints in Layers

Preserve raw counts; derive probabilities on demand so smoothing and diagnostics remain possible.
Sample by cumulative probability with one seeded random generator, and test boundary draws.
Split the corpus before building counts; otherwise held-out likelihood measures memorization on training data.

Common Pitfalls and Debugging

Symptom: generated text stops immediately or crashes. Cause: prompt state was unseen or has no outgoing transition. Fix: implement backoff, restart, or explicit end-state handling.
Symptom: probabilities do not sum to one. Cause: wrong denominator or mixed state orders. Fix: assert each nonterminal row independently.
Symptom: high-order evaluation looks excellent. Cause: train/test leakage or copied spans. Fix: split first and inspect overlap and unseen-state rate.

Definition of Done

Counts and probabilities match a hand-worked corpus.
Every nonterminal state’s transition probabilities normalize within tolerance.
Seeded generation is reproducible for multiple n-gram orders.
Unknown states and sequence endings have explicit behavior.
Held-out scoring and sparsity/unseen-state statistics compare model orders.
A small transition graph and simulation demonstrate general Markov-chain behavior.

Project 41: Complete ML Pipeline and House-Price Predictor

Difficulty: Level 5: Master
Time: 4 weeks
Language: Python (NumPy; Pandas may be used only for data loading)
Prerequisites: Projects 14, 30, 34, 36, and at least two of Projects 13 and 38-39
Merged from: ML-Math P20; ML-Found P19

What You Will Build

Build a reproducible command-line pipeline that ingests raw housing data, validates its schema, profiles missing values and outliers, creates leakage-safe train/validation/test splits, fits preprocessing only on training folds, and compares several models implemented in earlier projects. The final artifact is not merely a predictor: it is a versioned decision report containing the data contract, transformations, cross-validation results, error slices, selected model, test metrics, configuration, seed, and serialized inference bundle. Add a small prediction API only after the offline evaluation is trustworthy.

Real World Outcome

Running one command on a housing CSV produces a validation report, saved split indices, feature pipeline, cross-validation comparison, final RMSE/R² with uncertainty, residual plots, model file, and a sample API response. Repeating the run with the same data and seed reproduces the metrics within a documented tolerance.

Core Question

How do you turn messy raw data into a prediction that can be reproduced, audited, and safely used for a decision?

Concepts You Must Understand First

Data contracts, missingness, categorical encoding, scaling, and feature engineering — Designing Machine Learning Systems by Chip Huyen.
Leakage boundaries: preprocessing, feature selection, and imputation must be fitted inside each training fold.
Cross-validation and model selection — An Introduction to Statistical Learning, Chapter 5.
Regression diagnostics: RMSE, MAE, R², residual structure, and subgroup error.
Reproducibility: seeds, immutable split indices, configuration snapshots, and dataset fingerprints.

Build Milestones

Implement schema checks, data profiling, deterministic splitting, and a baseline mean predictor.
Build fit/transform preprocessing components and prove that validation/test statistics never influence training.
Compare linear/ridge regression, a decision tree, and one additional eligible model with identical folds.
Retrain the selected configuration, evaluate once on the held-out test set, and generate error-slice plots.
Serialize preprocessing plus model together and expose a validated prediction command or HTTP endpoint.

Hints in Layers

Start with one numeric-only end-to-end slice before adding categoricals and engineered features.
Give every transformer fit and transform phases; log which row IDs each phase sees.
Hash the raw file, store fold indices, and rerun from the saved manifest to test reproducibility.

Common Pitfalls and Debugging

Validation looks excellent but test error jumps: preprocessing or selection saw validation/test data. Fit every learned transformation within training folds and audit row IDs.
Metrics change between identical runs: random state or row ordering is uncontrolled. Seed all generators and persist shuffled indices and configuration.
API predictions differ from notebook predictions: serving uses a different transformation path. Serialize one composite preprocessing-model artifact and test it against offline examples.

Definition of Done

Schema violations and malformed requests fail with actionable diagnostics.
A naive baseline and at least three model configurations use identical cross-validation folds.
Leakage tests prove that held-out rows never influence fitted transformations.
The final report includes uncertainty, residual analysis, and at least two meaningful error slices.
A clean process can reproduce the selected model and metrics from the saved manifest.
Serialized inference returns the same predictions as the evaluation pipeline.

Phase 7 — Neural Networks from First Principles (Projects 42-50)

Project 42: Manual Neuron and Single-Neuron Backpropagation

Difficulty: Level 3: Advanced
Time: 1-2 weeks
Language: Python without ML frameworks
Prerequisites: Projects 26, 33, and 36
Merged from: ML-Math P11; NN P01

What You Will Build

Implement one inspectable neuron from scalar arithmetic: weighted sum, bias, activation, loss, local derivatives, parameter gradients, and gradient-descent updates. Train it first with a threshold/perceptron rule on linearly separable Boolean gates, then with sigmoid and differentiable loss so every chain-rule factor can be printed. The program must expose the forward cache and backward calculation for an individual example, compare analytical gradients with centered finite differences, learn AND/OR/NAND/NOR, and demonstrate—rather than hide—why a single neuron cannot solve XOR.

Real World Outcome

A CLI training run prints the equation for one forward/backward step, a decreasing loss curve, learned decision boundary, final truth-table predictions, and a gradient-check report. A separate XOR run terminates with a clear “not linearly separable” diagnosis instead of pretending that more epochs will fix the model.

Core Question

How does changing a weight change the prediction and loss, and how does that local sensitivity become learning?

Concepts You Must Understand First

Linear decision boundaries and weighted sums — Grokking Deep Learning by Andrew Trask.
Sigmoid, step, and identity activations; saturation and derivative behavior.
Mean-squared or binary cross-entropy loss and the chain rule — Neural Networks and Deep Learning by Michael Nielsen.
Gradient descent, learning rate, and finite-difference derivative checks.
Linear separability and the geometric reason XOR needs a hidden layer.

Build Milestones

Implement deterministic forward prediction and truth-table tests for a hand-configured threshold neuron.
Add loss and explicitly derive gradients for weights, bias, and pre-activation.
Train differentiable and perceptron variants on AND/OR; plot points and the learned boundary.
Add centered finite-difference checks for every parameter and test gradient accumulation over a batch.
Run XOR, visualize the failed boundary, and explain the representational limitation.

Hints in Layers

Keep all variables scalar for the first version and name each intermediate quantity from the derivation.
Check gradients on one example before summing a batch; use relative error when values are near zero.
If learning stalls, print activation and derivative together—sigmoid may be saturated.

Common Pitfalls and Debugging

Loss increases or oscillates: update sign is reversed or learning rate is too large. Verify parameter -= rate × gradient on a one-parameter quadratic.
Gradient check fails: the backward pass uses recomputed or overwritten intermediates. Cache exactly the values from the corresponding forward pass.
XOR never converges: this is a capacity limit, not necessarily a bug. Plot the four points and prove no single line separates the labels.

Definition of Done

Forward output matches hand calculations for at least three parameter/input cases.
Analytical gradients pass centered finite-difference checks under a stated tolerance.
The neuron learns at least four linearly separable Boolean gates from more than one seed.
A trace explains every factor in one backward update.
Decision-boundary and loss plots make learning observable.
The XOR experiment documents the limitation and motivates an MLP.

Project 43: Matrix Calculus Backpropagation Workbench

Difficulty: Level 4: Expert
Time: 2-3 weeks
Language: Python with NumPy
Prerequisites: Projects 20, 26, and 42
Source: ML-Math P22

What You Will Build

Create a shape-aware notebook or CLI laboratory that turns scalar backpropagation into batched matrix calculus. Begin with an affine layer, then compose activation and stable softmax-cross-entropy operations. Every operation publishes input/output shapes, its local derivative contract, and a vector-Jacobian product rather than materializing enormous Jacobians unnecessarily. Derive gradients for weights, bias, inputs, and batch reductions; compare each parameter block with centered finite differences; and emit diagnostics that identify transpose, broadcasting, and reduction mistakes by layer and tensor index.

Real World Outcome

For a two-layer MLP and a fixed mini-batch, the workbench prints the forward loss, shape trace, analytical/numerical relative errors for W1, b1, W2, b2, and inputs, plus a pass/fail chain-rule report. Deliberately enabling a transpose or mean/sum bug points to the offending block.

Core Question

How does backpropagation work when inputs, outputs, and derivatives are vectors and matrices rather than scalars?

Concepts You Must Understand First

Jacobians, gradients, differentials, and a consistent numerator/denominator layout convention.
Matrix chain rule and vector-Jacobian products — Parr and Howard, The Matrix Calculus You Need for Deep Learning.
Broadcasting and the need to reduce gradients over expanded axes.
Stable softmax-cross-entropy and batch mean versus batch sum conventions.
Centered finite differences and robust relative-error denominators.

Build Milestones

Derive and implement affine forward/backward operations with exhaustive shape assertions.
Add ReLU or sigmoid and stable softmax-cross-entropy, documenting each local derivative.
Compose a two-layer batched network using reverse-mode vector-Jacobian products.
Build per-tensor and sampled per-element numerical gradient checks.
Add fault injection for transpose, broadcast-reduction, and batch-scaling errors.

Hints in Layers

Write the shape beside every symbol before writing algebra or code.
Seed the upstream gradient with the loss derivative and work backward one operator at a time.
Test non-square dimensions and batch size greater than one; symmetric shapes conceal transpose bugs.

Common Pitfalls and Debugging

Gradients have plausible values but wrong shapes: broadcasting was reversed incorrectly. Sum over axes that were added or had size one in the forward pass.
Errors differ by exactly the batch size: one path averages while the other sums. Choose and document one loss-reduction convention.
Checks fail only near zero: raw relative error is unstable. Combine absolute and relative tolerances and sample away from activation kinks.

Definition of Done

Every operator documents and asserts its forward/backward shape contract.
All parameter and input gradients pass numerical checks on non-square batches.
The derivation clearly connects local Jacobians to efficient vector-Jacobian products.
Stable softmax-cross-entropy agrees with a trusted reference on extreme logits.
Fault injection produces a localized, useful mismatch report.

Project 44: Autograd Engine

Difficulty: Level 4: Expert
Time: 2 weeks
Language: Python
Prerequisites: Projects 2 and 42-43
Source: NN P05

What You Will Build

Implement a small define-by-run automatic differentiation engine. A Value or tensor object stores data, accumulated gradient, parent nodes, operation metadata, and a local backward closure. Arithmetic and nonlinear operators construct a directed acyclic computation graph dynamically; backward() topologically sorts reachable nodes and executes local rules in reverse. Support repeated operands and branching, where gradients must accumulate rather than overwrite. Add graph visualization, zeroing semantics, finite-difference verification, and enough operations to train a tiny multilayer perceptron.

Real World Outcome

A user can compose an ordinary expression such as a tiny neural loss, inspect its computation graph, call one backward method, and see correct derivatives for every leaf. The engine passes numerical checks on branched expressions and trains a small classifier while logging graph size, loss, and gradients.

Core Question

How can a program record arbitrary calculations and automatically apply the chain rule backward through them?

Concepts You Must Understand First

Directed acyclic graphs, reachability, and topological ordering.
Reverse-mode automatic differentiation and local chain-rule composition — Baydin et al., Automatic Differentiation in Machine Learning: a Survey.
Gradient accumulation when one value contributes through multiple paths.
Operator overloading and closure capture — Fluent Python by Luciano Ramalho.
Numerical gradient checking and non-differentiable points.

Build Milestones

Implement scalar nodes and overloaded addition, multiplication, power, negation, and division.
Add tanh, ReLU, exponential, and logarithm with local backward rules.
Implement deterministic topological traversal, reverse execution, accumulation, and gradient reset.
Verify nested, branched, repeated-node, and shared-subexpression graphs against finite differences.
Add neuron/layer/module abstractions and train a small binary classifier using the engine.

Hints in Layers

Draw the graph for a*a + a; it exposes both repeated-use and accumulation requirements.
The backward seed for a scalar output is 1 because its derivative with respect to itself is 1.
Capture operands, not mutable loop variables, inside each backward closure.

Common Pitfalls and Debugging

A reused variable has a gradient that is too small: local backward rules overwrite .grad. Add contributions with +=.
Some nodes never receive gradients: traversal order is wrong or parents are omitted. Build a postorder list, then reverse it.
A second training step explodes: old gradients remain. Make zero_grad() explicit and test repeated backward behavior.

Definition of Done

Supported operators pass analytical and finite-difference unit tests.
Repeated operands and branched graphs accumulate every gradient contribution.
Topological execution is deterministic and handles shared subexpressions once.
Graph visualization labels values, operations, and gradients.
A small MLP trained through the engine reduces loss on a nonlinear dataset.
Gradient reset and repeated-backward semantics are documented and tested.

Project 45: MLP Classifier from First Principles

Difficulty: Level 4: Expert
Time: 3-4 weeks
Language: Python with NumPy only
Prerequisites: Projects 30, 36-37, and 42-44
Merged from: ML-Math P19; Prog Final Neural Network; NN P06, NN P08; ML-Found P17

What You Will Build

Build a vectorized multilayer perceptron without TensorFlow or PyTorch. Progress from a hidden layer that solves XOR to a configurable stack of dense layers trained by mini-batch gradient descent. Implement ReLU/sigmoid/tanh, stable softmax and cross-entropy, initialization, forward caches, matrix-form backward passes, SGD, regularization, class weighting, and checkpointing. Validate first on a small imbalanced fraud-style binary dataset, where precision/recall matter, then train a 784→128→64→10 network on MNIST and analyze both correct and mistaken digits.

Real World Outcome

One training command prints architecture and parameter counts, learning curves, confusion matrix, class-sensitive metrics, and gradient-check status. The binary model beats a majority baseline without ignoring the rare class; the MNIST model reaches a defensible target such as 95%+ test accuracy and saves reproducible weights and misclassification plots.

Core Question

How do linear layers, nonlinear activations, a loss function, and backpropagation combine into a network that learns nonlinear decisions?

Concepts You Must Understand First

Dense-layer matrix multiplication and batched matrix calculus — Project 43.
Reverse-mode backpropagation and parameter ownership — Project 44.
ReLU, sigmoid, tanh, softmax, and cross-entropy — Deep Learning, Chapter 6.
Xavier/He initialization, saturation, and symmetry breaking.
Imbalanced classification metrics, class weighting, and threshold selection.

Build Milestones

Implement dense layers and activations; prove a one-hidden-layer network learns XOR.
Add stable softmax/cross-entropy, vectorized mini-batches, initialization, and per-layer gradient checks.
Train a binary MLP with class weighting; report precision, recall, PR curve, and confusion matrix.
Train the configurable MLP on MNIST; save checkpoints and inspect misclassified digits.
Compare activations, depths, initialization schemes, and seeded runs in one experiment report.

Hints in Layers

Overfit a tiny batch before attempting the full dataset; failure here indicates implementation, not generalization.
Subtract each row’s maximum logit before exponentiating and combine softmax with cross-entropy backward.
Track activation and gradient statistics per layer to detect saturation or dead ReLUs.

Common Pitfalls and Debugging

Loss is nan: softmax overflow or log(0). Use log-sum-exp stabilization and clip only at the final logarithm boundary.
Accuracy is high but fraud recall is zero: class imbalance rewards predicting the majority. Use class weights and report PR metrics.
Every hidden unit behaves identically: weights were initialized equally. Use seeded random initialization scaled by fan-in.

Definition of Done

XOR is learned consistently by a hidden-layer network.
Every layer’s gradients pass finite-difference checks on a tiny batch.
Training is vectorized, seeded, checkpointed, and recoverable.
Binary evaluation includes minority-class recall, precision, and threshold analysis.
MNIST performance meets a declared target and includes error inspection.
An experiment report explains activation, initialization, and architecture tradeoffs.

Project 46: Backpropagation Flow Visualizer

Difficulty: Level 3: Advanced
Time: 2 weeks
Language: Python with Plotly or Matplotlib
Prerequisites: Project 45
Source: ML-Found P18

What You Will Build

Instrument the MLP from Project 45 to capture per-layer activations, pre-activations, parameters, incoming gradients, parameter gradients, and update magnitudes during training. Present an interactive network view in which node/edge color encodes gradient magnitude, plots show activation distributions and layer-wise norms, and a step control reveals forward and backward flow. Add controlled experiments for sigmoid saturation, dead ReLUs, poor initialization, excessive learning rate, and increasing depth so the dashboard teaches why training succeeds or fails.

Real World Outcome

An interactive session can pause on any mini-batch, animate one optimization step, inspect a neuron’s activation and gradient, and compare healthy ReLU/He training with a saturated sigmoid or exploding-gradient configuration. The dashboard flags suspect layers and exports a diagnostic snapshot alongside the loss curve.

Core Question

What happens to signal and gradient magnitude as information travels forward and error travels backward through a deep network?

Concepts You Must Understand First

Forward caches, computational graphs, and matrix backpropagation — Projects 43-45.
Activation saturation and dead neurons — Deep Learning, Chapters 6 and 8.
Vanishing/exploding gradients and initialization scaling.
Gradient norms, update-to-weight ratios, and distribution summaries.
Skip connections as alternate gradient paths — He et al., Deep Residual Learning.

Build Milestones

Add non-invasive hooks that record layer statistics without changing computed gradients.
Draw architecture, weight magnitude, activations, and gradients for a single training step.
Add time-series views for loss, gradient norms, saturation, dead-unit fraction, and update ratios.
Create reproducible pathology presets and compare activations, depth, and initialization.
Implement warnings with evidence, not opaque labels, and export a diagnostic report.

Hints in Layers

Store summaries by default; retaining every tensor for every step will exhaust memory.
Use log-scaled gradient magnitude because layers may differ by many orders of magnitude.
Verify instrumentation by training with hooks enabled and disabled under the same seed.

Common Pitfalls and Debugging

Training changes when visualization is enabled: hooks mutate arrays or retain graph state. Record copies or read-only summaries and test trajectory equivalence.
All gradients look equally dark: linear color scaling hides small values. Use a log scale with an explicit legend and zero handling.
Dashboard freezes on long runs: full tensors are retained. Sample batches/steps and aggregate histograms online.

Definition of Done

Instrumented and uninstrumented runs match under an identical seed.
Users can inspect activation, gradient, parameter, and update statistics per layer.
At least four reproducible pathologies have visibly distinct signatures.
The view supports pause, single-step, comparison, and exported snapshots.
Warning rules cite the measured statistic and threshold that triggered them.

Project 47: Convolutional Kernel Explorer

Difficulty: Level 2: Intermediate
Time: 8-16 hours
Language: Python with NumPy and an image library
Prerequisites: Projects 21, 25, and 45
Source: NN P07

What You Will Build

Create an image laboratory that implements discrete 2D convolution directly from nested loops before optimizing it. Users can load grayscale or RGB images, choose or edit kernels, and inspect padded input, each sliding window, elementwise products, accumulated output, and final feature map. Include identity, box blur, Gaussian blur, sharpen, emboss, horizontal/vertical Sobel, and Laplacian filters. Support padding, stride, multiple channels, normalization/display policies, and comparison against a trusted image-library implementation.

Real World Outcome

An interactive CLI or small GUI applies selected kernels and displays original image, kernel matrix, intermediate response, and normalized output side by side. A probe mode clicking one output pixel shows the exact source patch and arithmetic that produced it; benchmark and correctness reports compare your result with a reference convolution.

Core Question

How can one small matrix detect or transform a visual pattern everywhere in an image?

Concepts You Must Understand First

Discrete convolution versus cross-correlation and the meaning of kernel flipping.
Sliding windows, receptive fields, padding, stride, and output-size formulas.
Linear image filters and separability; Gaussian and Sobel kernels.
RGB channel aggregation and multi-output feature maps.
Pixel range, clipping, normalization, signed edge responses, and dtype overflow.

Build Milestones

Implement valid single-channel convolution and verify every output on tiny hand-computed arrays.
Add same/valid padding, stride, output-shape validation, and probeable window arithmetic.
Add the preset kernel gallery and explain the visual role of each filter.
Extend to color images and multiple kernels, displaying every feature map.
Compare correctness and runtime with a trusted library; optionally add a vectorized path.

Hints in Layers

Begin with a 4×4 array and 2×2 kernel where every result can be calculated on paper.
Compute expected output dimensions before allocating; reject non-integral stride placements explicitly.
Preserve floating-point signed responses until display time—edge filters naturally produce negatives.

Common Pitfalls and Debugging

Features appear mirrored or signs reverse: your code and reference disagree on convolution versus correlation. State the convention and flip the kernel only when required.
Borders shift or output size is wrong: padding is asymmetric or the size formula is incorrect. Print padded shape and first/last window coordinates.
Edges become black or wrap around: unsigned integer arithmetic overflowed. Convert to float before multiplying and choose an explicit visualization transform.

Definition of Done

Tiny-array tests verify values, padding, stride, and output dimensions.
At least eight named kernels produce explainable image effects.
Probe mode reveals the exact arithmetic for any selected output location.
Grayscale and RGB paths have documented channel semantics.
Results match a trusted reference within a justified tolerance.
Signed/filter outputs are displayed without silently corrupting values.

Project 48: CNN from Scratch

Difficulty: Level 5: Master
Time: 3 weeks
Language: Python with NumPy only
Prerequisites: Projects 43-47
Source: NN P09

What You Will Build

Implement and train a convolutional neural network without a deep-learning framework. Build Conv2D, ReLU, max-pooling, flatten, dense, and softmax-cross-entropy layers with complete forward and backward passes. Start with a literal loop implementation whose indices are easy to audit, then optionally add im2col for speed. Train a compact LeNet-style model on MNIST, visualize learned kernels and feature maps, and compare it fairly with the dense MLP from Project 45 to explain how local connectivity and parameter sharing change image learning.

Real World Outcome

A command trains the CNN, reports architecture/output shapes and parameter count, plots train/validation loss and accuracy, saves checkpoints, and produces a gallery of input images, intermediate feature maps, predictions, and errors. Every convolution and pooling gradient passes numerical checks on tiny tensors, and the final model reaches a declared MNIST target.

Core Question

How do local receptive fields and shared kernels learn spatial features while using fewer parameters than dense image models?

Concepts You Must Understand First

Convolution, channels, padding, stride, receptive fields, and feature maps — Project 47.
Parameter sharing and translation tolerance — Deep Learning, Chapter 9.
Max-pooling forward routing and backward argmax masks.
Tensor shape transformations between convolutional and dense stages.
Matrix backpropagation, stable loss, and gradient checking — Projects 43-45.

Build Milestones

Implement loop-based Conv2D forward and validate shapes/values on hand-computed tensors.
Derive and implement gradients for inputs, kernels, and biases; numerically check each one.
Add max-pooling backward, ReLU, flatten, dense, and stable classification loss.
Overfit a tiny MNIST subset, then train a seeded LeNet-style network on full data.
Visualize feature maps/kernels and compare parameters, speed, and errors with the dense MLP.

Hints in Layers

Use tiny non-square inputs and kernels; square symmetric cases hide axis mistakes.
Cache pooling argmax positions and convolution input patches needed by backward.
Optimize only after the loop version passes checks; use it as an oracle for im2col.

Common Pitfalls and Debugging

Backward shapes match but gradients fail: kernel/input axes were transposed. Write every tensor shape beside the summation and test one element manually.
Pooling gradient appears in several cells: ties or masks are mishandled. Define a deterministic tie policy and route each upstream value accordingly.
Training is impossibly slow: loops dominate. First reduce the test dataset, then add a verified im2col path.

Definition of Done

Convolution and pooling forward outputs pass hand-computed tests.
Input, weight, bias, and pooling gradients pass finite-difference checks.
A tiny batch can be deliberately overfit before full training.
The full model meets a declared MNIST accuracy target reproducibly.
Feature-map and kernel visualizations support a written interpretation.
Dense-versus-CNN comparison includes parameters, runtime, accuracy, and error examples.

Project 49: Recurrent Character Generator

Difficulty: Level 5: Master
Time: 3 weeks
Language: Python with NumPy only
Prerequisites: Projects 6, 40, and 43-45
Source: NN P10

What You Will Build

Build a character-level recurrent neural network that reads text one character at a time, maintains hidden state, predicts the next-character distribution, and generates seeded samples. Implement vocabulary encoding, sequence batching, recurrent forward equations, stable softmax loss, backpropagation through time (BPTT), gradient clipping, hidden-state handling, checkpointing, and temperature-controlled sampling. Use a small corpus such as Shakespeare, names, or source code and make the temporal mechanics inspectable rather than delegating them to a framework.

Real World Outcome

Training reports loss, perplexity, gradient norms, and periodic generated samples. A generation command loads a checkpoint and accepts seed text, length, temperature, and random seed. Diagnostic mode can show hidden-state norms and per-timestep gradients, making vanishing or exploding behavior visible.

Core Question

How can a network carry information through time, learn sequence dependencies, and assign probabilities to the next symbol?

Concepts You Must Understand First

Markov sequence models and conditional next-token distributions — Project 40.
Recurrent state equation, unrolling through time, and shared parameters.
BPTT and gradient accumulation across timesteps — Deep Learning, Chapter 10.
Cross-entropy, perplexity, categorical sampling, and temperature.
Vanishing/exploding gradients, clipping, and truncated BPTT.

Build Milestones

Build deterministic vocabulary maps and a baseline frequency/Markov generator.
Implement one recurrent step and unrolled forward loss on very short sequences.
Derive BPTT for all parameters; check gradients with sequence length one, then several steps.
Add clipping, mini-batching or stream chunks, state-reset policy, checkpoints, and sampling.
Train on a real corpus and analyze how temperature and context length affect output.

Hints in Layers

With sequence length one, BPTT reduces to ordinary backpropagation and is much easier to verify.
Remember that recurrent parameters are reused, so their gradients sum contributions from every timestep.
Sample from probabilities rather than taking argmax; temperature below one sharpens and above one diversifies.

Common Pitfalls and Debugging

Gradient check passes for one step but fails for sequences: shared-parameter contributions are overwritten. Accumulate gradients across all timesteps.
Loss becomes nan: exploding gradients or unstable softmax. Track norms, clip by global norm, and use log-sum-exp.
Generated text repeats forever: argmax decoding or overly low temperature collapses diversity. Use categorical sampling and inspect learned probabilities.

Definition of Done

Vocabulary encoding/decoding round-trips arbitrary supported text.
Multi-timestep recurrent gradients pass numerical checks on a tiny problem.
Training logs loss, perplexity, gradient norms, and deterministic checkpoints.
Generation supports seed text, temperature, length, and reproducible random sampling.
Samples improve visibly over a Markov/frequency baseline.
A diagnostic explains at least one vanishing/exploding-gradient experiment.

Project 50: Neural Network Framework

Difficulty: Level 5: Master
Time: 4-8 weeks
Language: Python with NumPy
Prerequisites: Projects 44-49
Merged from: NN BrainInABox; ML-Found P20

What You Will Build

Turn the ad hoc components from earlier neural projects into a cohesive Mini-PyTorch/BrainInABox library. Define tensors with autograd, trainable parameters, nested modules, linear and convolutional layers, BatchNorm, activations, losses, Sequential, optimizers such as SGD and Adam, data loaders, train/eval modes, registered parameters and buffers, state dictionaries, serialization, and a reusable training loop. The public API should feel small and consistent while preserving correct parameter discovery, gradient accumulation, broadcasting, normalization state, and numerical stability. Prove the framework by rebuilding the earlier MLP and a CNN without model-specific training code.

Real World Outcome

A separate example program imports your package, prints a model summary and parameter count, trains an MLP on MNIST and a BatchNorm-equipped CNN on CIFAR-10, saves a checkpoint, loads it in a fresh process, and reproduces predictions. Training and evaluation runs demonstrate that BatchNorm uses batch statistics only while training and saved running statistics during evaluation. The test suite compares selected operations, gradients, buffers, and optimizer updates with trusted numerical/reference results.

Core Question

How do deep-learning frameworks organize differentiation, parameters, layers, optimization, data, and persistence into one safe composable system?

Concepts You Must Understand First

Dynamic autograd graphs and tensor operations — Project 44.
Module composition and parameter ownership — PyTorch nn.Module design as a reference.
Dense/convolutional forward and backward contracts — Projects 45 and 48.
Batch normalization: per-channel statistics, learnable scale/shift, running mean/variance buffers, epsilon, and distinct train/eval behavior.
Optimizer state, momentum, Adam moments, and zero-gradient lifecycle.
Python data model/operator overloading — Fluent Python by Luciano Ramalho.

Build Milestones

Stabilize tensor/autograd behavior, broadcasting rules, dtypes, shape validation, and gradient tests.
Implement Parameter, Module, recursive parameter registration, Linear, activations, losses, and containers.
Add SGD/Adam, data loader iteration, train/eval mode, and a generic fit/evaluate loop.
Add Conv2D, pooling, BatchNorm forward/backward behavior, registered running-statistic buffers, model summaries, state dictionaries, and safe save/load round trips.
Port the MNIST MLP and train a CIFAR-10 CNN through the generic API; publish accuracy targets, API documentation, benchmarks, and limitations.

Hints in Layers

Write the user-facing example before the internals; it becomes the API contract.
Keep parameter discovery recursive and identity-based so shared parameters are not duplicated.
Test serialization in a fresh process, not merely by reusing in-memory objects; verify BatchNorm output stays stable in evaluation mode after reload.

Common Pitfalls and Debugging

Optimizer updates miss some weights: parameters were stored in ordinary containers or not recursively registered. Test nested modules and lists explicitly.
Broadcasted operations return wrong gradients: backward must reduce expanded axes. Centralize an “unbroadcast” helper and test it independently.
Loaded model predicts differently: buffers, architecture metadata, or dtype were omitted. Version the state format and validate keys/shapes on load.
Evaluation changes with batch composition: BatchNorm is still using current-batch statistics. Propagate train/eval mode recursively and use saved running buffers during inference.

Definition of Done

Tensor/autograd operations pass unit and finite-difference tests, including broadcasting.
Nested/shared parameters are discovered exactly once and zeroed predictably.
SGD and Adam match hand-computed updates for several steps.
MLP and CNN examples train through one generic loop to declared accuracy targets.
BatchNorm gradients pass numerical checks, running statistics update only during training, and train/eval outputs match a trusted reference.
A CIFAR-10 CNN trains end to end through the public framework API and documents its reproducible validation accuracy.
Save/load in a fresh process preserves parameters and predictions.
Public API docs, model summary, tests, benchmarks, and known limitations are included.

Phase 8 — Advanced and Research-Level Mathematics (Projects 51-61)

Project 51: Convex Optimization and KKT Solver

Difficulty: Level 4: Expert
Time: 2-3 weeks
Language: Python
Prerequisites: Projects 20, 26, 33-34, and 36
Source: ML-Math P25

What You Will Build

Implement a small constrained convex-optimization system that reports mathematical certificates, not only a low objective value. Support at least two problem families, such as L2-regularized logistic regression under an L1 budget and a convex quadratic program with affine inequalities. Validate declared problem structure, run a projected, primal-dual, or barrier-based method, and record primal/dual objectives, feasibility, stationarity, complementary slackness, and duality gap each iteration. Include deliberately infeasible and non-convex inputs so the solver distinguishes bad models from failed convergence.

Real World Outcome

A run returns a candidate solution plus a certificate report: objective values, duality gap, every KKT residual, active constraints, stopping reason, and pass/fail tolerance. Convergence plots show whether optimization, feasibility, and certification improve together; infeasible cases receive explicit diagnostics.

Core Question

How can you certify that a constrained solution is genuinely optimal rather than merely the best point your algorithm happened to find?

Concepts You Must Understand First

Convex sets/functions and composition rules — Boyd and Vandenberghe, Convex Optimization.
Lagrangians, dual functions, weak/strong duality, and Slater’s condition.
KKT stationarity, primal feasibility, dual feasibility, and complementary slackness.
Projected/primal-dual gradient methods, line search, and stopping criteria.
Conditioning and stable linear solves — Projects 29-30.

Build Milestones

Define typed objectives/constraints with gradient, shape, and convexity checks.
Implement one solver and test it first on a one- or two-dimensional problem with known optimum.
Derive dual quantities and compute normalized KKT residuals and duality gap every iteration.
Add a second constrained ML problem plus infeasible and invalid-problem cases.
Compare solutions with a trusted solver and generate a certificate-focused report.

Hints in Layers

Separate “candidate generation” from “candidate certification” so either component can be tested alone.
Normalize residuals by relevant data/solution scales before comparing them with tolerances.
Plot objective and feasibility together; objective improvement can hide constraint violation.

Common Pitfalls and Debugging

Objective falls while the answer remains invalid: feasibility is ignored. Track maximum constraint violation and reject uncertified output.
KKT residual stays large near a trusted optimum: Lagrangian sign or multiplier convention is inconsistent. Verify a hand-solvable inequality case.
Duality gap is negative unexpectedly: primal/dual formulas or numerical tolerances are wrong. Recheck minimization bounds and report tiny roundoff separately.

Definition of Done

Two constrained convex problem families are supported and validated.
Every run reports all KKT components, primal/dual values, and stopping reason.
Known analytic examples and a trusted solver agree within justified tolerances.
Infeasible and non-convex inputs do not receive false optimality claims.
Convergence plots distinguish objective, feasibility, residual, and gap behavior.
The tolerance/certification policy is documented with scale-aware reasoning.

Project 52: Real Analysis Foundations Lab

Difficulty: Level 4: Expert
Time: 2 weeks
Language: Python
Prerequisites: Projects 6, 26, and 29-30
Source: foundations portion of ML-Math P27

What You Will Build

Create a counterexample-driven numerical lab for limits, continuity, completeness intuition, compact domains, and pointwise versus uniform convergence of function sequences. A function-family runner samples domains, refines adaptively near suspicious regions, compares pointwise trajectories with supremum error, and records the exact domain and norm behind every claim. Include canonical examples such as x^n on [0,1], moving spikes, continuous approximations to discontinuous limits, and families whose behavior changes when an endpoint is removed.

Real World Outcome

For a selected family, the lab plots several functions, pointwise errors, estimated supremum error versus index, and refinement locations. It produces a report such as “pointwise convergence passes on sampled points; uniform convergence fails because worst-case error remains one near the endpoint,” with an explicit counterexample witness.

Core Question

What kind of convergence or continuity are you claiming, on which domain, and what guarantee actually follows?

Concepts You Must Understand First

Epsilon definitions of sequence/function limits and continuity — Stephen Abbott, Understanding Analysis.
Pointwise versus uniform convergence and the supremum norm.
Compactness, completeness, Cauchy sequences, and why domain assumptions matter.
Lipschitz continuity and sensitivity bounds.
Counterexamples as tests of overgeneralized conjectures.

Build Milestones

Implement a function-family/domain interface and pointwise trajectory explorer.
Estimate supremum error and add adaptive refinement around peaks, endpoints, and discontinuities.
Reproduce at least four examples separating pointwise and uniform behavior.
Add continuity/Lipschitz diagnostics and experiments that alter domain assumptions.
Generate a report that states hypotheses, evidence, limitations, and a witness for failed claims.

Hints in Layers

A finite grid cannot prove a universal statement; frame results as diagnostics backed by theory.
For x^n, sample increasingly close to one or optimize the error instead of trusting a fixed grid.
Always display the norm and domain beside an error curve.

Common Pitfalls and Debugging

Lab incorrectly reports uniform convergence: coarse sampling misses a narrowing boundary region. Add adaptive refinement and a known analytic witness.
Changing the grid changes the conclusion: the numerical claim lacks stability. Report grid-sensitivity and avoid labeling it proof.
Average error is small but worst case is large: mean error was substituted for supremum error. Compute and visualize both.

Definition of Done

At least four function families include known limits and domain assumptions.
Two pointwise-but-not-uniform examples are correctly diagnosed.
Supremum estimates are tested against grid refinement or analytic values.
Compactness/endpoint changes produce an explained change in behavior.
Every report distinguishes numerical evidence from mathematical proof.
A final note records at least one misconception corrected by a counterexample.

Project 53: Sequence and Series Convergence Lab

Difficulty: Level 2: Intermediate
Time: 1 week
Language: Python
Prerequisites: Projects 6, 26, 29, and 52
Source: ML-Math P21

What You Will Build

Build a convergence workbench for arithmetic, geometric, harmonic, alternating, power, and user-defined sequences or series. Track terms and partial sums, apply appropriate comparison/ratio/root/alternating diagnostics, and estimate truncation error when theory supplies a bound. Stopping rules must combine evidence across a window instead of declaring convergence from one small step. Add naive and compensated summation, multiple floating-point precisions where available, log-scale plots, and adversarial examples that look converged over short windows but are not.

Real World Outcome

A CLI accepts a family and tolerance, plots terms/partial sums/error, and reports likely convergence or divergence, theoretical test applicability, estimated limit, justified error bound, number of terms, precision mode, and stopping reason. A geometric series matches its closed form while harmonic and oscillatory counterexamples expose false stopping rules.

Core Question

When does an infinite process become a reliable finite computation, and what evidence justifies stopping?

Concepts You Must Understand First

Formal sequence/series convergence and Cauchy intuition — Concrete Mathematics by Graham, Knuth, and Patashnik.
Partial sums and geometric/alternating remainder bounds.
Comparison, ratio, root, and integral tests with their applicability conditions.
Absolute versus conditional convergence and rearrangement risk.
Floating-point cancellation, resolution limits, and compensated summation.

Build Milestones

Implement generators and partial-sum tracking for at least six representative families.
Add rolling numerical diagnostics and theory-aware convergence tests with “not applicable” states.
Implement available remainder/error bounds and adaptive term budgeting for a requested tolerance.
Compare naive and compensated summation across magnitude/precision regimes.
Build false-convergence and divergence cases and document each failure signature.

Hints in Layers

Separate mathematical classification from a numerical stopping decision; neither automatically proves the other.
A single small delta is weak evidence—use windows, trends, and known bounds.
Plot absolute error or term magnitude on a log scale to distinguish rates.

Common Pitfalls and Debugging

A divergent series is marked converged: increments temporarily fall below tolerance. Require sustained behavior and theory-aware warnings.
Partial sums stop changing: floating-point resolution, not convergence, may be responsible. Compare precision modes and compensated summation.
Ratio test returns one and code decides anyway: the test is inconclusive. Preserve an explicit inconclusive state and try another method.

Definition of Done

At least six convergent/divergent/oscillatory families are supported.
Diagnostics distinguish pass, fail, inconclusive, and test-not-applicable.
Known remainder bounds drive an adaptive tolerance mode.
False-convergence counterexamples defeat naive stopping rules.
Compensated versus naive summation is measured and explained.
Reports state family, precision, evidence, bound, and stopping reason.

Project 54: Measure-Theoretic Probability Sandbox

Difficulty: Level 4: Expert
Time: 2 weeks
Language: Python
Prerequisites: Projects 12, 14, 26, 29, and 52-53
Source: ML-Math P26

What You Will Build

Construct a finite-world sandbox where formal probability objects are executable and inspectable. Represent finite sample spaces, event families, generated sigma-algebras, and probability measures; validate empty/full events, complements, unions, normalization, non-negativity, and additivity. Let users define random variables and target measurable sets, then test measurability through preimages. Compute distributions and expectations both from atoms and induced values, and add experiments illustrating almost-sure, in-probability, and expectation behavior without claiming finite simulations prove infinite theorems.

Real World Outcome

Given a finite coin tree or dice experiment, the program prints the generated sigma-algebra, measure-validation results, preimage evidence for measurability, induced distribution, and matching expectation calculations. Invalid event collections and non-additive weights produce precise counterexamples showing the failed axiom.

Core Question

What structural assumptions make probability statements, random variables, and expectations mathematically valid?

Concepts You Must Understand First

Sigma-algebras and probability measures — Patrick Billingsley, Probability and Measure.
Generated event systems, closure, atoms, and finite additivity/countable-additivity intuition.
Measurable functions through preimages of measurable sets.
Expectation as a Lebesgue integral, reduced to weighted sums in finite spaces.
Almost-sure, in-probability, and mean convergence as distinct modes.

Build Milestones

Represent finite sample spaces/events and generate closure under complement and union.
Validate sigma-algebra axioms and probability-measure normalization/additivity with counterexamples.
Implement random variables, induced distributions, and explicit preimage measurability checks.
Compute expectation from sample atoms and grouped output values; require agreement.
Add convergence-mode demonstrations and a report connecting assumptions to ML probability claims.

Hints in Layers

Use immutable sets (frozenset) so events can themselves be members of collections.
In finite spaces, generate a sigma-algebra iteratively until no new complements/unions appear.
Start measurability with every subset of a finite codomain, then try a coarser target sigma-algebra.

Common Pitfalls and Debugging

Any collection of events is accepted: closure was not checked to a fixed point. Return the missing complement or union as a witness.
Random variable is assumed measurable: its preimages were never tested. Enumerate target measurable sets and report the first invalid preimage.
Two expectation methods disagree: probabilities were not grouped correctly or measure additivity failed. Validate the measure first and print per-value contributions.

Definition of Done

Sigma-algebra generation and validation work on several finite spaces.
Invalid structures return a concrete failed axiom and witness events.
Measurability is decided through explicit preimage tests.
Atom-based and induced-distribution expectations agree within tolerance.
At least two convergence modes are contrasted with honest simulation limitations.
A short report connects formal assumptions to one ML convergence statement.

Project 55: Statistical Learning and Generalization-Bounds Lab

Difficulty: Level 4: Expert
Time: 2 weeks
Language: Python
Prerequisites: Projects 37, 41, 52, and 54
Source: generalization-bounds portion of ML-Math P27, begun in Project 52

What You Will Build

Build a controlled experimental lab that compares empirical train/test gaps with theoretical concentration and capacity-based bounds. Generate or load classification problems, define nested hypothesis classes such as bounded linear predictors and decision trees of increasing depth, and estimate empirical risk over repeated samples. Compute Hoeffding-style fixed-hypothesis bounds and at least one class-capacity bound or defensible proxy, then vary sample size, class complexity, regularization, confidence level, and data noise. The report must show when a bound is valid but loose, when assumptions fail, and why observing one split does not establish generalization.

Real World Outcome

A reproducible sweep produces learning curves, observed generalization gaps, bound curves, violation-frequency checks over many trials, and a table of assumptions. For each configuration, the report states whether the bound covered the observed gap, how loose it was, and what complexity/sample-size tradeoff explains the result.

Core Question

How can finite training performance support a claim about unseen data, and what price do model capacity and confidence demand?

Concepts You Must Understand First

Empirical versus population risk and hypothesis classes.
Concentration inequalities such as Hoeffding’s inequality — Understanding Machine Learning by Shalev-Shwartz and Ben-David.
Uniform convergence, supremum deviations, and the distinction from pointwise convergence — Project 52.
Capacity measures: finite-class union bounds, VC dimension, or empirical complexity proxies.
IID sampling assumptions, confidence parameters, multiple trials, and selection bias.

Build Milestones

Implement repeated seeded sampling and empirical/population-risk estimates on a distribution with known test behavior.
Add a fixed-hypothesis concentration bound and verify its claimed failure frequency across trials.
Add nested model classes and a capacity-aware bound or clearly labeled empirical proxy.
Sweep sample size, complexity, noise, and regularization; plot gaps and bounds together.
Produce an assumption ledger and examples of valid-but-vacuous and invalidly-applied bounds.

Hints in Layers

Begin with bounded 0/1 loss; many simple concentration formulas require bounded variables.
Use a large independent evaluation sample as an approximation to population risk and label it honestly.
A bound above one for classification can be mathematically valid but practically vacuous—show that explicitly.

Common Pitfalls and Debugging

Bound is violated far more than its confidence permits: assumptions, constants, or repeated-selection effects are wrong. Freeze the hypothesis before evaluation and test the simplest case.
Bound improves when model class grows: capacity term has the wrong sign or class size. Unit-test monotonicity with synthetic class counts.
One successful trial is presented as proof: coverage is probabilistic. Run many independent trials and compare empirical violation rate with delta.

Definition of Done

Fixed-hypothesis and class-level guarantees are clearly distinguished.
Bound implementation passes analytic sanity and monotonicity tests.
Repeated trials measure coverage/violation frequency against claimed confidence.
Sweeps reveal sample-size, complexity, noise, and regularization effects.
At least one vacuous bound and one broken-assumption case are explained.
Every chart states loss range, hypothesis class, sampling assumptions, and confidence.

Project 56: Functional Analysis and RKHS Kernel Lab

Difficulty: Level 5: Master
Time: 3 weeks
Language: Python
Prerequisites: Projects 20, 24, 51-52, and 55
Source: ML-Math P28

What You Will Build

Create a kernel-method workbench that turns Hilbert-space language into testable computations. Implement linear, polynomial, and RBF Gram matrices; verify symmetry and positive semidefiniteness numerically; and show how kernels define similarity without explicitly constructing high-dimensional features. Build dual kernel ridge regression, compare it with an explicit primal feature map where one exists, and sweep regularization to plot prediction error against the RKHS norm proxy. Include ill-conditioned and intentionally invalid kernels, scaling diagnostics, and at least one approximation such as Nyström or random Fourier features.

Real World Outcome

Given a small nonlinear dataset, the lab reports kernel eigenvalues/PSD status, condition number, selected regularization, train/test error, RKHS norm, and primal-versus-dual prediction difference. It plots fitted functions or decision surfaces and demonstrates how kernel width and regularization control smoothness and generalization.

Core Question

How can learning in an infinite-dimensional function space reduce to a finite kernel matrix computation?

Concepts You Must Understand First

Normed, inner-product, and Hilbert spaces; bounded linear operators.
Positive semidefinite kernels and Gram matrices — Schölkopf and Smola, Learning with Kernels.
Reproducing property and the representer theorem.
Ridge regularization as control of function-space norm.
Eigenvalues, conditioning, duality, and stable solves — Projects 24, 30, and 51.

Build Milestones

Implement kernel interfaces and PSD/symmetry/conditioning diagnostics.
Implement dual kernel ridge fitting and prediction with stable linear solves.
Verify primal/dual equivalence for linear and finite polynomial features.
Sweep kernel hyperparameters and lambda; plot norm, error, smoothness, and conditioning.
Add an invalid-kernel test and one scalable approximation with quality/runtime comparison.

Hints in Layers

Never compute a matrix inverse explicitly; solve the regularized linear system.
Tiny negative eigenvalues may be roundoff, while substantial negatives indicate an invalid kernel—use scale-aware tolerances.
Fit feature scaling only on training data; RBF distance is extremely scale-sensitive.

Common Pitfalls and Debugging

Solver becomes unstable as lambda approaches zero: Gram matrix is ill-conditioned. Report condition number and use Cholesky/robust solves with regularization.
RBF predicts almost a constant or memorizes every point: kernel width is at an extreme. Visualize the Gram matrix and sweep width on a log scale.
Kernel declared valid despite negative eigenvalues: PSD tolerance ignores matrix scale. Normalize the tolerance to the largest eigenvalue/norm.

Definition of Done

Linear, polynomial, and RBF kernels pass symmetry and PSD diagnostics.
Dual kernel ridge regression works without explicit inversion.
Explicit-feature primal and kernel dual predictions agree where comparable.
Regularization/kernel sweeps report error, norm, smoothness, and conditioning.
Invalid and near-PSD numerical cases are distinguished responsibly.
One approximation method includes accuracy, runtime, and memory tradeoffs.

Project 57: Graph Message-Passing Playground

Difficulty: Level 3: Advanced
Time: 1-2 weeks
Language: Python
Prerequisites: Projects 9, 20, 24, and 45
Source: ML-Math P29

What You Will Build

Build a sparse graph-learning playground for node signals and neighborhood aggregation. Parse edge lists and node features, compute degree/component statistics and simple centrality baselines, then implement normalized message-passing layers with configurable self-loops, aggregation, weights, activation, and depth. Train a small node classifier or perform fraud/risk signal propagation while exposing each node’s incoming messages. Compare raw, mean, symmetric-normalized, and learned transformations; measure hub bias, isolated-node behavior, receptive-field growth, and over-smoothing as layers increase.

Real World Outcome

On a citation, social, or payment graph, the program reports node/edge/component counts, task metrics, top-ranked nodes, and performance by degree bucket. An inspection view explains a selected node’s updated representation from its neighbors, while hop-wise plots warn when embeddings collapse toward indistinguishable values.

Core Question

How does relational structure change the evidence available for a prediction, and how should neighborhood information be aggregated safely?

Concepts You Must Understand First

Graphs, adjacency lists/matrices, connected components, degree, and sparse storage.
Permutation invariance/equivariance of node aggregation.
Message-passing neural networks and neighborhood aggregation.
Degree normalization, self-loops, and hub bias.
Node-level evaluation, leakage through graph splits, and over-smoothing.

Build Milestones

Parse sparse graphs/features and validate duplicates, self-loops, components, and node IDs.
Implement transparent one-hop aggregation and compare normalization choices on hand-built graphs.
Add trainable message/update functions and a node-classification or ranking task.
Evaluate by hop depth, degree bucket, connected component, and random seed.
Visualize selected message flows and quantify embedding similarity/over-smoothing.

Hints in Layers

Test permutation behavior by renumbering nodes; outputs should renumber correspondingly.
Use sparse edge-index operations instead of materializing a dense adjacency matrix.
Include isolated nodes and high-degree hubs in unit tests because normalization edge cases live there.

Common Pitfalls and Debugging

Memory explodes on a moderate graph: dense adjacency was constructed. Store edges/CSR data and aggregate only present edges.
High-degree nodes dominate predictions: raw summation introduces scale bias. Compare mean and symmetric normalization and report degree slices.
Validation is suspiciously high: graph edges or labels leak across the split. Define the inductive/transductive protocol and audit preprocessing visibility.

Definition of Done

Sparse representation handles a realistically sized graph without dense adjacency.
Hand-built graph tests verify normalization, self-loops, isolated nodes, and permutation behavior.
A node task beats feature-only and structure-only baselines where appropriate.
Evaluation includes degree/component slices and multiple seeds.
Message inspection explains one prediction through concrete neighbors.
Over-smoothing is measured and tied to a recommended depth.

Project 58: Spectral Graph and Laplacian Clusterer

Difficulty: Level 4: Expert
Time: 2 weeks
Language: Python
Prerequisites: Projects 9, 24-25, and 38
Source: ML-Math P30

What You Will Build

Implement spectral clustering from sparse graph structure. Construct adjacency and degree matrices plus unnormalized, random-walk, and symmetric-normalized Laplacians. Compute the smallest eigenpairs, identify trivial zero modes and connected components, use nontrivial eigenvectors as a spectral embedding, and apply your Project 38 k-means implementation. Add eigengap-based suggestions for cluster count, row normalization where appropriate, cut/modularity/silhouette metrics, and repeated-seed stability analysis. Compare spectral clusters with k-means on raw node features or adjacency summaries.

Real World Outcome

Given a network edge list, the clusterer displays the spectrum and eigengaps, recommends k, plots nodes in spectral coordinates, colors recovered communities, and reports normalized cut, modularity, embedding silhouette, and seed stability. Synthetic two-moons and stochastic-block graphs show structures that ordinary Euclidean k-means misses.

Core Question

Why do low-frequency eigenvectors of a graph Laplacian reveal communities hidden in the raw edge representation?

Concepts You Must Understand First

Adjacency/degree matrices, graph cuts, and connected components.
Eigenvalues/eigenvectors and invariant subspaces — Projects 24-25.
Unnormalized and normalized graph Laplacians — Fan Chung, Spectral Graph Theory.
Rayleigh quotient, Fiedler vector, spectral embedding, and eigengaps.
K-means initialization, seed sensitivity, and clustering metrics — Project 38.

Build Milestones

Construct and validate all three Laplacian variants on small known graphs.
Compute smallest sparse eigenpairs and interpret zero modes/components and the Fiedler vector.
Implement spectral embedding, optional row normalization, and k-means clustering.
Add eigengap selection, cut/modularity/silhouette, and multi-seed stability reports.
Compare variants and baselines on synthetic and one real graph.

Hints in Layers

The number of near-zero Laplacian eigenvalues should match the number of connected components.
Exclude trivial constant eigenvectors from clustering features; disconnected graphs may have several.
Eigenvector signs can flip between runs without changing meaning; compare subspaces or clusters, not raw signs.

Common Pitfalls and Debugging

Clusters reflect degree rather than community: normalization is unsuitable for skewed degrees. Compare symmetric/random-walk Laplacians.
Results change wildly across runs: weak eigengap or k-means instability. Repeat seeds and report adjusted agreement/stability.
Eigensolver is slow or runs out of memory: dense decomposition is used. Keep Laplacians sparse and request only the smallest needed eigenpairs.

Definition of Done

Laplacian variants pass symmetry, row-sum, PSD, and component tests where applicable.
Sparse eigenpair computation avoids full dense decomposition.
Trivial modes are correctly excluded and eigengap choice is explained.
Quality report includes cut/modularity, embedding metric, and seed stability.
Spectral clustering beats a raw-space baseline on a non-convex synthetic case.
Normalization tradeoffs are demonstrated on a degree-imbalanced graph.

Project 59: Random Matrix Theory for High-Dimensional ML

Difficulty: Level 5: Master
Time: 2-3 weeks
Language: Python
Prerequisites: Projects 12, 24-25, 30, and 54
Source: ML-Math P31

What You Will Build

Build a simulation lab that separates covariance signal from high-dimensional noise. Generate Gaussian noise and spiked-covariance datasets across sample/feature aspect ratios, compute correctly normalized sample covariance spectra, and overlay empirical eigenvalue histograms with Marchenko-Pastur bulk support. Detect eigenvalues beyond the theoretical edge, measure false positives on pure-noise trials, and test how signal strength and finite sample size affect detection. Use spectrum-informed component retention or shrinkage in a downstream prediction/reconstruction task and compare it with naive explained-variance PCA.

Real World Outcome

A configured run reports n, p, aspect ratio, predicted bulk edges, observed extreme eigenvalues, detected spikes, and bootstrap-calibrated false-positive evidence. Plots show bulk/outliers across regimes, and a decision table demonstrates whether theory-guided truncation improves held-out performance or covariance estimation.

Core Question

When the number of dimensions is comparable to the number of observations, how can you distinguish genuine low-rank structure from noise eigenvalues?

Concepts You Must Understand First

Sample covariance, eigenvalue spectra, PCA, and aspect ratio.
Concentration in high dimensions — Roman Vershynin, High-Dimensional Probability.
Marchenko-Pastur bulk law and its asymptotic support.
Spiked covariance models, spectral outliers, and detectability thresholds.
Shrinkage, finite-sample calibration, and train/test evaluation.

Build Milestones

Simulate pure-noise matrices over several p/n regimes and validate covariance normalization.
Overlay empirical spectra with Marchenko-Pastur density/support and track extreme eigenvalues.
Add controllable low-rank spikes and measure detection rate versus strength/sample size.
Estimate false-positive rates and add bootstrap or repeated-simulation edge calibration.
Compare spectrum-guided truncation/shrinkage with naive PCA on a downstream task.

Hints in Layers

State whether data matrix entries and covariance use 1/n or 1/(n-1); formulas must match.
Keep variance scaling consistent before comparing with the theoretical bulk edges.
A finite-sample eigenvalue just beyond the asymptotic edge is not automatically signal—measure null fluctuations.

Common Pitfalls and Debugging

Pure noise produces many “signals”: covariance scaling or bulk formula is mismatched. Validate on large synthetic Gaussian cases first.
Results disagree after transposing data: sample and feature axes were confused. Assert shapes and define covariance orientation explicitly.
All outliers are retained without validation: finite-sample edge variation is ignored. Calibrate false-positive rate through null simulations.

Definition of Done

Pure-noise spectra match theoretical bulk behavior across multiple aspect ratios.
Covariance normalization and axis conventions are explicit and tested.
Spiked experiments report detection and false-positive rates over repeated trials.
Finite-sample edge uncertainty is measured rather than dismissed.
Theory-guided retention/shrinkage is compared with naive PCA on held-out data.
The final report distinguishes asymptotic prediction, simulation evidence, and practical recommendation.

Project 60: Decision-Ready ML Math Workbench

Difficulty: Level 5: Master
Time: 3-4 weeks
Language: Python
Prerequisites: Projects 14, 25, 30, 37, 41, 51, and 58-59
Source: ML-Math Final Decision-Ready ML Math Workbench

What You Will Build

Integrate the strongest diagnostics from the curriculum into one reproducible decision workbench. It ingests a versioned tabular dataset, enforces a schema/data contract, creates leakage-safe splits, trains a stable probabilistic model, verifies loss and gradients, evaluates discrimination and calibration, and—when applicable—adds convex/KKT optimization certificates. For relational or high-dimensional features, run spectral graph and random-matrix risk checks. The output must be a decision report with assumptions, evidence, uncertainty, threshold/cost analysis, failure flags, model/data hashes, and a recommendation that can also be “do not deploy.”

Real World Outcome

One command creates an immutable run directory containing configuration, dataset fingerprint, split IDs, model artifact, metrics, calibration/reliability plots, numerical-stability probes, optional KKT and spectral reports, threshold decision table, and a concise deploy/hold verdict. A replay command reproduces the report from the manifest.

Core Question

What mathematical and numerical evidence is sufficient to turn a model score into an auditable real-world decision?

Concepts You Must Understand First

Leakage-safe ML pipelines and reproducibility contracts — Project 41.
Loss engineering, calibration, proper scoring rules, and threshold costs — Project 37.
Conditioning, stable optimization, and adversarial scale tests — Projects 29-30.
Convex certificates and KKT residuals — Project 51.
Spectral graph and high-dimensional noise diagnostics — Projects 58-59.

Build Milestones

Define the decision, stakeholders, cost matrix, data contract, and explicit abstain/hold conditions.
Build deterministic ingestion/splitting and train a baseline plus candidate model with stability checks.
Add discrimination, calibration, subgroup/error-slice, uncertainty, and threshold-value analyses.
Integrate applicable KKT, conditioning, graph, and random-matrix diagnostics behind clear capability checks.
Produce signed/hash-linked artifacts, a replay command, and an evidence-based recommendation template.

Hints in Layers

Start from the final decision table and work backward to the evidence each column requires.
Treat optional diagnostics as declared “not applicable,” never silently absent.
Create deliberate bad-data, leakage, miscalibration, and ill-conditioning drills to test safeguards.

Common Pitfalls and Debugging

Report is long but cannot support a decision: metrics are disconnected from costs and thresholds. Tie each metric to an action or risk.
Replay gives different results: data, splits, dependencies, or configuration are not fully captured. Hash and persist every effective input.
All checks pass on a deliberately corrupted case: diagnostics are decorative. Add automated failure drills with expected blocking outcomes.

Definition of Done

Dataset, configuration, splits, code/model version, and effective dependencies are traceable.
Report includes baseline comparison, uncertainty, calibration, threshold costs, and error slices.
Numerical, KKT, graph, and spectral checks run when applicable or state why not.
At least four failure drills trigger the intended warning or deployment block.
Replay reproduces material outputs within documented tolerances.
Final recommendation cites evidence and supports a legitimate “hold” verdict.

Project 61: GPT from Scratch

Difficulty: Level 5: Master
Time: 3+ months
Language: Python using the Project 50 framework; NumPy backend first
Prerequisites: Projects 37, 44, and 49-50; Project 60 is a recommended final readiness check
Source: ML-Found Final GPT from Scratch

What You Will Build

Build a small decoder-only transformer language model from first principles. Begin with a character tokenizer for correctness, then optionally implement byte-pair encoding. Add token/positional embeddings, scaled dot-product causal self-attention, multi-head attention, feed-forward blocks, residual connections, layer normalization, dropout if supported, final vocabulary logits, stable cross-entropy, autoregressive batching, Adam training, checkpoints, and temperature/top-k generation. Implement the missing tensor operations in your Project 50 framework and numerically verify each new backward rule before composing the full model.

Real World Outcome

A training command prints vocabulary, context length, architecture, parameter count, train/validation loss, perplexity, throughput, gradient norms, and periodic samples. A generation command loads a checkpoint and accepts prompt, length, temperature, top-k, and seed. Attention-mask tests prove tokens cannot see the future, and small-corpus output becomes measurably better than unigram/Markov baselines.

Core Question

How can causal self-attention, repeated transformer blocks, and next-token likelihood produce a model that learns and generates text?

Concepts You Must Understand First

Tokenization, embeddings, conditional language modeling, cross-entropy, and perplexity.
Query-key-value scaled dot-product attention — Vaswani et al., Attention Is All You Need.
Causal masks, multi-head attention, residual streams, and layer normalization.
Reverse-mode autograd/framework design — Projects 44 and 50.
Autoregressive sequence training and sampling — Project 49.

Build Milestones

Implement tokenizer, shifted next-token batches, and unigram/Markov baselines.
Implement embeddings, masked single-head attention, and hand-calculated causal-mask tests.
Add multi-head attention, feed-forward block, residuals, layer norm, and gradient checks.
Stack decoder blocks, overfit a tiny batch, then train a small model with checkpoint/resume.
Add deterministic generation controls, attention visualization, evaluation, and scaling benchmarks.

Hints in Layers

Use tiny dimensions and a sequence of three tokens to calculate attention scores and masks by hand.
Before full training, force the model to memorize one small batch; inability means an implementation or optimization bug.
Subtract row maxima in softmax, mask future logits before softmax, and test that forbidden probabilities are exactly zero within tolerance.

Common Pitfalls and Debugging

Validation loss is implausibly low: targets leaked through an incorrect causal mask or batch shift. Perturb future tokens and prove earlier logits are unchanged.
Training produces nan: unstable softmax/layer norm or exploding gradients. Inspect per-block norms, use epsilon correctly, and clip gradients if necessary.
Generation repeats or becomes nonsense: decoding settings or undertraining dominate. Compare temperatures/top-k and report validation loss before judging samples.

Definition of Done

Tokenizer round-trips text and batching aligns every input with its next-token target.
Causal-mask tests prove no output depends on future tokens.
New attention/layer-normalization autograd operations pass numerical checks.
The complete model deliberately overfits a tiny batch before corpus training.
Checkpoint/resume preserves optimizer state and reproduces the loss trajectory.
Generation is seeded/configurable and evaluation compares against simpler language baselines.

Project Comparison Table

Range	Typical difficulty	Typical time	Main evidence of mastery
1-7	Beginner	4-10 hours each	Correct transcripts, domains, residuals, and explanations
8-14	Beginner to intermediate	8-16 hours each	Search traces, seeded experiments, uncertainty reports
15-25	Intermediate	10-20 hours each	Interactive geometry, matrix checks, reconstruction errors
26-32	Intermediate	12-24 hours each	Error curves, convergence studies, stable simulations
33-41	Intermediate to advanced	16-30 hours each	Reproducible training, held-out metrics, leakage controls
42-50	Advanced	20-40 hours each	Gradient checks, training diagnostics, reusable modules
51-61	Advanced to research	24-60 hours each	Proof-oriented experiments, certificates, bounds, spectra, generation

Recommendation

If you are unsure where to start, begin with Project 1 and move sequentially. Experienced developers should still run the Definition of Done checks rather than skipping by familiarity: the early projects establish the deterministic-output, residual-checking, and reproducibility habits used later.

Do not race toward Project 61. A transformer is a composition of ideas developed throughout the path—token representation, matrix multiplication, probability, softmax, optimization, normalization, autograd, and training-system design. The final build is much more educational when those components are already yours.

Final Overall Project: Mathematical Systems Portfolio

Create one repository that presents the curriculum as a coherent engineering portfolio rather than 61 disconnected exercises.

Give every project a consistent CLI, notebook, or web entry point.
Publish a generated catalog containing project purpose, prerequisites, commands, example output, and validation status.
Share reusable modules for parsing, plotting, random seeding, numeric comparison, datasets, metrics, and experiment records.
Add a reproducibility command that reruns representative tests from every phase.
Include a technical essay tracing one idea—such as gradients, eigenvectors, or probability—from its first appearance to GPT.

Success criteria: a new reader can select any completed project, reproduce its evidence, understand its assumptions, and navigate backward to every prerequisite without consulting another guide.

From Learning to Production

Learning artifact	Production analogue	Additional production work
Calculator and symbolic tools	Computer algebra system	Grammar coverage, exact arithmetic, security limits
Simulators and numerical labs	Scientific computing package	Solver selection, adaptive methods, performance profiling
Classical ML pipeline	Training and model-selection platform	Data governance, lineage, monitoring, deployment
Autograd and neural framework	PyTorch/JAX-style runtime	Accelerators, kernels, memory planning, distributed execution
GPT from scratch	Language-model training stack	Large datasets, safety, scaling, evaluation, serving

Summary

This guide consolidates 93 original project entries into 61 ordered, self-contained projects. Each project now includes the build specification, visible outcome, conceptual checkpoints, milestones, layered hints, failure modes, and testable completion criteria. The default route moves from exact reasoning and observable arithmetic through numerical approximation, statistical learning, neural systems, rigorous mathematics, and transformer integration.

The strongest outcome is not the number of projects completed. It is the ability to turn a mathematical claim into a representation, preserve its invariants, make failure visible, and explain why the resulting software deserves trust.

Additional Resources and References

Official technical references

Primary papers and open books

Original-guide provenance