← Back to all projects

LEARN MATH FOR PROGRAMMING

While you can write a lot of code without thinking about math, a deep understanding of the underlying mathematics allows you to:

Learn Math for Programming: From Foundations to Advanced Applications

Goal: To build a strong, practical foundation in the mathematical concepts that are most critical for software engineering. You will learn by building projects that directly apply mathematical principles to solve real-world programming challenges.

Why Learn Math for Programming?

While you can write a lot of code without thinking about math, a deep understanding of the underlying mathematics allows you to:

Write More Efficient Algorithms: Understand complexity, optimize performance, and choose the right data structures.
Unlock New Fields: Dive into machine learning, computer graphics, cryptography, and scientific computing.
Solve Harder Problems: Break down complex problems into logical, solvable mathematical components.
Think More Rigorously: Improve your ability to reason about code, state, and logic.

After completing these projects, you won’t just use libraries for graphics or machine learning; you’ll understand the principles they are built on.

Core Concept Analysis

The Programmer’s Mathematical Toolkit

┌─────────────────────────────────────────────────────────────────────────┐
│                          Computer Science Problems                       │
│      (Graphics, AI, Security, Search, Simulation, Optimization)          │
└─────────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼ Application of Mathematical Fields
┌──────────────────┬──────────────────┬──────────────────┬──────────────────┐
│ DISCRETE MATH    │  LINEAR ALGEBRA  │    CALCULUS      │   PROBABILITY &  │
│                  │                  │                  │    STATISTICS    │
│ • Logic          │ • Vectors        │ • Derivatives    │ • Bayes' Theorem │
│ • Set Theory     │ • Matrices       │ • Integrals      │ • Distributions  │
│ • Graph Theory   │ • Transformations│ • Optimization   │ • Regression     │
│ • Combinatorics  │ • Vector Spaces  │ (Gradient Descent)│ • Monte Carlo    │
├──────────────────┴──────────────────┴──────────────────┴──────────────────┤
│                  FOUNDATIONAL & CROSS-CUTTING CONCEPTS                   │
│                                                                          │
│ • Number Theory (Modular Arithmetic, Primes) - for Cryptography, Hashing │
│ • Boolean Algebra (Bitwise Operations) - for Low-Level Optimization      │
└─────────────────────────────────────────────────────────────────────────┘

Key Concepts Explained

1. Discrete Mathematics

This is the math of computer science. Unlike continuous math (like calculus), it deals with distinct, countable objects.

Logic & Boolean Algebra: The foundation of all computation, control flow (if/else), and bitwise operations.
Set Theory: The basis for data structures like Set, and for reasoning about groups of items.
Graph Theory: The study of nodes and edges. It’s how you model networks, social connections, dependencies, and navigation.
Combinatorics: The art of counting. Used in analyzing algorithm complexity and probability.

2. Linear Algebra

The math of data. It provides the tools to work with multi-dimensional data and geometric transformations.

Vectors: Represent points in space, or lists of numbers (e.g., a feature vector in ML).
Matrices: Represent transformations (rotate, scale, translate), systems of linear equations, or datasets.
Key Operations: Dot product, cross product, matrix multiplication.

3. Calculus

The math of change. It’s essential for any problem involving continuous systems, simulation, and optimization.

Derivatives: Measure the rate of change. The core of gradient descent, the optimization algorithm that powers modern machine learning.
Integrals: Calculate the total accumulation or area under a curve. Used in physics engines to calculate motion from velocity and acceleration.

4. Probability & Statistics

The math of uncertainty and data. It allows you to make predictions and draw conclusions from data.

Probability: The likelihood of events. Used in randomized algorithms and analyzing risk.
Bayes’ Theorem: A way to update beliefs in light of new evidence. The engine behind spam filters and many diagnostic systems.
Statistics: Tools for describing and analyzing data (mean, median, variance, regression).

Concept Summary Table

Concept Cluster	What You Need to Internalize
Logic & Boolean Algebra	Every `if` statement, every `&&` and `\|\|` is an application of Boolean algebra. Understanding truth tables and logical operations is understanding how computers make decisions.
Set Theory	Sets are the foundation of data structures. Understanding union, intersection, and set operations helps you reason about collections, uniqueness, and membership.
Graph Theory	Networks, dependencies, social connections, state machines—they’re all graphs. Understanding nodes, edges, paths, and cycles is essential for modeling relationships.
Combinatorics	Counting isn’t just for kids. It’s how you analyze algorithm complexity, calculate probabilities, and understand why some problems are computationally hard.
Vectors & Matrices	Data is multi-dimensional. Vectors represent points, directions, and features. Matrices represent transformations, systems of equations, and datasets. Master these and you unlock graphics, ML, and scientific computing.
Matrix Operations	Dot products measure similarity. Cross products give perpendicularity. Matrix multiplication chains transformations. These operations are the vocabulary of linear algebra.
Derivatives	Derivatives measure rates of change. They tell you how a function behaves locally. This is the engine of optimization—finding the best solution by following the gradient.
Integrals	Integrals accumulate change. They calculate areas, volumes, and totals. In physics engines, they turn acceleration into velocity and velocity into position.
Gradient Descent	The algorithm that powers modern AI. By following the negative gradient (the direction of steepest descent), you can minimize any differentiable function.
Probability	The math of uncertainty. Understanding random events, distributions, and expected values lets you build algorithms that work with incomplete information.
Conditional Probability	The probability of A given B has occurred. This is the foundation of Bayesian reasoning and machine learning classification.
Bayes’ Theorem	Update your beliefs based on evidence. It’s how spam filters work, how medical diagnosis systems reason, and how AI makes predictions.
Prime Numbers	The atoms of number theory. They’re the foundation of RSA encryption and many hash functions. Understanding primes means understanding security.
Modular Arithmetic	Math with wraparound. It’s how clocks work, how hash tables avoid collisions, and how cryptography keeps secrets.
GCD & Extended Euclidean	Finding the greatest common divisor and modular inverses. These are the tools that make RSA key generation possible.

Deep Dive Reading By Concept

This section maps each mathematical concept to specific book chapters for deeper understanding. Read these before or alongside the projects to build strong mental models.

Discrete Mathematics (Logic, Sets, Graphs, Combinatorics)

Concept	Book & Chapter
Boolean Logic & Truth Tables	Grokking Algorithms by Aditya Bhargava — Introduction & Ch. 1: “Introduction to Algorithms”
Set Theory Basics	Concrete Mathematics by Graham, Knuth & Patashnik — Ch. 1: “Recurrent Problems” (section 1.2 on sums)
Graph Representation	A Common-Sense Guide to Data Structures and Algorithms by Jay Wengrow — Ch. 18: “Graphs”
Graph Traversal (BFS/DFS)	Grokking Algorithms by Aditya Bhargava — Ch. 6: “Breadth-First Search”
Shortest Path Algorithms	Algorithms, Fourth Edition by Sedgewick & Wayne — Ch. 4: “Graphs” (section 4.4 on shortest paths)
Recursion & Backtracking	Grokking Algorithms by Aditya Bhargava — Ch. 3: “Recursion” & Ch. 4: “Quicksort”
Combinatorics & Counting	Concrete Mathematics by Graham, Knuth & Patashnik — Ch. 5: “Binomial Coefficients”

Linear Algebra (Vectors, Matrices, Transformations)

Concept	Book & Chapter
Vectors & Vector Operations	Math for Programmers by Paul Orland — Ch. 1: “Learning Math with Code” & Ch. 2: “Drawing with 2D Vectors”
Matrices & Matrix Multiplication	Math for Programmers by Paul Orland — Ch. 4: “Transforming Vectors with Matrices”
3D Transformations	Computer Graphics from Scratch by Gabriel Gambetta — Ch. 3: “Perspective Projection” & Ch. 7: “Filled Triangles”
Linear Transformations	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 2: “Linear Algebra” (sections 2.1-2.3)
Dot Product & Cross Product	Math for Programmers by Paul Orland — Ch. 3: “Angles, Triangles, and Trigonometry”
Eigenvalues & Eigenvectors	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 2.7: “Matrix Decomposition”

Calculus (Derivatives, Integrals, Optimization)

Concept	Book & Chapter
Derivatives & Rates of Change	Math for Programmers by Paul Orland — Ch. 10: “Derivatives”
Gradient Descent	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 7: “Optimization”
The Chain Rule	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 5.2: “Differentiation of Univariate Functions”
Numerical Integration	Math for Programmers by Paul Orland — Ch. 11: “Integrals”
Partial Derivatives	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 5.3: “Partial Differentiation and Gradients”
Backpropagation Math	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 5.5: “Backpropagation”

Probability & Statistics

Concept	Book & Chapter
Basic Probability	Data Science for Business by Provost & Fawcett — Ch. 4: “Fitting a Model to Data”
Conditional Probability	Data Science for Business by Provost & Fawcett — Ch. 9: “Evidence and Probabilities”
Bayes’ Theorem	Data Science for Business by Provost & Fawcett — Ch. 9: “Evidence and Probabilities” (Naive Bayes section)
Probability Distributions	Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 6: “Probability and Distributions”
Statistical Inference	Data Science for Business by Provost & Fawcett — Ch. 5: “Overfitting and Its Avoidance”
Monte Carlo Methods	Math for Programmers by Paul Orland — Ch. 13: “Simulating with Random Sampling”

Number Theory & Cryptography

Concept	Book & Chapter
Prime Numbers & Primality Testing	Serious Cryptography by Jean-Philippe Aumasson — Ch. 4: “Public-Key Encryption”
Modular Arithmetic	Serious Cryptography by Jean-Philippe Aumasson — Ch. 4: “Public-Key Encryption” (modular exponentiation section)
GCD & Extended Euclidean Algorithm	Concrete Mathematics by Graham, Knuth & Patashnik — Ch. 4: “Number Theory” (section 4.1-4.3)
RSA Algorithm	Serious Cryptography by Jean-Philippe Aumasson — Ch. 4: “Public-Key Encryption” (RSA section)
Hash Functions	Serious Cryptography by Jean-Philippe Aumasson — Ch. 2: “Randomness” & Ch. 5: “Cryptographic Hashing”

Essential Reading Order

For maximum comprehension, read in this order:

Foundation (Week 1-2):
- Grokking Algorithms Ch. 1-4 (basic algorithms and recursion)
- Math for Programmers Ch. 1-3 (vectors and basic operations)
- A Common-Sense Guide to Data Structures and Algorithms Ch. 1-5 (fundamentals)
Linear Algebra (Week 3-4):
- Math for Programmers Ch. 4 (matrices and transformations)
- Computer Graphics from Scratch Ch. 1-3 (3D basics)
- Mathematics for Machine Learning Ch. 2 (linear algebra foundations)
Calculus & Optimization (Week 5-6):
- Math for Programmers Ch. 10-11 (derivatives and integrals)
- Mathematics for Machine Learning Ch. 5 (differentiation)
- Mathematics for Machine Learning Ch. 7 (optimization)
Probability & Statistics (Week 7-8):
- Data Science for Business Ch. 4, 9 (probability and Bayes)
- Mathematics for Machine Learning Ch. 6 (probability distributions)
Advanced Topics (Week 9+):
- Serious Cryptography Ch. 2, 4, 5 (cryptography fundamentals)
- Concrete Mathematics Ch. 4-5 (number theory and combinatorics)
- Mathematics for Machine Learning Ch. 8-12 (machine learning theory)

Project List

These projects are designed to teach you these mathematical concepts through practical, hands-on coding.

Project 1: Sudoku Solver

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: Go
Alternative Programming Languages: Python, C++, Rust
Coolness Level: Level 3: Genuinely Clever
Business Potential: 1. The “Resume Gold”
Difficulty: Level 2: Intermediate
Knowledge Area: Discrete Math / Algorithms
Software or Tool: A command-line Sudoku solver.
Main Book: “Grokking Algorithms” by Aditya Bhargava (for recursion/backtracking).

What you’ll build: A program that can take any valid (and solvable) Sudoku puzzle as input and print its unique solution.

Why it teaches math: This is a perfect introduction to computational logic and discrete math. Sudoku is a pure constraint satisfaction problem. You’ll learn to think about problems in terms of rules, states, and searching a solution space.

Core challenges you’ll face:

Representing the board → maps to using a 2D array or matrix
Defining the rules of Sudoku in code → maps to implementing logical constraints
Implementing a backtracking algorithm → maps to recursive searching and state management
Finding the next empty cell efficiently → maps to algorithmic thinking

Key Concepts:

Recursion: “Grokking Algorithms” Ch. 3.
Backtracking: A common algorithm for finding solutions by incrementally building candidates and abandoning a candidate (“backtracking”) as soon as it determines that it cannot possibly be completed to a valid solution.
Constraint Satisfaction: The core problem type, where you have a set of variables and a set of constraints they must satisfy.

Difficulty: Intermediate Time estimate: Weekend Prerequisites: Solid understanding of loops, functions, and recursion.

Real world outcome: A command-line tool that takes a string representing a puzzle and prints the solved grid.

$ ./sudoku-solver "53..7....6..195....98....6.8...6...34..8.3..17...2...6.6....28....419..5....8..79"

3 4 | 6 7 8 | 9 1 2
7 2 | 1 9 5 | 3 4 8
9 8 | 3 4 2 | 5 6 7
------+-------+------
5 9 | 7 6 1 | 4 2 3
2 6 | 8 5 3 | 7 9 1
1 3 | 9 2 4 | 8 5 6
------+-------+------
6 1 | 5 3 7 | 2 8 4
8 7 | 4 1 9 | 6 3 5
4 5 | 2 8 6 | 1 7 9

Implementation Hints:

Create a function solve(board).
Find the first empty cell on the board. If no cells are empty, the puzzle is solved; return true.
Try placing a number from 1 to 9 in that empty cell.
For each number, check if placing it there violates the Sudoku rules (is it valid in its row, column, and 3x3 square?).
If it’s a valid placement, recursively call solve(board).
If the recursive call returns true, it means a solution was found, so you should also return true.
If the recursive call returns false, it means this number didn’t lead to a solution. “Undo” your choice (set the cell back to empty) and try the next number.
If you try all numbers from 1 to 9 and none of them lead to a solution, return false.

Learning milestones:

Board representation and rule-checking work → You’ve translated logical rules into code.
The solver can fill in a few missing numbers → Your basic algorithm is on the right track.
The solver can solve easy puzzles → Your backtracking implementation is correct.
The solver can handle difficult puzzles quickly → Your algorithm is reasonably efficient.

The Core Question You’re Answering

How do you teach a computer to think like a human solving a logic puzzle, and why does exploring “wrong” paths sometimes lead to the right answer?

This project reveals the fundamental trade-off in computational problem-solving: exhaustive search vs. intelligent pruning. You’ll understand why constraint satisfaction problems are everywhere in CS (from scheduling to circuit design), and why backtracking—despite seeming inefficient—is often the most elegant solution when the problem space is too large for brute force but too irregular for greedy algorithms.

Concepts You Must Understand First

Recursion as a State Machine
- Recursion isn’t just “a function calling itself”—it’s a way to traverse a tree of possibilities where each function call represents a decision point
- The call stack IS your state; when you return, you’re unwinding a decision
- Every recursive call should move toward a base case; infinite recursion means you haven’t defined your problem boundaries
Check your understanding:
- Can you trace a recursive factorial function and draw the call stack at each step?
- What happens to local variables when a recursive call returns?
- How would you convert a recursive function to an iterative one using an explicit stack?
Read: Grokking Algorithms — Chapter 3: “Recursion”
The Backtracking Pattern
- Backtracking = try a choice, recurse, undo the choice if it doesn’t work
- It’s depth-first search with pruning: you abandon branches that violate constraints early
- The three critical steps: choose, explore, unchoose (also called “make move, recurse, unmake move”)
Check your understanding:
- Why is backtracking more efficient than generating all possible boards and checking each one?
- What’s the difference between backtracking and brute force?
- When would backtracking be a poor choice for a problem?
Read: Grokking Algorithms — Chapter 4: “Quicksort” (divide and conquer section)
Constraint Satisfaction Problems (CSP)
- A CSP has variables (empty cells), domains (possible values 1-9), and constraints (Sudoku rules)
- The key insight: check constraints BEFORE recursing, not after
- Early pruning is what makes CSP solvers practical
Check your understanding:
- Can you name three other problems that are CSPs? (Hint: think scheduling, coloring, seating)
- What’s the difference between checking validity after filling the board vs. before each placement?
- How would you represent Sudoku constraints as a set of boolean expressions?
Read: A Common-Sense Guide to Data Structures and Algorithms — Chapter 11: “Recursive Algorithms for Speed”
2D Array Indexing and Iteration
- A 9x9 board is board[row][col], but you also need to think in terms of 3x3 boxes
- Converting between (row, col) and box number requires modular arithmetic: boxIndex = (row / 3) * 3 + (col / 3)
- Efficient Sudoku validation means checking row, column, AND box simultaneously
Check your understanding:
- Given cell board[5][7], which 3x3 box is it in?
- How would you iterate through all cells in box number 4?
- What’s the time complexity of checking if a single placement is valid?
Read: A Common-Sense Guide to Data Structures and Algorithms — Chapter 8: “Arrays”

Questions to Guide Your Design

How will you represent an empty cell?
- Should you use 0, null, -1, or a special character like '.'?
- This choice affects every conditional in your code—choose wisely based on your language’s idioms.
What’s the most efficient way to find the next empty cell?
- Linear scan from top-left? Keep a list of empty cells? Use the first empty cell you find?
- Does the order matter for correctness? For performance?
When should you check if a number placement is valid?
- Before placing it (fail fast) or after placing it (simpler code)?
- How does this choice affect your recursion structure?
How do you validate a placement without checking the entire board?
- You only need to check 27 cells maximum (9 in row + 9 in column + 9 in box, with overlap)
- Can you write a isValid(board, row, col, num) function that’s O(1) with respect to board size?
What should your function return?
- Should solve(board) return true/false, or should it return the solved board?
- Should it modify the board in-place or create a copy?
- How does this affect the “unchoose” step of backtracking?
How will you handle unsolvable puzzles?
- What should happen if the puzzle has no solution or multiple solutions?
- Should you detect this early or just let the algorithm exhaust all possibilities?

Thinking Exercise

Before writing ANY code, trace this minimal backtracking example by hand:

// Problem: Fill array with numbers 1-3 such that no adjacent numbers are the same
func solve(arr []int, index int) bool {
    if index == len(arr) {
        fmt.Println(arr)  // Found a solution
        return true
    }

    for num := 1; num <= 3; num++ {
        // Check constraint: not same as previous
        if index == 0 || arr[index-1] != num {
            arr[index] = num           // Choose
            if solve(arr, index+1) {   // Explore
                return true
            }
            arr[index] = 0             // Unchoose
        }
    }
    return false
}

// Initial call: solve([]int{0, 0, 0}, 0)

Answer these questions as you trace:

Draw the recursion tree. How many nodes does it have?
At what point does the first solution get found? What is it?
Which branches get pruned by the constraint check?
How many times does the function call itself before finding a solution?
What would happen if you removed the “unchoose” line? Would it still work?
How would you modify this to find ALL solutions instead of just one?

Now apply this mental model to Sudoku: The pattern is IDENTICAL, just with more complex constraints.

The Interview Questions They’ll Ask

“What’s the time complexity of your Sudoku solver?”
- Answer: O(9^k) where k is the number of empty cells, worst case O(9^81)
- Follow-up: How could you optimize this? (Hint: choosing the cell with fewest valid options next)
“Can you modify your solver to find ALL solutions to a Sudoku puzzle?”
- Answer: Remove the early return; collect solutions in a list instead
“How would you parallelize Sudoku solving?”
- Answer: For the first empty cell, try all 9 numbers in parallel threads; each explores its subtree independently
“Why is backtracking better than generating all 9^81 possible boards?”
- Answer: Early pruning—we abandon invalid branches without exploring them fully
“How is solving Sudoku different from solving N-Queens?”
- Answer: Sudoku starts partially filled; N-Queens is placing items. But the backtracking pattern is identical.
“Could you use dynamic programming for Sudoku?”
- Answer: No—there’s no optimal substructure or overlapping subproblems. Each cell’s value depends on context.
“What if you needed to generate Sudoku puzzles instead of solving them?”
- Answer: Start with a solved board, remove numbers randomly while ensuring unique solution (use your solver to verify)

Books That Will Help

Topic	Book	Chapter
Recursion fundamentals	Grokking Algorithms by Aditya Bhargava	Ch. 3: “Recursion”
Divide and conquer thinking	Grokking Algorithms by Aditya Bhargava	Ch. 4: “Quicksort”
Recursive algorithm patterns	A Common-Sense Guide to Data Structures and Algorithms by Jay Wengrow	Ch. 11: “Recursive Algorithms for Speed”
Backtracking algorithms	Algorithms, Fourth Edition by Sedgewick & Wayne	Ch. 1.3: “Bags, Queues, and Stacks”
Array manipulation	A Common-Sense Guide to Data Structures and Algorithms by Jay Wengrow	Ch. 8: “Arrays”
Constraint satisfaction	Artificial Intelligence: A Modern Approach by Russell & Norvig	Ch. 6: “Constraint Satisfaction Problems” (if available)
Problem-solving strategies	Think Like a Programmer by V. Anton Spraul	Ch. 4: “Solving Problems with Recursion”

Project 2: A 3D Rendering Engine from Scratch

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: Python (with a simple graphics library like Pygame or Pillow)
Alternative Programming Languages: C++, Rust, JavaScript (with HTML Canvas)
Coolness Level: Level 5: Pure Magic (Super Cool)
Business Potential: 1. The “Resume Gold”
Difficulty: Level 3: Advanced
Knowledge Area: Linear Algebra / Computer Graphics
Software or Tool: A program that renders a spinning 3D cube.
Main Book: “Computer Graphics from Scratch” by Gabriel Gambetta.

What you’ll build: A program that renders a 3D object (like a wireframe cube) onto a 2D screen, including perspective projection and rotation, without using a 3D graphics library like OpenGL.

Why it teaches math: This project is a masterclass in applied linear algebra. You’ll stop thinking of vectors and matrices as abstract concepts and start seeing them for what they are: powerful tools for manipulating points in space.

Core challenges you’ll face:

Representing 3D points → maps to using 3D vectors
Applying rotations → maps to matrix multiplication with rotation matrices
Projecting 3D points onto a 2D screen → maps to perspective projection transformation
Connecting the projected points to draw lines → maps to basic rasterization

Key Concepts:

Vectors: Representing vertices of your 3D model.
Matrices: 4x4 transformation matrices are the standard for 3D graphics (combining translation, rotation, and scale).
Matrix Multiplication: The operation used to apply transformations to vectors.
Perspective Projection: The matrix transformation that simulates how objects further away appear smaller.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Strong grasp of functions, loops, and basic data structures.

Real world outcome: A window displaying a wireframe cube (or another 3D shape) that you can rotate in real-time.

Implementation Hints:

Define your 3D cube using a list of 8 vertices (vectors) and a list of 12 edges (pairs of vertices to connect).
Create rotation matrices for the X, Y, and Z axes. The formulas are readily available online.
In your main loop: a. Create a rotation matrix based on the elapsed time (to make it spin). b. For each vertex of the cube, apply the rotation by multiplying the vertex vector by the rotation matrix. c. Create a projection matrix. Apply it to the rotated vertices to get 2D points. d. These 2D points are in “normalized device coordinates” (usually -1 to 1). You’ll need to scale them to your screen’s pixel dimensions. e. For each edge of the cube, draw a line between its two projected (and scaled) 2D vertices.

Learning milestones:

You can draw a 2D projection of a static 3D cube → You understand vector representation and basic projection.
You can rotate the cube on a single axis → You’ve implemented matrix-vector multiplication correctly.
The cube spins smoothly on multiple axes → You understand how to combine transformation matrices.
The perspective projection looks correct → You’ve implemented a full 3D graphics pipeline.

The Core Question You’re Answering

How do you translate the three-dimensional world onto a flat screen in a way that preserves the illusion of depth, and why is matrix multiplication the secret language of geometric transformations?

This project demystifies the magic behind every 3D game, CAD program, and animated film. You’ll discover that “graphics programming” is really just applied linear algebra—that rotating, scaling, and projecting are all instances of the same mathematical operation. By the end, you’ll never look at a rotation the same way again; you’ll see the matrix behind it.

Concepts You Must Understand First

Vectors as Positions and Directions
- A 3D vector [x, y, z] can represent a point in space OR a direction/displacement
- Vector addition moves a point: point + displacement = new_point
- The magnitude (length) of a vector is sqrt(x² + y² + z²)
- A unit vector has magnitude 1 and represents pure direction
Check your understanding:
- What’s the difference between the point (3, 4, 0) and the vector [3, 4, 0]? (Hint: trick question—mathematically they’re the same)
- If you add two position vectors, what does the result represent?
- How do you normalize a vector (make it unit length)?
Read: Math for Programmers by Paul Orland — Ch. 2: “Drawing with 2D Vectors”
Matrix-Vector Multiplication as Transformation
- Multiplying a vector by a matrix transforms it: v' = M × v
- Each row of the matrix produces one component of the output vector
- The matrix encodes the transformation: rotation matrices rotate, scale matrices scale
- Matrix multiplication is NOT commutative: A × B ≠ B × A (order matters!)
Check your understanding:
- What does the identity matrix do when you multiply a vector by it?
- If matrix A rotates 90° and matrix B scales by 2, what’s the difference between A × B × v and B × A × v?
- How do you multiply a 3x3 matrix by a 3x1 vector by hand?
Read: Math for Programmers by Paul Orland — Ch. 4: “Transforming Vectors with Matrices”
Rotation Matrices
- A rotation around the Z-axis by angle θ is:
```
[cos(θ)  -sin(θ)  0]
[sin(θ)   cos(θ)  0]
[0        0       1]
```
- Rotations around X and Y axes follow similar patterns (each leaves one axis unchanged)
- To rotate around multiple axes, multiply the rotation matrices together
- The order of matrix multiplication determines the order of rotations
Check your understanding:
- What happens when θ = 0? When θ = 90°?
- Why does rotation preserve the length of a vector?
- If you rotate by θ then by -θ, do you get back to the original vector?
Read: Computer Graphics from Scratch by Gabriel Gambetta — Ch. 7: “Filled Triangles” (transformation section)
Perspective Projection
- Objects farther away appear smaller—that’s perspective
- The projection divides by the Z coordinate: screen_x = x / z, screen_y = y / z
- Points with the same x/z ratio project to the same screen X coordinate
- This creates the “vanishing point” effect where parallel lines appear to converge
Check your understanding:
- Why do we divide by Z and not by X or Y?
- What happens if Z is negative (object is behind the camera)?
- How would you prevent division by zero?
Read: Computer Graphics from Scratch by Gabriel Gambetta — Ch. 3: “Perspective Projection”
Homogeneous Coordinates (4D vectors for 3D graphics)
- We represent 3D points as 4D vectors: [x, y, z, 1] for positions, [x, y, z, 0] for directions
- This lets us represent translation as matrix multiplication (impossible with 3x3 matrices)
- A 4x4 transformation matrix can combine rotation, scale, AND translation in one operation
- The final perspective divide (x/w, y/w, z/w) happens after the projection matrix
Check your understanding:
- Why can’t a 3x3 matrix represent translation?
- What’s the difference between [2, 3, 4, 1] and [2, 3, 4, 0]?
- After multiplying by a projection matrix, why do we divide by the w component?
Read: Mathematics for Machine Learning by Deisenroth, Faisal & Ong — Ch. 2.7: “Affine Spaces”

Questions to Guide Your Design

How will you represent your 3D model?
- As a list of vertices and a separate list of edges (pairs of vertex indices)?
- Should you store faces (triangles) instead of edges for filled rendering later?
- What coordinate system will you use? (Right-handed vs. left-handed, where is the camera?)
Should you implement your own matrix multiplication or use a library?
- Writing it yourself teaches you what’s happening, but NumPy/similar is faster
- How would you structure a 3x3 or 4x4 matrix in your language? (Nested arrays? Flat array with index math?)
What order will you apply transformations?
- Scale, then rotate, then translate? Or translate first?
- Does your cube rotate around its center or around the world origin?
- How do you make an object rotate around its own center? (Hint: translate to origin, rotate, translate back)
How will you handle the projection from 3D to 2D?
- Simple orthographic (just drop the Z coordinate) or perspective projection?
- What’s your field of view? How far is the camera from the object?
- After projection, you have coordinates from -1 to 1 (normalized device coordinates)—how do you map these to pixel coordinates?
How will you animate the rotation?
- Should the rotation angle increase based on time or based on frame count?
- How do you ensure smooth rotation even if frame rate varies? (Hint: delta time)
- Will you rotate at a constant speed or accelerate/decelerate?
What will you do about hidden line removal?
- For a wireframe cube, should you draw all 12 edges, or hide the ones “behind” the cube?
- How do you determine which edges are visible? (This is non-trivial for wireframes!)
- Is it okay to just draw everything for now and tackle z-buffering in version 2?

Thinking Exercise

Before implementing the full renderer, trace this 2D rotation example:

import math

# Rotate point (1, 0) by 90 degrees around origin
def rotate_2d(x, y, angle_degrees):
    angle = math.radians(angle_degrees)

    # Rotation matrix
    # [cos(θ)  -sin(θ)]
    # [sin(θ)   cos(θ)]

    new_x = x * math.cos(angle) - y * math.sin(angle)
    new_y = x * math.sin(angle) + y * math.cos(angle)

    return (new_x, new_y)

# Test cases
print(rotate_2d(1, 0, 0))    # Should be (1, 0) - no rotation
print(rotate_2d(1, 0, 90))   # Should be (0, 1) - 90° rotation
print(rotate_2d(1, 0, 180))  # Should be (-1, 0) - 180° rotation
print(rotate_2d(1, 1, 45))   # Should be (0, sqrt(2)) - 45° rotation

Answer these questions:

Manually calculate rotate_2d(1, 0, 90) using the formula. What do you get?
What does rotating (1, 0) by 90° mean geometrically?
What’s the rotation matrix for 0 degrees? Why is this called the “identity matrix”?
If you rotate by 90° four times, where do you end up?
Draw the vector (1, 0) and then its rotated version (0, 1) on paper. Does it look right?
Extend this to 3D: How would you rotate the point (1, 0, 0) by 90° around the Z-axis?

Now apply this to your cube: Each vertex is just a 3D vector that you’ll rotate and project.

The Interview Questions They’ll Ask

“Explain the difference between orthographic and perspective projection.”
- Answer: Orthographic preserves parallel lines (no vanishing point), perspective divides by depth (objects get smaller with distance)
“Why do we use 4x4 matrices instead of 3x3 for 3D graphics?”
- Answer: To represent translation as matrix multiplication (homogeneous coordinates)
“What’s the difference between model, view, and projection matrices?”
- Answer: Model transforms object space to world space, view transforms world to camera space, projection transforms 3D camera space to 2D screen space
“How would you implement a camera that can move and look around?”
- Answer: Create a view matrix that translates by -cameraPosition and rotates by -cameraOrientation
“What’s a quaternion and why might you use one instead of rotation matrices?”
- Answer: Quaternions represent rotations without gimbal lock, interpolate smoothly, and use less memory (4 numbers vs. 9)
“How does the Z-buffer algorithm work for hidden surface removal?”
- Answer: For each pixel, store the depth of the closest fragment; only draw new fragments if their depth is less than the stored value
“What’s the difference between row-major and column-major matrix ordering?”
- Answer: How matrix elements are stored in memory; affects whether you do M × v or v × M. OpenGL uses column-major, DirectX uses row-major.

Books That Will Help

Topic	Book	Chapter
Vector basics and operations	Math for Programmers by Paul Orland	Ch. 2: “Drawing with 2D Vectors”
3D vectors and geometry	Math for Programmers by Paul Orland	Ch. 3: “Angles, Triangles, and Trigonometry”
Matrix transformations	Math for Programmers by Paul Orland	Ch. 4: “Transforming Vectors with Matrices”
3D to 2D projection	Computer Graphics from Scratch by Gabriel Gambetta	Ch. 3: “Perspective Projection”
Rasterization and lines	Computer Graphics from Scratch by Gabriel Gambetta	Ch. 7: “Filled Triangles”
Linear algebra theory	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Ch. 2: “Linear Algebra”
Affine transformations	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Ch. 2.7: “Affine Spaces”
Coordinate systems	Computer Graphics from Scratch by Gabriel Gambetta	Ch. 1: “Introductory Concepts”
3D rendering pipeline	Computer Graphics from Scratch by Gabriel Gambetta	Ch. 9: “Shading” (pipeline overview)
Matrix multiplication	Concrete Mathematics by Graham, Knuth & Patashnik	Ch. 5: “Binomial Coefficients” (section on matrix operations)

Project 3: RSA Cryptosystem from Scratch

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: Python
Alternative Programming Languages: Go
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 1. The “Resume Gold”
Difficulty: Level 3: Advanced
Knowledge Area: Number Theory / Cryptography
Software or Tool: An encryption/decryption tool.
Main Book: “Serious Cryptography” by Jean-Philippe Aumasson.

What you’ll build: A simplified implementation of the RSA algorithm to encrypt and decrypt messages.

Why it teaches math: This project is a direct application of number theory. You’ll implement algorithms for finding prime numbers, calculating the greatest common divisor, and performing modular exponentiation. It demystifies public-key cryptography.

Core challenges you’ll face:

Generating large prime numbers → maps to implementing a primality test (like Miller-Rabin)
Finding modular multiplicative inverse → maps to using the Extended Euclidean Algorithm
Performing modular exponentiation efficiently → maps to implementing the “exponentiation by squaring” algorithm
Putting it all together to generate keys and encrypt/decrypt → maps to understanding the RSA algorithm itself

Key Concepts:

Prime Numbers: The foundation of RSA’s security.
Modular Arithmetic: Performing arithmetic where numbers “wrap around” a modulus. (a * b) mod n.
Greatest Common Divisor (GCD) and the Extended Euclidean Algorithm: Used to find the private key d.
Euler’s Totient Function: Used in the key generation process.

Difficulty: Advanced Time estimate: 1-2 weeks

Prerequisites: Comfort with programming fundamentals. No prior number theory knowledge is assumed, as that’s what you’ll be learning.

Real world outcome: A program that can generate a public/private key pair, encrypt a message with the public key, and decrypt the ciphertext back to the original message with the private key.

$ ./rsa-tool
Generated Public Key (e, n): (65537, 274877906944)
Generated Private Key (d, n): (168478116413, 274877906944)

Enter message to encrypt: Hello, math!
Encrypted (ciphertext): 832040123456...
Decrypted (plaintext): Hello, math!

(Note: This is for educational purposes ONLY. Do NOT use your implementation for real security.)

Implementation Hints:

Key Generation: a. Find two large, distinct prime numbers, p and q. b. Calculate n = p * q and phi(n) = (p - 1) * (q - 1). c. Choose a public exponent e (commonly 65537) that is coprime to phi(n). d. Calculate the private exponent d as the modular multiplicative inverse of e modulo phi(n). This is where you use the Extended Euclidean Algorithm.
Encryption: ciphertext = (plaintext ^ e) mod n.
Decryption: plaintext = (ciphertext ^ d) mod n.
You will need a function for modular exponentiation that can handle very large numbers without overflowing standard integer types.

Learning milestones:

You can generate large prime numbers → You’ve implemented a primality test.
You can find the modular inverse → Your Euclidean algorithm implementation is correct.
Key generation works → You can successfully generate valid public/private key pairs.
A message can be encrypted and decrypted correctly → Your full RSA implementation is working.

The Core Question You’re Answering

How can two people securely communicate over an insecure channel when they’ve never met to exchange a secret key?

This question gets to the heart of modern cryptography. Before RSA, encryption required both parties to somehow secretly share a key. RSA solves this by using a mathematical trapdoor: it’s easy to multiply two large primes together, but extremely hard to factor the result back into those primes. You’ll understand why public-key cryptography is considered one of the most important inventions in computer science.

Concepts You Must Understand First

Prime Numbers and Why They Matter
- A prime is a number greater than 1 that has no divisors except 1 and itself
- Primes are the “atoms” of number theory—every number can be uniquely factored into primes
- The security of RSA relies on the fact that multiplying primes is easy, but factoring is hard
Guiding Questions:
- Why is the number 1 not considered prime?
- If you have a 200-digit number that’s the product of two 100-digit primes, why can’t even supercomputers factor it quickly?
- What makes the distribution of primes unpredictable?
Book Reference: Serious Cryptography by Jean-Philippe Aumasson — Ch. 4: “Public-Key Encryption”
Modular Arithmetic (Clock Arithmetic)
- When you do math “mod n”, numbers wrap around after reaching n (like a clock wraps at 12)
- 17 mod 5 = 2 because 17 = 3×5 + 2
- Properties: (a + b) mod n = ((a mod n) + (b mod n)) mod n, and similarly for multiplication
Guiding Questions:
- What is 100 mod 7? Can you explain why without using a calculator?
- If it’s 10 o’clock now, what time will it be in 100 hours? (This is modular arithmetic!)
- Why does (a × b) mod n = ((a mod n) × (b mod n)) mod n matter for large number calculations?
Book Reference: Concrete Mathematics by Graham, Knuth & Patashnik — Ch. 4: “Number Theory” (sections 4.1-4.3)
Greatest Common Divisor (GCD) and Coprimality
- The GCD of two numbers is the largest number that divides both
- Two numbers are coprime if their GCD is 1 (they share no common factors)
- The Euclidean Algorithm efficiently finds GCD by repeated division
Guiding Questions:
- What is GCD(48, 18)? Can you find it using the Euclidean algorithm?
- Why must RSA’s public exponent e be coprime to φ(n)?
- How does the Extended Euclidean Algorithm give you not just GCD, but also the modular inverse?
Book Reference: Concrete Mathematics by Graham, Knuth & Patashnik — Ch. 4: “Number Theory” (section 4.1)
Modular Multiplicative Inverse
- The inverse of a modulo n is a number b such that (a × b) mod n = 1
- Not every number has a modular inverse (a must be coprime to n)
- Found using the Extended Euclidean Algorithm
Guiding Questions:
- What is the multiplicative inverse of 3 modulo 7? (Hint: find b where 3×b mod 7 = 1)
- Why doesn’t 6 have a multiplicative inverse modulo 9?
- How is finding d (the private exponent) in RSA really just finding the modular inverse of e?
Book Reference: Serious Cryptography by Jean-Philippe Aumasson — Ch. 4: “Public-Key Encryption” (modular exponentiation section)
Euler’s Totient Function φ(n)
- φ(n) counts how many numbers less than n are coprime to n
- For a prime p: φ(p) = p - 1 (all numbers less than a prime are coprime to it)
- For two primes p and q: φ(p×q) = (p-1)×(q-1)
Guiding Questions:
- What is φ(10)? (Count: 1, 3, 7, 9 are coprime to 10, so φ(10) = 4)
- Why does φ(pq) = (p-1)(q-1) when p and q are prime?
- Why does RSA use φ(n) = (p-1)(q-1) in the key generation formula?
Book Reference: Concrete Mathematics by Graham, Knuth & Patashnik — Ch. 4: “Number Theory” (section 4.3)
Modular Exponentiation and Exponentiation by Squaring
- Computing a^b mod n naively overflows for large numbers
- Exponentiation by squaring uses the binary representation of the exponent
- Example: a^13 = a^8 × a^4 × a^1 (13 in binary is 1101)
Guiding Questions:
- Why can’t you just compute a^b and then take mod n when b is huge (like b = 65537)?
- How does breaking the exponent into powers of 2 help you compute this efficiently?
- What is the time complexity of exponentiation by squaring?
Book Reference: Serious Cryptography by Jean-Philippe Aumasson — Ch. 4: “Public-Key Encryption”

Questions to Guide Your Design

How will you test if a large number is prime?
- Trial division (checking divisors up to √n) is too slow for large numbers
- You’ll need a probabilistic test like Miller-Rabin
- Consider: How many test rounds balance speed vs. certainty?
How will you generate random prime numbers of a specific bit-length?
- You need primes of similar size (e.g., both 512 bits for a 1024-bit modulus)
- Will you generate random odd numbers and test them, or use a different strategy?
- How will you ensure p ≠ q?
How will you handle numbers too large for your language’s native integer types?
- Python has arbitrary-precision integers built-in
- Other languages might need BigInteger libraries
- How will this affect your modular exponentiation implementation?
What’s your strategy for converting strings to numbers and back?
- RSA operates on numbers, but users input text
- Will you encrypt character-by-character, or in blocks?
- What encoding will you use (ASCII, UTF-8)?
How will you implement the Extended Euclidean Algorithm to find the modular inverse?
- The algorithm needs to track coefficients (Bézout’s identity)
- Edge case: What happens if GCD(e, φ(n)) ≠ 1?
- Will you use iteration or recursion?
How will you verify that your key generation is correct?
- The relationship (m^e)^d mod n = m must hold for any message m
- Can you test this with small primes first before scaling up?
- What invariants can you check during key generation?

Thinking Exercise

Before writing any code, trace through a miniature RSA example by hand:

Let p = 5, q = 11 (unrealistically small primes, but manageable by hand).

Part 1: Key Generation

Calculate n = p × q = ?
Calculate φ(n) = (p-1) × (q-1) = ?
Choose e = 3 (verify that GCD(3, φ(n)) = 1)
Find d such that (e × d) mod φ(n) = 1
- Use Extended Euclidean Algorithm or trial: try d = 1, 2, 3… until (3 × d) mod φ(n) = 1
- What value of d works?

Part 2: Encryption and Decryption

Let’s encrypt the message m = 2
- Compute ciphertext c = m^e mod n = 2^3 mod n = ?
Now decrypt c to recover m
- Compute plaintext = c^d mod n = ?
- Did you get m = 2 back?

Part 3: Understanding the Math

Why does this work? The magic is in Euler’s theorem:
- For any m coprime to n: m^φ(n) mod n = 1
- Since e × d ≡ 1 (mod φ(n)), we know e × d = 1 + k × φ(n) for some integer k
- Therefore: c^d = (m^e)^d = m^(ed) = m^(1 + k×φ(n)) = m × (m^φ(n))^k ≡ m × 1^k ≡ m (mod n)

Questions to Answer:

What happens if you try to encrypt a message m > n?
What happens if you choose an e that’s NOT coprime to φ(n)?
If an attacker knows n and e (the public key), why can’t they easily find d?

The Interview Questions They’ll Ask

“Explain how RSA encryption works to a non-technical person.”
- Focus on the asymmetric key concept and the analogy of a public lock/private key
“What is the computational complexity of factoring versus multiplication, and why does this matter for RSA security?”
- Discuss the one-way function property and current factoring algorithms
“Why do we use large prime numbers in RSA? What would happen if we used composite numbers or small primes?”
- Explain the factorization problem and key length recommendations
“What is the difference between RSA and symmetric encryption algorithms like AES?”
- Discuss key distribution, performance trade-offs, and typical use cases
“Walk me through the mathematical steps of RSA key generation.”
- Be ready to explain p, q, n, φ(n), e, and d, and their relationships
“What is a padding scheme and why is it necessary for secure RSA implementation?”
- Discuss vulnerabilities of textbook RSA and standards like OAEP
“If I gave you n and φ(n), could you break RSA? Why or why not?”
- Explain that knowing φ(n) lets you factor n easily: p and q are roots of x² - (n-φ(n)+1)x + n = 0

Books That Will Help

Topic	Book	Chapter
RSA Algorithm Overview	Serious Cryptography by Jean-Philippe Aumasson	Ch. 4: “Public-Key Encryption” (RSA section)
Modular Arithmetic Fundamentals	Concrete Mathematics by Graham, Knuth & Patashnik	Ch. 4: “Number Theory” (sections 4.1-4.3)
Prime Numbers and Primality Testing	Serious Cryptography by Jean-Philippe Aumasson	Ch. 4: “Public-Key Encryption”
Modular Exponentiation	Serious Cryptography by Jean-Philippe Aumasson	Ch. 4: “Public-Key Encryption”
Extended Euclidean Algorithm	Concrete Mathematics by Graham, Knuth & Patashnik	Ch. 4: “Number Theory” (section 4.1)
Euler’s Totient Function	Concrete Mathematics by Graham, Knuth & Patashnik	Ch. 4: “Number Theory” (section 4.3)
Cryptographic Best Practices	Serious Cryptography by Jean-Philippe Aumasson	Ch. 4: “Public-Key Encryption” (padding and attacks)
Number Theory Problems	Concrete Mathematics by Graham, Knuth & Patashnik	Ch. 4: “Number Theory” (exercises)

Project 4: A Simple Physics Engine

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: Python (with Pygame for visualization)
Alternative Programming Languages: JavaScript (with Canvas), C++
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 1. The “Resume Gold”
Difficulty: Level 3: Advanced
Knowledge Area: Calculus / Linear Algebra / Physics Simulation
Software or Tool: A simulation of bouncing balls in a box.
Main Book: “Math for Programmers” by Paul Orland.

What you’ll build: A 2D physics simulation where multiple balls move under gravity, collide with each other, and bounce off the walls of a container.

Why it teaches math: This project makes calculus tangible. You’ll use simple numerical integration to update object positions and velocities over time. It also heavily relies on linear algebra for vector operations (position, velocity, collision response).

Core challenges you’ll face:

Simulating motion over time → maps to numerical integration (Euler integration)
Detecting collisions (ball-wall, ball-ball) → maps to vector math and distance calculations
Calculating collision response → maps to conservation of momentum and vector projection
Managing the state of multiple objects → maps to algorithmic structure and game loop design

Key Concepts:

Euler Integration: A simple method for numerical integration. position = position + velocity * deltaTime. velocity = velocity + acceleration * deltaTime.
Vectors: Used for position, velocity, and acceleration of each ball.
Vector Operations: Dot product is essential for calculating collision responses.
Conservation of Momentum: The physical principle governing how objects behave when they collide.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Project 2 (Rendering Engine) is helpful for the vector concepts.

Real world outcome: A graphical simulation where you can watch dozens of balls realistically bouncing and colliding in a confined space.

Implementation Hints:

Create an object class that stores its position (vector), velocity (vector), acceleration (vector, e.g., (0, 9.8) for gravity), and radius.
In your main game loop, for each time step (deltaTime): a. Integration: For each object, update its velocity and then its position using Euler integration. b. Collision Detection: i. Check if any ball is colliding with the walls. If so, invert its velocity component perpendicular to the wall. ii. Check every pair of balls to see if they are colliding (is the distance between their centers less than the sum of their radii?). c. Collision Response: If two balls are colliding, use the 1D collision formula (available online) on the components of their velocities along the line connecting their centers. This requires using vector projection (dot products). d. Drawing: Draw each ball at its new position.

Learning milestones:

A single ball moves under gravity and bounces off the floor → You’ve implemented Euler integration.
The ball bounces off all four walls correctly → You understand basic collision detection and response.
Two balls collide and bounce off each other realistically → You’ve implemented vector-based collision response.
The simulation is stable with many balls → Your engine is robust.

The Core Question You’re Answering

How do you simulate continuous physical motion in a world that updates in discrete time steps, and why is calculus the key to making virtual objects move realistically?

This project bridges the gap between mathematics and reality. You’ll discover that physics engines don’t “know” physics—they approximate it using numerical methods. Every game, animation, and simulation you’ve ever seen uses these same techniques. By the end, you’ll understand why integration isn’t just abstract math; it’s literally how we model the passage of time in code.

Concepts You Must Understand First

Vectors as Physical Quantities
- Position vector: where an object is in space (x, y)
- Velocity vector: how fast and in what direction it’s moving (vx, vy)
- Acceleration vector: how the velocity is changing (ax, ay) (e.g., gravity = (0, 9.8))
- All three use the same representation but mean different things
Guiding Questions:
- If position is (10, 5) and velocity is (2, -3), where will the object be 1 second later?
- What’s the difference between speed (scalar) and velocity (vector)?
- Why is acceleration due to gravity always downward, regardless of current velocity?
Book Reference: Math for Programmers by Paul Orland — Ch. 2: “Drawing with 2D Vectors”
Numerical Integration (Euler Method)
- Integration in calculus: finding the area under a curve, or reversing differentiation
- In physics: velocity is the integral of acceleration, position is the integral of velocity
- Euler method: approximate the integral by taking small time steps
- Formula: new_value = old_value + rate_of_change × Δt
Guiding Questions:
- Why does position += velocity × deltaTime make sense? (Hint: if you’re moving at 5 m/s for 2 seconds, how far do you go?)
- What happens if deltaTime is too large? Will the simulation still be accurate?
- Why do we update velocity BEFORE updating position?
Book Reference: Math for Programmers by Paul Orland — Ch. 11: “Integrals”
The Dot Product and Vector Projection
- Dot product: a · b = ax×bx + ay×by = |a| × |b| × cos(θ) where θ is angle between vectors
- If dot product is 0, vectors are perpendicular; if positive, they point somewhat the same direction
- Vector projection: decomposing a vector into components parallel and perpendicular to another vector
- Essential for collision response—you need to split velocity into “along collision normal” and “perpendicular to it”
Guiding Questions:
- What is (3, 4) · (1, 0)? What does this tell you?
- If two velocity vectors have a negative dot product, are the objects moving toward or away from each other?
- How would you project vector a onto vector b?
Book Reference: Math for Programmers by Paul Orland — Ch. 3: “Angles, Triangles, and Trigonometry”
Distance Formula and Circle Collision Detection
- Distance between two points: d = sqrt((x2-x1)² + (y2-y1)²)
- This is just the magnitude of the difference vector: |position2 - position1|
- Two circles collide if distance between centers < sum of radii
- For efficiency, compare squared distances to avoid the sqrt
Guiding Questions:
- Why is checking distance < r1 + r2 the same as checking if circles overlap?
- How can you check collision without computing the expensive square root?
- What happens at the exact moment distance == r1 + r2?
Book Reference: Math for Programmers by Paul Orland — Ch. 2: “Drawing with 2D Vectors”
Conservation of Momentum
- Momentum = mass × velocity (a vector!)
- In a collision, total momentum before = total momentum after
- For equal masses: the velocities exchange along the collision normal
- For unequal masses: lighter object bounces more, heavier object barely moves
Guiding Questions:
- If a moving ball hits a stationary ball of equal mass head-on, what happens?
- Why does a ping-pong ball bounce off a bowling ball, but the bowling ball barely moves?
- What’s the difference between elastic (bouncy) and inelastic (sticky) collisions?
Book Reference: Math for Programmers by Paul Orland — Ch. 11: “Integrals” (motion section)
The Game Loop and Delta Time
- The game loop runs continuously: update physics, then render, repeat
- Delta time (Δt): the time elapsed since the last frame
- Physics must scale with Δt to be frame-rate independent
- Small Δt = more accurate simulation but more computation
Guiding Questions:
- Why does velocity += gravity without multiplying by Δt make objects fall faster on slow computers?
- What’s the trade-off between a fixed time step (e.g., always 1/60 second) vs. variable time step?
- How do you handle the case where Δt is huge (e.g., user pauses for a minute)?
Book Reference: Math for Programmers by Paul Orland — Ch. 10: “Derivatives”

Questions to Guide Your Design

How will you represent a ball’s state?
- Class with position, velocity, acceleration, mass, and radius?
- Should color and drawing info be part of the physics object or separate?
- Will you use a 2D vector class or just separate x/y variables?
What coordinate system will you use?
- Is (0,0) the top-left (screen coordinates) or bottom-left (math coordinates)?
- Does positive Y point up or down?
- How does this affect your gravity vector and collision normal calculations?
How will you structure your game loop?
- Fixed time step (always simulate 1/60 sec) or variable (use actual elapsed time)?
- Update all positions first, then check collisions, or interleave?
- How many physics updates per render frame?
How will you detect ball-ball collisions efficiently?
- Naive approach: check every pair (O(n²) for n balls)
- Can you use spatial hashing to only check nearby balls?
- Is it worth optimizing for your use case, or is O(n²) fast enough?
How will you handle the collision response?
- Just reflect velocity along the collision normal, or account for conservation of momentum?
- What if balls are already overlapping when you detect the collision?
- Should you separate the balls first, then update velocities?
How will you make the simulation realistic?
- Add damping (air resistance) to prevent eternal bouncing?
- Add friction when balls collide?
- Model restitution coefficient (bounciness)?

Thinking Exercise

Before coding the full engine, trace through this simple example by hand:

# A ball falling under gravity
position_y = 100    # meters above ground
velocity_y = 0      # m/s (starting at rest)
gravity = -9.8      # m/s² (negative because down)
deltaTime = 0.1     # 100ms time steps

# Simulate 5 time steps
for step in range(5):
    # Euler integration
    velocity_y = velocity_y + gravity * deltaTime
    position_y = position_y + velocity_y * deltaTime

    # Check ground collision
    if position_y <= 0:
        position_y = 0
        velocity_y = -velocity_y * 0.8  # bounce with 80% energy retained

    print(f"Step {step}: pos={position_y:.2f}m, vel={velocity_y:.2f}m/s")

Trace this by hand:

What is velocity_y after step 0? After step 1?
What is position_y after step 0? After step 1?
At which step does the ball hit the ground?
What is the ball’s velocity immediately after the first bounce?
Does the ball eventually come to rest? Why or why not?

Now think about 2D:

If you have velocity vector (5, -5) and you hit a vertical wall (normal = (1, 0)), what should the new velocity be?
For two balls colliding, how do you find the collision normal? (Hint: it’s the direction from one center to the other)

The Interview Questions They’ll Ask

“Explain the difference between Euler integration and more advanced methods like Runge-Kutta. Why might Euler be insufficient for some simulations?”
- Discuss accuracy, stability, and when simple methods break down
“How would you detect and resolve a situation where two balls are overlapping?”
- Talk about separation before velocity update, and preventing tunneling
“Your physics simulation runs fast on your computer but in slow motion on a slower machine. What went wrong?”
- Explain the importance of delta time and frame-rate independence
“How would you optimize collision detection for 1000 balls instead of 10?”
- Discuss spatial partitioning, quadtrees, or broad-phase/narrow-phase approaches
“Walk me through the math of a perfectly elastic collision between two balls of different masses.”
- Be ready to explain conservation of momentum and energy
“How would you add rotation to your physics engine?”
- Discuss angular velocity, torque, and moment of inertia
“What’s the relationship between force, acceleration, and mass? How do you model forces in code?”
- Explain F = ma and how forces are accumulated each frame then converted to acceleration

Books That Will Help

Topic	Book	Chapter
Vectors and Vector Operations	Math for Programmers by Paul Orland	Ch. 2: “Drawing with 2D Vectors”
Dot Product and Projections	Math for Programmers by Paul Orland	Ch. 3: “Angles, Triangles, and Trigonometry”
Derivatives and Rates of Change	Math for Programmers by Paul Orland	Ch. 10: “Derivatives”
Numerical Integration (Euler Method)	Math for Programmers by Paul Orland	Ch. 11: “Integrals”
Simulating Motion	Math for Programmers by Paul Orland	Ch. 11: “Integrals” (physics simulation section)
Linear Algebra for Physics	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Ch. 2: “Linear Algebra” (sections 2.1-2.3)
Collision Detection Geometry	Computer Graphics from Scratch by Gabriel Gambetta	Ch. 7: “Filled Triangles” (geometric tests)
Game Loop Architecture	Game Programming Patterns by Robert Nystrom	Ch. 3: “Game Loop” (if available)

Project 5: Bayesian Spam Filter

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: Python
Alternative Programming Languages: Go, Java
Coolness Level: Level 3: Genuinely Clever
Business Potential: 2. The “Micro-SaaS / Pro Tool”
Difficulty: Level 2: Intermediate
Knowledge Area: Probability & Statistics
Software or Tool: A tool to classify text as spam or not-spam.
Main Book: “Data Science for Business” by Provost and Fawcett.

What you’ll build: A program that can be “trained” on a dataset of spam and non-spam (ham) emails, and can then predict whether a new, unseen email is spam.

Why it teaches math: This is the classic, intuitive introduction to Bayesian statistics. You’ll learn how to calculate conditional probabilities and use Bayes’ theorem to make predictions based on evidence (the words in an email).

Core challenges you’ll face:

Tokenizing text → maps to breaking emails down into a list of words
Counting word frequencies in spam vs. ham → maps to building a probabilistic model
Applying Bayes’ theorem → maps to combining probabilities to classify a new email
Handling unknown words and avoiding floating-point underflow → maps to Laplace smoothing and using log probabilities

Key Concepts:

Bayes’ Theorem: P(A|B) = [P(B|A) * P(A)] / P(B). In our case, P(Spam | Words).
Conditional Probability: The probability of an event given that another event has occurred. P(word | Spam) is the probability that a given word appears in a spam email.
Laplace Smoothing: A technique to handle words that haven’t been seen in the training data by adding 1 to all counts.

Difficulty: Intermediate Time estimate: Weekend Prerequisites: Basic data structures, especially dictionaries/hash maps.

Real world outcome: A command-line tool that can classify messages.

# First, train the model
$ ./spam-filter --train --spam spam_folder/ --ham inbox/
Model trained successfully.

# Now, classify a new message
$ ./spam-filter --classify "free money now click here"
Result: SPAM (Probability: 99.8%)

$ ./spam-filter --classify "Hi mom, see you on Sunday. -Bob"
Result: HAM (Probability: 98.5%)

Implementation Hints:

Training: a. Build a vocabulary of all words seen in the training emails. b. Create two frequency maps (dictionaries): one for words in spam emails, one for words in ham emails. c. Calculate the base probabilities P(Spam) and P(Ham) (e.g., number of spam emails / total emails).
Classification: a. For a new email, tokenize it into words. b. Start with the base probability P(Spam). c. For each word in the email, multiply the current probability by P(word | Spam). Do the same for P(Ham). d. P(word | Spam) is (count of word in spam + 1) / (total words in spam + vocab size). The +1 is Laplace smoothing. e. Pro-tip: Probabilities are small numbers. Multiplying many of them together will lead to floating-point underflow. Instead, add their logarithms: log(P1*P2) = log(P1) + log(P2). Compare the final log probabilities to decide.

Learning milestones:

You can count word frequencies from a text file → Your data processing pipeline is working.
The model can calculate P(word | Spam) and P(word | Ham) → You’ve built the core statistical model.
The classifier works for simple cases → Your Bayes’ theorem implementation is correct.
The classifier is robust to new words and avoids underflow → You’ve handled the practical edge cases of probabilistic models.

The Core Question You’re Answering

How can we use evidence (the words in a message) to update our belief about whether that message is spam, and why does multiplying conditional probabilities give us a classification decision?

Concepts You Must Understand First

Basic Probability (Sample Spaces and Events)
- Probability is the measure of how likely an event is to occur, ranging from 0 (impossible) to 1 (certain)
- A sample space is the set of all possible outcomes; an event is a subset of that sample space
- The probability of mutually exclusive events is additive: P(A or B) = P(A) + P(B)
Guiding questions:
- If you flip a fair coin twice, what is the sample space? What is the probability of getting at least one heads?
- If 30% of all emails are spam and 70% are ham, what is P(Spam)? What is P(Ham)?
- Why must P(Spam) + P(Ham) = 1 in a binary classification problem?
Book reference: “Data Science for Business” by Provost & Fawcett — Ch. 4: “Fitting a Model to Data”

Conditional Probability

P(A B) means “the probability of A given that B has already occurred”
It’s calculated as P(A B) = P(A and B) / P(B), assuming P(B) > 0
Conditional probabilities allow us to update our beliefs based on new information

Guiding questions:

What does P(word=”free” Spam) mean in plain English? How is it different from P(Spam)?

If the word “free” appears in 80% of spam emails but only 10% of ham emails, what is P(“free”

Spam) and P(“free”

Ham)?

Why is conditional probability the key to classification? What information does it give you that regular probability doesn’t?

Book reference: “Data Science for Business” by Provost & Fawcett — Ch. 9: “Evidence and Probabilities”

Bayes’ Theorem
- The formula: P(A B) = [P(B A) × P(A)] / P(B)
- It allows us to “reverse” conditional probabilities: if we know P(B A), we can calculate P(A B)
- In classification, it lets us go from P(words Spam) to P(Spam words), which is what we actually want
Guiding questions:
- Why can’t we directly calculate P(Spam words) from our training data? What do we calculate instead?
- In the Bayes formula for spam filtering, what is P(A)? What is P(B)? What is P(B A)?
- Why do we call P(Spam) the “prior” probability and P(Spam words) the “posterior” probability?
Book reference: “Data Science for Business” by Provost & Fawcett — Ch. 9: “Evidence and Probabilities”
Independence Assumption (The “Naive” in Naive Bayes)
- The assumption that the presence of one word doesn’t affect the probability of another word appearing
- Mathematically: P(word1 and word2 Spam) = P(word1 Spam) × P(word2 Spam)
- This assumption is often wrong in reality, but the algorithm still works surprisingly well
Guiding questions:
- Is the independence assumption realistic? Can you think of words that tend to appear together?
- If we didn’t assume independence, how would the formula change? Why would it be harder to calculate?
- Why does Naive Bayes still work well even though its core assumption is violated?
Book reference: “Data Science for Business” by Provost & Fawcett — Ch. 9: “Evidence and Probabilities”

Laplace Smoothing (Add-One Smoothing)

A technique to handle zero probabilities by adding 1 to all word counts
Without it, a single unseen word would make P(words Spam) = 0, breaking the classifier
The formula becomes: P(word Spam) = (count of word in spam + 1) / (total words in spam + vocabulary size)

Guiding questions:

What happens if a word appears in your test email but never appeared in training? Why is P(word

Spam) = 0 a problem?

How does adding 1 to all counts solve this problem? Does it introduce any bias?
Why do we add the vocabulary size to the denominator? What would happen if we only added 1 to the numerator?

Book reference: “Data Science for Business” by Provost & Fawcett — Ch. 9: “Evidence and Probabilities”

Log Probabilities (Avoiding Numerical Underflow)
- Multiplying many small probabilities (like 0.001 × 0.002 × 0.0001…) causes floating-point underflow
- Using logarithms converts multiplication to addition: log(a × b) = log(a) + log(b)
- We compare log P(Spam words) vs log P(Ham words) instead of the probabilities themselves
Guiding questions:
- What is floating-point underflow? At what point do computers start treating very small numbers as zero?
- Why does log(a × b) = log(a) + log(b)? How does this help us avoid underflow?
- If log P(Spam words) = -15.3 and log P(Ham words) = -18.7, which is more likely? How do you know?
Book reference: “Mathematics for Machine Learning” by Deisenroth, Faisal & Ong — Ch. 6.2: “Discrete and Continuous Probabilities”

Questions to Guide Your Design

How will you represent the training data?
- Will you store every email, or just summary statistics (word counts)?
- Should you use a hash map/dictionary for word frequencies? What are the keys and values?
- How will you handle case sensitivity (is “Free” the same as “free”)?
What counts as a “word” when tokenizing?
- Should you remove punctuation? What about numbers?
- Should you filter out common words like “the” and “is” (stop words)?
- How will you handle email-specific features like sender addresses, subject lines, and HTML tags?
**How will you calculate P(Spam words) for a multi-word email?**
- Do you multiply P(word1 Spam) × P(word2 Spam) × … for all words?
- How do you incorporate the prior probability P(Spam)?
- Should you consider word frequency in the email (e.g., “free” appears 5 times), or just presence/absence?
How will you handle words that appear in both spam and ham?
- If “meeting” appears equally in spam and ham, should it affect the classification?
- What if a word is slightly more common in spam (60% vs 40%)? How does this change the probability?
- Can you think of neutral words that shouldn’t influence the decision much?
How will you evaluate your classifier’s accuracy?
- Should you test on the same emails you trained on? Why or why not?
- What metrics matter: accuracy, false positives (ham classified as spam), or false negatives (spam classified as ham)?
- How will you split your data into training and testing sets?
How will you decide the threshold for classification?
- If P(Spam words) = 0.55, is that spam or ham?
- Should the threshold be 0.5, or should you bias toward avoiding false positives?
- How does changing the threshold affect precision vs recall?

Thinking Exercise

Before writing any code, work through this example by hand:

Training Data:

Spam emails: 3 total
- Email 1: “buy now free”
- Email 2: “free money now”
- Email 3: “click here free”
Ham emails: 2 total
- Email 4: “meeting now”
- Email 5: “project update here”

Step 1: Calculate Prior Probabilities

P(Spam) = ?
P(Ham) = ?

Step 2: Build the Vocabulary and Count Words

Complete vocabulary: {buy, now, free, money, click, here, meeting, project, update}
Vocabulary size = ?
Create a table showing the count of each word in spam and ham emails

Step 3: Calculate Conditional Probabilities (with Laplace Smoothing)

Total words in spam emails = ?
Total words in ham emails = ?
Calculate P(“free” Spam) using: (count of “free” in spam + 1) / (total words in spam + vocab size)
Calculate P(“free” Ham) using the same formula
Calculate P(“meeting” Spam) and P(“meeting” Ham)

Step 4: Classify a New Email New email: “free meeting”

Calculate the probability it’s spam:

Start with log P(Spam)
Add log P(“free” Spam)
Add log P(“meeting” Spam)
Final: log P(Spam “free meeting”) = ?

Calculate the probability it’s ham:

Start with log P(Ham)
Add log P(“free” Ham)
Add log P(“meeting” Ham)
Final: log P(Ham “free meeting”) = ?

Questions to answer:

Which probability is higher? Is the email classified as spam or ham?
Why did we use logarithms instead of multiplying the probabilities directly?
What happens if the new email contains the word “discount” which never appeared in training?
How would your classification change if the prior P(Spam) was 0.9 instead of 0.6?

The Interview Questions They’ll Ask

Explain Bayes’ theorem in plain English and give an example of how it’s used in spam filtering.

Expected: Describe how we use evidence to update beliefs, and explain the reversal from P(words

Spam) to P(Spam

words)

Why is the Naive Bayes classifier called “naive”? What assumption does it make?
- Expected: Discuss the independence assumption and give examples where it’s violated but the algorithm still works
What is Laplace smoothing and why is it necessary?
- Expected: Explain the zero-probability problem and how adding 1 to all counts solves it
How would you handle numerical underflow when multiplying many small probabilities?
- Expected: Describe using log probabilities and converting multiplication to addition
What’s the difference between false positives and false negatives in spam filtering, and which is worse?
- Expected: False positive = ham marked as spam (user misses important email); false negative = spam marked as ham (user sees spam). Discuss trade-offs.
How would you improve the basic Naive Bayes classifier?
- Expected: Mention using n-grams instead of individual words, incorporating metadata (sender, time), or using TF-IDF weighting
Why can’t you just count how many spam words vs ham words are in an email?
- Expected: Explain that not all words are equally informative; probabilities weight the evidence appropriately

Books That Will Help

Topic	Book	Chapter
Basic Probability Theory	Data Science for Business	Ch. 4: “Fitting a Model to Data”
Conditional Probability	Data Science for Business	Ch. 9: “Evidence and Probabilities”
Bayes’ Theorem & Naive Bayes	Data Science for Business	Ch. 9: “Evidence and Probabilities”
Probability Distributions	Mathematics for Machine Learning	Ch. 6: “Probability and Distributions”
Text Classification	Data Science for Business	Ch. 10: “Representing and Mining Text”
Log Probabilities & Numerical Stability	Mathematics for Machine Learning	Ch. 6.2: “Discrete and Continuous Probabilities”
Model Evaluation (Precision/Recall)	Data Science for Business	Ch. 7: “Decision Analytic Thinking I”
Smoothing Techniques	Data Science for Business	Ch. 9: “Evidence and Probabilities”

Project 6: Graph Traversal Visualizer

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: JavaScript (with a library like D3.js or p5.js)
Alternative Programming Languages: Python (with NetworkX and Matplotlib)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 1. The “Resume Gold”
Difficulty: Level 2: Intermediate
Knowledge Area: Discrete Math / Graph Theory
Software or Tool: An interactive web page that visualizes graph algorithms.
Main Book: “A Common-Sense Guide to Data Structures and Algorithms” by Jay Wengrow.

What you’ll build: An interactive web application where users can create a graph (nodes and edges) and then visualize pathfinding algorithms like Breadth-First Search (BFS), Depth-First Search (DFS), and Dijkstra’s Algorithm in action.

Why it teaches math: It turns the abstract concepts of graph theory into a concrete, visual experience. You’ll see exactly how these fundamental algorithms explore a network, which is crucial for understanding everything from social networks to internet routing.

Core challenges you’ll face:

Representing a graph in code → maps to adjacency lists or adjacency matrices
Implementing BFS and DFS → maps to using queues and stacks (or recursion) to traverse a graph
Implementing Dijkstra’s algorithm → maps to using a priority queue to find the shortest path in a weighted graph
Visualizing the algorithm’s progress step-by-step → maps to managing state and using a rendering loop

Key Concepts:

Graphs: A set of vertices (nodes) and edges that connect them.
Adjacency List: A common way to represent a graph, where each vertex has a list of its neighbors.
BFS (Breadth-First Search): Explores a graph layer by layer. Uses a queue.
DFS (Depth-First Search): Explores a graph by going as deep as possible down one path before backtracking. Uses a stack or recursion.
Dijkstra’s Algorithm: An algorithm for finding the shortest paths between nodes in a weighted graph.

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Familiarity with basic data structures (queues, stacks, priority queues).

Real world outcome: A web page where you can click to create nodes and edges, select a start and end node, and watch the pathfinding algorithm color the nodes as it explores them, finally highlighting the shortest path.

Implementation Hints:

Use an adjacency list (e.g., a hash map where keys are node IDs and values are lists of neighbor IDs) to store your graph.
For visualization, you’ll need to separate the algorithm’s logic from its execution. Instead of running the whole algorithm at once, make it return a list of “steps” (e.g., [{node: 5, status: 'visiting'}, {node: 7, status: 'visited'}]).
Your JavaScript rendering loop (requestAnimationFrame) can then pull one step from this list on each frame and update the colors of the nodes on the canvas, creating an animation.
BFS: Use a queue. Add the start node. While the queue is not empty, dequeue a node, process it, and enqueue all its unvisited neighbors.
DFS: Use a stack (or just use recursion, which uses the call stack). Add the start node. While the stack is not empty, pop a node, process it, and push all its unvisited neighbors.
Dijkstra: Use a priority queue to always explore the next-closest node from the start.

Learning milestones:

You can draw a user-created graph → Your data representation and rendering are working.
BFS and DFS correctly traverse the entire graph → You’ve implemented the core traversal algorithms.
Dijkstra’s algorithm finds the shortest path in a weighted graph → You’ve implemented a more complex pathfinding algorithm.
The visualization is clear and animated → You have a polished, educational tool.

The Core Question You’re Answering

How do different graph traversal algorithms explore a network, and why does the order in which you visit nodes matter for finding paths, detecting cycles, and understanding connectivity?

Concepts You Must Understand First

Graph Fundamentals (Nodes, Edges, and Relationships)
- A graph G = (V, E) consists of a set of vertices V and a set of edges E that connect them
- Graphs can be directed (edges have direction) or undirected (edges are bidirectional)
- Graphs can be weighted (edges have costs/distances) or unweighted (all edges are equal)
Guiding questions:
- How would you represent a social network as a graph? What are the nodes and edges?
- What’s the difference between a directed and undirected graph? Give examples of each.
- In a weighted graph representing a road network, what might the edge weights represent?
Book reference: “A Common-Sense Guide to Data Structures and Algorithms” by Jay Wengrow — Ch. 18: “Graphs”
Graph Representation (Adjacency List vs Adjacency Matrix)
- An adjacency list stores each vertex with a list of its neighboring vertices
- An adjacency matrix is a 2D array where matrix[i][j] indicates if there’s an edge from vertex i to vertex j
- Adjacency lists are space-efficient for sparse graphs; matrices are better for dense graphs
Guiding questions:
- For a graph with 1000 nodes and only 2000 edges, which representation uses less memory? Why?
- How do you check if there’s an edge between two specific nodes in each representation? What’s the time complexity?
- How do you find all neighbors of a node in each representation? What’s the time complexity?
Book reference: “Grokking Algorithms” by Aditya Bhargava — Ch. 6: “Breadth-First Search”
Queues and Stacks (The Data Structures Behind Traversal)
- A queue is FIFO (First-In-First-Out): elements are removed in the order they were added
- A stack is LIFO (Last-In-First-Out): the most recently added element is removed first
- The choice between queue and stack fundamentally changes how you explore a graph
Guiding questions:
- If you add nodes [A, B, C] to a queue, in what order do you remove them?
- If you add nodes [A, B, C] to a stack, in what order do you remove them?
- Why does using a queue lead to “breadth-first” exploration while a stack leads to “depth-first”?
Book reference: “A Common-Sense Guide to Data Structures and Algorithms” by Jay Wengrow — Ch. 9: “Stacks and Queues”
Breadth-First Search (BFS) - Layer-by-Layer Exploration
- BFS explores all nodes at distance k before exploring nodes at distance k+1
- It uses a queue to keep track of which nodes to visit next
- BFS finds the shortest path in unweighted graphs
Guiding questions:
- Starting from node A, if A connects to [B, C] and B connects to [D, E], in what order does BFS visit all nodes?
- Why does BFS guarantee the shortest path in an unweighted graph?
- What’s the time complexity of BFS? How many times do you visit each edge?
Book reference: “Grokking Algorithms” by Aditya Bhargava — Ch. 6: “Breadth-First Search”
Depth-First Search (DFS) - Exploring As Deep As Possible
- DFS follows a path as far as possible before backtracking
- It can be implemented with a stack or recursively (using the call stack)
- DFS is useful for detecting cycles, topological sorting, and maze solving
Guiding questions:
- Starting from node A, if A connects to [B, C], B connects to [D, E], and C connects to [F, G], in what order might DFS visit all nodes?
- Why is DFS often implemented recursively? What does the recursive call stack represent?
- How can you modify DFS to detect if a graph contains a cycle?
Book reference: “Algorithms, Fourth Edition” by Sedgewick & Wayne — Ch. 4: “Graphs” (section 4.1 on DFS)
Dijkstra’s Algorithm (Finding Shortest Paths in Weighted Graphs)
- Dijkstra’s algorithm finds the shortest path from a source node to all other nodes in a weighted graph
- It uses a priority queue to always explore the next closest unvisited node
- It only works with non-negative edge weights
Guiding questions:
- Why doesn’t BFS work for finding the shortest path in a weighted graph?
- What invariant does Dijkstra’s algorithm maintain? (Hint: once a node is visited, its shortest distance is final)
- Why must all edge weights be non-negative for Dijkstra’s to work correctly?
Book reference: “Algorithms, Fourth Edition” by Sedgewick & Wayne — Ch. 4: “Graphs” (section 4.4 on shortest paths)

Questions to Guide Your Design

How will you represent the graph data structure?
- Will you use an adjacency list or adjacency matrix? Why?
- How will you store node positions for visualization (x, y coordinates)?
- How will you handle weighted vs unweighted edges?
How will you make the algorithm visualization step-by-step?
- Should the algorithm run to completion first, then replay the steps? Or should it pause between steps?
- What information do you need to store for each step (node being visited, current distance, parent node)?
- How will you represent node states (unvisited, visiting, visited) visually?
How will you handle user interaction?
- How do users create nodes and edges? Click to add nodes, drag between nodes to create edges?
- How do users select the start and end nodes for pathfinding?
- Should users be able to adjust edge weights? If so, how?
How will you implement the BFS algorithm?
- What data structure will you use to track which nodes to visit next?
- How will you mark nodes as visited to avoid revisiting them?
- How will you reconstruct the path from start to end after BFS completes?
How will you implement the DFS algorithm?
- Will you use an explicit stack or recursion?
- How does the visiting order differ from BFS? Can you predict which nodes will be visited in what order?
- How will you handle graphs with cycles to prevent infinite loops?
How will you implement Dijkstra’s algorithm?
- What data structure will you use for the priority queue (min-heap)?
- How will you track the current shortest distance to each node?
- How will you handle updating distances when you find a shorter path to a node?

Thinking Exercise

Before coding, trace through these algorithms by hand on this graph:

Graph (undirected, weighted):
    A ---2--- B
    |         |
    1         3
    |         |
    C ---1--- D ---2--- E

Exercise 1: Breadth-First Search from A

Initialize: Queue = [A], Visited = {}
Step by step, show the state of the queue and visited set after each node is processed
In what order are nodes visited?
Draw the BFS tree (which edges were used?)

Exercise 2: Depth-First Search from A

Initialize: Stack = [A], Visited = {}
Step by step, show the state of the stack and visited set after each node is processed
In what order are nodes visited? (Note: may differ from BFS)
Draw the DFS tree (which edges were used?)

Exercise 3: Dijkstra’s Algorithm from A

Initialize: dist[A] = 0, dist[others] = ∞, Priority Queue = [(0, A)]
For each step, show:
- Current node being processed
- Priority queue contents
- Current distances to all nodes
- Which distances get updated
What’s the shortest path from A to E? What’s the total distance?
In what order are nodes finalized (removed from priority queue)?

Questions to answer:

Why does BFS visit nodes in a different order than DFS?
Does BFS find the shortest path from A to E in this weighted graph? Why or why not?
How does Dijkstra’s algorithm know when it has found the shortest path to a node?
If all edges had weight 1, would BFS, DFS, and Dijkstra’s all find the same shortest path?

The Interview Questions They’ll Ask

Explain the difference between BFS and DFS. When would you use each?
- Expected: BFS for shortest path in unweighted graphs, level-order traversal; DFS for cycle detection, topological sort, exploring all paths
What data structures do BFS and DFS use, and why?
- Expected: BFS uses a queue for FIFO ordering; DFS uses a stack (or recursion) for LIFO ordering
Why doesn’t BFS work for finding shortest paths in weighted graphs?
- Expected: BFS assumes all edges have equal cost; in weighted graphs, a path with more edges might have lower total weight
How does Dijkstra’s algorithm differ from BFS?
- Expected: Dijkstra’s uses a priority queue to always explore the closest unvisited node; handles weighted edges
What’s the time complexity of BFS and DFS? What about Dijkstra’s?
- Expected: BFS and DFS are O(V + E); Dijkstra’s is O((V + E) log V) with a binary heap
How would you detect a cycle in a graph using DFS?
- Expected: If you encounter a node that’s currently being visited (on the recursion stack), there’s a cycle
Can Dijkstra’s algorithm handle negative edge weights? Why or why not?
- Expected: No, negative weights can break the greedy property; a longer path might become shorter after exploring negative edges

Books That Will Help

Topic	Book	Chapter
Graph fundamentals	A Common-Sense Guide to Data Structures and Algorithms	Ch. 18: “Graphs”
Graph representation	Grokking Algorithms	Ch. 6: “Breadth-First Search”
BFS algorithm	Grokking Algorithms	Ch. 6: “Breadth-First Search”
DFS algorithm	Algorithms, Fourth Edition	Ch. 4.1: “Undirected Graphs” (DFS section)
Dijkstra’s algorithm	Algorithms, Fourth Edition	Ch. 4.4: “Shortest Paths”
Priority queues	Algorithms, Fourth Edition	Ch. 2.4: “Priority Queues”
Graph applications	A Common-Sense Guide to Data Structures and Algorithms	Ch. 18: “Graphs”
Weighted graphs	Algorithms, Fourth Edition	Ch. 4.4: “Shortest Paths”
Topological sort	Algorithms, Fourth Edition	Ch. 4.2: “Directed Graphs”

Final Project: A Neural Network from Scratch

File: LEARN_MATH_FOR_PROGRAMMING.md
Main Programming Language: Python (with NumPy)
Alternative Programming Languages: Go, Rust
Coolness Level: Level 5: Pure Magic (Super Cool)
Business Potential: 3. The “Service & Support” Model
Difficulty: Level 5: Master
Knowledge Area: Linear Algebra / Calculus / Machine Learning
Software or Tool: A program that recognizes handwritten digits.
Main Book: “Mathematics for Machine Learning” by Deisenroth, Faisal, and Ong.

What you’ll build: A simple neural network that can learn to recognize handwritten digits from the MNIST dataset, using only basic matrix operations and calculus.

Why it’s the final project: This project is the ultimate fusion of programming and mathematics. It combines linear algebra (for the network’s structure and forward propagation), calculus (for training the network via backpropagation with gradient descent), and probability/statistics (for the loss function and accuracy measurement). It’s the “hello world” of deep learning, built from first principles.

Core challenges you’ll face:

Implementing the network layers → maps to matrix multiplication and activation functions
The forward pass → maps to passing data through the network to get a prediction
Calculating the loss → maps to quantifying how wrong the network’s prediction is
The backward pass (backpropagation) → maps to using the chain rule from calculus to calculate the gradient of the loss with respect to each weight in the network
Updating the weights → maps to using gradient descent to improve the network

Key Concepts:

Matrix Multiplication: The fundamental operation of a neural network layer.
Activation Functions (Sigmoid, ReLU): Non-linear functions that allow networks to learn complex patterns.
Loss Function (Cross-Entropy Loss): A function that measures the error of the network’s predictions.
Gradient Descent: An optimization algorithm that minimizes the loss by taking steps in the opposite direction of the gradient.
The Chain Rule: The calculus rule that makes backpropagation possible, allowing you to calculate the derivative of a complex, nested function.

Difficulty: Master Time estimate: 1 month+ Prerequisites: All the mathematical concepts from the previous projects, especially linear algebra and a conceptual understanding of derivatives.

Real world outcome: A program that trains on the MNIST dataset and can then take a new image of a handwritten digit and correctly predict what number it is, achieving >90% accuracy.

$ python ./neural-net.py --train
Epoch 1/10, Loss: 0.352, Accuracy: 90.1%
Epoch 2/10, Loss: 0.168, Accuracy: 94.5%
...
Epoch 10/10, Loss: 0.082, Accuracy: 97.8%
Training complete.

$ python ./neural-net.py --predict my_digit.png
Prediction: 7

Implementation Hints:

Structure: The network will be a list of layers. Each layer has a weight matrix and a bias vector.
Forward Pass: For each layer, the output is activation_function(input @ W + b), where @ is matrix multiplication. The output of one layer is the input to the next.
Loss: The final output will be a vector of 10 probabilities. Compare this to the “true” label (a one-hot encoded vector) using a loss function.
Backward Pass: This is the hardest part. You start at the loss and work backward. For each layer, you calculate how much its weights and biases contributed to the error. This requires finding the derivative of the loss function with respect to the layer’s weights (dL/dW). The chain rule is key: dL/dW = dL/dOutput * dOutput/dZ * dZ/dW, where Z = input @ W + b.
Update: For each weight W in the network, update it: W = W - learning_rate * dL/dW.
Repeat this process for many “epochs” (passes over the training data).

Learning milestones:

The forward pass works and produces a (random) prediction → Your network structure and matrix math are correct.
You can calculate the loss for a single prediction → You’ve defined your objective function.
Backpropagation correctly calculates gradients for a single layer → You’ve mastered the chain rule.
The network’s loss decreases over several training epochs → Your gradient descent implementation is working, and the network is learning!
The network achieves high accuracy on the test set → You have successfully built and trained a neural network from scratch.

The Core Question You’re Answering

How does a machine learn from examples to recognize patterns it has never seen before?

This is not just about neural networks—it’s about understanding the mathematical foundation of intelligence itself. When you complete this project, you’ll grasp how optimization, calculus, and linear algebra converge to create systems that improve through experience. You’ll understand why deep learning works, not just how to use it. You’ll see that “learning” is fundamentally about minimizing error through systematic exploration of a high-dimensional parameter space, guided by gradients that point toward better solutions.

This project answers the deeper questions: What does it mean to “learn”? How can we quantify wrongness? How can we systematically reduce that wrongness? And why does repeatedly adjusting millions of numbers according to their gradients lead to emergent intelligence?

Concepts You Must Understand First

Before you write a single line of code, ensure you deeply understand these foundational concepts. Each builds on the previous ones, forming the mathematical scaffolding of neural networks.

1. Matrix Operations and Dimensionality

What it means: Neural networks are fundamentally matrix multiplication machines. Understanding how matrices transform data, how dimensions must align, and how to reason about shapes is critical. A weight matrix with shape (128, 64) transforms a 128-dimensional input into a 64-dimensional output.

Guiding questions:

If you multiply a matrix of shape (m, n) by a matrix of shape (n, p), what is the resulting shape?
Why must the number of columns in the first matrix equal the number of rows in the second?
How does transposing a matrix change the direction of information flow during backpropagation?

Book reference: Mathematics for Machine Learning Chapter 2 (Linear Algebra), sections 2.1-2.3; Math for Programmers Chapter 4 (Transforming Vectors with Matrices)

2. Forward Propagation

What it means: The process of passing input data through layers of the network to produce an output prediction. Each layer applies a linear transformation (matrix multiplication plus bias) followed by a non-linear activation function. Forward propagation is your network’s “thinking” process.

Guiding questions:

Why do we need both matrix multiplication AND activation functions? What happens if you remove activation functions?
How does the output of one layer become the input to the next?
Why is the final layer’s output interpreted as probabilities for classification tasks?

Book reference: Mathematics for Machine Learning Chapter 8.2 (Deep Neural Networks); Math for Programmers Chapter 4

3. Activation Functions (Sigmoid, ReLU, Softmax)

What it means: Non-linear functions applied after the linear transformation in each layer. Without them, stacking layers would be pointless—any number of linear transformations composes to a single linear transformation. Activation functions give networks the power to learn complex, non-linear patterns.

Guiding questions:

Why does a network without activation functions reduce to linear regression, no matter how many layers it has?
What are the practical differences between sigmoid, ReLU, and tanh? When would you choose each?
Why is softmax specifically used in the output layer for classification?

Book reference: Mathematics for Machine Learning Chapter 8.2.2 (Activation Functions); Deep Learning by Goodfellow et al., Chapter 6.3

4. Loss Functions (Cross-Entropy, Mean Squared Error)

What it means: A mathematical function that quantifies how wrong your network’s predictions are. The loss is a single number that captures the gap between what your network predicted and what it should have predicted. Minimizing this number is the entire goal of training.

Guiding questions:

Why is cross-entropy loss preferred over mean squared error for classification tasks?
How does the loss function connect the network’s output to the true labels?
What does it mean when the loss is 0? Is that always achievable or desirable?

Book reference: Mathematics for Machine Learning Chapter 8.2.4 (Loss Functions for Training); Data Science for Business Chapter 4

5. The Chain Rule of Calculus

What it means: The fundamental theorem that makes backpropagation possible. It states that the derivative of a composite function equals the product of the derivatives of its constituent functions: d/dx[f(g(x))] = f’(g(x)) · g’(x). In neural networks, the loss is a deeply nested composition of functions, and the chain rule lets you “unpack” it layer by layer.

Guiding questions:

If f(x) = (2x + 3)², what is df/dx using the chain rule?
Why can’t you compute gradients for deep networks without the chain rule?
How does the chain rule relate to the concept of “error flowing backward” through the network?

Book reference: Mathematics for Machine Learning Chapter 5.2 (Differentiation of Univariate Functions); Math for Programmers Chapter 10 (Derivatives)

6. Partial Derivatives and Gradients

What it means: When a function has multiple inputs, the partial derivative with respect to one variable measures how the function changes when only that variable changes. The gradient is the vector of all partial derivatives—it points in the direction of steepest increase.

Guiding questions:

If L(w₁, w₂) = w₁² + w₂², what are ∂L/∂w₁ and ∂L/∂w₂?
Why is the gradient a vector, not a single number?
In what direction should you move in parameter space to decrease the loss most quickly?

Book reference: Mathematics for Machine Learning Chapter 5.3 (Partial Differentiation and Gradients)

7. Backpropagation

What it means: The algorithm that efficiently computes the gradient of the loss with respect to every weight in the network. It works backward from the output, using the chain rule to propagate error gradients through each layer. Backpropagation is what makes training deep networks computationally feasible.

Guiding questions:

Why is it more efficient to compute gradients backward from the loss rather than forward from the inputs?
How does each layer use the gradient from the layer above it to compute its own gradient?
What information must you store during the forward pass to compute gradients during the backward pass?

Book reference: Mathematics for Machine Learning Chapter 5.5 (Backpropagation); Deep Learning by Goodfellow et al., Chapter 6.5

8. Gradient Descent Optimization

What it means: The iterative algorithm that updates network weights to minimize the loss. At each step, you compute the gradient (the direction of steepest ascent) and move a small step in the opposite direction (steepest descent). Over many iterations, this converges to a local minimum.

Guiding questions:

Why do we subtract the gradient from the weights rather than add it?
What is the “learning rate” and why does it matter?
Why might gradient descent get stuck in local minima, and why is that often okay in practice?

Book reference: Mathematics for Machine Learning Chapter 7 (Continuous Optimization); Math for Programmers Chapter 10

9. Vectorization and Broadcasting

What it means: Processing entire batches of data at once using matrix operations instead of looping over individual examples. Vectorization is crucial for performance—NumPy/GPU operations on matrices are orders of magnitude faster than Python loops.

Guiding questions:

How do you transform a loop that processes one training example at a time into a single matrix operation that processes a batch?
What does it mean for an operation to be “vectorized”?
How does broadcasting allow you to add a bias vector to every row of a matrix?

Book reference: Math for Programmers Chapter 4; NumPy documentation on broadcasting

10. Numerical Stability and Gradient Flow

What it means: In practice, neural networks can suffer from exploding or vanishing gradients, overflow/underflow in exponentials, and other numerical issues. Understanding these pitfalls and how to avoid them (gradient clipping, careful initialization, normalization) is essential for robust implementations.

Guiding questions:

Why might gradients become vanishingly small in deep networks with sigmoid activations?
What causes numerical overflow when computing softmax, and how do you fix it?
Why is proper weight initialization important for gradient flow?

Book reference: Deep Learning by Goodfellow et al., Chapter 8 (Optimization for Training Deep Models)

Questions to Guide Your Design

These questions should guide your implementation decisions. Think deeply about each one before you start coding.

How will you represent the architecture of your network? Will you hardcode a specific structure, or create a flexible system where layers can be added/removed? How will you store weights, biases, and intermediate activations?
What activation function will you use in hidden layers, and why? ReLU is popular for its simplicity and gradient properties, but sigmoid has nicer mathematical properties for understanding. What are the trade-offs?
How will you handle the forward pass for a batch of inputs simultaneously? Will you process one image at a time or leverage matrix operations to process 32, 64, or 128 images in parallel?
How will you compute and verify your gradients? Can you implement gradient checking using finite differences to ensure your backpropagation math is correct? This is crucial—subtle bugs in gradient computation can cause slow learning or complete failure.
What learning rate will you choose, and how might you adjust it during training? Too large and training diverges; too small and learning is glacially slow. How can you find the sweet spot?
How will you prevent overfitting? Should you implement regularization (L2 penalty on weights)? Early stopping? Dropout? How will you split your data into training and validation sets?
How will you initialize your weights? Random initialization breaks symmetry, but how random? Too large and activations saturate; too small and gradients vanish. What distribution will you sample from?
How will you measure and track progress during training? What metrics beyond loss are meaningful? Accuracy? Precision/recall for each digit? How often should you evaluate on the validation set?

Thinking Exercise

Before writing any code, work through this exercise by hand to build intuition for how neural networks compute and learn.

Manual Forward Pass Calculation

Consider a tiny neural network:

Input layer: 2 neurons (x₁, x₂)
Hidden layer: 2 neurons with sigmoid activation
Output layer: 1 neuron with sigmoid activation

Weights and biases:

W₁ = [[0.5, -0.3],    b₁ = [0.1, -0.2]  (input to hidden)
      [0.2,  0.4]]

W₂ = [[0.6],          b₂ = [0.15]        (hidden to output)
      [-0.5]]

Input: x = [1.0, 0.5] True label: y = 1

Compute these by hand (use sigmoid(z) = 1/(1+e^(-z))):

Hidden layer activations:
- z₁⁽¹⁾ = w₁₁x₁ + w₁₂x₂ + b₁ = ?
- z₂⁽¹⁾ = w₂₁x₁ + w₂₂x₂ + b₂ = ?
- a₁⁽¹⁾ = sigmoid(z₁⁽¹⁾) = ?
- a₂⁽¹⁾ = sigmoid(z₂⁽¹⁾) = ?
Output layer activation:
- z⁽²⁾ = w₁a₁⁽¹⁾ + w₂a₂⁽¹⁾ + b = ?
- ŷ = sigmoid(z⁽²⁾) = ?
Loss (using mean squared error for simplicity):
- L = ½(y - ŷ)² = ?

Manual Backward Pass Calculation

Now compute gradients to see how the network would update:

Output layer gradient:
- dL/dŷ = ŷ - y = ?
- dŷ/dz⁽²⁾ = ŷ(1 - ŷ) = ? (derivative of sigmoid)
- δ⁽²⁾ = dL/dz⁽²⁾ = (dL/dŷ)(dŷ/dz⁽²⁾) = ? (chain rule!)
Output layer weight gradients:
- dL/dW₂ = δ⁽²⁾ · a⁽¹⁾ᵀ = ?
- dL/db₂ = δ⁽²⁾ = ?
Hidden layer gradient (backpropagate through W₂):
- dL/da⁽¹⁾ = W₂ᵀ · δ⁽²⁾ = ?
- dL/dz⁽¹⁾ = dL/da⁽¹⁾ ⊙ a⁽¹⁾(1 - a⁽¹⁾) = ? (⊙ means element-wise multiplication)
Hidden layer weight gradients:
- dL/dW₁ = dL/dz⁽¹⁾ · xᵀ = ?
- dL/db₁ = dL/dz⁽¹⁾ = ?

Questions to answer:

Which weights have the largest gradients? Why?
If the learning rate is 0.1, what are the new values for all weights after one update?
How would the prediction ŷ change after this update? Would it be closer to the true label y = 1?

The Interview Questions They’ll Ask

These are real questions you should be able to answer confidently after completing this project.

Explain backpropagation to someone who understands calculus but has never studied machine learning. Focus on the chain rule and the computational graph perspective.
Why do we use mini-batches for training instead of computing the gradient on the entire dataset at once? Discuss computational efficiency, memory constraints, and stochastic gradient descent’s regularization effect.
What is the vanishing gradient problem, and how does ReLU activation help mitigate it? Explain how gradients can become exponentially small in deep networks with sigmoid activations.
Walk me through the dimensions of all matrices during forward and backward propagation for a network with architecture [784, 128, 64, 10]. Be specific about input batches, weight matrices, activation vectors, and gradient shapes.
How would you debug a neural network that isn’t learning? Discuss gradient checking, visualizing loss curves, checking for dead ReLUs, examining weight distributions, and verifying data preprocessing.
What’s the difference between a parameter and a hyperparameter? Give examples of each. Parameters are learned (weights, biases); hyperparameters are set before training (learning rate, batch size, number of layers).
Explain the bias-variance tradeoff in the context of neural networks. How does model complexity, training data size, and regularization affect this tradeoff?
Why do we need a separate validation set in addition to the test set? Discuss the danger of overfitting to the validation set through hyperparameter tuning.
How does the choice of loss function affect what the network learns? Compare cross-entropy vs. MSE for classification, and explain why cross-entropy is preferred.
Implement the derivative of the softmax function combined with cross-entropy loss. Why is this combination mathematically elegant? The gradient simplifies to (ŷ - y), which has a beautiful interpretation.

Books That Will Help

This is your reading list for mastering the mathematics and implementation of neural networks from scratch. Read these chapters in order, alongside your coding.

Topic	Book	Chapter
Linear Algebra Foundations	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 2: Linear Algebra
Matrix Operations & Transformations	Math for Programmers by Paul Orland	Chapter 4: Transforming Vectors with Matrices
Vectors and Dot Products	Math for Programmers by Paul Orland	Chapter 2: Drawing with 2D Vectors
Derivatives and Rates of Change	Math for Programmers by Paul Orland	Chapter 10: Derivatives
The Chain Rule	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 5.2: Differentiation of Univariate Functions
Partial Derivatives & Gradients	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 5.3: Partial Differentiation and Gradients
Backpropagation Theory	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 5.5: Backpropagation
Gradient Descent & Optimization	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 7: Continuous Optimization
Neural Network Architecture	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 8.2: Deep Neural Networks
Activation Functions	Deep Learning by Goodfellow, Bengio & Courville	Chapter 6.3: Activation Functions
Loss Functions & Training	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 8.2.4: Loss Functions
Numerical Stability	Deep Learning by Goodfellow, Bengio & Courville	Chapter 8: Optimization for Training Deep Models
Probability for Classification	Mathematics for Machine Learning by Deisenroth, Faisal & Ong	Chapter 6: Probability and Distributions
Overfitting Prevention	Data Science for Business by Provost & Fawcett	Chapter 5: Overfitting and Its Avoidance
Practical ML Implementation	Grokking Deep Learning by Andrew Trask	Chapters 1-8 (entire book is building from scratch)
NumPy for Neural Networks	Python Data Science Handbook by Jake VanderPlas	Chapter 2: Introduction to NumPy

Recommended Reading Path:

Week 1-2: Foundation

Math for Programmers Ch. 2, 4 (vectors and matrices)
Mathematics for Machine Learning Ch. 2 (linear algebra)
Start implementing basic matrix operations in NumPy

Week 3: Calculus

Math for Programmers Ch. 10 (derivatives)
Mathematics for Machine Learning Ch. 5.2-5.3 (chain rule and gradients)
Implement forward propagation

Week 4: Backpropagation

Mathematics for Machine Learning Ch. 5.5 (backpropagation)
Grokking Deep Learning Ch. 4-6 (gradient descent and backprop)
Implement backward propagation and gradient checking

Week 5: Optimization & Training

Mathematics for Machine Learning Ch. 7, 8.2 (optimization and neural networks)
Deep Learning Ch. 6.3, 8 (activation functions and training)
Train your network on MNIST

Week 6+: Refinement

Data Science for Business Ch. 5 (avoiding overfitting)
Experiment with architecture, hyperparameters, and regularization
Achieve >95% test accuracy

Project Comparison Table

Project	Difficulty	Time	Math Area	Fun Factor
Sudoku Solver	2	Weekend	Discrete Math	★★★☆☆
3D Rendering Engine	3	1-2 weeks	Linear Algebra	★★★★★
RSA Cryptosystem	3	1-2 weeks	Number Theory	★★★★☆
Physics Engine	3	1-2 weeks	Calculus, LinAlg	★★★★★
Spam Filter	2	Weekend	Probability	★★★☆☆
Graph Visualizer	2	1-2 weeks	Discrete Math	★★★★☆
Neural Network	5	1 month+	LinAlg, Calculus	★★★★★

Recommendation

A great path through this material would be:

Start with the Sudoku Solver. It’s a fun, satisfying introduction to algorithmic thinking and discrete math without being overwhelming.
Next, build the Bayesian Spam Filter. It’s a quick and practical introduction to probability that yields a genuinely useful tool.
Then, tackle the 3D Rendering Engine. This is the best project for making linear algebra “click.” It’s challenging but incredibly rewarding.
With those under your belt, you’ll be well-prepared to take on the more advanced projects like the Physics Engine, RSA Cryptosystem, or the final boss: the Neural Network from Scratch.

This path builds from logic to probability to linear algebra, giving you a solid and ever-increasing foundation of mathematical tools.

Summary

Project 1: Sudoku Solver: Go
Project 2: A 3D Rendering Engine from Scratch: Python
Project 3: RSA Cryptosystem from Scratch: Python
Project 4: A Simple Physics Engine: Python
Project 5: Bayesian Spam Filter: Python
Project 6: Graph Traversal Visualizer: JavaScript
Final Project: A Neural Network from Scratch: Python