← Back to all projects

LINEAR ALGEBRA LEARNING PROJECTS

Learning Linear Algebra Through Building

Linear algebra is the mathematics of transformations and spaces. It’s the language that computers use to manipulate graphics, train AI, process signals, and simulate physics. To truly internalize it, you need to build systems where these concepts aren’t abstractions—they’re the actual machinery.

Core Concept Analysis

Linear algebra breaks down into these fundamental building blocks:

Concept What It Really Means
Vectors Arrows in space (direction + magnitude), or ordered lists of numbers
Matrices Transformation machines that stretch, rotate, shear, or project vectors
Linear Transformations Functions that preserve lines and the origin
Systems of Equations Finding where multiple constraints intersect
Eigenvalues/Eigenvectors The “natural axes” of a transformation—directions that only scale
Dot/Cross Products Measuring alignment (dot) and perpendicularity (cross)
Determinants How much a transformation scales area/volume
Matrix Decompositions Breaking matrices into simpler, meaningful pieces (LU, QR, SVD)

Project 1: Software 3D Renderer from Scratch

  • File: LINEAR_ALGEBRA_LEARNING_PROJECTS.md
  • Programming Language: C
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Computer Graphics / Linear Algebra
  • Software or Tool: Graphics Pipeline
  • Main Book: “Computer Graphics from Scratch” by Gabriel Gambetta

What you’ll build: A program that renders 3D wireframe models to 2D screen coordinates, with rotation, scaling, and perspective projection—no graphics libraries.

Why it teaches linear algebra: Every single operation in 3D graphics IS linear algebra. You can’t fake it—the math is literally what makes pixels appear in the right place. Rotating a cube? That’s a rotation matrix. Perspective? That’s a projection matrix. Camera movement? Translation and transformation composition.

Core challenges you’ll face:

  • Representing 3D points as vectors and transforming them (maps to vector operations)
  • Building rotation matrices for X, Y, Z axes (maps to matrix construction and multiplication)
  • Composing multiple transformations in the correct order (maps to matrix multiplication non-commutativity)
  • Projecting 3D coordinates to 2D screen space (maps to projection matrices)
  • Implementing a camera/view matrix (maps to change of basis)

Key Concepts:

  • Vectors and coordinates: “Computer Graphics from Scratch” by Gabriel Gambetta - Chapter 1
  • Transformation matrices: “3Blue1Brown: Essence of Linear Algebra” - Episode 3 (YouTube)
  • Matrix multiplication: “Math for Programmers” by Paul Orland - Chapter 4
  • Homogeneous coordinates: “Computer Graphics from Scratch” by Gabriel Gambetta - Chapter 12
  • Projection: “Math for Programmers” by Paul Orland - Chapter 5

Difficulty: Intermediate Time estimate: 2-3 weeks Prerequisites: Basic programming, trigonometry fundamentals

Real world outcome: You will see a rotating 3D wireframe cube (or any OBJ model you load) rendered on your screen. You can move a virtual camera around the scene. When you change a single number in your rotation matrix, you’ll SEE the object rotate—making the abstract concrete.

Learning milestones:

  1. Render a static 2D projection of a 3D cube → You understand vectors as coordinates
  2. Rotate the cube smoothly with keyboard input → You understand transformation matrices
  3. Add perspective (things farther away appear smaller) → You understand projection matrices
  4. Move a camera through the scene → You understand change of basis and inverse transformations

Project 2: Image Compression Using SVD

  • File: LINEAR_ALGEBRA_LEARNING_PROJECTS.md
  • Programming Language: Python (or C)
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Linear Algebra / Image Processing
  • Software or Tool: SVD
  • Main Book: “Math for Programmers” by Paul Orland

What you’ll build: A tool that compresses images by decomposing them with Singular Value Decomposition (SVD), letting you visually see how much information each “rank” captures.

Why it teaches linear algebra: Images are matrices. SVD reveals their hidden structure—the most important “directions” of variation. You’ll viscerally understand that a matrix can be broken into simpler pieces, and that some pieces matter more than others. This is the foundation for dimensionality reduction, recommendation systems, and data analysis.

Core challenges you’ll face:

  • Representing images as matrices of pixel values (maps to matrix representation)
  • Implementing or using SVD decomposition (maps to matrix factorization)
  • Understanding what U, Σ, V^T actually represent (maps to eigenvectors and singular values)
  • Reconstructing images from partial decompositions (maps to low-rank approximation)
  • Measuring compression quality vs. file size (maps to understanding rank and information)

Key Concepts:

  • Matrix representation of data: “Math for Programmers” by Paul Orland - Chapter 6
  • SVD intuition: “3Blue1Brown: Essence of Linear Algebra” - Singular Value Decomposition (YouTube)
  • Eigenvalues and eigenvectors: “Grokking Algorithms” by Aditya Bhargava - Appendix on Linear Algebra
  • Low-rank approximation: “Hands-On Machine Learning” by Aurélien Géron - Chapter 8

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Basic programming, understanding of what matrices are

Real world outcome: You’ll have a CLI tool where you input an image and a “rank” parameter. At rank 1, you see a blurry smear. At rank 10, recognizable shapes. At rank 50, nearly perfect quality at 10% the data. You’ll literally SEE linear algebra compressing information.

Learning milestones:

  1. Load an image as a matrix of numbers → You understand matrices as data
  2. Apply SVD and reconstruct the original → You understand matrix decomposition
  3. Reconstruct with only the top k singular values → You understand rank and information content
  4. Compare compression ratios and quality → You understand the power of eigenvalue decomposition

Project 3: Simple Neural Network from Scratch

  • File: LINEAR_ALGEBRA_LEARNING_PROJECTS.md
  • Programming Language: Python (or C)
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Machine Learning / Linear Algebra
  • Software or Tool: Matrix Calculus
  • Main Book: “Math for Programmers” by Paul Orland

What you’ll build: A neural network that classifies handwritten digits (MNIST), implementing forward propagation, backpropagation, and training—using only matrix operations.

Why it teaches linear algebra: Neural networks ARE linear algebra + nonlinearity. Every layer is a matrix multiplication. Backpropagation is the chain rule applied to matrices. You can’t understand deep learning without understanding that it’s just matrices transforming vectors through high-dimensional space.

Core challenges you’ll face:

  • Representing weights as matrices and inputs as vectors (maps to matrix-vector multiplication)
  • Computing layer outputs through matrix operations (maps to linear transformations)
  • Implementing backpropagation via matrix calculus (maps to gradients and transposes)
  • Understanding why weight initialization matters (maps to eigenvalue distribution)
  • Batch processing multiple inputs efficiently (maps to matrix-matrix multiplication)

Key Concepts:

  • Matrix multiplication as transformation: “Math for Programmers” by Paul Orland - Chapter 4
  • Gradient descent fundamentals: “Hands-On Machine Learning” by Aurélien Géron - Chapter 4
  • Neural network mathematics: “Grokking Algorithms” by Aditya Bhargava - Chapter on ML
  • Backpropagation mechanics: “3Blue1Brown: Neural Networks” - Episode 4 (YouTube)

Difficulty: Intermediate-Advanced Time estimate: 2-3 weeks Prerequisites: Calculus basics (derivatives), matrix multiplication

Real world outcome: You draw a digit on screen (or feed in MNIST test images), and your network correctly classifies it. You can visualize the weight matrices as images—seeing what “features” each neuron learned. When accuracy improves during training, you’re watching gradient descent navigate a high-dimensional landscape.

Learning milestones:

  1. Implement forward pass with matrix multiplication → You understand linear transformations
  2. Compute gradients via backpropagation → You understand matrix calculus and transposes
  3. Train and see accuracy improve → You understand optimization in high-dimensional space
  4. Visualize learned weights → You understand what matrices “encode”

Project 4: Physics Simulation with Constraints

  • File: LINEAR_ALGEBRA_LEARNING_PROJECTS.md
  • Programming Language: C
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 4: Expert
  • Knowledge Area: Physics Simulation / Linear Algebra
  • Software or Tool: Physics Engine
  • Main Book: “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron (Mathematical foundations)

What you’ll build: A 2D physics engine that simulates rigid body dynamics with constraints (springs, joints, collisions)—requiring you to solve systems of linear equations each frame.

Why it teaches linear algebra: Physics simulation requires solving Ax = b every frame. Collision response requires projections. Constraint solving requires understanding null spaces and least squares. You’ll see linear algebra as the engine of physical reality.

Core challenges you’ll face:

  • Representing position and velocity as vectors (maps to vectors as state)
  • Solving systems of equations for constraint forces (maps to solving Ax = b)
  • Detecting and resolving collisions via projections (maps to projections and dot products)
  • Implementing stable integration (maps to matrix conditioning)
  • Handling over/under-constrained systems (maps to rank, null space, least squares)

Key Concepts:

  • Vectors as physical quantities: “Math for Programmers” by Paul Orland - Chapter 2
  • Solving linear systems: “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron - Mathematical foundations
  • Projections and orthogonality: “3Blue1Brown: Essence of Linear Algebra” - Episode on Dot Products
  • Numerical methods: “Numerical Recipes in C” by Press et al. - Chapter 2 (Gaussian elimination)

Difficulty: Advanced Time estimate: 3-4 weeks Prerequisites: Basic physics intuition, matrix operations

Real world outcome: You’ll see balls bouncing, chains swinging, and objects stacking realistically on screen. When you add a constraint, you’re adding an equation. When things explode (they will at first), you’ll debug by examining matrix conditioning—making abstract stability concepts visceral.

Learning milestones:

  1. Simulate unconstrained motion (gravity, velocity) → You understand vectors as state
  2. Add collision detection and response → You understand projections and dot products
  3. Implement spring/joint constraints → You understand solving systems of equations
  4. Handle stability and stacking → You understand matrix conditioning and numerical stability

Project 5: Recommendation Engine with Matrix Factorization

  • File: LINEAR_ALGEBRA_LEARNING_PROJECTS.md
  • Programming Language: Python (or C)
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Machine Learning / Linear Algebra
  • Software or Tool: Matrix Factorization
  • Main Book: “Data Science for Business” by Provost & Fawcett

What you’ll build: A movie/music recommendation system that uses matrix factorization to predict user preferences from sparse ratings data.

Why it teaches linear algebra: Recommendation systems treat users and items as vectors in a latent space. Factorizing the ratings matrix reveals hidden structure—what “dimensions” of taste exist. This is applied linear algebra for real business value.

Core challenges you’ll face:

  • Representing the user-item ratings as a sparse matrix (maps to matrix representation)
  • Factorizing into user and item embedding matrices (maps to matrix factorization)
  • Minimizing reconstruction error (maps to optimization and Frobenius norm)
  • Handling missing values (maps to sparse matrices and regularization)
  • Finding similar users/items via vector similarity (maps to dot products and cosine similarity)

Key Concepts:

  • Matrix factorization intuition: “Data Science for Business” by Provost & Fawcett - Chapter on recommendations
  • Optimization and loss functions: “Hands-On Machine Learning” by Aurélien Géron - Chapter 4
  • Similarity metrics: “Math for Programmers” by Paul Orland - Chapter 6

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Basic matrix operations, some optimization intuition

Real world outcome: You input your movie ratings, and the system recommends films you’ll likely enjoy. You can inspect the learned “latent factors”—discovering that one dimension might correspond to “artsy vs. blockbuster” and another to “action vs. romance.” Abstract vectors become interpretable preferences.

Learning milestones:

  1. Build the ratings matrix from data → You understand sparse matrix representation
  2. Implement matrix factorization → You understand low-rank approximation
  3. Predict missing ratings → You understand reconstruction from factors
  4. Find similar items via embeddings → You understand vector similarity in learned spaces

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor Primary Concepts
3D Renderer Intermediate 2-3 weeks ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ Transformations, matrices, projections
SVD Image Compression Intermediate 1-2 weeks ⭐⭐⭐⭐ ⭐⭐⭐⭐ Decomposition, eigenvalues, rank
Neural Network Intermediate-Advanced 2-3 weeks ⭐⭐⭐⭐ ⭐⭐⭐⭐ Matrix multiplication, gradients
Physics Simulation Advanced 3-4 weeks ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ Systems of equations, projections
Recommendation Engine Intermediate 1-2 weeks ⭐⭐⭐ ⭐⭐⭐ Factorization, similarity, sparse matrices

Recommendation

Start with the 3D Renderer.

Here’s why:

  1. Visual feedback loop: Every matrix operation produces a visible result. Mess up a rotation matrix? The cube warps. Get it right? It spins smoothly. This immediate feedback burns the concepts into your brain.

  2. Forces foundational understanding: You can’t skip ahead. You must understand vectors before transformations, transformations before composition, composition before projection.

  3. Historical authenticity: This is how linear algebra was developed for computers—to solve graphics problems. You’re learning it the way it was meant to be learned.

  4. Gateway to everything else: Once you understand transformation matrices, neural networks (just more transformations) and physics (just vectors and constraints) become natural extensions.

Supplementary resource to work through in parallel: Watch 3Blue1Brown’s “Essence of Linear Algebra” series (YouTube, ~3 hours total). It provides the geometric intuition that textbooks often lack. Watch one episode, then implement that concept in your renderer.

Final Capstone Project: Real-Time Ray Tracer with Global Illumination

What you’ll build: A ray tracer that renders photorealistic 3D scenes by simulating light bouncing through the environment, with reflections, refractions, and soft shadows—all from pure linear algebra.

Why it teaches the complete picture: This project synthesizes EVERYTHING:

  • Vectors: Rays are origin points + direction vectors
  • Dot products: Determine angles for lighting calculations
  • Cross products: Compute surface normals
  • Transformations: Object positioning and camera movement
  • Systems of equations: Ray-object intersection
  • Matrix inverses: Transforming rays into object space
  • Projections: Reflection and refraction vectors
  • Orthonormal bases: Building coordinate frames for sampling

Core challenges you’ll face:

  • Computing ray-sphere and ray-plane intersections (maps to solving quadratic/linear systems)
  • Implementing reflection and refraction vectors (maps to projections and vector decomposition)
  • Building camera transformation matrices (maps to change of basis)
  • Computing surface normals for arbitrary geometry (maps to cross products and normalization)
  • Implementing Monte Carlo sampling for soft effects (maps to basis construction and hemispheres)

Key Concepts:

  • Ray-geometry intersection: “Computer Graphics from Scratch” by Gabriel Gambetta - Chapters on Raytracing
  • Reflection/refraction physics: “Physically Based Rendering” by Pharr & Humphreys - Chapter 8
  • Transformation matrices: “Math for Programmers” by Paul Orland - Chapter 5
  • Monte Carlo methods: “Physically Based Rendering” by Pharr & Humphreys - Chapter 2

Difficulty: Advanced Time estimate: 1-2 months Prerequisites: Complete at least the 3D Renderer project first

Real world outcome: You’ll generate photorealistic images—shiny metal spheres reflecting checkered floors, glass balls refracting light, soft shadows from area lights. Each stunning image is proof that you’ve internalized linear algebra deeply enough to simulate light itself. You can even render animations.

Learning milestones:

  1. Cast rays and detect sphere intersections → You’ve mastered vector equations
  2. Add Phong shading with proper lighting → You understand dot products as projection
  3. Implement reflections and refractions → You understand vector decomposition
  4. Add transforms for camera and objects → You understand matrix inverses and change of basis
  5. Implement soft shadows and global illumination → You understand sampling in 3D spaces

Getting Started Today

  1. Set up: Choose a language (C is great for understanding, Python for speed of iteration)
  2. Watch: 3Blue1Brown Episode 1 - “Vectors, what even are they?”
  3. Build: Start the 3D Renderer—draw a single projected point on screen
  4. Iterate: Each day, add one transformation. See it work. Understand why.

The goal isn’t to rush through projects—it’s to reach the point where you see matrices as transformations, feel eigenvectors as natural axes, and think in terms of vector spaces. Building makes this happen in a way that reading never can.