Project 18: Logistic Regression Classifier

A binary classifier using logistic regression with gradient descent. Train on labeled data, learn the decision boundary, and visualize the sigmoid probability outputs.

Quick Reference

Attribute Value
Difficulty Level 3: Advanced (The Engineer)
Main Programming Language Python
Alternative Programming Languages C, Julia, Rust
Coolness Level Level 3: Genuinely Clever
Business Potential 2. The “Micro-SaaS / Pro Tool” (Solo-Preneur Potential)
Knowledge Area Classification / Optimization
Software or Tool Logistic Classifier
Main Book “Hands-On Machine Learning” by Aurélien Géron

1. Learning Objectives

By completing this project, you will:

  1. Translate math definitions into deterministic implementation steps.
  2. Build validation checks that make correctness observable.
  3. Diagnose numerical, logical, and data-shape failures early.
  4. Explain tradeoffs in interviews using evidence from your own build.

2. All Theory Needed (Per-Concept Breakdown)

This project applies the following theory clusters:

  • Symbolic-to-numeric translation (expressions, data shapes, invariants)
  • Stability constraints (precision, scaling, stopping criteria)
  • Optimization or inference logic (depending on project objective)
  • Evaluation discipline (error analysis, test coverage, reproducibility)

Concept A: Mathematical Representation Discipline

Fundamentals A math expression is not executable until you define representation, ordering, and domain constraints. The same equation can be represented as a token stream, tree, matrix pipeline, or probability graph. Choosing representation determines what bugs you can catch early.

Deep Dive into the concept Most project failures begin before algorithm selection: they start with ambiguous representation. If your parser cannot distinguish unary minus from subtraction, your calculator fails. If your matrix dimensions are implicit rather than validated, your linear algebra pipeline fails silently. If your probabilistic assumptions (independence, stationarity, or class priors) are not explicit, your inference can look accurate on one split and collapse on another. The core implementation move is to treat representation as a contract. Define each object with shape, domain, and semantic intent. Then enforce invariants at boundaries: input parser, preprocessing, training loop, evaluation stage. This makes debugging local instead of global.

How this fits this project You will encode each operation with explicit contracts and invariant checks.

Definitions & key terms

  • Invariant: Property that must hold before and after each operation.
  • Shape contract: Expected dimensional structure of vectors/matrices/tensors.
  • Domain constraint: Allowed value range (for example log input > 0).

Mental model diagram

User Input -> Representation Layer -> Validated Operation -> Observable Output
              (tokens/shapes)        (invariants pass)       (tests/plots/logs)

How it works

  1. Parse/ingest data into typed structures.
  2. Validate shape/domain invariants.
  3. Execute operation.
  4. Compare observed output with expected behavior.
  5. Record failure signature if mismatch appears.

Minimal concrete example

PSEUDOCODE
read expression
tokenize with precedence rules
if token sequence invalid -> return syntax error
evaluate tree
if domain violation -> return bounded diagnostic
print value and confidence check

Common misconceptions

  • “If it runs once, representation is correct.” -> false.
  • “Type checks are enough without shape checks.” -> false.

Check-your-understanding questions

  1. Which invariant catches division-by-zero earliest?
  2. Why does shape validation belong at boundaries rather than only in core logic?
  3. Predict failure if tokenization ignores unary minus.

Check-your-understanding answers

  1. Domain check on denominator before operation execution.
  2. Boundary validation keeps errors local and diagnostic.
  3. Expressions like -2^2 get misinterpreted and produce wrong precedence behavior.

Real-world applications Feature preprocessing, model-serving input validation, and experiment-tracking schema enforcement.

Where you’ll apply it This project and every downstream project in the sprint.

References

  • CSAPP (Bryant & O’Hallaron), floating-point chapter
  • Math for Programmers (Paul Orland), representation-oriented chapters

Key insight Correct representation reduces the complexity of every later decision.

Summary Stable ML math implementations start with explicit contracts, not implicit assumptions.

Homework/Exercises

  1. Write five invariants for your project.
  2. Build a failing test input for each invariant.

Solutions

  1. Include at least one shape, one domain, one convergence, one reproducibility, and one output-range invariant.
  2. Each failing input should trigger exactly one diagnostic to keep root-cause analysis clean.

3. Build Blueprint

  1. Scope the smallest end-to-end slice that produces visible output.
  2. Add deterministic tests and edge-case probes.
  3. Layer complexity only after baseline behavior is stable.
  4. Add metrics logging before optimization.
  5. Run failure drills: perturb inputs, scale values, and check stability.

4. Real-World Outcome (Target)

$ python logistic.py train iris_binary.csv

Training logistic regression on Iris dataset (setosa vs non-setosa)
Features: sepal_length, sepal_width
Samples: 150 (50 setosa, 100 non-setosa)

Training...
Epoch 100:  Loss = 0.423, Accuracy = 92%
Epoch 500:  Loss = 0.187, Accuracy = 97%
Epoch 1000: Loss = 0.124, Accuracy = 99%

Learned weights:
  w_sepal_length = -2.34
  w_sepal_width  =  4.12
  bias           = -1.56

Decision boundary: sepal_width = 0.57 * sepal_length + 0.38

[2D plot: points colored by class, linear decision boundary shown]
[Probability surface: darker = more confident]

$ python logistic.py predict "sepal_length=5.0, sepal_width=3.5"
P(setosa) = 0.94
Classification: setosa (high confidence)

Implementation Hints: Forward pass:

z = X @ w + b
prob = 1 / (1 + np.exp(-z))  # sigmoid

Cross-entropy loss:

loss = -np.mean(y * np.log(prob + 1e-10) + (1-y) * np.log(1-prob + 1e-10))

Gradient (beautifully simple!):

gradient_w = X.T @ (prob - y) / n_samples
gradient_b = np.mean(prob - y)

The gradient has the same form as linear regression—this is not a coincidence!

Learning milestones:

  1. Classifier achieves high accuracy → You understand logistic regression
  2. Decision boundary is correct → You understand linear separability
  3. Probability outputs are calibrated → You understand probabilistic classification

5. Core Design Notes from Main Guide

Core Question

“How do you turn a line into a decision?”

Linear regression predicts continuous values, but what if you need to predict yes or no, spam or not spam, cat or dog? You cannot just use a line because lines extend to infinity in both directions. The insight is to “squash” the linear output through a sigmoid function, transforming any real number into a probability between 0 and 1. This simple idea–applying a nonlinear transformation to a linear model–is the foundation of neural networks. By building logistic regression, you understand the key transition from regression to classification, from predicting “how much” to predicting “which one.”

Concepts You Must Understand First

Stop and research these before coding:

  1. The Sigmoid Function
    • What is the formula for sigmoid: 1 / (1 + exp(-z))?
    • Why does it squash all real numbers to (0, 1)?
    • What is the derivative of sigmoid? Why is it so elegant?
    • Book Reference: “Neural Networks and Deep Learning” Chapter 1 - Michael Nielsen
  2. Binary Cross-Entropy Loss
    • Why do we use -ylog(p) - (1-y)log(1-p) instead of squared error?
    • What happens to the loss when prediction is confident and wrong?
    • How does this relate to maximum likelihood estimation?
    • Book Reference: “Deep Learning” Chapter 3 - Goodfellow et al.
  3. Decision Boundaries
    • What is the equation of the decision boundary for logistic regression?
    • Why is the boundary always linear (a hyperplane)?
    • What does it mean for data to be “linearly separable”?
    • Book Reference: “Hands-On Machine Learning” Chapter 4 - Aurelien Geron
  4. Gradient of Cross-Entropy Loss
    • Why does the gradient simplify to (prediction - label) * input?
    • How is this similar to the gradient for linear regression?
    • What makes this mathematical coincidence significant?
    • Book Reference: “Machine Learning” (Coursera) Week 3 - Andrew Ng
  5. Regularization (L1 and L2)
    • What is the difference between L1 (lasso) and L2 (ridge) regularization?
    • Why does regularization prevent overfitting?
    • How does it affect the decision boundary?
    • Book Reference: “The Elements of Statistical Learning” Chapter 3 - Hastie et al.
  6. Probability Calibration
    • What does it mean for predicted probabilities to be “calibrated”?
    • How do you check if your model’s 80% predictions are actually correct 80% of the time?
    • Why is calibration important for real applications?
    • Book Reference: “Probabilistic Machine Learning” Chapter 5 - Kevin Murphy

Questions to Guide Your Design

Before implementing, think through these:

  1. Numerical stability: What happens when exp(-z) overflows? How do you handle very large or very small values of z?

  2. Learning rate selection: How do you choose an appropriate learning rate? What symptoms indicate it is too high or too low?

  3. Convergence criteria: When do you stop training? Fixed epochs? Loss threshold? Validation accuracy plateau?

  4. Handling imbalanced data: What if 95% of your data belongs to one class? How does this affect training?

  5. Multiclass extension: How would you extend binary logistic regression to handle more than two classes?

  6. Feature importance: After training, how can you interpret which features matter most for the classification?

Thinking Exercise

Work through sigmoid and its derivative by hand:

  1. Compute sigmoid for these values:
    • z = 0: sigmoid(0) = 1/(1+e^0) = 1/(1+1) = 0.5
    • z = 10: sigmoid(10) = 1/(1+e^(-10)) is approximately 0.99995
    • z = -10: sigmoid(-10) is approximately 0.00005
  2. Prove that the derivative of sigmoid is sigmoid(z) * (1 - sigmoid(z)):
    • Let s = 1/(1+e^(-z))
    • ds/dz = … (work through the calculus)
  3. Verify gradient descent update: Given one data point: x = [1, 2], y = 1 (positive class), current weights w = [0.1, 0.2]
    • z = w^T x = 0.11 + 0.22 = 0.5
    • p = sigmoid(0.5) is approximately 0.622
    • loss = -1log(0.622) - 0log(0.378) is approximately 0.475
    • gradient = (p - y) * x = (0.622 - 1) * [1, 2] = [-0.378, -0.756]
    • new_w = [0.1, 0.2] - 0.1 * [-0.378, -0.756] = [0.138, 0.276]

Notice how the weights moved in the direction that makes the prediction closer to 1!

Interview Questions

  1. “Explain the intuition behind logistic regression.”
    • Expected answer: It is linear regression followed by a sigmoid function. The linear part creates a weighted sum, sigmoid converts it to probability, and we minimize cross-entropy loss.
  2. “Why do we use cross-entropy loss instead of MSE for classification?”
    • Expected answer: Cross-entropy has stronger gradients when predictions are wrong (log(small number) is very negative). MSE gradients vanish near 0 and 1 due to sigmoid saturation.
  3. “What is the equation of the decision boundary for logistic regression?”
    • Expected answer: w^T x + b = 0, which is a hyperplane. Points above the hyperplane are classified as positive (sigmoid > 0.5).
  4. “How would you handle a dataset where one class has 100x more samples?”
    • Expected answer: Class weights, oversampling minority class (SMOTE), undersampling majority, or adjusting the decision threshold.
  5. “What happens if your features are linearly dependent?”
    • Expected answer: The model still trains, but weights are not unique (infinitely many solutions). Regularization helps by preferring smaller weights.
  6. “How do you interpret the coefficients of a logistic regression model?”
    • Expected answer: exp(w_i) is the odds ratio–how much the odds multiply when feature i increases by 1, holding other features constant.
  7. “When would logistic regression fail compared to more complex models?”
    • Expected answer: When the true decision boundary is nonlinear. Logistic regression can only draw straight lines (hyperplanes).

Hints in Layers (Treat as pseudocode guidance)

Hint 1: The sigmoid function is your foundation:

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

But beware: large negative z causes overflow. Use np.clip(z, -500, 500) for stability.

Hint 2: Cross-entropy loss with numerical stability:

def cross_entropy(y_true, y_pred):
    epsilon = 1e-15  # Prevent log(0)
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

Hint 3: The gradient is beautifully simple:

predictions = sigmoid(X @ w + b)
gradient_w = (1/n) * X.T @ (predictions - y)
gradient_b = np.mean(predictions - y)

Hint 4: For the decision boundary visualization (2D features):

# Decision boundary: w1*x1 + w2*x2 + b = 0
# Solve for x2: x2 = -(w1*x1 + b) / w2
x1_range = np.linspace(X[:, 0].min(), X[:, 0].max(), 100)
x2_boundary = -(w[0] * x1_range + b) / w[1]
plt.plot(x1_range, x2_boundary, 'k--', label='Decision Boundary')

Hint 5: Add L2 regularization:

# Regularization term: lambda * ||w||^2 / 2
# Add to loss: loss + lambda * np.sum(w**2) / 2
# Add to gradient: gradient_w + lambda * w

Books That Will Help

Topic Book Chapter
Logistic Regression Theory “Hands-On Machine Learning” by Aurelien Geron Chapter 4: Training Models
Cross-Entropy and Maximum Likelihood “Deep Learning” by Goodfellow et al. Chapter 3: Probability
Sigmoid and Activation Functions “Neural Networks and Deep Learning” by Michael Nielsen Chapter 1: Using Neural Nets
Regularization “The Elements of Statistical Learning” by Hastie et al. Chapter 3: Linear Methods
Gradient Descent for Classification “Machine Learning” (Coursera) by Andrew Ng Week 3: Logistic Regression
Probability Calibration “Probabilistic Machine Learning” by Kevin Murphy Chapter 5: Decision Theory

6. Validation, Pitfalls, and Completion

Common Pitfalls and Debugging

Problem 1: “Outputs drift after a few iterations”

  • Why: Hidden numerical instability (unscaled features, aggressive step size, or repeated subtraction of nearly equal values).
  • Fix: Normalize inputs, reduce step size, and track relative error rather than only absolute error.
  • Quick test: Run the same task with two scales of input (for example x and 10x) and compare normalized error curves.

Problem 2: “Results are inconsistent across runs”

  • Why: Random seeds, data split randomness, or non-deterministic ordering are uncontrolled.
  • Fix: Set seeds, log configuration, and store split indices and hyperparameters with each run.
  • Quick test: Re-run three times with the same seed and confirm metrics remain inside a tight tolerance band.

Problem 3: “The project works on the demo case but fails on edge cases”

  • Why: Tests only cover happy-path inputs.
  • Fix: Add adversarial inputs (empty values, extreme ranges, near-singular matrices, rare classes).
  • Quick test: Build an edge-case test matrix and ensure every scenario reports expected behavior.

Definition of Done

  • Core functionality works on reference inputs
  • Edge cases are tested and documented
  • Results are reproducible (seeded and versioned configuration)
  • Performance or convergence behavior is measured and explained
  • A short retrospective explains what failed first and how you fixed it

7. Extension Ideas

  1. Add a stress-test mode with adversarial inputs.
  2. Add a short benchmark report (runtime + memory + error trend).
  3. Add a reproducibility bundle (seed, config, and fixed test corpus).

8. Why This Project Matters

Not specified

This project is valuable because it creates observable evidence of mathematical reasoning under real implementation constraints.