Project 14: A/B Testing and Uncertainty Dashboard

A self-contained deep-dive from the canonical Math from Foundations to Machine Learning curriculum.

  • Difficulty: Intermediate
  • Time: 1-2 weeks
  • Language: Python (alternatives: R, JavaScript, Julia, Go)
  • Prerequisites: Project 12
  • Merged from: ML-Math P15; ML-Found P11

What You Will Build

Build a reproducible analyzer for two-variant conversion experiments. Accept visitor and conversion counts or row-level Bernoulli data, validate assignment and missingness assumptions, compute rates, absolute/relative lift, pooled standard error, a two-sided proportion z-test, p-value, and confidence interval for the rate difference. Add power and minimum-detectable-effect calculations, warnings for small samples or peeking, and a decision panel that separates statistical significance from practical significance. A simulation mode should reveal false positives, low power, and interval coverage across repeated experiments.

Real World Outcome

Given control 203/4532 and treatment 248/4621, the dashboard shows rates, lift, test statistic, p-value, and a confidence interval, then states what the result does and does not justify. Error-bar and sampling-distribution plots show uncertainty. The recommendation incorporates a user-provided minimum worthwhile effect, sample-size adequacy, and data-quality warnings rather than reducing the decision to p < .05.

Core Question

When observed conversion rates differ, how much evidence is there for a real effect, how large might it be, and is it useful enough to act on?

Concepts You Must Understand First

  1. Sampling variability and Bernoulli/binomial models: observed proportions fluctuate even with no effect.
  2. Null/alternative hypotheses and z statistics: standardize a difference under the null model.
  3. P-values: probability of equally or more extreme data under the null, not probability the null is true. See Reinhart, Statistics Done Wrong, Chapter 1.
  4. Confidence intervals and effect size: quantify plausible magnitude, not only threshold crossing. See James et al., An Introduction to Statistical Learning, Chapter 2.
  5. Power and experimental validity: randomization, independence, predeclared analysis, and adequate sample size. See Wheelan, Naked Statistics, Chapter 10.

Build Milestones

  1. Validate counts/data and compute conversion rates, absolute lift, relative lift, and standard errors.
  2. Implement a two-proportion z-test and confidence interval, checked against trusted software.
  3. Add practical-effect thresholds, power, sample-size, and minimum-detectable-effect calculations.
  4. Build plots and warnings for imbalance, low expected counts, repeated peeking, and missing data.
  5. Simulate null and alternative experiments to verify Type I error, power, and interval coverage.

Hints in Layers

  1. Use the pooled proportion for the null-test variance but an unpooled variance for the ordinary difference interval; document the choice.
  2. Create exact boundary fixtures: identical rates, zero conversions, total conversion, and tiny samples.
  3. Keep “statistically detectable” and “worth shipping” as independent dashboard fields.

Common Pitfalls and Debugging

  • Symptom: p-value is reported as the chance the treatment is ineffective. Cause: null-conditional probability was reversed. Fix: include a fixed plain-language interpretation template.
  • Symptom: relative lift explodes. Cause: control rate is near zero. Fix: always report absolute difference and flag unstable relative measures.
  • Symptom: simulation false positives exceed the selected alpha. Cause: repeated looks or incorrect standard error. Fix: freeze one analysis point and compare formulas with a reference package.

Definition of Done

  • Input validation catches impossible counts and documents analysis assumptions.
  • Test statistics, p-values, and intervals match trusted reference cases.
  • Reports include absolute/relative effect, uncertainty, power, and practical threshold.
  • Visuals and language avoid common p-value and confidence-interval misinterpretations.
  • Simulations demonstrate Type I error, power, and coverage behavior.
  • Decision output includes warnings rather than an unconditional ship/no-ship command.

Previous: Project 13 · Complete learning path · Next: Project 15