Project 10: Property-Based Testing Framework

Project 10: Property-Based Testing Framework

Building QuickCheck from Scratch: Where Randomness Meets Rigor

  • Main Programming Language: Haskell
  • Alternative Languages: Scala (ScalaCheck), F# (FsCheck), Python (Hypothesis)
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Testing / Generators / Type Classes
  • Estimated Time: 2-3 weeks
  • Prerequisites: Projects 5 (Algebraic Data Types), Project 6 (Functor/Applicative/Monad), understanding of randomness

Learning Objectives

After completing this project, you will be able to:

  1. Implement the Gen monad from scratch - Build a monadic random value generator that composes elegantly using do-notation
  2. Design and implement the Arbitrary type class - Create a polymorphic interface for generating random values of any type
  3. Build shrinking algorithms - Implement strategies to find minimal counterexamples when properties fail
  4. Compose properties using combinators - Design higher-order functions that combine and modify properties
  5. Implement the core QuickCheck loop - Build the machinery that runs tests, catches failures, and reports results
  6. Apply property-based testing to real code - Identify invariants in your code and express them as testable properties
  7. Understand the relationship between types and generators - See how types guide the generation of test data

Conceptual Foundation

The Revolution in Testing

In 1999, Koen Claessen and John Hughes published “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs,” and it changed how we think about testing. Before QuickCheck, testing meant writing example-based tests:

-- Traditional unit tests
test1 = reverse [1,2,3] == [3,2,1]
test2 = reverse [] == []
test3 = reverse [1] == [1]

These tests check specific examples, but they can never cover all cases. QuickCheck introduced a fundamentally different idea: instead of testing specific examples, test properties that should hold for ALL inputs.

-- Property-based test
prop_reverse_reverse :: [Int] -> Bool
prop_reverse_reverse xs = reverse (reverse xs) == xs
-- This property is tested with HUNDREDS of random inputs!

This shift from “does my code produce the right output for these examples?” to “does my code satisfy this invariant for all inputs?” is profound. It’s the difference between testing instances and testing laws.

What Is a Property?

A property is a boolean predicate that should hold for all inputs of a given type. Properties capture invariants—things that must always be true about your code.

Consider the reverse function. What invariants should it satisfy?

  1. Involution: reverse (reverse xs) == xs
  2. Length preservation: length (reverse xs) == length xs
  3. First becomes last: head (reverse xs) == last xs (for non-empty lists)
  4. Concatenation reversal: reverse (xs ++ ys) == reverse ys ++ reverse xs

Each of these is a property. If reverse is implemented correctly, all these properties should hold for ANY list. QuickCheck tests this by generating hundreds of random lists and checking each property.

The Power of Random Testing

Why is random testing so effective? Consider testing a sorting function:

sort :: Ord a => [a] -> [a]

Properties of a correct sort:

  1. Output is sorted: isSorted (sort xs)
  2. Output is a permutation: sort xs \isPermutationOf` xs`
  3. Idempotent: sort (sort xs) == sort xs
  4. Length preserved: length (sort xs) == length xs

If you write example-based tests, you might test sort [3,1,2], sort [], sort [1], etc. But what about:

  • Lists with duplicates?
  • Lists that are already sorted?
  • Lists sorted in reverse?
  • Very long lists?
  • Lists with negative numbers?

Random testing automatically explores this space. The generator produces all sorts of weird edge cases you might not think of. This is why property-based tests often find bugs that example-based tests miss.

The Gen Monad: Generating Random Values

At the heart of QuickCheck is the Gen monad—a way to describe random value generation compositionally. Let’s understand why it needs to be a monad.

The Problem with Naive Random Generation

Imagine you want to generate a random person:

data Person = Person { name :: String, age :: Int }

With imperative randomness:

# Imperative approach
import random

def random_person():
    name = random_string()  # Uses global random state
    age = random.randint(0, 100)  # Uses global random state
    return Person(name, age)

This works but has problems:

  1. Hidden state: The random state is global and mutable
  2. Not reproducible: Can’t replay a specific test run
  3. Not composable: Hard to combine generators in complex ways

The Monadic Solution

Haskell’s Gen monad makes randomness explicit and composable:

newtype Gen a = Gen { runGen :: StdGen -> (a, StdGen) }

A Gen a is a stateful computation that takes a random generator and produces a value a along with a new generator state. This is exactly the State monad pattern!

Why is this beautiful?

  1. Explicit threading: The random state is threaded through explicitly
  2. Pure: Given the same seed, you get the same sequence of values
  3. Composable: Monadic bind (>>=) automatically threads the state
instance Monad Gen where
  return x = Gen $ \g -> (x, g)

  (Gen f) >>= k = Gen $ \g ->
    let (a, g') = f g        -- Run first generator
        Gen h = k a           -- Get second generator from result
    in h g'                   -- Run second generator with new state

Now generating a Person becomes:

genPerson :: Gen Person
genPerson = do
  name <- genString
  age  <- choose (0, 100)
  return $ Person name age

The do notation hides the state threading. Each generator consumes some randomness and passes the rest along.

Combinators: Building Complex Generators from Simple Ones

The real power of Gen comes from combinators—functions that build complex generators from simple ones.

Basic Combinators

-- Choose from a range
choose :: Random a => (a, a) -> Gen a
choose (lo, hi) = Gen $ \g -> randomR (lo, hi) g

-- Pick from a list
elements :: [a] -> Gen a
elements xs = do
  i <- choose (0, length xs - 1)
  return (xs !! i)

-- One of several generators
oneof :: [Gen a] -> Gen a
oneof gens = do
  i <- choose (0, length gens - 1)
  gens !! i

-- Weighted choice
frequency :: [(Int, Gen a)] -> Gen a
frequency xs = do
  let total = sum (map fst xs)
  n <- choose (1, total)
  pick n xs
  where
    pick n ((k,g):rest)
      | n <= k    = g
      | otherwise = pick (n-k) rest

Structural Recursion for Recursive Types

Generating recursive data types (like lists or trees) requires careful handling to ensure termination:

-- Generate a list (with bounded recursion)
genList :: Gen a -> Gen [a]
genList genA = do
  len <- choose (0, 30)  -- Bound the length!
  replicateM len genA

-- Generate a tree (using sized to control depth)
genTree :: Gen a -> Gen (Tree a)
genTree genA = sized $ \n ->
  if n <= 0
    then Leaf <$> genA
    else oneof
      [ Leaf <$> genA
      , Branch <$> resize (n `div` 2) (genTree genA)
               <*> resize (n `div` 2) (genTree genA)
      ]

The sized combinator passes a “size” parameter that decreases with recursion depth, ensuring termination.

The Arbitrary Type Class: Polymorphic Generation

Different types need different generators. The Arbitrary type class provides a uniform interface:

class Arbitrary a where
  arbitrary :: Gen a
  shrink :: a -> [a]
  shrink _ = []  -- Default: no shrinking

Any type with an Arbitrary instance can be randomly generated. This is type class polymorphism in action—the same interface works for any type:

instance Arbitrary Int where
  arbitrary = choose (-1000, 1000)
  shrink 0 = []
  shrink n = [0, n `div` 2]  -- Shrink toward 0

instance Arbitrary Bool where
  arbitrary = elements [True, False]
  shrink True = [False]
  shrink False = []

instance Arbitrary a => Arbitrary [a] where
  arbitrary = do
    len <- choose (0, 30)
    replicateM len arbitrary
  shrink [] = []
  shrink (x:xs) =
    [] :                           -- Remove all elements
    xs :                           -- Remove first element
    [x':xs | x' <- shrink x] ++    -- Shrink first element
    [x:xs' | xs' <- shrink xs]     -- Shrink rest

Notice how Arbitrary [a] requires Arbitrary a—this is the power of type class constraints propagating through instances.

Shrinking: Finding Minimal Counterexamples

When a property fails, the failing input might be huge and hard to understand. Shrinking finds a minimal counterexample.

Example: Suppose testing prop_bad_sort fails on [7, -3, 12, 4, -8, 1, 9]. That’s a lot to debug! But after shrinking, we might find the minimal failure is [1, 0]—much easier to understand.

How Shrinking Works

The shrink function returns a list of “smaller” values:

shrink :: a -> [a]

For example:

  • shrink 10 might return [0, 5, 8, 9] (smaller integers)
  • shrink [1,2,3] might return [[], [2,3], [1,3], [1,2], ...] (shorter lists or smaller elements)

The shrinking algorithm:

shrinkLoop :: (a -> Bool) -> a -> a
shrinkLoop prop value =
  case filter (not . prop) (shrink value) of
    []        -> value  -- Can't shrink further; this is minimal
    (smaller:_) -> shrinkLoop prop smaller  -- Found a smaller failure!

This is a greedy search: keep finding smaller failing inputs until none can be found. The result is a local minimum—the smallest failure in the neighborhood defined by shrink.

Designing Good Shrink Functions

A good shrink function should:

  1. Move toward “simpler” values: Smaller numbers, shorter lists, simpler structures
  2. Preserve type invariants: Don’t generate invalid values
  3. Make progress: Eventually terminate (shrink values should be strictly “smaller”)
  4. Be thorough: Explore multiple shrinking directions

For lists, shrinking can:

  • Remove elements (makes it shorter)
  • Shrink individual elements (makes elements simpler)
shrink (x:xs) =
  -- Remove the first element
  xs :
  -- Remove the first element completely
  [] :
  -- Keep first, shrink rest
  [x:xs' | xs' <- shrink xs] ++
  -- Shrink first, keep rest
  [x':xs | x' <- shrink x]

The Property-Checking Loop

The core loop that ties everything together:

quickCheck :: Testable prop => prop -> IO ()
quickCheck prop = do
  seed <- newStdGen
  let result = runTests 100 seed
  case result of
    Success n -> putStrLn $ "OK, passed " ++ show n ++ " tests."
    Failure input -> do
      putStrLn $ "FAILED: " ++ show input
      putStrLn $ "Shrinking..."
      let minimal = shrinkLoop prop input
      putStrLn $ "Minimal counterexample: " ++ show minimal

The Testable class abstracts over things that can be tested:

class Testable prop where
  property :: prop -> Property

instance Testable Bool where
  property b = Result b

instance (Arbitrary a, Show a, Testable prop) => Testable (a -> prop) where
  property f = forAll arbitrary $ \x -> property (f x)

This instance for functions is the key insight: a function a -> Bool is testable by generating random a values!

Why Property-Based Testing Matters

Property-based testing changes how you think about correctness:

  1. Forces you to articulate invariants: What should always be true?
  2. Tests edge cases automatically: Random generation explores the space
  3. Finds subtle bugs: Counterexamples you’d never think to test
  4. Documents behavior: Properties serve as executable specifications
  5. Compositional: Combine generators and properties elegantly

Real bugs found by QuickCheck:

  • Concurrency bugs in Erlang/OTP (John Hughes’ famous work)
  • Protocol conformance issues
  • Off-by-one errors in array algorithms
  • Unicode handling bugs

The Mathematical Foundation

Property-based testing connects to several deep ideas:

Free Theorems: In polymorphic code, types impose constraints that lead to properties. For example, any function f :: [a] -> [a] must satisfy:

map g . f = f . map g

This is a “free theorem” derivable purely from the type!

Algebraic Laws: Properties often express algebraic laws:

  • Monoid: x <> mempty = x, mempty <> x = x, (x <> y) <> z = x <> (y <> z)
  • Functor: fmap id = id, fmap (f . g) = fmap f . fmap g

QuickCheck tests these laws for specific instances.

Specification as Code: Properties are executable specifications. Instead of writing informal documentation (“reverse should undo itself”), you write code that is the specification.

Historical Context

QuickCheck emerged from the functional programming community’s focus on equational reasoning. In Haskell, we reason about code by substituting equals for equals:

-- If we know:
reverse (reverse xs) = xs

-- Then we can transform:
reverse (reverse (reverse ys))
= reverse ys  -- by the law

QuickCheck lets us test these laws empirically. It bridges the gap between formal proofs and practical testing.

The idea spread rapidly:

  • ScalaCheck (2007): QuickCheck for Scala
  • FsCheck (2008): For F#/.NET
  • Hypothesis (2015): For Python, adding advanced shrinking
  • PropEr (2011): For Erlang
  • fast-check (2018): For JavaScript/TypeScript

Every major language now has a property-based testing library, all descended from the original Haskell QuickCheck.


Project Specification

You will build a property-based testing framework from scratch. Your framework should support:

Core Requirements

  1. Gen Monad
    • Random value generation with composable generators
    • Combinators: choose, elements, oneof, frequency, listOf, vectorOf
    • Size control with sized and resize
    • Monad instance for do-notation support
  2. Arbitrary Type Class
    • Instances for: Bool, Int, Integer, Char, [a], Maybe a, Either a b, (a, b)
    • Shrink function for each type
    • Derived generators using type class constraints
  3. Property Testing
    • quickCheck function that runs tests
    • Configurable number of tests
    • Seed control for reproducibility
    • Failure reporting with the failing input
  4. Shrinking
    • Automatic shrinking of failing inputs
    • Report the minimal counterexample
    • Track shrinking steps
  5. Property Combinators
    • (==>) for conditional properties (implications)
    • forAll for explicit generator specification
    • classify for input classification/statistics
    • label for labeling test cases

Stretch Goals

  • Coverage checking: Ensure properties are tested with diverse inputs
  • Parallel testing: Run tests in parallel for speed
  • Stateful testing: Test stateful systems with sequences of commands
  • Custom shrinking strategies: Allow users to define domain-specific shrinking

Solution Architecture

Module Structure

src/
  QuickCheck/
    Gen.hs        -- Gen monad and combinators
    Arbitrary.hs  -- Arbitrary class and instances
    Property.hs   -- Property type and combinators
    Test.hs       -- Testing loop and runners
    Shrink.hs     -- Shrinking algorithms
    Random.hs     -- Random number utilities
  QuickCheck.hs   -- Re-exports public API

Core Data Types

-- The generator monad
newtype Gen a = Gen { unGen :: StdGen -> Int -> (a, StdGen) }
-- The Int is the "size" parameter for recursive generators

-- The arbitrary class
class Arbitrary a where
  arbitrary :: Gen a
  shrink :: a -> [a]
  shrink _ = []

-- Property result
data Result = Success | Failure String | Gave Up String

-- Property wrapper
newtype Property = MkProperty { unProperty :: Gen Result }

-- Test configuration
data Config = Config
  { numTests :: Int
  , maxDiscards :: Int
  , seed :: Maybe Int
  }

Key Design Decisions

  1. Gen includes size: The size parameter allows recursive generators to terminate
  2. Result includes messages: For better error reporting
  3. Property is Gen Result: Properties are themselves generators of results
  4. Shrink returns list: Multiple shrink candidates allow thorough searching

Implementation Guide

Phase 1: The Gen Monad (Days 1-2)

Goal: Implement the generator monad with basic combinators.

Milestone 1.1: Basic Gen type and Monad instance

newtype Gen a = Gen { runGen :: StdGen -> (a, StdGen) }

instance Functor Gen where
  fmap f (Gen g) = Gen $ \s -> let (a, s') = g s in (f a, s')

instance Applicative Gen where
  pure a = Gen $ \s -> (a, s)
  Gen f <*> Gen a = Gen $ \s ->
    let (fn, s')  = f s
        (x,  s'') = a s'
    in (fn x, s'')

instance Monad Gen where
  Gen m >>= k = Gen $ \s ->
    let (a, s') = m s
    in runGen (k a) s'

Milestone 1.2: Basic combinators

choose :: Random a => (a, a) -> Gen a
elements :: [a] -> Gen a
oneof :: [Gen a] -> Gen a
frequency :: [(Int, Gen a)] -> Gen a

Milestone 1.3: Size-aware generation

sized :: (Int -> Gen a) -> Gen a
resize :: Int -> Gen a -> Gen a

Test: Generate random integers and lists, verify distribution.

Phase 2: The Arbitrary Class (Days 3-4)

Goal: Implement the type class and instances for common types.

Milestone 2.1: Class definition and primitive instances

class Arbitrary a where
  arbitrary :: Gen a
  shrink :: a -> [a]
  shrink _ = []

instance Arbitrary Bool where ...
instance Arbitrary Int where ...
instance Arbitrary Char where ...

Milestone 2.2: Compound type instances

instance Arbitrary a => Arbitrary [a] where ...
instance Arbitrary a => Arbitrary (Maybe a) where ...
instance (Arbitrary a, Arbitrary b) => Arbitrary (Either a b) where ...
instance (Arbitrary a, Arbitrary b) => Arbitrary (a, b) where ...

Milestone 2.3: Shrinking for all types

shrink :: Int -> [Int]  -- Shrink toward 0
shrink :: [a] -> [[a]]  -- Remove elements, shrink elements

Test: Generate values, verify they’re in expected ranges, test shrinking produces smaller values.

Phase 3: Properties and Testing (Days 5-7)

Goal: Build the property checking machinery.

Milestone 3.1: Property type

data Result = Success | Failure String | GaveUp
newtype Property = Property { unProperty :: Gen Result }

Milestone 3.2: The Testable class

class Testable a where
  property :: a -> Property

instance Testable Bool where ...
instance Testable Result where ...
instance (Arbitrary a, Show a, Testable b) => Testable (a -> b) where ...

Milestone 3.3: The quickCheck function

quickCheck :: Testable prop => prop -> IO ()

Test: Run simple properties, verify they pass/fail correctly.

Phase 4: Shrinking Loop (Days 8-10)

Goal: Implement automatic shrinking of counterexamples.

Milestone 4.1: Shrinking algorithm

shrinkLoop :: (a -> Bool) -> a -> a
-- Find minimal failing input

Milestone 4.2: Integration with quickCheck

  • When a test fails, automatically shrink
  • Report shrinking progress
  • Show minimal counterexample

Milestone 4.3: Shrink tracing

  • Count number of shrink steps
  • Show intermediate values (optional verbose mode)

Test: Verify that shrinking finds known minimal counterexamples.

Phase 5: Advanced Combinators (Days 11-14)

Goal: Add property combinators for expressiveness.

Milestone 5.1: Implication

(==>) :: Testable prop => Bool -> prop -> Property
-- Only test when precondition holds

Milestone 5.2: forAll

forAll :: (Show a, Testable prop) => Gen a -> (a -> prop) -> Property
-- Use custom generator

Milestone 5.3: Classification

classify :: Testable prop => Bool -> String -> prop -> Property
label :: Testable prop => String -> prop -> Property
collect :: (Show a, Testable prop) => a -> prop -> Property
-- For gathering statistics on test inputs

Test: Use combinators in real properties, verify statistics collection.


Testing Strategy

Unit Tests for Gen

-- Distribution tests
test_choose_bounds = do
  let values = generate 1000 $ choose (0, 10)
  assert $ all (\x -> x >= 0 && x <= 10) values

test_elements_coverage = do
  let values = generate 1000 $ elements ['a'..'z']
  assert $ all (`elem` ['a'..'z']) values

Property Tests for Your Framework

Use your framework to test itself! (Meta-property testing)

-- Shrinking properties
prop_shrink_smaller :: Int -> Bool
prop_shrink_smaller n = all (< abs n) (map abs $ shrink n) || n == 0

-- Generator distribution
prop_choose_uniform :: Int -> Int -> Property
prop_choose_uniform lo hi = lo < hi ==>
  let gen = choose (lo, hi)
  in forAll gen $ \x -> x >= lo && x <= hi

Known Failure Cases

Create functions with known bugs and verify your framework catches them:

-- Bug: fails to handle negative numbers
badSort :: [Int] -> [Int]
badSort = sort . map abs  -- Oops! Loses negative info

prop_sort_preserves :: [Int] -> Bool
prop_sort_preserves xs = sort xs `isPermutationOf` xs
-- Should fail with minimal case like [-1]

Common Pitfalls

1. Non-Terminating Generators

Problem: Recursive generators that don’t decrease size:

-- WRONG: Infinite recursion
genTree :: Gen Tree
genTree = oneof [Leaf <$> arbitrary, Branch <$> genTree <*> genTree]

Solution: Use sized to bound recursion:

genTree :: Gen Tree
genTree = sized go
  where
    go 0 = Leaf <$> arbitrary
    go n = oneof
      [ Leaf <$> arbitrary
      , Branch <$> go (n `div` 2) <*> go (n `div` 2)
      ]

2. Biased Generators

Problem: Generators that favor certain values:

-- WRONG: Empty lists almost never generated
genList :: Arbitrary a => Gen [a]
genList = do
  x <- arbitrary
  xs <- genList
  return (x:xs)  -- Always adds at least one element!

Solution: Include base cases with appropriate frequency:

genList = frequency [(1, return []), (4, (:) <$> arbitrary <*> genList)]

3. Shrinking Creates Invalid Values

Problem: Shrink produces values that violate invariants:

-- Positive integers represented as regular Int
newtype Positive = Positive Int

-- WRONG: Can shrink to 0 or negative!
shrink (Positive n) = map Positive (shrink n)

Solution: Filter invalid shrinks:

shrink (Positive n) = [Positive m | m <- shrink n, m > 0]

4. Weak Properties

Problem: Properties that don’t actually test anything:

-- WRONG: Always true, tests nothing useful
prop_sort_is_list :: [Int] -> Bool
prop_sort_is_list xs = length (sort xs) >= 0

Solution: Write properties that would fail on buggy implementations:

-- BETTER: Actually tests sorting behavior
prop_sort_ordered xs = isSorted (sort xs)
prop_sort_permutation xs = sort xs `isPermutationOf` xs

5. State Threading Bugs

Problem: Forgetting to thread the random generator state:

-- WRONG: Reuses same generator state
genPair :: Gen (Int, Int)
genPair = Gen $ \g ->
  let (a, _) = runGen arbitrary g  -- Ignores new state!
      (b, g') = runGen arbitrary g -- Same g, same value!
  in ((a, b), g')

Solution: Thread state properly (or use monadic bind):

genPair = do
  a <- arbitrary
  b <- arbitrary
  return (a, b)

Extensions and Challenges

1. Stateful Testing

Test stateful systems by generating sequences of commands:

data Command = Push Int | Pop | Peek
type Model = [Int]

prop_stack_model :: [Command] -> Bool
prop_stack_model cmds = runModel [] cmds == runReal newStack cmds

2. Coverage-Guided Generation

Track which branches are tested and guide generation accordingly.

3. Parallel Test Execution

Run independent tests in parallel for speed.

4. Golden Master Testing

Combine with snapshot testing: generate inputs, save expected outputs.

5. Mutation Testing Integration

Generate properties that detect code mutations.


Real-World Connections

Where Property-Based Testing Appears

  1. Cryptocurrency: Formal verification of smart contracts
  2. Distributed Systems: Testing consensus protocols (Jepsen-style)
  3. Compilers: Testing optimizations preserve semantics
  4. Databases: Testing query planners and executors
  5. Serialization: Testing encode/decode roundtrips

Industry Adoption

  • Quviq AB: John Hughes’ company uses QuickCheck for industrial testing
  • Dropbox: Tests file sync with property-based testing
  • Amazon: Uses property-based testing for S3 and other services
  • Basho: Used QuickCheck extensively for Riak database

How This Connects to Other Concepts

  • Formal Methods: Properties are like lightweight specifications
  • Fuzzing: Random generation is related to fuzz testing
  • Model Checking: Stateful testing explores state spaces
  • Dependent Types: Types that guarantee properties at compile time

Interview Questions

Prepare to answer these after completing the project:

  1. “What is property-based testing and how does it differ from example-based testing?”
    • Answer: Property-based testing tests invariants that should hold for all inputs, while example-based testing checks specific input/output pairs. Property-based testing uses random generation to explore the input space automatically.
  2. “Explain how the Gen monad works and why it needs to be a monad.”
    • Answer: Gen wraps a function from random seed to (value, new seed). It’s a monad because we need to thread random state through composed generators. Monadic bind automatically handles state threading.
  3. “What is shrinking and why is it important?”
    • Answer: Shrinking finds minimal counterexamples when tests fail. It repeatedly tries “smaller” inputs until it finds one that still fails but has no smaller failing version. This makes debugging much easier.
  4. “How would you test a sorting function with properties?”
    • Answer: Key properties include: output is sorted, output is a permutation of input, length is preserved, idempotence (sort . sort = sort), and stability (for stable sorts).
  5. “What makes a good property? What makes a weak property?”
    • Answer: Good properties would fail on buggy implementations and capture essential invariants. Weak properties are always true or rarely fail. Example: “length xs >= 0” is weak; “reverse (reverse xs) == xs” is strong.
  6. “How do you handle recursive data types in generation?”
    • Answer: Use the sized combinator to pass a decreasing size parameter. At size 0, generate base cases only. This ensures termination while still generating deep structures for larger sizes.
  7. “How does the Arbitrary type class enable polymorphic testing?”
    • Answer: Arbitrary provides a uniform interface for generating any type. A property like prop :: [a] -> Bool is automatically testable for any a with an Arbitrary instance, thanks to the instance for functions.

Self-Assessment Checklist

You’ve mastered this project when you can:

  • Implement the Gen monad from scratch with correct state threading
  • Explain why Gen is a monad and what each typeclass law means
  • Write Arbitrary instances for compound types using constraints
  • Design shrink functions that make progress and preserve invariants
  • Implement the main testing loop with failure detection
  • Add shrinking that finds minimal counterexamples
  • Write properties that would catch real bugs
  • Use combinators like ==>, forAll, and classify
  • Generate recursive data without infinite loops
  • Explain the tradeoffs of random testing vs. exhaustive testing
  • Debug your framework when tests don’t behave as expected
  • Extend the framework with new features (coverage, parallelism)

Resources

Primary References

Topic Resource Specific Section
Original QuickCheck paper “QuickCheck: A Lightweight Tool for Random Testing” by Claessen & Hughes Full paper
QuickCheck implementation “Real World Haskell” by O’Sullivan, Stewart, Goerzen Chapter 11
Monads and state “Haskell Programming from First Principles” Chapter 18-23
Type classes “Haskell Programming from First Principles” Chapter 6
Random generation “Learn You a Haskell for Great Good!” Chapter 9

Advanced Reading

Topic Resource
State machine testing “Testing the Hard Stuff and Staying Sane” by John Hughes (talk)
Shrinking strategies “Integrated vs Type-Based Shrinking” by Hughes
Hypothesis design Hypothesis documentation on shrinking
Industrial applications “QuickCheck: A Lightweight Tool” case studies

Online Resources

  • QuickCheck source code on Hackage
  • Hypothesis (Python) documentation for modern shrinking ideas
  • PropEr (Erlang) for stateful testing patterns
  • John Hughes’ talks on YouTube

“Don’t write tests. Write specifications and let the computer generate tests.” - The philosophy behind property-based testing.

After building this framework, you’ll never look at testing the same way again.