Project 8: Monorepo Task Runner — Build a Mini Turborepo

A monorepo task runner (like a mini Turborepo or Nx) that detects which packages changed, runs only affected tests, and caches results to avoid redundant work.

Quick Reference

Attribute	Value
Difficulty	Expert
Time Estimate	1 month+
Main Programming Language	Rust
Alternative Programming Languages	Go, TypeScript, Python
Coolness Level	Level 5: Pure Magic
Business Potential	4. The “Open Core” Infrastructure
Prerequisites	Projects 1-7 completed, graph algorithms, understanding of build systems
Key Topics	Package Dependency Graphs, Affected Detection, Task Caching

1. Learning Objectives

By completing this project, you will:

Implement a working version of: A monorepo task runner (like a mini Turborepo or Nx) that detects which packages changed, runs only affected tests, and caches results to avoid redundant work..
Explain the core Git workflow tradeoff this project is designed to surface.
Design deterministic checks so results can be verified and reproduced.
Document operational failure modes and safe recovery actions.

2. All Theory Needed (Per-Concept Breakdown)

Package Dependency Graphs

Fundamentals This concept matters in this project because your implementation will fail or become non-deterministic without a precise model of Package Dependency Graphs. You should define what the concept controls, what invariants must hold, and which actions are safe versus destructive. Treat this concept as a production concern, not a tutorial checkbox.

Deep Dive into the concept When applying Package Dependency Graphs in this project, reason in three passes: data shape, state transitions, and enforcement. First, identify which artifacts are authoritative (commit objects, refs, metadata, policy config, CI status, or scan findings). Second, map how those artifacts change when your tool runs. Third, define failure behavior explicitly. In Git tooling, silent partial success is dangerous: you need either complete success with evidence or an explicit failure state with remediation guidance. Also account for scale behavior. A workflow that works on a toy repo may fail on large history depth, concurrent updates, or mixed branch policies. Include trace logs for every irreversible action, and separate simulation mode from write mode. For interview readiness, be able to explain how this concept protects delivery speed while reducing operational risk.

How this fit on projects In this project, Package Dependency Graphs is directly used in design decisions, implementation constraints, and verification criteria.

Definitions & key terms

Package Dependency Graphs invariant: A condition that must remain true before and after every operation.
Safety boundary: The point where actions become destructive unless guarded.
Verification signal: Evidence proving the action behaved as expected.

Mental model diagram

Input state -> Validate invariant -> Apply change -> Verify output -> Record evidence

How it works

Capture current state and constraints.
Evaluate whether Package Dependency Graphs preconditions are satisfied.
Execute the minimal safe transition.
Verify postconditions and publish an auditable result.

Failure modes: stale state, partial writes, race conditions, ambiguous output contracts.

Minimal concrete example

Plan -> dry-run -> execute -> verify -> rollback/forward-fix decision

Common misconceptions

Assuming local success implies team-safe behavior.
Treating policy violations as warnings instead of merge blockers.
Skipping deterministic verification because the output appears correct.

Check-your-understanding questions

Which invariant is most likely to break first under concurrency?
What output proves your tool handled an edge case correctly?
Where should enforcement happen: local hook, CI, or protected branch gate?

Check-your-understanding answers

The invariant tied to mutable refs or policy-dependent merge eligibility.
A deterministic transcript showing both success and controlled failure behavior.
Layered enforcement: fast local checks plus non-bypassable server-side gates.

Real-world applications

Change-management tooling for fast-moving teams.
Incident-safe release workflows with traceable rollback paths.
Compliance-ready source-control automation.

Where you’ll apply it This project and its immediate adjacent projects in this sprint.

References

https://git-scm.com/docs
https://dora.dev/capabilities/trunk-based-development/

Key insights Package Dependency Graphs is only valuable when its invariants are encoded into tooling and checks.

Summary Mastering Package Dependency Graphs here gives you transferable patterns for larger workflow systems.

Homework/Exercises to practice the concept

Write one failing scenario and expected detection output.
Define one invariant and one explicit violation test.

Solutions to the homework/exercises

Use a stale branch or invalid metadata case and assert deterministic error reporting.
Invariant: protected branch must not accept unchecked changes; violation test: bypass attempt should fail fast.

Affected Detection

Fundamentals This concept matters in this project because your implementation will fail or become non-deterministic without a precise model of Affected Detection. You should define what the concept controls, what invariants must hold, and which actions are safe versus destructive. Treat this concept as a production concern, not a tutorial checkbox.

Deep Dive into the concept When applying Affected Detection in this project, reason in three passes: data shape, state transitions, and enforcement. First, identify which artifacts are authoritative (commit objects, refs, metadata, policy config, CI status, or scan findings). Second, map how those artifacts change when your tool runs. Third, define failure behavior explicitly. In Git tooling, silent partial success is dangerous: you need either complete success with evidence or an explicit failure state with remediation guidance. Also account for scale behavior. A workflow that works on a toy repo may fail on large history depth, concurrent updates, or mixed branch policies. Include trace logs for every irreversible action, and separate simulation mode from write mode. For interview readiness, be able to explain how this concept protects delivery speed while reducing operational risk.

How this fit on projects In this project, Affected Detection is directly used in design decisions, implementation constraints, and verification criteria.

Definitions & key terms

Affected Detection invariant: A condition that must remain true before and after every operation.
Safety boundary: The point where actions become destructive unless guarded.
Verification signal: Evidence proving the action behaved as expected.

Mental model diagram

Input state -> Validate invariant -> Apply change -> Verify output -> Record evidence

How it works

Capture current state and constraints.
Evaluate whether Affected Detection preconditions are satisfied.
Execute the minimal safe transition.
Verify postconditions and publish an auditable result.

Failure modes: stale state, partial writes, race conditions, ambiguous output contracts.

Minimal concrete example

Plan -> dry-run -> execute -> verify -> rollback/forward-fix decision

Common misconceptions

Assuming local success implies team-safe behavior.
Treating policy violations as warnings instead of merge blockers.
Skipping deterministic verification because the output appears correct.

Check-your-understanding questions

Which invariant is most likely to break first under concurrency?
What output proves your tool handled an edge case correctly?
Where should enforcement happen: local hook, CI, or protected branch gate?

Check-your-understanding answers

The invariant tied to mutable refs or policy-dependent merge eligibility.
A deterministic transcript showing both success and controlled failure behavior.
Layered enforcement: fast local checks plus non-bypassable server-side gates.

Real-world applications

Change-management tooling for fast-moving teams.
Incident-safe release workflows with traceable rollback paths.
Compliance-ready source-control automation.

Where you’ll apply it This project and its immediate adjacent projects in this sprint.

References

https://git-scm.com/docs
https://dora.dev/capabilities/trunk-based-development/

Key insights Affected Detection is only valuable when its invariants are encoded into tooling and checks.

Summary Mastering Affected Detection here gives you transferable patterns for larger workflow systems.

Homework/Exercises to practice the concept

Write one failing scenario and expected detection output.
Define one invariant and one explicit violation test.

Solutions to the homework/exercises

Use a stale branch or invalid metadata case and assert deterministic error reporting.
Invariant: protected branch must not accept unchecked changes; violation test: bypass attempt should fail fast.

Task Caching

Fundamentals This concept matters in this project because your implementation will fail or become non-deterministic without a precise model of Task Caching. You should define what the concept controls, what invariants must hold, and which actions are safe versus destructive. Treat this concept as a production concern, not a tutorial checkbox.

Deep Dive into the concept When applying Task Caching in this project, reason in three passes: data shape, state transitions, and enforcement. First, identify which artifacts are authoritative (commit objects, refs, metadata, policy config, CI status, or scan findings). Second, map how those artifacts change when your tool runs. Third, define failure behavior explicitly. In Git tooling, silent partial success is dangerous: you need either complete success with evidence or an explicit failure state with remediation guidance. Also account for scale behavior. A workflow that works on a toy repo may fail on large history depth, concurrent updates, or mixed branch policies. Include trace logs for every irreversible action, and separate simulation mode from write mode. For interview readiness, be able to explain how this concept protects delivery speed while reducing operational risk.

How this fit on projects In this project, Task Caching is directly used in design decisions, implementation constraints, and verification criteria.

Definitions & key terms

Task Caching invariant: A condition that must remain true before and after every operation.
Safety boundary: The point where actions become destructive unless guarded.
Verification signal: Evidence proving the action behaved as expected.

Mental model diagram

Input state -> Validate invariant -> Apply change -> Verify output -> Record evidence

How it works

Capture current state and constraints.
Evaluate whether Task Caching preconditions are satisfied.
Execute the minimal safe transition.
Verify postconditions and publish an auditable result.

Failure modes: stale state, partial writes, race conditions, ambiguous output contracts.

Minimal concrete example

Plan -> dry-run -> execute -> verify -> rollback/forward-fix decision

Common misconceptions

Assuming local success implies team-safe behavior.
Treating policy violations as warnings instead of merge blockers.
Skipping deterministic verification because the output appears correct.

Check-your-understanding questions

Which invariant is most likely to break first under concurrency?
What output proves your tool handled an edge case correctly?
Where should enforcement happen: local hook, CI, or protected branch gate?

Check-your-understanding answers

The invariant tied to mutable refs or policy-dependent merge eligibility.
A deterministic transcript showing both success and controlled failure behavior.
Layered enforcement: fast local checks plus non-bypassable server-side gates.

Real-world applications

Change-management tooling for fast-moving teams.
Incident-safe release workflows with traceable rollback paths.
Compliance-ready source-control automation.

Where you’ll apply it This project and its immediate adjacent projects in this sprint.

References

https://git-scm.com/docs
https://dora.dev/capabilities/trunk-based-development/

Key insights Task Caching is only valuable when its invariants are encoded into tooling and checks.

Summary Mastering Task Caching here gives you transferable patterns for larger workflow systems.

Homework/Exercises to practice the concept

Write one failing scenario and expected detection output.
Define one invariant and one explicit violation test.

Solutions to the homework/exercises

Use a stale branch or invalid metadata case and assert deterministic error reporting.
Invariant: protected branch must not accept unchecked changes; violation test: bypass attempt should fail fast.

3. Project Specification

3.1 What You Will Build

A monorepo task runner (like a mini Turborepo or Nx) that detects which packages changed, runs only affected tests, and caches results to avoid redundant work.

3.2 Functional Requirements

Scope control: Deliver a deterministic and testable implementation.
Correctness: Preserve Git invariants and policy constraints.

3.3 Non-Functional Requirements

Performance: Deterministic execution with documented runtime behavior on representative history sizes.
Reliability: Repeated runs on the same input produce identical outputs.
Usability: Clear CLI or report output for both success and failure cases.

3.4 Example Usage / Output

You’ll have a task runner that makes monorepos manageable:

Example Output:

$ mono status
=== Monorepo Status ===

Packages (5):
  packages/core       - library, no changes
  packages/utils      - library, 2 files changed
  packages/api        - app, depends on core, utils
  packages/web        - app, depends on core, utils
  packages/cli        - app, depends on core

Dependency graph:
  api ──→ core
    └───→ utils
  web ──→ core
    └───→ utils
  cli ──→ core

$ mono affected --base=main
Analyzing changes from main...

Changed files:
  packages/utils/src/string.ts (+12 -3)
  packages/utils/src/date.ts (+5 -2)

Affected packages (3):
  packages/utils      - directly changed
  packages/api        - depends on utils
  packages/web        - depends on utils

Unaffected packages (2):
  packages/core       - no dependency on changed files
  packages/cli        - no dependency on changed files

$ mono test --affected
Running tests for affected packages...

[1/3] Testing utils...
  Using cached result from abc123 (12 tests, 0.8s ago)
  ✓ Skipped (cache hit)

Wait, utils changed! Invalidating cache...
  Running 12 tests...
  ✓ 12 passed (2.3s)
  Cache stored: def456

[2/3] Testing api...
  Dependency utils changed, cache invalidated
  Running 47 tests...
  ✓ 47 passed (8.1s)
  Cache stored: ghi789

[3/3] Testing web...
  Dependency utils changed, cache invalidated
  Running 83 tests...
  ✓ 83 passed (12.4s)
  Cache stored: jkl012

Summary:
  3 packages tested
  142 tests passed
  Total time: 22.8s (without cache: ~45s)
  Cache hit rate: 0% (invalidated by changes)

$ mono test --affected  # Run again, nothing changed
All 3 affected packages have valid cache entries.
✓ Nothing to run (22.8s saved)

3.5 Data Formats / Schemas / Protocols

Describe input repository assumptions, output report shape, and any policy/config schema consumed by the tool.

3.6 Edge Cases

Empty repository or shallow clone state.
Detached HEAD or rewritten history during execution.
Invalid metadata/policy configuration.

3.7 Real World Outcome

You’ll have a task runner that makes monorepos manageable:

Example Output:

$ mono status
=== Monorepo Status ===

Packages (5):
  packages/core       - library, no changes
  packages/utils      - library, 2 files changed
  packages/api        - app, depends on core, utils
  packages/web        - app, depends on core, utils
  packages/cli        - app, depends on core

Dependency graph:
  api ──→ core
    └───→ utils
  web ──→ core
    └───→ utils
  cli ──→ core

$ mono affected --base=main
Analyzing changes from main...

Changed files:
  packages/utils/src/string.ts (+12 -3)
  packages/utils/src/date.ts (+5 -2)

Affected packages (3):
  packages/utils      - directly changed
  packages/api        - depends on utils
  packages/web        - depends on utils

Unaffected packages (2):
  packages/core       - no dependency on changed files
  packages/cli        - no dependency on changed files

$ mono test --affected
Running tests for affected packages...

[1/3] Testing utils...
  Using cached result from abc123 (12 tests, 0.8s ago)
  ✓ Skipped (cache hit)

Wait, utils changed! Invalidating cache...
  Running 12 tests...
  ✓ 12 passed (2.3s)
  Cache stored: def456

[2/3] Testing api...
  Dependency utils changed, cache invalidated
  Running 47 tests...
  ✓ 47 passed (8.1s)
  Cache stored: ghi789

[3/3] Testing web...
  Dependency utils changed, cache invalidated
  Running 83 tests...
  ✓ 83 passed (12.4s)
  Cache stored: jkl012

Summary:
  3 packages tested
  142 tests passed
  Total time: 22.8s (without cache: ~45s)
  Cache hit rate: 0% (invalidated by changes)

$ mono test --affected  # Run again, nothing changed
All 3 affected packages have valid cache entries.
✓ Nothing to run (22.8s saved)

4. Solution Architecture

4.1 High-Level Design

Inputs -> Validation -> Core Engine -> Output Formatter -> Verification Report

4.2 Key Components

Component	Responsibility	Key Decisions
Input loader	Discover commits/refs/config inputs	Deterministic ordering and clear failure messages
Core engine	Compute project-specific logic	Separate read-only simulation from mutating actions
Reporter	Produce user-facing output and evidence	Include machine-readable and human-readable forms

4.4 Data Structures (No Full Code)

ProjectState { refs, commits, policy, findings, metrics }
Result { status, evidence, warnings, next_actions }

4.4 Algorithm Overview

Collect state from repository and configuration.
Evaluate invariants and policy preconditions.
Execute core transformation or analysis logic.
Verify postconditions and emit deterministic report.

Complexity Analysis:

Time: O(history + affected scope)
Space: O(active graph window + report size)

5. Implementation Guide

5.1 Development Environment Setup

Use the environment defined in the main guide. Pin tool versions and fixture data to keep outputs reproducible.

5.2 Project Structure

project-root/
├── fixtures/
├── src/
├── tests/
├── docs/
└── README.md

5.3 The Core Question You’re Answering

“How do you build and test only what changed in a codebase with hundreds of packages?”

Before you write any code, sit with this question. The answer combines Git diff to know what changed, a dependency graph to know what’s affected, and content-addressable caching to skip redundant work.

5.4 Concepts You Must Understand First

Stop and research these before coding:

Package Dependency Graphs
- How do you model dependencies between packages?
- What’s the difference between dependencies and devDependencies?
- How do you detect circular dependencies?
- Book Reference: “Grokking Algorithms” Ch. 6 — Bhargava
Affected Detection
- How do you use git diff to find changed files?
- How do you map files to packages?
- How do you propagate “affected” through the dependency graph?
- Book Reference: “Software Engineering at Google” Ch. 17 — Winters et al.
Task Caching
- What inputs determine a task’s cache key?
- How do you store and retrieve cache entries?
- When is it safe to reuse a cached result?
- Resource: Turborepo documentation on caching

5.5 Questions to Guide Your Design

Before implementing, think through these:

Package Discovery
- How will you find packages in the repo (package.json, Cargo.toml, etc.)?
- How will you extract dependencies?
- How will you handle different package managers?
Cache Key Calculation
- What inputs affect a task’s output (source files, dependencies, config)?
- How do you hash all these inputs efficiently?
- Should the cache be local, remote, or both?
Task Orchestration
- How do you respect dependency order (topological sort)?
- How do you parallelize independent tasks?
- How do you handle task failures?

5.6 Thinking Exercise

Trace a Change Through Dependencies

Map out a monorepo change manually:

packages/
  shared-utils/    (dependency of everything)
  auth-service/    (depends on: shared-utils)
  user-api/        (depends on: shared-utils, auth-service)
  web-app/         (depends on: shared-utils, user-api)
  cli-tool/        (depends on: shared-utils)

Change: Edit packages/shared-utils/src/format.ts

Questions while tracing:

Which packages need to be rebuilt?
In what order should they be rebuilt?
If you had cached builds from yesterday, which caches are now invalid?
How many packages could you build in parallel?

5.7 The Interview Questions They’ll Ask

Prepare to answer these:

“How would you design a build system for a monorepo with 100 packages?”
“Explain how you would calculate a cache key for a build task.”
“What’s the time complexity of detecting affected packages?”
“How do you handle diamond dependencies in a monorepo?”
“What are the tradeoffs between monorepos and polyrepos?”

5.8 Hints in Layers

Hint 1: Starting Point Start with package discovery. Glob for package.json files, parse them, extract dependencies.

Hint 2: Dependency Graph Build an adjacency list where graph[pkg] = list of packages that depend on pkg. For affected detection, traverse from changed packages.

Hint 3: Cache Key Compute: hash(source_files_hash + dependencies_cache_keys + config_hash). If any input changes, the cache is invalid.

Hint 4: Execution Order Use Kahn’s algorithm for topological sort. Build a queue of packages with no pending dependencies; process and add newly unblocked packages.

5.9 Books That Will Help

Topic	Book	Chapter
Monorepo at scale	“Software Engineering at Google” by Winters et al.	Ch. 16-18
Graph algorithms	“Grokking Algorithms” by Bhargava	Ch. 6
Build system design	“The Bazel Book” (online)	Ch. 1-3

5.10 Implementation Phases

Phase 1: Foundation (1-2 sessions)

Define fixtures, expected outputs, and invariant checks.
Build read-only analysis path.

Phase 2: Core Functionality (2-4 sessions)

Implement project-specific core logic and deterministic reporting.
Add policy and edge-case handling.

Phase 3: Polish and Edge Cases (1-2 sessions)

Add failure demos, performance notes, and usability improvements.
Finalize docs and validation transcripts.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Execution mode	direct write vs dry-run+write	dry-run+write	Safer and easier to debug
Output contract	free text vs structured+text	structured+text	Better automation and readability
Enforcement location	local only vs local+CI	local+CI	Prevents bypass in shared branches

6. Testing Strategy

6.1 Test Categories

Unit tests for parsing and policy logic.
Integration tests on fixture repositories.
Edge-case tests for stale refs, malformed metadata, and large histories.

6.2 Critical Test Cases

Deterministic golden-path scenario.
Policy violation hard-fail scenario.
Recovery path after partial or conflicting state.

6.3 Test Data

Use fixed repository fixtures with known commit graphs and expected outputs stored under version control.

7. Common Pitfalls & Debugging

Problem 1: “Output looks correct but history or metadata is inconsistent”

Why: Validation happens after mutation, not before.
Fix: Add a preflight invariant check and a post-write verification step.
Quick test: Run the same command twice on the same fixture and verify identical results.

Problem 2: “Tool works on small repo but times out on larger history”

Why: Full traversal is performed where selective traversal is possible.
Fix: Cache intermediate graph lookups and scope analysis to affected commits/paths.
Quick test: Compare runtime on small and large fixtures with a clear budget target.

Problem 3: “Policy check can be bypassed by local-only behavior”

Why: Enforcement is advisory, not server-authoritative.
Fix: Mirror critical checks in CI and protected branch rules.
Quick test: Attempt merge with failing policy in CI and confirm hard block.

8. Extensions & Challenges

8.1 Beginner Extensions

Add richer error messages with remediation hints.
Add fixture generation helpers for repeatable demos.

8.2 Intermediate Extensions

Add performance instrumentation and budget assertions.
Add policy configuration profiles by repository type.

8.3 Advanced Extensions

Add distributed execution support for large repositories.
Add signed evidence exports for compliance workflows.

9. Real-World Connections

9.1 Industry Applications

Internal developer portals.
Enterprise repository governance systems.
Release safety and incident diagnostics tooling.

Git core: https://git-scm.com/
GitHub CLI: https://github.com/cli/cli
pre-commit framework: https://pre-commit.com/

9.3 Interview Relevance

This project prepares you for architecture and debugging interviews that focus on merge policy, CI gates, and workflow reliability tradeoffs.

10. Resources

10.1 Essential Reading

Pro Git (Internals and Workflows chapters)
Software Engineering at Google (Version control and build chapters)
Accelerate (delivery performance practices)

10.2 Video Resources

Git internals talks from Git Merge conference archives.
DORA and delivery metrics conference sessions.

10.3 Tools and Documentation

https://git-scm.com/docs
https://docs.github.com/
https://dora.dev/

11. Self-Assessment Checklist

11.1 Understanding

I can explain the primary invariant this project enforces.
I can explain one failure mode and one safe recovery path.

11.2 Implementation

Functional requirements are met on deterministic fixtures.
Critical edge cases are tested and documented.

11.3 Growth

I can describe tradeoffs in an interview setting.
I documented what I would change in a production version.

12. Submission / Completion Criteria

Minimum Viable Completion:

Deterministic golden-path output exists.
One failure scenario is handled with clear output.
Core workflow objective is demonstrably met.

Full Completion:

Minimum criteria plus policy validation, structured reporting, and edge-case coverage.

Excellence:

Full completion plus measurable performance budget and production-hardening notes.

Project 8: Monorepo Task Runner — Build a Mini Turborepo

Quick Reference

1. Learning Objectives

2. All Theory Needed (Per-Concept Breakdown)

Package Dependency Graphs

Affected Detection

Task Caching

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Data Formats / Schemas / Protocols

3.6 Edge Cases

3.7 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.4 Data Structures (No Full Code)

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

Trace a Change Through Dependencies

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: Foundation (1-2 sessions)

Phase 2: Core Functionality (2-4 sessions)

Phase 3: Polish and Edge Cases (1-2 sessions)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls & Debugging

8. Extensions & Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools and Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria