Project 11: Async Runtime from Scratch (Understand Futures and Executors)

A minimal async runtime like a tiny tokio—implementing the executor that polls futures, the reactor that handles I/O events, and the waker system that schedules tasks.

Quick Reference

Attribute Value
Primary Language Rust
Alternative Languages None (deep Rust internals)
Difficulty Level 5: Master
Time Estimate 1 month+
Knowledge Area Async / Futures / Executors / Polling
Tooling std::future, waker, context
Prerequisites All prior projects, deep understanding of ownership and lifetimes

What You Will Build

A minimal async runtime like a tiny tokio—implementing the executor that polls futures, the reactor that handles I/O events, and the waker system that schedules tasks.

Why It Matters

This project builds core skills that appear repeatedly in real-world systems and tooling.

Core Challenges

  • Understanding the Future trait and poll method → maps to how async desugars
  • Implementing a waker and context → maps to how tasks get rescheduled
  • Building an event loop with epoll/kqueue → maps to OS-level async I/O
  • Managing task queues and scheduling → maps to executor design

Key Concepts

  • Future trait: “Asynchronous Programming in Rust” - Rust Async Book
  • Wakers and Context: “Rust for Rustaceans” Chapter 8 - Jon Gjengset
  • State machine desugaring: “Programming Rust, 2nd Edition” Chapter 20 - Jim Blandy
  • epoll/kqueue: “The Linux Programming Interface” Chapter 63 - Michael Kerrisk

Real-World Outcome

// Your runtime powers this async code:
fn main() {
    MyRuntime::block_on(async {
        println!("Starting async operations...");

        // Spawn concurrent tasks
        let task1 = spawn(async {
            sleep(Duration::from_secs(1)).await;
            "Task 1 complete"
        });

        let task2 = spawn(async {
            sleep(Duration::from_millis(500)).await;
            "Task 2 complete"
        });

        // Wait for both
        let (r1, r2) = join!(task1, task2);
        println!("{}, {}", r1, r2);
    });
}

// Output:
// Starting async operations...
// Task 2 complete, Task 1 complete
// (Task 2 finishes first because it sleeps less)

// Benchmark vs tokio:
// Your runtime: 50k tasks/sec
// Tokio: 500k tasks/sec
// Not bad for learning!

Implementation Guide

  1. Reproduce the simplest happy-path scenario.
  2. Build the smallest working version of the core feature.
  3. Add input validation and error handling.
  4. Add instrumentation/logging to confirm behavior.
  5. Refactor into clean modules with tests.

Milestones

  • Milestone 1: Minimal working program that runs end-to-end.
  • Milestone 2: Correct outputs for typical inputs.
  • Milestone 3: Robust handling of edge cases.
  • Milestone 4: Clean structure and documented usage.

Validation Checklist

  • Output matches the real-world outcome example
  • Handles invalid inputs safely
  • Provides clear errors and exit codes
  • Repeatable results across runs

References

  • Main guide: LEARN_RUST_DEEP_DIVE.md
  • “Asynchronous Programming in Rust” (Rust Async Book) + “Rust for Rustaceans”