Project 5: Pipeline System

A system that executes cmd1 | cmd2 | cmd3 by creating pipes between processes, correctly wiring stdout of each process to stdin of the next.

Quick Reference

Attribute Value
Primary Language C
Alternative Languages Rust, Zig, Go
Difficulty Level 3: Advanced (The Engineer)
Time Estimate 1 week
Knowledge Area Operating Systems / IPC
Tooling Unix Shell
Prerequisites Projects 1-4, solid understanding of file descriptors

What You Will Build

A system that executes cmd1 | cmd2 | cmd3 by creating pipes between processes, correctly wiring stdout of each process to stdin of the next.

Why It Matters

This project builds core skills that appear repeatedly in real-world systems and tooling.

Core Challenges

  • Creating pipes (pipe() returns two file descriptors) → maps to IPC mechanisms
  • Wiring file descriptors (dup2 to redirect stdout/stdin) → maps to fd manipulation
  • Closing unused pipe ends (critical for proper EOF) → maps to resource management
  • Managing multiple children (fork N processes for N-command pipeline) → maps to process coordination
  • Waiting for all children (pipeline exit status is last command’s) → maps to process lifecycle

Key Concepts

  • pipe() system call: “The Linux Programming Interface” Chapter 44 - Kerrisk
  • dup2() for redirection: “Advanced Programming in the UNIX Environment” Chapter 3 - Stevens
  • Pipeline implementation: “Operating Systems: Three Easy Pieces” Chapter 5 - Arpaci-Dusseau

Real-World Outcome

$ ./mysh
mysh> ls -la | head -5
total 48
drwxr-xr-x  5 douglas  staff   160 Dec 20 10:00 .
drwxr-xr-x  3 douglas  staff    96 Dec 20 09:00 ..
-rw-r--r--  1 douglas  staff  1234 Dec 20 10:00 main.c
-rw-r--r--  1 douglas  staff   567 Dec 20 10:00 lexer.c
mysh> cat /etc/passwd | grep douglas | cut -d: -f1
douglas
mysh> seq 1 1000000 | wc -l
1000000

Implementation Guide

  1. Reproduce the simplest happy-path scenario.
  2. Build the smallest working version of the core feature.
  3. Add input validation and error handling.
  4. Add instrumentation/logging to confirm behavior.
  5. Refactor into clean modules with tests.

Milestones

  • Milestone 1: Minimal working program that runs end-to-end.
  • Milestone 2: Correct outputs for typical inputs.
  • Milestone 3: Robust handling of edge cases.
  • Milestone 4: Clean structure and documented usage.

Validation Checklist

  • Output matches the real-world outcome example
  • Handles invalid inputs safely
  • Provides clear errors and exit codes
  • Repeatable results across runs

References

  • Main guide: SHELL_INTERNALS_DEEP_DIVE_PROJECTS.md
  • “The Linux Programming Interface” by Michael Kerrisk