Project 4: stream-viz

Build a streaming CLI with a live progress bar and clean signal handling.

Quick Reference

Attribute Value
Difficulty Level 3 (Advanced)
Time Estimate 1 week
Language Rust (Alternatives: Go, C)
Prerequisites Project 1, basic signals, buffered I/O
Key Topics SIGINT handling, progress UI, streaming

1. Learning Objectives

By completing this project, you will:

  1. Handle SIGINT without corrupting output or state.
  2. Render progress updates without breaking pipes.
  3. Measure throughput and elapsed time accurately.
  4. Build a graceful shutdown that summarizes work done.
  5. Combine streaming I/O with periodic UI updates.

2. Theoretical Foundation

2.1 Core Concepts

  • Signals: SIGINT is asynchronous. You cannot safely do heavy work inside a signal handler; use a flag and let the main loop stop.
  • TTY vs Pipe: Progress bars belong on terminals. When output is a pipe, UI should be disabled or redirected to STDERR.
  • Throughput: Real-time rates should use monotonic clocks to avoid time jumps.
  • Backpressure: If downstream closes the pipe, your tool may receive SIGPIPE and should exit cleanly.

2.2 Why This Matters

Long-running CLI tools are judged by how they behave under interruption. A clean exit and accurate progress display are often the difference between a professional tool and a toy.

2.3 Historical Context / Background

Tools like pv and rsync pioneered progress UI conventions. They only draw progress when connected to a TTY and never contaminate output data streams.

2.4 Common Misconceptions

  • “SIGINT just kills the process”: You can intercept it and finalize cleanly.
  • “Progress bars are harmless”: They break pipelines if written to STDOUT.

3. Project Specification

3.1 What You Will Build

A CLI tool that reads a stream of data (file or STDIN), writes it to STDOUT, and shows progress on the terminal. On SIGINT, it stops, prints a summary, and exits gracefully.

3.2 Functional Requirements

  1. Streaming pass-through: Data is written unchanged to STDOUT.
  2. Progress UI: Displays bytes processed, percentage, and rate.
  3. Signal handling: On SIGINT, stop reading and print summary.
  4. Output modes: Disable UI if STDOUT is not a TTY.

3.3 Non-Functional Requirements

  • Reliability: No corrupted output.
  • Performance: Avoid per-byte overhead.
  • Compatibility: Works in pipelines and with large files.

3.4 Example Usage / Output

$ cat bigfile.bin | stream-viz > /dev/null

3.5 Real World Outcome

Run in a terminal with a large file. You see a progress bar updating in place and a final summary:

$ stream-viz bigfile.bin > /dev/null
[###########---------] 62%  1.2 GB / 2.0 GB  142 MB/s  ETA 00:05
^C
Processed: 1.2 GB in 8.4s (142 MB/s)

If you pipe output into another tool, no progress UI appears and the output remains clean.

4. Solution Architecture

4.1 High-Level Design

+-------------+     +----------------+     +------------------+
| Reader      | --> | Progress Meter | --> | Writer (STDOUT)  |
+-------------+     +----------------+     +------------------+
         |                  |
         +------------------+-- Signal handler (sets stop flag)

4.2 Key Components

Component Responsibility Key Decisions
Reader Buffered input chunk size
Meter Track bytes/time sampling interval
Renderer Progress output TTY-only, STDERR
Signal Handler Stop loop atomic flag

4.3 Data Structures

struct Progress {
    bytes: u64,
    start: Instant,
    last_update: Instant,
}

4.4 Algorithm Overview

Key Algorithm: Stream loop with interrupt flag

  1. Read chunk into buffer.
  2. Write chunk to STDOUT.
  3. Update byte count and UI (if TTY).
  4. Check atomic stop flag set by SIGINT.

Complexity Analysis:

  • Time: O(N) for N bytes
  • Space: O(B) for buffer size

5. Implementation Guide

5.1 Development Environment Setup

cargo new stream-viz
cargo add atty indicatif ctrlc

5.2 Project Structure

stream-viz/
├── src/
│   ├── main.rs
│   ├── progress.rs
│   └── signal.rs
└── README.md

5.3 The Core Question You Are Answering

“How do I build a streaming tool that stays responsive to interrupts without breaking pipelines?”

5.4 Concepts You Must Understand First

  1. Signal handling basics
  2. TTY detection
  3. Buffered I/O loops
  4. Monotonic time vs wall clock

5.5 Questions to Guide Your Design

  1. How will you update progress without flooding the terminal?
  2. What exit code should SIGINT produce?
  3. How do you avoid partial output when interrupted?

5.6 Thinking Exercise

Draw the data path when stream-viz is used in a pipe chain. Where should the progress output go?

5.7 The Interview Questions They Will Ask

  1. How do you handle SIGINT safely?
  2. Why should progress bars be disabled in pipes?
  3. How do you measure throughput accurately?

5.8 Hints in Layers

Hint 1: Use an atomic boolean to signal shutdown.

Hint 2: Render progress to STDERR, not STDOUT.

Hint 3: Throttle updates to 5-10 per second.

5.9 Books That Will Help

Topic Book Chapter
Signals “The Linux Programming Interface” Ch. 20
Terminal I/O “Advanced Programming in the UNIX Environment” Ch. 18

5.10 Implementation Phases

Phase 1: Foundation (2-3 days)

Goals:

  • Streaming pass-through
  • Basic progress counter

Checkpoint: stream-viz file > /dev/null works.

Phase 2: Signal Handling (2 days)

Goals:

  • Clean SIGINT exit
  • Summary print

Checkpoint: Ctrl+C prints final summary.

Phase 3: Polish (1-2 days)

Goals:

  • TTY detection
  • Nice progress display

Checkpoint: No UI when piped.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
UI output stream STDOUT vs STDERR STDERR Preserve data stream
Update interval fixed vs adaptive fixed Predictable behavior
Buffer size 4KB, 64KB, 1MB 64KB Good throughput

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests Progress math rate calculation
Integration Tests Stream pass-through compare input/output
Signal Tests Interrupt handling send SIGINT

6.2 Critical Test Cases

  1. Pipe input to output and verify equality (checksum).
  2. Redirect STDOUT to file and ensure no UI output.
  3. Trigger SIGINT and verify summary output.

6.3 Test Data

# generate a sample file
head -c 1048576 /dev/urandom > test.bin

Expected:

  • stream-viz test.bin > out.bin produces identical data

7. Common Pitfalls and Debugging

Pitfall Symptom Solution
UI written to STDOUT Corrupt pipeline Use STDERR
Too frequent updates Slow throughput Throttle updates
Ignoring SIGPIPE Crash on downstream close Handle SIGPIPE gracefully

8. Extensions and Challenges

8.1 Beginner Extensions

  • Add --bytes to show raw count only
  • Add --quiet mode

8.2 Intermediate Extensions

  • Add ETA calculation
  • Add percent when file size known

8.3 Advanced Extensions

  • Support multiple input files
  • Add JSON summary output

9. Real-World Connections

  • pv and rsync style progress meters
  • Stream processing pipelines in data engineering

10. Resources

  • pv source code for inspiration
  • indicatif or mpb docs

11. Self-Assessment Checklist

  • I can explain why progress output belongs on STDERR
  • I can handle SIGINT without corrupting output

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Stream pass-through works
  • SIGINT prints summary

Full Completion:

  • Progress UI is clean and TTY-aware

Excellence (Going Above and Beyond):

  • ETA, JSON summary, multiple inputs