Project 7: Watch Commander

Build a multi-command watcher with per-command intervals and change alerts.

Quick Reference

Attribute Value
Difficulty Beginner
Time Estimate Weekend
Language Bash (Alternatives: Python, Go, Rust)
Prerequisites Basic shell scripting
Key Topics polling loops, diffing output, alerts

1. Learning Objectives

By completing this project, you will:

  1. Schedule multiple commands with different intervals.
  2. Detect output changes between runs.
  3. Trigger alerts based on thresholds.
  4. Render a simple multi-panel terminal view.

2. Theoretical Foundation

2.1 Core Concepts

  • Polling: Periodic sampling is simple but must balance noise and overhead.
  • Change detection: Comparing outputs or extracted fields across samples.
  • Alerting: Thresholds and rate-limiting reduce false positives.

2.2 Why This Matters

Most operational monitoring is still polling under the hood. This project teaches disciplined observation.

2.3 Historical Context / Background

watch is a classic Unix tool, but modern monitoring requires multiple signals at different cadences.

2.4 Common Misconceptions

  • “Polling is easy”: Schedules and output comparisons introduce complexity.
  • “All checks run at same rate”: They should not.

3. Project Specification

3.1 What You Will Build

A CLI tool that runs multiple commands on a schedule, highlights changes, and logs alerts.

3.2 Functional Requirements

  1. Read a config file of commands and intervals.
  2. Run each command on its own schedule.
  3. Detect changes and highlight them.
  4. Log alerts to a file.

3.3 Non-Functional Requirements

  • Performance: Avoid overlapping runs.
  • Reliability: Handle command failures gracefully.
  • Usability: Simple config and readable output.

3.4 Example Usage / Output

$ ./watch-commander --config monitors.yaml
[CPU] load: 0.45 0.67 0.89
[DISK] /home 89% used (warn)

3.5 Real World Outcome

You will see multiple checks updating independently in a single terminal view. Example:

$ ./watch-commander --config monitors.yaml
[CPU] load: 0.45 0.67 0.89
[DISK] /home 89% used (warn)

4. Solution Architecture

4.1 High-Level Design

config -> scheduler -> run commands -> diff -> render -> log alerts

4.2 Key Components

Component Responsibility Key Decisions
Scheduler Track next run times 1s base tick
Runner Execute commands Capture stdout/stderr
Diff Compare outputs Store previous output
Renderer Display panels Fixed layout

4.3 Data Structures

NEXT_RUN[cmd_id]=timestamp
LAST_OUT[cmd_id]=output

4.4 Algorithm Overview

Key Algorithm: Scheduler Loop

  1. For each command, check if due.
  2. Run command and capture output.
  3. Compare to previous output and update view.

Complexity Analysis:

  • Time: O(k) per tick, k = number of commands
  • Space: O(k) stored outputs

5. Implementation Guide

5.1 Development Environment Setup

bash --version

5.2 Project Structure

project-root/
├── watch_commander.sh
├── monitors.yaml
└── README.md

5.3 The Core Question You’re Answering

“How do I watch several signals at different frequencies without losing my mind?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. Loops and sleep
    • Handling long-running commands.
  2. Diffing output
    • Use diff or hash comparison.
  3. Thresholds
    • Parse numeric values from output.

5.5 Questions to Guide Your Design

Before implementing, think through these:

  1. What happens if a command takes longer than its interval?
  2. How will you log alerts without spamming?
  3. How will you persist last output between runs?

5.6 Thinking Exercise

Manual watch

Run two watch commands side by side and note how awkward it is. Use that pain to design a better UX.

5.7 The Interview Questions They’ll Ask

Prepare to answer these:

  1. “What are the tradeoffs of polling?”
  2. “How do you avoid alert storms?”
  3. “How do you detect change between runs?”

5.8 Hints in Layers

Hint 1: Start with one command Then generalize to a list.

Hint 2: Use timestamps Track next_run per command.

Hint 3: Persist output Store previous output in /tmp or memory.

5.9 Books That Will Help

Topic Book Chapter
Bash loops “The Linux Command Line” Ch. 29
Scripting patterns “Bash Cookbook” Ch. 17
Monitoring “Effective Shell” Ch. 12

5.10 Implementation Phases

Phase 1: Foundation (Half day)

Goals:

  • Run a single command on a timer.

Tasks:

  1. Loop with sleep.
  2. Print output.

Checkpoint: Output updates on interval.

Phase 2: Core Functionality (1 day)

Goals:

  • Add multiple commands with schedules.

Tasks:

  1. Parse config.
  2. Track next-run timestamps.

Checkpoint: Commands update independently.

Phase 3: Polish & Edge Cases (Half day)

Goals:

  • Add change detection and alerts.

Tasks:

  1. Compare output to previous.
  2. Log alert if threshold exceeded.

Checkpoint: Alerts fire only on real changes.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Config format env vs YAML simple YAML Human-friendly
Diff method full diff vs hash hash Faster on large output

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Scheduling Validate intervals Two commands, different rates
Diffing Detect changes date output
Alerts Thresholds disk usage > 80%

6.2 Critical Test Cases

  1. Command fails; tool continues.
  2. Output change detected and highlighted.
  3. Alerts rate-limited.

6.3 Test Data

Command: uptime

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Drift in timing Commands run late Use monotonic time
Overlapping runs CPU spikes Skip if still running
Spamming alerts Too many logs Add cooldown

7.2 Debugging Strategies

  • Log timestamps for each run.
  • Test with sleep 2 commands.

7.3 Performance Traps

Running too many commands too often creates monitoring overhead.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add color highlighting.
  • Add command tags.

8.2 Intermediate Extensions

  • Export to JSON logs.
  • Add interactive pause/quit.

8.3 Advanced Extensions

  • Build a curses-based multi-panel UI.
  • Add plug-in checks.

9. Real-World Connections

9.1 Industry Applications

  • Simple health checks and ad-hoc monitoring.
  • watch: https://gitlab.com/procps-ng/procps
  • monit: https://mmonit.com/monit

9.3 Interview Relevance

  • Polling and alerting patterns are common in SRE interviews.

10. Resources

10.1 Essential Reading

  • watch(1) - man 1 watch

10.2 Video Resources

  • Monitoring basics talks (search “polling vs event”)

10.3 Tools & Documentation

  • diff(1) - man 1 diff

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain polling tradeoffs.
  • I can detect output changes.
  • I can manage intervals per command.

11.2 Implementation

  • Commands run on independent schedules.
  • Changes are detected and displayed.
  • Alerts are rate-limited.

11.3 Growth

  • I can add new checks quickly.
  • I can explain my alert logic.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Run multiple commands on different intervals.

Full Completion:

  • Detect changes and log alerts.

Excellence (Going Above & Beyond):

  • Provide a curses UI and JSON output.

This guide was generated from LINUX_SYSTEM_TOOLS_MASTERY.md. For the complete learning path, see the parent directory.