Project 1: Website Status Checker

Build a robust CLI tool that reads a list of URLs, checks HTTP availability and response time, and emits a clear, deterministic report with actionable failure categories.

Quick Reference

Attribute Value
Difficulty Level 1: Beginner
Time Estimate 6 to 10 hours
Main Programming Language Bash (POSIX shell acceptable)
Alternative Programming Languages Python, Go, Ruby
Coolness Level Level 2: Practical but Useful
Business Potential Level 2: Internal Ops Tool
Prerequisites Basic shell usage, basic HTTP vocabulary, ability to read error messages
Key Topics Shell parsing and exit codes, HTTP status codes, curl usage, DNS/TCP/TLS failure modes, deterministic reporting

1. Learning Objectives

By completing this project, you will:

  1. Build a CLI script that safely parses input, handles comments, and never breaks on spaces or special characters.
  2. Distinguish HTTP failures from network failures using curl exit codes and status codes.
  3. Measure and report response time with consistent, deterministic output formatting.
  4. Design clear exit codes for your own CLI tools and document them.
  5. Create a monitoring report that is useful for humans and automation alike.

2. All Theory Needed (Per-Concept Breakdown)

Concept 1: Shell Execution, Quoting, and Exit Codes

Fundamentals

The shell is a language that transforms what you type into an executable command. That transformation is not trivial: it tokenizes text, performs expansions, splits words, and only then executes programs. In a monitoring script, most mistakes come from the shell, not from curl: unquoted variables can become multiple arguments, filenames can expand unexpectedly, and conditional checks can misinterpret a failure as success. Exit codes are the numeric contract between programs; they let you build control flow that reacts to failure instead of ignoring it. You must know that zero means success and any non-zero value means failure, and you must learn to propagate that meaning into your own script. The shell provides tools like set -e and set -o pipefail, but they have sharp edges and must be used intentionally. The goal of this concept is to make the shell predictable so that your monitoring script behaves exactly the same every time it runs.

Deep Dive into the Concept

When bash reads a line, it applies a precise order of operations: tokenization, expansions, word splitting, and globbing. This is why url=$line is safe but curl $url is not: the unquoted variable undergoes word splitting, so a URL containing & or spaces becomes multiple arguments. The rule for safe scripting is simple: always quote variables unless you explicitly want word splitting. If your script reads from a file, you must also protect against empty lines and comment lines, which means a while IFS= read -r line; do ... done loop. The -r flag prevents backslash escapes from being interpreted, and IFS= prevents trimming leading or trailing whitespace. If you choose for url in $(cat file) you are choosing an error-prone path, because command substitution forces word splitting on whitespace, which destroys URLs and any line with spaces.

Exit codes are the glue between commands. curl uses exit codes to indicate network failures (DNS failure, connection refused, timeout) and HTTP status codes to indicate application-layer responses. If you ignore exit codes, you will misinterpret failures. A solid CLI tool defines its own exit codes and documents them. For example, exit code 0 for success, 1 for usage errors, 2 if any URL failed, and 3 for internal errors. That allows automation and cron jobs to detect failure without parsing human-readable output. If you use pipelines, you must decide whether the pipeline’s status should reflect the last command or any failed command. set -o pipefail changes pipeline exit status to the last non-zero command, but it can also cause your script to exit when an intermediate tool fails, which might be acceptable or not. The safe pattern is to capture exit codes explicitly (rc=$?) after each critical command and handle them intentionally.

Shell functions and local variables help isolate logic. A function like check_url() can take a URL, run curl, parse output, and return both a status string and an exit code. You can communicate structured data by printing CSV or JSON lines, while returning exit codes as the control signal. Avoid eval at all costs; it re-parses strings as code and can allow URLs to become commands. Use arrays if your shell supports them, especially for building argument lists, because arrays preserve spacing correctly. Another tool is printf, which is more predictable than echo and allows you to align columns. Combining printf with fixed timestamps (via TZ=UTC and date -u) makes your output deterministic and testable.

Finally, consider input validation. A simple regex can check if a line starts with http:// or https:// before attempting curl. Do not over-validate; let curl be the validator for complex cases. The goal is to avoid obvious errors while still handling real-world URLs. Logging is part of this concept too: if your script fails, you need to know why. Emit errors to stderr, keep data on stdout, and keep the formats stable. This separation is what makes your script composable in pipelines and cron jobs. A monitoring script is a small program, and programs need clear contracts.

How This Fits in Projects

This is the foundation for Project 1. You will apply it in §3.2 (Functional Requirements), §5.2 (Project Structure), and §6.2 (Critical Test Cases). The same shell safety patterns also apply to P02 Log File Analyzer and P03 Automated Backup Script.

Definitions & Key Terms

  • Exit code: Numeric status of a command, where 0 means success.
  • Word splitting: Breaking a string into multiple arguments based on whitespace.
  • Globbing: Shell expansion of wildcard patterns like *.
  • Stdout/stderr: Output streams for data and errors, respectively.
  • Pipefail: A shell option that changes pipeline exit status behavior.

Mental Model Diagram (ASCII)

Input line -> tokenize -> expand -> split -> glob -> exec
                     |                      |
                     v                      v
                argv[]                 exit code

How It Works (Step-by-Step)

  1. Read one line safely using IFS= read -r.
  2. Validate basic URL shape (optional, lightweight).
  3. Call curl with quoted URL and fixed options.
  4. Capture curl exit code and output variables.
  5. Format the report line with printf.
  6. Accumulate summary counts and compute exit code.

Invariants: always quote variables; stdout is for data, stderr for errors. Failure modes: unquoted variables causing argument splitting, ignoring non-zero exit codes, mixing stderr into structured output.

Minimal Concrete Example

url="https://example.com/?q=a&b=c"
code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
rc=$?
printf "status=%s http=%s url=%s\n" "$rc" "$code" "$url"

Common Misconceptions

  • “If curl returns 404, the network failed.” -> HTTP 404 is an application response, not a network error.
  • “echo is fine for formatting.” -> printf is safer and consistent.
  • “Unquoted variables are okay if they look simple.” -> They will break on unexpected input.

Check-Your-Understanding Questions

  1. Why is for u in $(cat urls.txt) unsafe?
  2. What does rc=$? capture after a command?
  3. Why should errors go to stderr?
  4. What does set -o pipefail change?

Check-Your-Understanding Answers

  1. It forces word splitting on whitespace and breaks URLs with spaces or special characters.
  2. The exit code of the most recently executed command.
  3. To keep structured data on stdout for pipelines and make errors separable.
  4. It makes a pipeline fail if any command in it fails, not just the last.

Real-World Applications

  • Monitoring scripts run by cron
  • Deployment checks and smoke tests
  • Health checks in CI pipelines

Where You’ll Apply It

References

  • The Linux Command Line, 2nd Edition (Shotts), Ch. 10-11
  • Effective Shell (Kerr), Ch. 2-4

Key Insight

Correct shell behavior is not luck; it is the product of disciplined quoting and exit-code handling.

Summary

Shell safety determines whether your monitoring script is reliable or a minefield of subtle bugs.

Homework/Exercises to Practice the Concept

  1. Write a one-liner that prints a variable containing spaces without splitting it.
  2. Create a pipeline that fails in the middle; observe the exit code with and without pipefail.
  3. Write a loop that reads lines from a file and prints them exactly as-is.

Solutions to the Homework/Exercises

  1. printf "%s\n" "$var"
  2. false | true; echo $?; set -o pipefail; false | true; echo $?
  3. while IFS= read -r line; do printf "%s\n" "$line"; done < file.txt

Concept 2: HTTP Requests, Status Codes, and curl Instrumentation

Fundamentals

HTTP is the application protocol that defines how a client requests a resource and how a server responds. It is built on top of TCP (and often TLS) but presents its own vocabulary: methods like GET and HEAD, response status codes, and headers. For a status checker, you do not need to download full content; you need to check whether the server responds and how it responds. curl is a versatile CLI HTTP client that can perform a request and report both the HTTP status and the timing information of the transfer. The HTTP status code indicates what the server decided; the curl exit code indicates whether the request could be made at all. Understanding the difference is essential, because it lets you classify failures accurately.

Deep Dive into the Concept

HTTP is a request-response protocol. The client sends a request line (method, path, protocol version), headers, and optionally a body. The server responds with a status line, headers, and optionally a body. Status codes are grouped by class: 2xx for success, 3xx for redirection, 4xx for client errors, and 5xx for server errors. A status checker should treat these classes differently. For example, 200 indicates healthy; 301 might be acceptable if you allow redirects; 404 likely indicates a missing endpoint; 500 suggests a server-side failure. Your tool should make those distinctions explicit rather than just saying “fail”.

curl provides both HTTP status and timing via the --write-out format. For example, -w "%{http_code} %{time_total}" prints the HTTP status and total request time in seconds. When you add -o /dev/null, you discard response bodies and reduce bandwidth. -I requests only headers, but some servers treat HEAD differently, so a safe approach is to use GET and discard the body. -s silences progress output so you can control output deterministically. --connect-timeout limits DNS and TCP connection time, while --max-time caps total time. This prevents your script from hanging on dead endpoints. Using -L can follow redirects, but you must decide whether redirects should be counted as success or not. A monitoring script should make this policy explicit.

Timing is subtle. time_namelookup captures DNS lookup time, time_connect is TCP connect time, time_appconnect is TLS handshake time, and time_total is overall time. Even if you only report total time, understanding these phases helps diagnose failures. For example, a DNS failure will give a curl exit code and no HTTP status; a TLS failure might occur after DNS success but before HTTP response. You can optionally log these timing fields in CSV if you want deeper diagnostics. But even a single number is useful if you treat it consistently and use a fixed timestamp for logging.

HTTP also supports status caching and proxy behavior. If you run from a corporate environment, you might see 403 or proxy errors. This is why you should allow a user-agent override and proxy environment variables (http_proxy, https_proxy). curl respects these environment variables by default. When you build a reusable tool, be clear about this behavior. You should also recognize that a DNS failure yields HTTP code 000. This is not an HTTP response; it is a placeholder for “no response”. Do not treat 000 as a status; treat it as a network failure. This distinction is essential for correct summaries.

Finally, because this tool is for learners, you should make your output meaningful. A line like OK 200 0.12s https://example.com is readable, but you should also include a summary such as “2 OK, 1 WARN (redirect), 1 FAIL (timeout)”. This is how monitoring tools convey information: quick per-target output plus a final rollup. For deterministic output, freeze time with TZ=UTC and allow SOURCE_DATE_EPOCH to override timestamps. This makes tests reproducible and avoids confusion about local timezones.

How This Fits in Projects

This concept drives the main functionality in §3 (Project Specification) and §4 (Solution Architecture). It is also directly used in the failure demos in §3.7.3. Understanding curl output and HTTP status handling is unique to Project 1, but the idea of extracting structured output applies to P02 Log File Analyzer.

Definitions & Key Terms

  • HTTP status code: Numeric response classifying server outcome.
  • HEAD request: HTTP method that returns headers without body.
  • User-Agent: HTTP header identifying the client.
  • curl exit code: Numeric code representing network or client errors.
  • time_total: Total time for the request in curl metrics.

Mental Model Diagram (ASCII)

Client (curl)
   |
   |-- DNS lookup --> IP
   |-- TCP connect --> socket
   |-- TLS handshake (optional)
   |-- HTTP request
   |<- HTTP response (status, headers, body)

How It Works (Step-by-Step)

  1. Resolve hostname to IP (DNS).
  2. Establish TCP connection (and TLS if HTTPS).
  3. Send HTTP request headers.
  4. Receive HTTP status and headers.
  5. Optionally read body (discarded in this tool).
  6. Record timing and status code.

Invariants: HTTP status exists only if request succeeds; curl exit code is always set. Failure modes: DNS failure, connect timeout, TLS failure, HTTP 5xx.

Minimal Concrete Example

curl -s -o /dev/null -w "%{http_code} %{time_total}" https://example.com

Common Misconceptions

  • “HTTP 000 is a real status.” -> It means no HTTP response was received.
  • “HEAD is always safe.” -> Some servers behave differently for HEAD.
  • “HTTP 301 is a failure.” -> It can be acceptable depending on policy.

Check-Your-Understanding Questions

  1. Why can curl exit code be non-zero even if HTTP would be 200?
  2. What is the difference between --connect-timeout and --max-time?
  3. Why might you prefer GET with -o /dev/null over HEAD?

Check-Your-Understanding Answers

  1. Because the request might fail before any HTTP response is received.
  2. Connect-timeout limits DNS/TCP setup; max-time caps total duration.
  3. Some servers mis-handle HEAD; GET is more representative of real behavior.

Real-World Applications

  • Uptime monitoring
  • API health checks
  • Latency tracking in CI

Where You’ll Apply It

References

  • HTTP Semantics (RFC 9110), status code classes
  • curl man page sections for --write-out and exit codes

Key Insight

HTTP status codes explain the server decision; curl exit codes explain the network path.

Summary

Reliable checks require interpreting both HTTP and transport-level signals.

Homework/Exercises to Practice the Concept

  1. Run curl with -w to print status and total time for three URLs.
  2. Force a DNS failure and record the exit code.
  3. Compare HEAD and GET results for the same URL.

Solutions to the Homework/Exercises

  1. curl -s -o /dev/null -w "%{http_code} %{time_total}\n" https://example.com https://example.net https://example.org
  2. curl -s -o /dev/null -w "%{http_code}\n" https://no-such-domain.invalid; echo $?
  3. curl -I https://example.com; curl -s -o /dev/null -w "%{http_code}\n" https://example.com

Concept 3: Network Failure Modes, Timeouts, and Retries

Fundamentals

Not all failures are equal. A DNS failure means the hostname could not be resolved; a TCP failure means the host did not accept a connection; a TLS failure means the secure handshake failed; an HTTP 5xx means the server responded but failed internally. A robust status checker must recognize these differences so the output is actionable. Timeouts and retries are the standard safety tools: timeouts prevent the script from hanging, and retries handle transient errors. In this project, you will learn to configure timeouts in curl and design a retry policy that is simple, predictable, and safe for monitoring.

Deep Dive into the Concept

Networking failures happen before HTTP even begins. DNS resolution can fail because the resolver is down, because the domain does not exist (NXDOMAIN), or because of a transient timeout. TCP connections can fail with “connection refused” when the host is reachable but no service is listening, or with “connection timed out” when packets are dropped. TLS failures can occur if the certificate is invalid, expired, or does not match the hostname. All of these failures result in a non-zero curl exit code, and they should be classified differently. For monitoring, this difference matters: a DNS failure suggests a domain configuration issue, while a connection timeout suggests a firewall or host outage.

Timeouts should be explicit. --connect-timeout limits the DNS + TCP connect phase, and --max-time limits the entire request. Setting these values prevents the script from hanging, which is especially important in cron jobs where a single hung check can block all others. A good default might be 5 seconds for connect and 10 seconds total, but you should allow override flags. Retries are trickier: if you retry too aggressively, you can amplify outages; if you never retry, a brief hiccup becomes a false alarm. A reasonable policy is one retry for transient errors like timeouts, with a short sleep between attempts. You can also limit retries to specific exit codes, such as 28 (timeout), 6 (could not resolve host), or 7 (failed to connect). Do not retry on HTTP 4xx, because those are usually deterministic client-side errors.

Another subtlety is that some failures are “soft” while others are “hard”. A redirect (301) is not necessarily an error, but it might indicate a stale URL list. A 503 could mean maintenance, which might be acceptable depending on your use case. The goal of this tool is not to decide for every scenario, but to give you a structured report so you can decide. That means categorizing results: OK, WARN (redirect), FAIL_HTTP (4xx/5xx), FAIL_NET (DNS/TCP/TLS/timeout). These categories are more informative than raw numbers.

Your script should also be deterministic. Use fixed timestamps in tests, and make sure your report format does not depend on locale or time zone. Set LC_ALL=C for consistent sorting and numeric formatting. For reproducible examples, allow SOURCE_DATE_EPOCH to override the timestamp. This is standard practice in reproducible builds, and it makes your output stable for testing. This is not just a nicety: it makes your monitoring output comparable across runs, which is crucial when you are trying to detect regressions.

Retries should be logged explicitly. If a URL fails once and then succeeds, the script should note that it succeeded on retry. This information helps diagnose flaky endpoints. A clean design is to measure each attempt, keep the best status, and record the number of attempts. For example, attempts=2 and last_status=200 is more informative than just “OK”. At minimum, the summary should include counts of failures by category. Your tool becomes useful when its output communicates why a failure happened, not just that it happened.

How This Fits in Projects

This concept informs §3.2 (Functional Requirements) and §3.7 (Real World Outcome), where you define and demonstrate error categories and exit codes. Similar retry and timeout logic is also useful in P03 Automated Backup Script for network syncs.

Definitions & Key Terms

  • DNS failure: Hostname could not be resolved to an IP.
  • Connection timeout: TCP connection did not complete in time.
  • TLS handshake: Cryptographic setup phase of HTTPS.
  • Retry policy: Rules for when and how to reattempt a request.
  • Transient error: Failure likely to succeed on retry.

Mental Model Diagram (ASCII)

URL -> DNS -> TCP -> TLS -> HTTP
  |      |      |      |     |
  |      |      |      |     +-- HTTP status (2xx/3xx/4xx/5xx)
  |      |      |      +-------- TLS errors
  |      |      +--------------- TCP errors/timeouts
  |      +---------------------- DNS errors
  +----------------------------- input validation

How It Works (Step-by-Step)

  1. Validate input URL shape.
  2. Attempt request with curl and timeouts.
  3. If exit code indicates transient failure, retry once.
  4. Classify result into OK/WARN/FAIL categories.
  5. Emit report line with attempt count and reason.

Invariants: timeouts must always be set; retries must be bounded. Failure modes: infinite retries, unclassified exit codes, ambiguous output.

Minimal Concrete Example

code=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 --max-time 10 "$url")
rc=$?
if [ $rc -ne 0 ]; then
  printf "FAIL_NET rc=%s url=%s\n" "$rc" "$url"
fi

Common Misconceptions

  • “Retries always help.” -> They can hide outages or overload a failing service.
  • “Timeouts are only for slow networks.” -> They protect automation from hanging forever.
  • “All non-200 codes are the same.” -> 301, 404, and 500 mean different things.

Check-Your-Understanding Questions

  1. Why is retrying HTTP 404 usually pointless?
  2. Which errors should be treated as network failures?
  3. Why must timeouts be explicit in automation?

Check-Your-Understanding Answers

  1. Because 404 indicates a missing resource, not a transient condition.
  2. DNS failures, connection errors, and TLS handshake failures.
  3. Because a hung check can block the entire script or cron job.

Real-World Applications

  • SLA monitoring
  • Incident triage and root-cause analysis
  • CI/CD health gates

Where You’ll Apply It

References

  • The Linux Command Line (Shotts), Ch. 10
  • curl documentation for exit codes and timeouts

Key Insight

Monitoring is only useful when it distinguishes between transport failures and HTTP failures.

Summary

Timeouts and error categories turn raw curl output into meaningful diagnostics.

Homework/Exercises to Practice the Concept

  1. Simulate a timeout with a non-routable IP and record the exit code.
  2. Create a retry loop that only retries exit code 28.
  3. Make a list of error categories and map them to curl exit codes.

Solutions to the Homework/Exercises

  1. curl --connect-timeout 1 --max-time 2 https://10.255.255.1; echo $?
  2. for i in 1 2; do curl --connect-timeout 1 --max-time 2 https://10.255.255.1; rc=$?; [ $rc -ne 28 ] && break; done; echo $rc
  3. Example mapping: 6 DNS, 7 connection failed, 28 timeout, 35 TLS, 60 certificate.

3. Project Specification

3.1 What You Will Build

You will build a CLI tool named status-checker that reads a file of URLs and prints a per-URL status line plus a summary. It will classify failures into clear categories, measure response time, and exit with a meaningful code. The tool will support configurable timeouts and a deterministic output mode for testing. It will not download or parse response bodies, perform parallel checks, or store long-term history. Its purpose is immediate visibility, not persistent monitoring.

3.2 Functional Requirements

  1. Input file parsing: Accept a text file containing one URL per line; skip empty lines and # comments.
  2. Safe handling: Preserve each line exactly; URLs with query strings or spaces must not break parsing.
  3. HTTP check: Perform an HTTP GET (or HEAD if configured) using curl and capture HTTP status and total time.
  4. Error classification: Distinguish HTTP errors (4xx/5xx) from network errors (DNS/TCP/TLS/timeouts).
  5. Retries: Retry once for timeout errors if --retries 1 is specified; default is no retry.
  6. Reporting: Print a deterministic, aligned report line for each URL.
  7. Summary: Print counts for OK, WARN (redirect), FAIL_HTTP, FAIL_NET.
  8. Exit codes: 0 if all URLs OK or WARN; 2 if any FAIL_HTTP or FAIL_NET; 1 for usage errors; 3 for internal errors.
  9. Optional formats: Support --csv output with a stable header.

3.3 Non-Functional Requirements

  • Performance: Per-URL check must respect --connect-timeout and --max-time.
  • Reliability: No unbounded loops; retries capped; errors are classified consistently.
  • Usability: Output is human-readable by default, machine-readable with --csv.

3.4 Example Usage / Output

$ ./status-checker urls.txt --connect-timeout 3 --max-time 6
[OK   ] 200  0.12s  https://example.com
[WARN ] 301  0.08s  http://example.org
[FAIL ] NET  0.00s  https://no-such-domain.invalid (rc=6)

Summary: ok=1 warn=1 fail_http=0 fail_net=1 total=3
Exit code: 2

3.5 Data Formats / Schemas / Protocols

Input file (urls.txt):

# One URL per line
https://example.com
http://example.org
https://no-such-domain.invalid

CSV output (--csv):

timestamp,url,http_code,category,elapsed_s,attempts,rc
2025-12-31T00:00:00Z,https://example.com,200,OK,0.12,1,0
2025-12-31T00:00:00Z,https://no-such-domain.invalid,000,FAIL_NET,0.00,1,6

3.6 Edge Cases

  • URL line contains spaces or & characters.
  • URL line is empty or starts with #.
  • DNS failure (curl exit code 6).
  • TLS failure (curl exit code 35 or 60).
  • Server returns 301 redirect when redirects are not followed.
  • Timeout (curl exit code 28).
  • Input file does not exist or is not readable.

3.7 Real World Outcome

This section is a golden reference. Compare your output directly.

3.7.1 How to Run (Copy/Paste)

cd /path/to/status-checker
chmod +x status-checker
TZ=UTC SOURCE_DATE_EPOCH=1767225600 ./status-checker urls.txt --connect-timeout 2 --max-time 5 --csv

3.7.2 Golden Path Demo (Deterministic)

With TZ=UTC and SOURCE_DATE_EPOCH=1767225600 (2025-12-31T00:00:00Z), you should see fixed timestamps in CSV output and stable formatting.

3.7.3 If CLI: Exact Terminal Transcript

$ cat urls.txt
https://example.com
http://example.org
https://no-such-domain.invalid

$ TZ=UTC SOURCE_DATE_EPOCH=1767225600 ./status-checker urls.txt --connect-timeout 2 --max-time 5
[OK   ] 200  0.12s  https://example.com
[WARN ] 301  0.08s  http://example.org
[FAIL ] NET  0.00s  https://no-such-domain.invalid (rc=6)

Summary: ok=1 warn=1 fail_http=0 fail_net=1 total=3
Exit code: 2

$ echo $?
2

3.7.4 Failure Demo (Bad Input)

$ ./status-checker missing.txt
ERROR: input file not found: missing.txt
Exit code: 1

$ echo $?
1

4. Solution Architecture

4.1 High-Level Design

+-------------------+
| Input Parser      |  reads urls.txt safely
+---------+---------+
          |
          v
+-------------------+
| URL Checker       |  curl + classification + timing
+---------+---------+
          |
          v
+-------------------+
| Formatter         |  text or CSV output
+---------+---------+
          |
          v
+-------------------+
| Summary + Exit    |  counts + exit code
+-------------------+

4.2 Key Components

Component Responsibility Key Decisions
Input Parser Read lines safely, skip comments Use IFS= read -r, skip # and empty
URL Checker Run curl, capture status/rc/time Use -s -o /dev/null -w
Classifier Map HTTP/rc to categories Separate HTTP vs NET errors
Formatter Output stable report printf with fixed widths
Summary Count categories and set exit code Define policy for WARN

4.3 Data Structures (No Full Code)

# Counters
ok=0
warn=0
fail_http=0
fail_net=0

# Per-line fields
url=""
http_code=""
elapsed=""
rc=0
attempts=1

4.4 Algorithm Overview

Key Algorithm: Per-URL Check

  1. Read URL line.
  2. Skip empty or comment lines.
  3. Run curl with timeouts and capture http_code/time_total.
  4. If curl exit code non-zero, classify as FAIL_NET.
  5. Else classify by HTTP status class.
  6. Print report line.
  7. Update counters.

Complexity Analysis:

  • Time: O(n * t) where n = number of URLs, t = time per request.
  • Space: O(1) besides output.

5. Implementation Guide

5.1 Development Environment Setup

# Verify dependencies
which bash curl
curl --version

5.2 Project Structure

status-checker/
├── status-checker
├── urls.txt
├── README.md
└── tests/
    ├── fixtures/
    └── test_status_checker.sh

5.3 The Core Question You’re Answering

“How can I reliably determine whether a URL is reachable and healthy using only standard CLI tools?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. Shell quoting and exit codes (Concept 1)
  2. HTTP status code classes (Concept 2)
  3. curl timeouts and exit codes (Concept 2 and 3)

5.5 Questions to Guide Your Design

  1. Do you treat redirects as OK or WARN?
  2. Which curl exit codes should be retried?
  3. How will you ensure output remains parseable by scripts?
  4. What is your default timeout policy?

5.6 Thinking Exercise

Write down how your script should behave for these cases:

  • A URL that times out.
  • A URL that returns 404.
  • A URL that redirects to HTTPS.

Describe the output category for each and why.

5.7 The Interview Questions They’ll Ask

  1. “How do you distinguish network failures from HTTP failures in curl?”
  2. “Why do you use printf instead of echo in scripts?”
  3. “What is the purpose of a non-zero exit code?”
  4. “How would you make your checker deterministic for tests?”

5.8 Hints in Layers

Hint 1: Safe loop

while IFS= read -r url; do
  [ -z "$url" ] && continue
  case "$url" in \#*) continue;; esac
  # check url...
 done < "$input"

Hint 2: curl output

out=$(curl -s -o /dev/null -w "%{http_code} %{time_total}" --connect-timeout "$ct" --max-time "$mt" "$url")

Hint 3: classify

if [ $rc -ne 0 ]; then
  category="FAIL_NET"
elif [ "$code" -ge 200 ] && [ "$code" -lt 300 ]; then
  category="OK"
elif [ "$code" -ge 300 ] && [ "$code" -lt 400 ]; then
  category="WARN"
else
  category="FAIL_HTTP"
fi

5.9 Books That Will Help

Topic Book Chapter
Shell scripting basics The Linux Command Line Ch. 10-11
Networking basics How Linux Works Ch. 6
Practical scripts Wicked Cool Shell Scripts Ch. 1-2

5.10 Implementation Phases

Phase 1: Foundation (2 hours)

Goals:

  • Read input file safely
  • Print one status line per URL

Tasks:

  1. Implement safe read loop.
  2. Run curl with timeouts.
  3. Print raw code/time.

Checkpoint: Script prints code/time for each URL without crashing on spaces.

Phase 2: Core Functionality (3 hours)

Goals:

  • Add classification and summary
  • Add exit codes

Tasks:

  1. Classify HTTP status classes.
  2. Count categories and emit summary.
  3. Implement exit code policy.

Checkpoint: Summary counts are correct; exit code is 2 when any failure.

Phase 3: Polish & Edge Cases (2 hours)

Goals:

  • Deterministic output
  • CSV mode
  • Retry logic

Tasks:

  1. Add CSV header and formatting.
  2. Add SOURCE_DATE_EPOCH override.
  3. Implement retry on timeout.

Checkpoint: CSV output matches golden sample; retries work as expected.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Redirect handling Treat as OK or WARN WARN Redirects are notable but not fatal
Request type HEAD vs GET GET with -o /dev/null HEAD can be unreliable
Output format Text only vs CSV Both Human + machine usage

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests Validate parsing and classification status class mapping, skip comments
Integration Tests Run against known URLs example.com, invalid domain
Edge Case Tests Handle special inputs spaces, empty lines, timeout

6.2 Critical Test Cases

  1. Valid URL: https://example.com returns HTTP 200 and OK category.
  2. Redirect: http://example.org returns 301 and WARN category.
  3. DNS failure: https://no-such-domain.invalid yields FAIL_NET with rc=6.
  4. Timeout: non-routable IP triggers retry and then FAIL_NET.
  5. Input file missing: exits 1 and prints error to stderr.

6.3 Test Data

urls.txt
# comment
https://example.com
http://example.org
https://no-such-domain.invalid

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Unquoted URL curl sees multiple args Quote the variable
Mixed stdout/stderr CSV output corrupted Send errors to stderr
Missing timeout Script hangs Use --connect-timeout and --max-time

7.2 Debugging Strategies

  • Print curl exit codes after each request to understand failures.
  • Use set -x temporarily to trace script execution.

7.3 Performance Traps

  • Long timeouts on unreachable hosts create slow, misleading runs. Keep defaults conservative.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a --method HEAD option.
  • Add a --user-agent flag.

8.2 Intermediate Extensions

  • Add --json output mode.
  • Add per-category counters by domain.

8.3 Advanced Extensions

  • Add parallel checks with xargs -P and keep output ordered.
  • Persist results and compute deltas between runs.

9. Real-World Connections

9.1 Industry Applications

  • Uptime monitoring: similar logic powers internal health checks.
  • CI smoke tests: quick verification after deployments.
  • curl: core HTTP client used by this tool.
  • nagios-plugins: classic monitoring checks with similar output semantics.

9.3 Interview Relevance

  • Shell scripting: quoting, exit codes, error handling.
  • Networking basics: DNS, HTTP status codes, timeouts.

10. Resources

10.1 Essential Reading

  • The Linux Command Line by William E. Shotts - Ch. 10-11
  • How Linux Works by Brian Ward - Ch. 6

10.2 Video Resources

  • “HTTP Basics” lecture series (any reputable CS networking course)

10.3 Tools & Documentation

  • curl: man curl for --write-out, exit codes, and timeouts
  • bash: help set, man bash for quoting and exit status

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain why curl exit codes are not HTTP status codes.
  • I can describe how quoting affects shell arguments.
  • I can explain what --connect-timeout does.

11.2 Implementation

  • All functional requirements are met.
  • Output is deterministic with fixed time settings.
  • Exit codes match documented policy.

11.3 Growth

  • I can explain this tool in one paragraph without notes.
  • I can identify at least one design improvement.
  • I documented lessons learned.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Input file parsing works and ignores comments/blank lines.
  • Each URL produces a status line with code and time.
  • Script exits 2 if any URL failed.

Full Completion:

  • All minimum criteria plus:
  • Retry logic for timeouts works.
  • CSV output mode exists and matches the schema.

Excellence (Going Above & Beyond):

  • Output includes per-category summaries by domain.
  • Parallel checks complete with ordered output and deterministic logs.