Project 2: The Log File Cleaner

Build a sed-powered pipeline that extracts, normalizes, and reformats log lines into a clean, structured report.

Quick Reference

Attribute	Value
Difficulty	Level 1: Beginner
Time Estimate	6-12 hours
Main Programming Language	sed (shell pipeline)
Alternative Programming Languages	awk, Python, Perl
Coolness Level	Level 3: Useful in the Real World
Business Potential	3: The “Service & Support” Model
Prerequisites	Regex basics, shell pipelines, file I/O
Key Topics	capture groups, backreferences, filtering, `-n`/`p`/`d`

1. Learning Objectives

By completing this project, you will:

Extract structured fields from noisy log lines using capture groups.
Reformat log entries into consistent CSV/TSV output.
Filter logs by level or component using addresses and selective printing.
Normalize whitespace and escape characters safely with sed.
Build deterministic, testable log transformations for automation.

2. All Theory Needed (Per-Concept Breakdown)

2.1 Capture Groups and Backreferences

Fundamentals

Capture groups let you “remember” parts of a regex match so you can reuse them in the replacement. In sed, capture groups are created with $ ... $ in BRE or ( ... ) in ERE (with -E). The captured text is referenced in the replacement with \1, \2, and so on. This is what makes sed viable for log cleaning: you can isolate a timestamp, log level, and message, then reorder them or wrap them in quotes. Without capture groups, you would only be able to replace fixed strings. With capture groups, you can transform variable, structured data.

Remember that capture groups are numbered from left to right. If you nest parentheses or add new ones, every backreference index can shift. That is why a short comment or a stable pattern is essential when you rely on , , and in production scripts.

Deep Dive into the Concept

Log lines are semi-structured: they look like natural text but usually follow a consistent pattern. Example: 2026-01-01T12:00:00Z INFO api: request_id=abc123 user=42 action=login A capture-group regex can extract each piece: the timestamp, level, component, and key/value payload. In BRE syntax, you might write something like: ^$[0-9TZ:-]*$ $[A-Z]*$ $[^:]*$: $.*$$ Then you can reformat with \1\t\2\t\3\t\4. The key idea is that capture groups are positional: \1 refers to the first group, \2 to the second. If you change the pattern, you must update the backreferences. That is why a stable, explicit regex is critical. In this project, you will keep the pattern narrow and document it in the README so future readers understand what is being captured.

In sed, capture groups are part of the pattern, not the replacement. The replacement is a string where backreferences are expanded. This means you cannot apply regex operators like + or ? in the replacement; instead, you must design the pattern to capture exactly what you need. For logs, this often means using greedy .* at the end, but only after you’ve captured the fixed fields. A safe pattern starts with a strict prefix (timestamp + level), then uses .* for the message tail. This is also where anchors (^ and $) matter: without anchors, your pattern might match partial lines and lead to incorrect captures.

Another subtlety is that sed does not support named groups; everything is positional. To avoid confusion, write your regex with comments in your documentation or break your script into multiple simpler substitutions. For example, you can first isolate the timestamp and replace it with a placeholder, then capture the rest. But for a single-pass pipeline, a single well-documented capture regex is acceptable.

There is also a portability issue: GNU sed supports -r or -E for ERE, while BSD sed uses -E. BRE is more portable but more verbose because you must escape parentheses and +. In a cross-platform script, consider using -E with a detection fallback or stick to BRE and accept extra backslashes. This is an important engineering trade-off: readability vs portability.

When you design capture groups for logs, you should also think about optional fields. Real logs sometimes omit a component or include extra tokens. In sed, optional groups are tricky in BRE, so a pragmatic approach is to capture the known prefix strictly and then treat the rest of the line as a single message field. This keeps the parser robust. If you need richer parsing, you can run multiple passes: first classify lines, then apply a second substitution only to those that match a stricter pattern. This layered approach makes your regexes simpler and reduces the chance of catastrophic over-matching.

Testing capture groups is also about fixtures. Build a small log file with one example of each format you expect, then run your regex and verify the captured fields with sed -n and p. This practice prevents fragile patterns from slipping into production because you are forced to articulate the intended structure of the logs.

How This Fits on Projects

This project relies heavily on capture groups to extract log fields and reorder them into a normalized format. In Project 3, you will reuse backreferences to convert Markdown syntax into HTML tags.

Definitions & Key Terms

Capture group: A parenthesized subpattern that stores matched text.
Backreference: A reference to a captured group in the replacement (\1, \2).
BRE vs ERE: Basic vs Extended Regular Expressions; ERE is more readable.
Greedy match: A regex that consumes as much text as possible.

Mental Model Diagram (ASCII)

Match line -> capture groups -> replacement template -> output line

[1]=timestamp  [2]=level  [3]=component  [4]=message

How It Works (Step-by-Step)

sed matches the regex pattern against the entire line.
Each parenthesized group captures a substring.
The replacement string inserts those groups in a new order.
The result is printed (or used for further commands).

Minimal Concrete Example

# Convert "DATE LEVEL msg" into "LEVEL\tDATE\tmsg"
sed -E 's/^([0-9-]+) ([A-Z]+) (.*)$/\2\t\1\t\3/' logs.txt

Common Misconceptions

Misconception: Backreferences work the same in pattern and replacement.
- Correction: Backreferences only expand in the replacement.
Misconception: .* is always safe.
- Correction: Greedy matches can swallow delimiters if not anchored.
Misconception: BRE and ERE are equivalent.
- Correction: ERE changes escaping rules and supported operators.

Check-Your-Understanding Questions

What does \1 refer to in a replacement string?
Why is ^ important when capturing log fields?
What happens if you reorder capture groups but forget to update the replacement?
Which is more portable: BRE or ERE?

Check-Your-Understanding Answers

The text matched by the first capture group in the pattern.
It anchors the match to the start of the line, preventing partial captures.
The output fields will be swapped or incorrect.
BRE is more portable, but ERE is more readable.

Real-World Applications

Reformatting web server access logs into CSV for analysis.
Extracting error summaries from application logs.
Normalizing log output before ingestion into monitoring tools.

Where You’ll Apply It

In this project: See §3.4 Example Usage / Output and §5.4 Concepts You Must Understand First.
Also used in: P03-markdown-to-html-converter.md.

References

“sed & awk” – capture groups and substitutions
“Mastering Regular Expressions” – group mechanics
GNU sed manual – backreferences

Key Insight

Capture groups turn messy log lines into structured data that you can reorder and analyze.

Summary

Backreferences are the bridge between matching and restructuring; they are the heart of log cleaning.

Homework/Exercises to Practice the Concept

Capture a date like 2026-01-01 and move it to the end of the line.
Extract a LEVEL token and wrap it in brackets.
Swap two colon-separated fields.

Solutions to the Homework/Exercises

sed -E 's/^([0-9-]+) (.*)$/\2 \1/' file.txt
sed -E 's/^([A-Z]+) (.*)$/[\1] \2/' file.txt
sed -E 's/^([^:]+):([^:]+)/\2:\1/' file.txt

2.2 Filtering, Printing, and Deleting with Addresses

Fundamentals

Log files are noisy. You rarely need every line, and you certainly do not want to reformat lines that you plan to discard. sed provides three fundamental commands for this: p (print), d (delete), and addresses to decide when they apply. With -n, you disable automatic printing and take control over output, effectively turning sed into a filter. This is crucial for log cleaning because you can restrict output to only ERROR or WARN lines, or exclude health checks and noise. Addressing makes the filter precise and predictable.

Filtering is not just about selecting lines; it is about when you select them. If you filter too late, you waste work transforming lines you will discard. If you filter too early, you might remove context you still needed.

Deep Dive into the Concept

The default sed behavior is to print every line after processing. This is convenient for simple substitutions but dangerous in filters because it gives you output even when you wanted to discard lines. The -n option changes the contract: nothing prints unless you explicitly p. In a log cleaner, you want explicit output control so you can guarantee that only lines matching a certain predicate are emitted. For example, sed -n '/ ERROR /p' prints only lines containing ERROR. You can also combine this with substitutions: sed -n '/ ERROR /{s/.../.../;p;}' to both transform and emit.

The d command is the opposite: it deletes the pattern space and immediately starts the next cycle. This short-circuits the rest of the script. This means that d is a control-flow tool, not just a deletion. If you place d early in your script, later commands will never run on those lines. In a log cleaner, you can use d to skip noise lines quickly, which improves performance on large files. The key is to order commands so that you discard early and transform late.

Addresses can be line numbers, regex patterns, or ranges. For logs, regex addresses are the most useful. You can target specific levels (/\bERROR\b/) or components (/auth-service/). sed does not support word boundaries in BRE, so you often approximate them with [[:space:]]ERROR[[:space:]] or (^|[[:space:]])ERROR($|[[:space:]]) depending on the log format. This is another place where portability matters. If you rely on GNU sed extensions, your script may not run on macOS without gsed.

A powerful pattern is to invert selection: /DEBUG/ d (delete debug lines), then let everything else print. This is safe only if you understand the default printing behavior. Many beginners forget -n and end up printing both original and modified lines. For this project, you should adopt the discipline of -n plus explicit p so your output is intentional and testable.

Performance and correctness are intertwined. A filter that uses d early can skip expensive substitutions. For example, you can delete all DEBUG lines before running heavy capture-group substitutions, saving CPU on large logs. Another useful technique is q to quit early when you only need the first N matches. This is not required in this project but illustrates how control-flow commands influence performance in real pipelines.

Another practical filter pattern is to normalize first, then filter. If your log level tokens appear in inconsistent positions, a normalization pass can bring them into a consistent column, and a second pass can filter by that column. This two-stage approach is slower but more reliable when input is messy or inconsistent.

You can also combine filters by nesting address blocks: /ERROR/ { /auth/ { ...; p; } }. This keeps the logic explicit and avoids brittle regex alternations. It is slightly verbose, but it reads like a decision tree, which is often easier to maintain.

When in doubt, favor clarity over brevity. A slightly longer sed script that cleanly separates filtering from formatting is easier to validate and less likely to mis-handle a log line that deviates from the norm.

How This Fits on Projects

Filtering is a core requirement of the log cleaner. You will use -n and p to output only the lines you care about, and d to skip noise. This will show up in the CLI’s --level or --component flags. In Project 4, you will use addressing over multi-line blocks, which is a more advanced version of the same idea.

Definitions & Key Terms

-n: Suppress automatic printing.
p: Print the current pattern space.
d: Delete the current pattern space and start the next cycle.
Address: A selector that decides whether a command runs.

Mental Model Diagram (ASCII)

Line -> address? -> { transform; p } -> output
        \-> d -> (skip line)

How It Works (Step-by-Step)

sed -n suppresses default printing.
The address tests each line against a regex.
If it matches, the p command prints the transformed line.
If it matches a deletion rule, d skips the line and restarts the cycle.

Minimal Concrete Example

# Print only ERROR lines and remove a prefix
sed -n '/ ERROR /{s/^.*ERROR /ERROR /;p;}' app.log

Common Misconceptions

Misconception: d only deletes text, not control flow.
- Correction: d jumps to the next cycle, skipping remaining commands.
Misconception: -n is optional in filters.
- Correction: Without -n, you may print both filtered and unfiltered lines.

Check-Your-Understanding Questions

What happens if you use p without -n?
Why does d prevent later commands from running?
How can you print only WARN or ERROR lines?

Check-Your-Understanding Answers

You get duplicate output because sed prints automatically and p prints again.
d ends the current cycle immediately and starts the next.
Use a regex address like / (WARN|ERROR) / with ERE or two separate addresses.

Real-World Applications

Filtering high-severity logs before paging on-call engineers.
Removing debug noise from production log pipelines.
Extracting only performance-related entries for analysis.

Where You’ll Apply It

In this project: See §3.2 Functional Requirements and §5.5 Questions to Guide Your Design.
Also used in: P04-multi-line-address-parser.md for block filters.

References

GNU sed manual – -n, p, d
“sed & awk” – filtering patterns

Key Insight

Filtering is not about deleting text; it is about controlling which lines are allowed to exist in output.

Summary

-n plus explicit p and d give you full control over log pipeline output.

Homework/Exercises to Practice the Concept

Print only lines containing WARN.
Delete any line containing healthcheck.
Print only the last 5 lines of a file (hint: use tail + sed).

Solutions to the Homework/Exercises

sed -n '/WARN/p' app.log
sed '/healthcheck/d' app.log
tail -n 5 app.log | sed -n 'p'

2.3 Whitespace Normalization and Delimiter Control

Fundamentals

Logs often contain inconsistent spacing, tabs, or extra punctuation. Normalization is the process of making those lines consistent so downstream tools can parse them. sed can normalize whitespace with regex like [[:space:]]+ and can replace separators like | or : with tabs. This is essential when you want to output clean CSV/TSV. Because sed operates line by line, you can normalize each line as it flows through the pipeline, avoiding whole-file parsing.

Normalization is most valuable when you know the next tool in the pipeline. If you plan to use cut -f or awk -F ' ', then tabs are the right delimiter. Choose the delimiter first, then normalize to it.

Deep Dive into the Concept

Whitespace normalization looks simple but hides tricky cases. Consider lines that mix tabs and spaces or include multiple spaces inside quoted strings. A naive s/[[:space:]]\+/ /g collapses all whitespace, but it may also change the meaning of fields if the original format uses alignment or indentation for semantics. The right approach is to normalize only the delimiters you care about. For example, if your log format is TIMESTAMP LEVEL component: message, you can replace the first two spaces with tabs while leaving the message intact. That requires capture groups or selective substitutions.

Character classes are the portable way to match whitespace. [[:space:]] matches spaces and tabs across POSIX systems. [[:blank:]] matches only spaces and tabs, not newlines. These classes are more portable than \s, which is not POSIX in sed. For delimiter control, you can also normalize punctuation. For example, if a component is api: and you want api, you can strip a trailing colon with s/:[[:space:]]*/\t/. If logs include brackets ([INFO]), you can remove them with s/^\[//; s/\]//.

Normalization also relates to idempotence. A good log cleaner should be safe to run multiple times without further changes. This means your normalization rules should reach a stable form. For instance, replacing , with , is idempotent, but replacing , with , might keep adding spaces if you are not careful. Always test by running the pipeline twice on its own output; it should be unchanged.

Finally, normalization is not just cosmetic. It determines whether your output can be reliably parsed by tools like cut, awk, or csvkit. A clean tab-separated file with no ambiguous whitespace makes downstream tooling robust.

Delimiter normalization also interacts with escaping. CSV is deceptively complex because fields can contain commas or quotes. TSV avoids some of that complexity, which is why this project uses tabs. Still, you should think about how to handle tabs inside the message field. A simple strategy is to replace any tabs inside the message with spaces before output, ensuring the TSV remains parseable. This is also a good place to enforce determinism: always collapse repeated whitespace the same way so a second run yields identical output.

If your message field can include tabs or commas, normalize them explicitly. For TSV, replace tabs in the message with spaces; for CSV, wrap the message in quotes and escape existing quotes. Sed can do this with multiple substitutions, but you should document the behavior so consumers of the output know exactly how fields are encoded.

Finally, normalization should include trimming trailing whitespace. Trailing spaces are invisible but can break exact comparisons or downstream parsers. A simple s/[[:space:]]+$// at the end of your pipeline stabilizes the output and improves determinism for tests and diffs.

Another practical stabilization step is to add a header row explicitly and ensure it is identical every run. This gives you a stable schema for tests and for downstream tooling that expects a fixed column order.

If you later export to CSV, document how commas and quotes are handled. Even if you stay with TSV, writing this down clarifies your contract with users and prevents silent parsing bugs downstream.

How This Fits on Projects

You will normalize spacing and delimiters in this project to output a clean, consistent TSV or CSV. In Project 3, you will also normalize markup syntax before transforming it.

Definitions & Key Terms

Normalization: Converting variations into a standard representation.
Delimiter: A character that separates fields (\t, ,, |).
Character class: POSIX class like [[:space:]].
Idempotence: Running the transformation multiple times yields the same output.

Mental Model Diagram (ASCII)

Raw line -> normalize whitespace -> normalize delimiters -> structured line

How It Works (Step-by-Step)

Identify the separators in the input format.
Replace variable whitespace with a single delimiter.
Strip or replace punctuation around fields.
Confirm the output format is consistent and stable.

Minimal Concrete Example

# Collapse multiple spaces and convert first delimiter to tab
sed -E 's/[[:space:]]+/ /g; s/ /\t/2' app.log

Common Misconceptions

Misconception: Replacing all whitespace is always safe.
- Correction: It can alter fields that legitimately contain spaces.
Misconception: \s is portable in sed.
- Correction: Use [[:space:]] for POSIX portability.

Check-Your-Understanding Questions

Why is [[:space:]] preferred over \s?
How do you ensure normalization is idempotent?
What is the difference between [[:blank:]] and [[:space:]]?

Check-Your-Understanding Answers

[[:space:]] is defined by POSIX; \s is not.
Run the transformation twice and ensure the output is unchanged.
[[:blank:]] matches only spaces and tabs; [[:space:]] includes newlines.

Real-World Applications

Preparing logs for import into spreadsheets or databases.
Normalizing audit logs before ingestion by SIEM tools.
Cleaning output of CLI tools for consistent diffing.

Where You’ll Apply It

In this project: See §3.5 Data Formats / Schemas and §6 Testing Strategy.
Also used in: P03-markdown-to-html-converter.md for cleanup passes.

References

POSIX regex character classes
“sed & awk” – substitution and classes

Key Insight

Whitespace normalization is the difference between human-readable logs and machine-parseable logs.

Summary

Normalize only what you must, and always test idempotence.

Homework/Exercises to Practice the Concept

Convert multiple spaces to a single tab.
Remove brackets around [INFO] tokens.
Replace | delimiters with commas.

Solutions to the Homework/Exercises

sed -E 's/[[:space:]]+/\t/g' file.txt
sed -E 's/^\[//; s/\]//' file.txt
sed 's/|/,/g' file.txt

3. Project Specification

3.1 What You Will Build

You will build a CLI tool named log-clean that reads a log file and outputs a normalized TSV report. It will extract four fields from each matching line: timestamp, level, component, and message. It will filter by log level and optionally by component, and it will skip lines that do not match the expected format. The tool will not parse JSON logs; it is designed for plain-text logs with consistent structure.

3.2 Functional Requirements

Parse fields: Extract timestamp, level, component, message.
Filter by level: --level INFO|WARN|ERROR.
Filter by component: --component api (optional).
Normalize output: Emit TSV with a fixed header row.
Dry run: --dry-run prints to stdout; otherwise write to a file.
Exit codes: Distinct codes for usage errors and missing input files.

3.3 Non-Functional Requirements

Performance: Process 100k lines in under 2 seconds.
Reliability: Skip malformed lines but count them for reporting.
Usability: Provide clear CLI help and examples.

3.4 Example Usage / Output

$ ./log-clean --file app.log --level ERROR --out report.tsv

# report.tsv
timestamp	level	component	message
2026-01-01T12:00:00Z	ERROR	auth	login failed user=42

3.5 Data Formats / Schemas / Protocols

Input line format (expected):

TIMESTAMP LEVEL component: message

Example:

2026-01-01T12:00:00Z ERROR auth: login failed user=42

Output format (TSV):

timestamp	level	component	message
2026-01-01T12:00:00Z	ERROR	auth	login failed user=42

3.6 Edge Cases

Lines without a component (no : separator).
Lowercase levels (error vs ERROR).
Extra spaces or tabs between fields.
Lines that match level but not the full format.
Empty files.

3.7 Real World Outcome

You will have a reusable pipeline that can turn raw logs into a clean report.

3.7.1 How to Run (Copy/Paste)

cat > app.log <<'EOF'
2026-01-01T12:00:00Z INFO api: ok request_id=abc
2026-01-01T12:00:05Z ERROR auth: login failed user=42
2026-01-01T12:00:06Z DEBUG health: ping
EOF

./log-clean --file app.log --level ERROR --out report.tsv

3.7.2 Golden Path Demo (Deterministic)

Expected output contains exactly one ERROR line and a header row.

3.7.3 CLI Transcript (Success)

$ ./log-clean --file app.log --level ERROR --out report.tsv
Wrote 1 rows to report.tsv (skipped 0 malformed lines)
$ echo $?
0
$ cat report.tsv
timestamp	level	component	message
2026-01-01T12:00:05Z	ERROR	auth	login failed user=42

3.7.4 CLI Transcript (Failure: Missing File)

$ ./log-clean --file missing.log --level ERROR --out report.tsv
Error: input file missing.log not found
$ echo $?
2

Exit codes:

0 success
1 usage error
2 input file missing

4. Solution Architecture

4.1 High-Level Design

input.log -> sed filter/transform -> TSV report
        \-> malformed counter (grep -v or sed branch)

4.2 Key Components

Component	Responsibility	Key Decisions
CLI parser	Parse flags and validate inputs	Use explicit `--file` and `--level`
sed pipeline	Match, capture, reformat	Use `-E` for readable groups
output writer	Write TSV with header	Always write header once

4.3 Data Structures (No Full Code)

record = {timestamp, level, component, message}

4.4 Algorithm Overview

Key Algorithm: Log Line Normalization

Validate inputs and ensure file exists.
Use a regex with capture groups to parse fields.
Filter by level and optional component address.
Emit a header, then output TSV rows.

Complexity Analysis:

Time: O(n) over number of lines
Space: O(1) streaming

5. Implementation Guide

5.1 Development Environment Setup

printf '2026-01-01T12:00:00Z INFO api: ok\n' > app.log

5.2 Project Structure

log-clean/
├── bin/
│   └── log-clean
├── tests/
│   └── test-log-clean.sh
└── README.md

5.3 The Core Question You’re Answering

“How can I turn noisy log files into reliable, structured data using only sed?”

5.4 Concepts You Must Understand First

Capture groups and backreferences.
-n with explicit p and d for filtering.
Whitespace normalization and delimiters.

5.5 Questions to Guide Your Design

What is the exact log format you expect?
Which lines should be filtered out before transformation?
How will you handle malformed lines?
How will you ensure idempotence?

5.6 Thinking Exercise

Trace this line by hand:

2026-01-01T12:00:05Z ERROR auth: login failed user=42

Write down the four captured fields and the final TSV output.

5.7 The Interview Questions They’ll Ask

“How would you extract fields from a log line using sed?”
“Why use -n and p instead of relying on default printing?”
“How do you avoid corrupting lines that don’t match the format?”

5.8 Hints in Layers

Hint 1: Start with a regex that matches the full line and prints captured fields.

Hint 2: Add a guard so only matching lines print.

Hint 3: Add normalization steps to stabilize whitespace.

5.9 Books That Will Help

Topic	Book	Chapter
Regular expressions	“Mastering Regular Expressions”	Ch. 1-4
`sed` scripting	“sed & awk”	Ch. 4-6
Log processing patterns	“The Unix Programming Environment”	Ch. 7

5.10 Implementation Phases

Phase 1: Foundation (2-3 hours)

Goals:

Parse timestamp and level with capture groups.
Output TSV without filtering.

Tasks:

Write a capture regex for four fields.
Output TSV format with tabs.

Checkpoint: Output is correctly formatted for all valid lines.

Phase 2: Core Functionality (2-4 hours)

Goals:

Add filtering by level and component.
Add header output.

Tasks:

Use addresses to filter by level.
Add header row exactly once.

Checkpoint: Report contains only desired rows.

Phase 3: Polish & Edge Cases (1-3 hours)

Goals:

Skip malformed lines cleanly.
Ensure idempotence and stable output.

Tasks:

Add a counter for skipped lines.
Test on empty files.

Checkpoint: Exit codes and counts are correct.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Regex flavor	BRE vs ERE	ERE with `-E`	More readable for groups
Output format	CSV vs TSV	TSV	Avoid comma escaping
Filtering	`p` with `-n` vs default	`-n` + explicit `p`	Prevents accidental output

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Unit Tests	Validate parsing regex	Normal and malformed lines
Integration Tests	End-to-end report	Filtered output correctness
Edge Case Tests	Robustness	Empty file, missing file

6.2 Critical Test Cases

Valid line parses into four columns.
Lines with DEBUG are removed when filtering for ERROR.
Malformed line is skipped and counted.

6.3 Test Data

2026-01-01T12:00:00Z INFO api: ok
2026-01-01T12:00:05Z ERROR auth: login failed
malformed line

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Missing anchors	Partial matches	Use `^` and `$`
Using `\s`	Fails on BSD	Use `[[:space:]]`
Forgetting `-n`	Duplicate output	Always use `-n` + `p`

7.2 Debugging Strategies

Print only matches: sed -n '/ERROR/p' to confirm filtering.
Use small fixtures: Test with 3-5 lines first.
Echo the regex: Document the expected line format.

7.3 Performance Traps

Avoid multiple passes over the same file; combine substitutions when possible.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a --stdout flag for immediate printing.
Add a --no-header option.

8.2 Intermediate Extensions

Support multiple --level filters.
Add a --since filter for timestamps.

8.3 Advanced Extensions

Emit JSON lines instead of TSV.
Build a streaming mode that reads stdin continuously.

9. Real-World Connections

9.1 Industry Applications

Incident response: extract critical errors quickly.
Analytics: prepare logs for aggregation.

Fluent Bit: log pipeline agent with similar filtering concepts.
Logstash: heavy-duty log pipeline; this project mimics its early stages.

9.3 Interview Relevance

Regex mastery: shows you can manipulate text data safely.
Automation: demonstrates practical log handling with Unix tools.

10. Resources

10.1 Essential Reading

“sed & awk” – chapters on regex and substitutions
“Mastering Regular Expressions” – capture groups and backreferences

10.2 Video Resources

Regex capture groups walkthrough (video)

10.3 Tools & Documentation

GNU sed manual – regex and substitution behavior
POSIX regex reference

Project 1: Config File Updater – teaches safe single-line edits.
Project 3: Markdown to HTML – uses backreferences for tag generation.

11. Self-Assessment Checklist

11.1 Understanding

I can explain how capture groups work in sed.
I can explain why -n is important for filtering.
I can normalize whitespace without breaking fields.

11.2 Implementation

Output has correct TSV header and rows.
Filtering by level works correctly.
Malformed lines are skipped without breaking the run.

11.3 Growth

I can explain this pipeline to a teammate.
I can extend it to another log format.

12. Submission / Completion Criteria

Minimum Viable Completion:

CLI reads a log file and outputs TSV with a header.
Filtering by level works as specified.
Missing file errors are handled cleanly.

Full Completion:

All minimum criteria plus:
Malformed line counting and reporting.
Tests covering filters and edge cases.

Excellence (Going Above & Beyond):

Supports multiple formats with a config file.
Provides JSON output with deterministic ordering.