Project 13: Pretty Printer / Code Formatter

Format AWK code with consistent indentation and spacing.

Quick Reference

Attribute Value
Difficulty Level 4: Advanced
Time Estimate 2-3 weeks
Main Programming Language AWK
Alternative Programming Languages Python
Coolness Level Level 4: Hardcore
Business Potential 2: Developer utility
Prerequisites Regex, state machines
Key Topics tokenization, indentation, strings/comments

1. Learning Objectives

  1. Build the core functionality described in the project.
  2. Apply AWK patterns, arrays, and formatting to real data.
  3. Handle errors and edge cases with deterministic output.
  4. Produce a CLI tool with clear usage and tests.

2. All Theory Needed (Per-Concept Breakdown)

Lexical Scanning

Fundamentals

Lexical Scanning is the core idea behind tokens and states. You use it to control how records are interpreted and how results are produced. In AWK, this concept is implemented with small, composable rules and arrays.

Deep Dive into the concept

Lexical Scanning matters because AWK processes data as a stream. You rarely have random access to records, so you must build small state machines that evolve line by line. This means your logic must be explicit about what is remembered, when it is updated, and how output is shaped. When you apply Lexical Scanning, you decide which data to keep in memory, how to identify important events, and how to emit deterministic results. The best AWK tools make these choices visible and testable. You should also consider edge cases: missing fields, malformed input, and ambiguous patterns. A robust implementation validates assumptions early, logs errors to stderr, and keeps output stable even with imperfect input.

For this project, Lexical Scanning interacts with other parts of the pipeline. It is not isolated: it affects how you parse fields, how you update aggregates, and how you format output. The deeper you understand it, the easier it becomes to reason about bugs, performance, and correctness.

How this fit on projects

It is a primary building block in the implementation and influences the overall behavior.

Definitions & key terms

  • Core term -> A concise definition related to Lexical Scanning
  • Invariant -> A property you expect to remain true
  • Failure mode -> A way the logic can break

Mental model diagram

input -> apply Lexical Scanning -> update state -> emit output

How it works (step-by-step)

  1. Read a record and extract required fields.
  2. Apply Lexical Scanning logic to update state.
  3. Decide whether to emit output.
  4. Repeat for all records, then summarize.

Minimal concrete example

# sketch
{ state[$1]++; if ($2 ~ /pattern/) print $0 }

Common misconceptions

  • “Lexical Scanning is only about syntax.” (It is about data flow and state.)
  • “Streaming means no structure.” (You can build structure with arrays.)

Check-your-understanding questions

  1. What state do you need to track for Lexical Scanning?
  2. What breaks if input does not match your assumptions?
  3. How do you make output deterministic?

Check-your-understanding answers

  1. Track minimal state required for correctness.
  2. You get missing fields, wrong matches, or unstable output.
  3. Use explicit ordering and fixed formatting.

Real-world applications

  • Production log processing
  • Data cleaning pipelines

Where you’ll apply it

  • See §5.4 and §5.10 Phase 2.

References

  • The AWK Programming Language
  • Effective awk Programming

Key insights

Lexical Scanning is a policy decision: correctness depends on explicit rules.

Summary

Lexical Scanning provides the structure needed to build reliable CLI tools.

Homework/Exercises to practice the concept

  1. Implement a tiny example that uses Lexical Scanning.
  2. Add a validation check to prevent a common failure mode.

Solutions to the homework/exercises

# simple validation example
if ($1=="") { print "missing key" > "/dev/stderr"; exit 2 }

Indentation and Formatting Rules

Fundamentals

Indentation and Formatting Rules is the core idea behind brace tracking. You use it to control how records are interpreted and how results are produced. In AWK, this concept is implemented with small, composable rules and arrays.

Deep Dive into the concept

Indentation and Formatting Rules matters because AWK processes data as a stream. You rarely have random access to records, so you must build small state machines that evolve line by line. This means your logic must be explicit about what is remembered, when it is updated, and how output is shaped. When you apply Indentation and Formatting Rules, you decide which data to keep in memory, how to identify important events, and how to emit deterministic results. The best AWK tools make these choices visible and testable. You should also consider edge cases: missing fields, malformed input, and ambiguous patterns. A robust implementation validates assumptions early, logs errors to stderr, and keeps output stable even with imperfect input.

For this project, Indentation and Formatting Rules interacts with other parts of the pipeline. It is not isolated: it affects how you parse fields, how you update aggregates, and how you format output. The deeper you understand it, the easier it becomes to reason about bugs, performance, and correctness.

How this fit on projects

It is a primary building block in the implementation and influences the overall behavior.

Definitions & key terms

  • Core term -> A concise definition related to Indentation and Formatting Rules
  • Invariant -> A property you expect to remain true
  • Failure mode -> A way the logic can break

Mental model diagram

input -> apply Indentation and Formatting Rules -> update state -> emit output

How it works (step-by-step)

  1. Read a record and extract required fields.
  2. Apply Indentation and Formatting Rules logic to update state.
  3. Decide whether to emit output.
  4. Repeat for all records, then summarize.

Minimal concrete example

# sketch
{ state[$1]++; if ($2 ~ /pattern/) print $0 }

Common misconceptions

  • “Indentation and Formatting Rules is only about syntax.” (It is about data flow and state.)
  • “Streaming means no structure.” (You can build structure with arrays.)

Check-your-understanding questions

  1. What state do you need to track for Indentation and Formatting Rules?
  2. What breaks if input does not match your assumptions?
  3. How do you make output deterministic?

Check-your-understanding answers

  1. Track minimal state required for correctness.
  2. You get missing fields, wrong matches, or unstable output.
  3. Use explicit ordering and fixed formatting.

Real-world applications

  • Production log processing
  • Data cleaning pipelines

Where you’ll apply it

  • See §5.4 and §5.10 Phase 2.

References

  • The AWK Programming Language
  • Effective awk Programming

Key insights

Indentation and Formatting Rules is a policy decision: correctness depends on explicit rules.

Summary

Indentation and Formatting Rules provides the structure needed to build reliable CLI tools.

Homework/Exercises to practice the concept

  1. Implement a tiny example that uses Indentation and Formatting Rules.
  2. Add a validation check to prevent a common failure mode.

Solutions to the homework/exercises

# simple validation example
if ($1=="") { print "missing key" > "/dev/stderr"; exit 2 }

3. Project Specification

3.1 What You Will Build

A CLI tool that implements: format awk code with consistent indentation and spacing.

3.2 Functional Requirements

  1. Parse input files and validate required arguments.
  2. Perform the core transformation or analysis.
  3. Emit deterministic output with stable ordering.
  4. Provide helpful errors with non-zero exit codes.

3.3 Non-Functional Requirements

  • Performance: Stream input without loading entire files.
  • Reliability: Deterministic output for identical inputs.
  • Usability: Clear help text and examples.

3.4 Example Usage / Output

$ ./tool --help

3.5 Data Formats / Schemas / Protocols

  • Input: line-based or delimited text
  • Output: line-based text or structured output

3.6 Edge Cases

  • Empty input
  • Missing fields
  • Invalid arguments

3.7 Real World Outcome

3.7.1 How to Run (Copy/Paste)

chmod +x tool
./tool input.txt

3.7.2 Golden Path Demo (Deterministic)

# input
sample 1
sample 2

./tool input.txt

sample 1
sample 2

3.7.3 Failure Demo (Deterministic)

$ ./tool --bad-flag
error: invalid argument
exit=2

3.7.4 Exit Codes

  • 0 success
  • 2 invalid arguments
  • 3 input unreadable

4. Solution Architecture

4.1 High-Level Design

input -> parse -> transform -> output

4.2 Key Components

Component Responsibility Key Decisions
Parser Read and validate input Fail fast on invalid data
Core Logic Apply transformations Keep state minimal
Output Format results Deterministic ordering

4.3 Data Structures (No Full Code)

state[key] = value

4.4 Algorithm Overview

  1. Read records.
  2. Update state.
  3. Emit results in END.

5. Implementation Guide

5.1 Development Environment Setup

awk --version | head -1

5.2 Project Structure

project/
├── tool
├── tool.awk
└── tests/

5.3 The Core Question You’re Answering

“How do I parse and reformat code using only text tools?”

5.4 Concepts You Must Understand First

  1. AWK patterns and actions
  2. Arrays and state
  3. Formatting output

5.5 Questions to Guide Your Design

  1. What is the minimal state needed to solve the problem?
  2. How will you validate inputs and handle errors?
  3. How will you make output deterministic?

5.6 Thinking Exercise

Sketch the data flow from input to output on paper.

5.7 The Interview Questions They’ll Ask

  1. How does AWK stream input?
  2. How do you avoid non-deterministic output?

5.8 Hints in Layers

Hint 1: Start with a single-rule AWK script.

Hint 2: Add state arrays for tracking.

Hint 3: Add validation and error handling.

5.9 Books That Will Help

Topic Book Chapter
AWK basics The AWK Programming Language Ch. 1-3

5.10 Implementation Phases

Phase 1: Foundation

  • Parse input and print basic output

Phase 2: Core Logic

  • Implement stateful processing

Phase 3: Polish

  • Add tests and CLI help

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
State storage arrays vs temp files arrays simpler and faster
Ordering input vs sorted input predictable for users

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit core logic state updates
Integration CLI output full pipeline
Edge error cases invalid args

6.2 Critical Test Cases

  1. Empty input
  2. Invalid arguments
  3. Deterministic output ordering

6.3 Test Data

sample 1
sample 2

7. Common Pitfalls & Debugging

  • Forgetting to validate input fields
  • Non-deterministic array iteration order

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a --verbose mode

8.2 Intermediate Extensions

  • Add configurable output formats

8.3 Advanced Extensions

  • Optimize for very large files

9. Real-World Connections

9.1 Industry Applications

  • Log processing and reporting
  • ETL and data preparation
  • GNU coreutils text tools

9.3 Interview Relevance

  • Streaming algorithms and data tooling

10. Resources

10.1 Essential Reading

  • The AWK Programming Language
  • Effective awk Programming

10.2 Video Resources

  • AWK tooling tutorials

10.3 Tools & Documentation

  • GNU awk manual

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain the data flow and state
  • I can describe how output is made deterministic

11.2 Implementation

  • All requirements are implemented
  • Tests cover edge cases

11.3 Growth

  • I can explain this project in an interview

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Core CLI works on fixtures

Full Completion:

  • Tests pass and output is deterministic

Excellence (Going Above & Beyond):

  • Extra features and robust validation

13. Additional Content Rules (Hard Requirements)

13.1 Determinism

  • Use fixed fixtures and explicit ordering.

13.2 Outcome Completeness

  • Provide success and failure demos with exit codes.

13.3 Cross-Linking

  • Link to related projects in §10.4.

13.4 No Placeholder Text

All sections are complete and concrete.