Project 18: Fixed-Width Settlement Parser and Property-Test Lab

Parse a realistic header/detail/trailer settlement format with binary patterns, validate cross-record invariants, and use generated adversarial inputs to prove the parser fails safely.

Quick Reference

Attribute	Value
Difficulty	Level 2 - Intermediate/Developer
Suggested Seniority	Mid-level
Time Estimate	12-18 hours
Main Programming Language	Elixir
Alternative Programming Languages	Erlang, Gleam
Coolness Level	Level 4 - Hardcore Tech Flex
Business Potential	Level 2 - Micro-SaaS/Pro Tool
Prerequisites	Pattern matching, guards, structs, ExUnit basics
Key Topics	Binaries/bitstrings, fixed-width records, recursive state, invariants, StreamData, shrinking

1. Learning Objectives

By completing this project, you will:

Use binary pattern matching and size modifiers to recognize fixed-width record shapes without unsafe indexing.
Separate lexical field extraction, semantic field validation, and file-level settlement invariants.
Accumulate line-aware errors while preserving valid records and deterministic totals.
Design StreamData generators for valid files and targeted corruptions.
State properties that prove the parser does not crash, does not accept a false trailer, and round-trips canonical records.

2. All Theory Needed (Per-Concept Breakdown)

Binary Contracts and Property-Based Parser Verification

Fundamentals

Elixir binaries are sequences of bytes, while bitstrings may contain a number of bits that is not divisible by eight. Binary pattern matching can split a record into fields of declared byte sizes and can require literal tags such as “H”, “D”, and “T”. This makes a fixed-width protocol’s shape visible in a function head or conceptual decoder, but size alone does not prove meaning. Decimal fields, dates, encodings, padding, signs, and cross-record totals require separate validation. Property-based testing complements example tests by generating many valid and invalid files. A generator describes structured inputs; a property states an invariant; shrinking searches for a smaller counterexample when the invariant fails. The goal is not random volume. It is to connect the grammar of the settlement format to generators and to prove safety and accounting rules across a broad input space.

Deep Dive into the concept

Begin with a written protocol contract. For this project, each physical record is 80 bytes excluding the line terminator. The first byte is a tag: H for one header, D for zero or more details, T for one trailer. Define exact byte ranges, ASCII padding, numeric encoding, date format, amount scale, and reserved bytes. State whether CRLF and LF are accepted and normalize line endings before record decoding without trimming significant field padding.

Binary patterns are a strong shape gate. A conceptual detail pattern can match the tag, settlement id, transaction id, date bytes, sign byte, amount digits, currency, reference, and reserved tail with exact sizes. An unmatched 79-byte or unknown-tag record returns a structural error containing line number, observed length, and safe prefix. Avoid conversions in the shape pattern. Extraction yields raw field binaries; semantic validators then strip allowed padding and convert values with explicit results.

Byte width is not character width for arbitrary UTF-8. Fixed-width financial formats commonly define an ASCII or single-byte encoding. Declare ASCII for the exercise and reject bytes outside the allowed range in textual fields. If UTF-8 were accepted, a visual character might occupy multiple bytes and violate positions. This is a protocol decision, not an implementation detail.

Money again uses integer minor units. Parse a sign plus fixed decimal digits into an integer; do not parse a decimal float. Reject non-digits and values exceeding the domain limit. Dates should become Date values only after exact YYYYMMDD validation. Reserved fields must be blank or match the specification; accepting arbitrary reserved bytes hides format evolution.

File-level structure is a small deterministic state machine. Before the header, only one header is allowed. After the header, details may accumulate. After the trailer, no records are allowed. The parser state contains header identity, detail count, signed amount total, seen transaction ids, detail records or a streaming sink, and errors. A trailer is valid only if its settlement id matches the header and its declared count/total match accumulated values. A structurally valid trailer with false totals must never produce an accepted file.

Decide fail-fast versus error accumulation. Structural corruption can make field positions unknown, but because physical records remain line-delimited, the parser can report that line and continue to discover independent errors. File acceptance still fails if any error exists. Duplicate transaction ids, mismatched currency, invalid dates, and unexpected post-trailer records become explicit stable error codes.

Property tests begin with a generator for valid domain structs, not arbitrary byte strings. A valid settlement generator creates one header, a list of unique details, and a trailer derived from the list. A renderer produces bytes. Core properties include: rendering then parsing yields the canonical domain value; parsed detail count and total equal generated values; parsing never raises for any binary; changing one trailer digit cannot be accepted as the original valid settlement; reordering records outside grammar is rejected.

Targeted corruption generators are more valuable than uniform random bytes. Mutations can truncate a record, replace a digit with a letter, flip the tag, duplicate an id, change a trailer count, alter a sign, insert a second header, or append a record after the trailer. Label/classify cases so test output proves each corruption family was exercised.

Shrinking must preserve enough structure to reach meaningful bugs. If a generated file with 1,000 details fails, StreamData should reduce the list and field values until perhaps one detail and a bad trailer remain. Overly manual opaque binary generators may shrink into unrelated garbage. Generate structured records, render late, and apply named corruptions so counterexamples remain understandable.

Example tests still matter. They document exact offsets, boundary values, and error messages. Properties add universal claims across generated ranges. A parser can pass “never crashes” while accepting everything, so pair safety properties with semantic ones. Likewise, “round trips” can pass if renderer and parser share the same bug; independent golden protocol fixtures and direct assertions on byte positions prevent circular confidence.

The invariants are: one header appears first; one trailer appears last; accepted records have exact width/tag; ids satisfy uniqueness; dates and currencies are valid; integer totals and counts match the trailer; any error prevents accepted-file status; arbitrary binary input never crashes the public parser. Failure modes include off-by-one sizes, byte/character confusion, trimming before width validation, floating money, integer conversion exceptions, duplicate overwrites, state reset after malformed records, false trailers, generators that never create edge cases, and properties too weak to detect acceptance bugs.

How this fits in the project

Binary patterns implement structural recognition, explicit validation turns fields into domain values, a recursive state enforces sequence/totals, and property tests attack both shape and semantics.

Definitions & key terms

Binary: Byte-aligned bitstring, commonly used for file/protocol data.
Binary segment: Sized or typed portion extracted by a pattern.
Structural validation: Checking record length, tag, and field boundaries.
Semantic validation: Checking decoded meaning such as a real date or allowed currency.
Property: General statement expected to hold over generated inputs.
Shrinking: Reducing a failing generated case to a smaller counterexample.

Mental model diagram

bytes -> physical lines -> exact-width/tag match -> raw fields
                                                    |
                                               field validators
                                                    |
             +---------------- recursive file state +----------------+
             | header seen | detail count/total | ids | trailer seen |
             +-------------------------+-------------------------------+
                                       |
                         accepted settlement OR errors

structured valid generator -> renderer -> parser -> property assertions
           |
     named corruption -> parser must reject without crashing

How it works

Normalize accepted line endings without stripping field padding.
Match record length and tag.
Extract exact byte segments.
Validate digits, dates, signs, currency, text encoding, and reserved bytes.
Transition the file parser state according to header/detail/trailer grammar.
Accumulate detail count, signed minor-unit total, and ids.
Validate trailer and final state.
Test golden examples and generated valid/corrupted files.

Minimal concrete example

grammar:
  file = header, zero-or-more details, trailer, end-of-input

parser state after detail:
  count = previous count + 1
  total = previous total + signed amount
  ids = ids plus transaction id, unless duplicate error

trailer acceptance:
  declared id == header id
  declared count == accumulated count
  declared total == accumulated total
  no previous errors

Common misconceptions

Exact width does not imply valid field content.
String length and byte size are not interchangeable under UTF-8.
A property named “does not crash” does not prove rejection correctness.
Renderer/parser round-trip alone can preserve the same defect on both sides.

Check-your-understanding questions

Why validate width before trimming padding?
Why generate valid structs before rendering binaries?
Which property stops a false trailer from being accepted?

Check-your-understanding answers

Padding is part of the positional protocol; trimming first destroys evidence of incorrect width.
Structured generators shrink meaningfully and guarantee known-valid inputs.
Derived trailer totals must equal parsed accumulated totals, and targeted total mutations must always reject.

Real-world applications

Payment settlements, mainframe feeds, ACH-like files, logistics manifests, telecom CDRs, government exchanges, and legacy integrations still use positional records.

Where you will apply it

Apply the concept to record patterns, validators, recursive grammar state, totals, error records, generators, corruptors, and properties. This does not overlap P3’s KV store or P12’s distributed reconciliation; it is a local protocol parser and verification laboratory.

References

Key insight

A binary pattern makes format structure executable, while properties make the parser’s safety and accounting promises executable.

Summary

This project teaches precise byte-level modeling, validation layers, recursive accumulation, stable errors, and a testing style that finds counterexamples beyond curated examples.

Homework/Exercises

Specify byte offsets for a detail record and identify every semantic validator.
Design five named corruptions and the reason each must produce.
Explain how to test arbitrary bytes without letting one test consume unbounded resources.

Solutions

Keep an offset table beside the implementation; validate digits, sign, date, currency, ASCII text, and reserved bytes after extraction.
Truncate, bad digit, duplicate id, bad trailer total, and post-trailer detail map to structural or invariant errors.
Bound generated binary size and parser line/record limits while asserting the public API returns rather than raises.

3. Project Specification

3.1 What You Will Build

Build “settlecheck”, a library and CLI for an invented but realistic 80-byte header/detail/trailer format. It prints accepted settlement totals or line-specific structural, field, sequencing, and trailer errors.

Scope boundary: Implement only the versioned fixture specification. Do not claim compatibility with a real payment network, process money, persist records, distribute parsing, or build a GenStage pipeline.

3.2 Functional Requirements

Parse LF and CRLF files with 80-byte records.
Recognize H/D/T tags and exact fields through binary patterns.
Decode dates, signed integer money, currencies, references, and reserved bytes.
Enforce header/detail/trailer grammar, id uniqueness, currency consistency, counts, and totals.
Return all safe line-aware errors without raising for malformed input.
Render valid domain fixtures and provide generated valid/corrupt tests.

3.3 Non-Functional Requirements

Public parse never raises for any bounded binary.
Error codes and line numbers are deterministic.
Money remains integer minor units.
Parsing documents whether details are retained or folded into a summary.
Limits prevent pathological line length/record count.

3.4 Example Usage / Output

$ mix settlecheck fixtures/settlement-ok.txt
settlement_id=20260715A records=4 details=2
currency=BRL declared_total=125050 calculated_total=125050
status=accepted

$ mix settlecheck fixtures/settlement-bad.txt
line=3 code=invalid_amount field=amount value="00A01000"
line=5 code=trailer_count_mismatch declared=3 calculated=2
status=rejected errors=2

3.5 Data Formats / Schemas / Protocols

The project includes a versioned field table with offset, byte size, encoding, padding, and rule for every H/D/T segment. Domain structs include Header, Detail, Trailer, Settlement, ParseError, and ParseSummary.

3.6 Edge Cases

Empty file, header only, trailer without header, second header, details after trailer.
79/81-byte records, unknown tags, non-ASCII text, bad padding.
Zero/negative/max amounts, impossible dates, unsupported currency.
Duplicate transaction ids, overflow policy, false counts/totals.
Missing final newline and mixed line endings.

3.7 Real World Outcome

3.7.1 How to Run

Run golden valid/invalid fixtures, then execute focused property tests with a reproducible seed. Preserve the smallest counterexample when deliberately injecting a parser bug.

3.7.2 Golden Path Demo

A valid file with two details is accepted. Corrupted variants for each field and file invariant are rejected with exact codes. Generated arbitrary binaries never raise.

3.7.3 Exact Terminal Transcript

$ mix test test/settlecheck_property_test.exs --seed 4242
property valid settlements round-trip
property arbitrary bounded binaries never crash
property corrupted trailer totals are rejected
property duplicate ids are rejected
4 properties, 0 failures

$ mix settlecheck fixtures/bad-trailer.txt
line=4 code=trailer_total_mismatch declared=120000 calculated=125050
status=rejected

4. Solution Architecture

4.1 High-Level Design

file bytes -> line framing -> record shape decoder -> field validators
                                                       |
                                                recursive file parser
                                                       |
                                           settlement OR ordered errors

generators -> domain records -> renderer -> parser -> properties
      \-> named corruptions -----------------------> rejection properties

4.2 Key Components

Versioned format specification
Physical line framer and record dispatcher
Header/detail/trailer field decoders
Recursive file-state reducer
Domain renderer for valid fixtures
StreamData generators/corruptors
CLI and error renderer

4.3 Data Structures

Use domain structs, a ParserState with phase/count/total/ids/errors, and stable ParseError values containing line, record tag, field, reason, and safe observed data.

4.4 Algorithm Overview

Frame lines, decode each record, validate fields, transition parser state, accumulate invariants, finalize at EOF, and return either accepted Settlement/Summary or deterministically ordered errors.

5. Implementation Guide

5.1 Development Environment Setup

Use supported Elixir/OTP, ExUnit, and StreamData. Commit the invented protocol specification before code so tests and implementation share an external contract, not hidden assumptions.

5.2 Project Structure

lib/settlecheck/
  format_v1
  record_decoder
  validators
  parser_state
  parser
  renderer
  cli
test/
  fixtures/
  generators/
  properties/

5.3 The Core Question You’re Answering

How can binary patterns and properties prove that a parser both accepts valid settlement data and rejects malformed or dishonest files without crashing?

5.4 Concepts You Must Understand First

Understand binary versus bitstring, byte_size, sized binary segments, guards, integer/date conversion results, recursive state, ExUnit assertions, generators, properties, classification, and shrinking.

5.5 Questions to Guide Your Design

Which checks are structural, field-semantic, or file-semantic?
Can parsing continue safely after a malformed record?
What does accepted status require at EOF?
Which properties are independent of the renderer?
How will generators exercise each corruption family?

5.6 Thinking Exercise

Write an 80-column ruler and place every detail field under it. Manually change one byte at each boundary and predict the outcome. Then draw the parser phases before-header, details, and after-trailer with valid/invalid transitions.

5.7 The Interview Questions They’ll Ask

How does binary pattern matching differ from string slicing?
Why separate extraction from semantic validation?
How would you avoid exceptions from integer conversion?
What makes a good property versus a large example test?
How does shrinking help diagnose failures?
Why can round-trip tests preserve a shared encoder/decoder bug?

5.8 Hints in Layers

Hint 1: Parse one exact detail record into raw fields.

Hint 2: Give every field validator one stable result/error contract.

Hint 3: Make file sequence and totals one explicit parser state.

Hint 4: Generate valid structs first, derive trailers, render late, and mutate one named rule at a time.

5.9 Books That Will Help

Topic	Book	Chapter
Binaries and pattern matching	“Programming Elixir >= 1.6” by Dave Thomas	Ch. 6, Modules and Named Functions; Ch. 11, Strings and Binaries
Testing discipline	“Test Driven Development: By Example” by Kent Beck	Part I, The Money Example; Ch. 25, Test-Driven Development Patterns
Integration contracts	“Enterprise Integration Patterns” by Hohpe and Woolf	Ch. 3, Messaging Systems; Ch. 4, Messaging Channels

5.10 Implementation Phases

Phase 1: Foundation (4-5 hours)

Write format table, domain structs, exact record decoders, validators, and golden examples.

Phase 2: Core Functionality (4-6 hours)

Implement parser state, totals/sequence/duplicate invariants, CLI output, and domain renderer.

Phase 3: Polish & Edge Cases (4-7 hours)

Add structured generators, named corruptions, shrinking-friendly properties, parser limits, and performance checks.

5.11 Key Implementation Decisions

Document encoding, line endings, width basis, money scale/sign, padding, reserved fields, error accumulation, detail retention, numeric limits, duplicate policy, and property run counts.

6. Testing Strategy

6.1 Test Categories

Exact offset/decoder unit tests
Table-driven field validator tests
File grammar/invariant tests
Golden protocol fixtures
Valid-domain round-trip properties
Named corruption rejection properties
Arbitrary bounded-binary safety property

6.2 Critical Test Cases

Every field boundary accepts exact width and rejects off-by-one records.
Non-ASCII bytes in ASCII fields reject.
Duplicate ids never disappear through a map overwrite.
False trailer count/total reject.
Detail after trailer rejects.
Arbitrary bounded input never raises.
Generated zero-detail valid settlement follows the specified policy.

6.3 Test Data

Maintain a field-boundary fixture matrix, valid one/two/many-detail files, each grammar violation, numeric boundaries, and generated structured records. Record the seed for any failure.

6.4 Definition of Done

The 80-byte format is documented independently of code.
Structural, semantic, and file-level errors are distinct.
Accepted totals/counts exactly match the trailer.
Public parsing never raises for bounded arbitrary binaries.
Valid settlements round-trip canonically.
Targeted corruptions are classified and rejected.
Smallest failing seeds are reproducible.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Offsets drift: Fields were trimmed/split before width matching. Match exact bytes first.

Unicode surprise: Character length was used for a byte protocol. Declare and enforce encoding.

Parser crashes: Bang conversions were used on untrusted fields. Convert through tagged results.

Properties pass trivially: Generator never produces meaningful data or parser accepts everything. Classify cases and assert semantics.

7.2 Debugging Strategies

Print a byte-offset ruler with safe hex for the failing record. Preserve line, tag, and field in every error. Replay property failures with seed and shrunk counterexample before adding another example.

7.3 Performance Traps

Repeated binary concatenation, copying entire files, retaining all details when only summary is needed, and unbounded generators cause waste. Parse line by line and choose retention deliberately.

8. Extensions & Challenges

8.1 Beginner Extensions

Add JSON error output, a format-explain command, and annotated byte rulers.

8.2 Intermediate Extensions

Support a second version selected by header, checksum fields, streaming summaries, and schema-driven fixture generation.

8.3 Advanced Extensions

Add stateful properties for chunked input, differential tests against an independent parser, mutation testing, and formal parser-state transition coverage.

9. Real-World Connections

9.1 Industry Applications

Bank settlement files, clearing systems, logistics manifests, EDI, telecom records, and government batch interfaces demand strict positional parsing and audit-quality errors.

StreamData illustrates generator composition and shrinking. OTP binary syntax documentation shows the runtime primitives. Real network specifications can inspire rigor but must not be claimed as supported.

9.3 Interview Relevance

This project demonstrates a deeper command of Elixir patterns, binary semantics, validation architecture, recursive state, accounting invariants, and property-based verification.