Project 5: City Mobility Data Pipeline

Build a typed, auditable Explorer pipeline that turns raw trip records into reconciled Parquet outputs without hiding rejected rows, join multiplication, or collection costs.

Quick Reference

Attribute	Value
Main language	Elixir
Comparison languages	SQL, Python
Primary tools	Livebook, Explorer, KinoExplorer, Parquet
Difficulty	Level 2 — Intermediate
Coolness	Level 3 — Very Cool
Business potential	Level 2 — Useful internal product
Estimated time	12–20 hours
Main book	Data Science for Business by Foster Provost and Tom Fawcett
Observable deliverables	Clean Parquet, quarantine Parquet, run manifest, reconciliation report

1. Learning Objectives

By the end of this project, you will be able to:

Declare a data contract for mobility trips and compare it with the schema inferred by a reader.
Separate invalid records into an explainable quarantine instead of silently dropping or repairing them.
Prove that a dimension join preserves the intended cardinality and population.
Keep transformations lazy until a deliberate, bounded collection boundary.
Reconcile source, accepted, and quarantined counts with explicit invariants.
Produce columnar outputs and a manifest that make a run reproducible and independently auditable.
Explain when a pipeline is transforming a plan, scanning data, materializing memory, or writing an artifact.

The project is complete only when the evidence is visible in the notebook. A correct-looking summary table is not proof that the upstream records were interpreted correctly.

2. Theoretical Foundation

2.1 Core Concepts

Declared and inferred dtypes. A reader can infer that a column resembles an integer, date, or string, but inference is an observation about a sample and parser rules—not a business contract. The pipeline needs a declared meaning for identifiers, UTC timestamps, monetary values, coordinates, and categorical fields. A zone identifier that happens to contain digits is still often an identifier, not a number to aggregate.

Lazy execution and collection boundaries. A lazy data frame represents a transformation plan. Operations can be combined and optimized without eagerly materializing every intermediate result. collect/1 is a semantic boundary: it asks the backend to execute the plan and return realized data. The important design question is not “should everything be lazy?” but “what is the smallest justified result that must cross into memory?”

Row reconciliation. Every input row must have a documented fate. For this lab, the central invariant is:

source_rows = accepted_rows + quarantined_rows

Rows repaired according to an explicit policy remain accepted, but the repair count and reason must be recorded separately. A silent filter breaks the invariant even if the final aggregate seems plausible.

Join cardinality. A trip-to-zone enrichment normally intends a many-to-one relationship: many trips may refer to one unique zone. Duplicate keys in the zone table turn that into many-to-many behavior and multiply trips. Validate dimension uniqueness before joining, then compare pre-join and post-join counts and report unmatched keys.

Columnar artifacts and manifests. Parquet preserves typed columns and supports projection and predicate pushdown in compatible readers. The manifest preserves run-level meaning: source identity, schema, policy version, counts, collection boundary, output checksums, and timestamps. The Parquet file is the data product; the manifest is its chain of custody.

2.2 Why This Matters

Analytical failures are frequently semantic rather than syntactic. A notebook can run successfully while parsing local timestamps as UTC, converting identifiers into numbers, multiplying rows in a join, or collecting millions of records too early. Those failures are dangerous because the output remains polished.

A mobility pipeline makes the risks concrete. Trip durations cross time zones and daylight-saving transitions. Zone metadata can contain duplicate or retired keys. Negative durations may indicate reversed timestamps, clock errors, or incorrect timezone assumptions. The disciplined response is to expose those decisions, not bury them in one transformation cell.

The same habits transfer to payments, logistics, telemetry, healthcare, and product analytics: contract-first ingestion, explicit reject paths, cardinality evidence, bounded materialization, and replayable artifacts.

2.3 Historical Context / Background

Early notebook workflows often loaded an entire CSV into process memory and then transformed it step by step. That was understandable for small exploratory datasets, but it blurred the distinction between a logical plan and physical execution. Modern dataframe engines increasingly use lazy plans, columnar formats, predicate pushdown, and query optimization so that the cost of analysis scales with the required columns and rows.

Explorer brings this model into Elixir. Its dataframe API can read tabular formats, build lazy transformations, and collect results at an intentional boundary. Parquet, standardized around a columnar layout, complements this approach by allowing readers to avoid scanning irrelevant columns and, depending on metadata and backend support, irrelevant row groups. Livebook contributes an executable narrative: assumptions, transformations, evidence, and artifacts remain next to one another.

2.4 Common Misconceptions

“The file loaded, so its schema is correct.” Loading proves parseability, not semantic correctness.
“A left join cannot increase row count.” It can when the right-side key is not unique.
“Lazy means no work ever happens.” Execution still occurs when an action requires data; laziness relocates and optimizes the boundary.
“Quarantining bad rows loses data.” A preserved quarantine with reasons is more recoverable than silent repair or deletion.
“Parquet guarantees reproducibility.” Reproducibility also requires stable inputs, policies, ordering rules, and manifest evidence.
“Matching row counts prove equivalence.” Values, types, ranges, null semantics, and checksums must also agree.

3. Project Specification

3.1 What You Will Build

Create a Livebook notebook that ingests a snapshot of city trip records plus a zone dimension, validates both against declared contracts, separates rejected rows, enriches accepted trips, computes service metrics lazily, and writes three artifacts:

mobility-clean.parquet — accepted, normalized, enriched trips.
mobility-quarantine.parquet — rejected records with one or more machine-readable reason codes.
mobility-run-manifest.json — source identity, policy version, schemas, reconciliation counts, join evidence, collection boundary, and output checksums.

The pipeline should compute a bounded daily_zone_summary before collection. Keep raw and accepted trip rows out of an eager notebook result unless a deliberately small sample is requested.

3.2 Functional Requirements

Accept a versioned trip snapshot and zone snapshot.
Display inferred dtypes next to the declared contract before transformation.
Normalize timestamps according to a documented timezone policy.
Validate required fields, coordinate ranges, duration rules, identifier formats, and categorical values.
Assign deterministic reason codes to every quarantined row.
Reconcile source, accepted, and quarantined counts.
Reject or stop on duplicate zone keys before enrichment.
Perform a many-to-one zone enrichment and report unmatched zone keys.
Produce daily per-zone counts, duration statistics, and service-quality rates.
Record the exact point where the lazy plan is collected and the number of rows collected.
Write clean and quarantine Parquet artifacts.
Calculate output checksums using a documented byte-level or canonical logical procedure.
Re-run the same immutable snapshot and obtain the same counts and checksums.

Suggested quarantine reason codes include missing_trip_id, invalid_timestamp, negative_duration, coordinate_out_of_range, unknown_vehicle_type, and unmatched_zone. Decide whether an unmatched zone is rejected or retained with an explicit sentinel; do not leave that behavior implicit.

3.3 Non-Functional Requirements

The notebook must remain usable with at least 2.4 million source rows.
No cell may render or retain an unbounded full dataset.
Each validation rule must be named, versioned, and observable in the manifest.
Output ordering used for logical checksums must be deterministic.
File paths, run identifiers, and timestamps must not make a replay checksum unstable unless they are intentionally part of the checksum contract.
Failure must be fail-closed: duplicate dimension keys, reconciliation mismatches, or output write failures prevent a success status.
The notebook must run from a fresh Livebook runtime using documented dependencies and inputs.

3.4 Example Usage / Output

The notebook’s final evidence panel should resemble this transcript:

CITY MOBILITY PIPELINE
Source rows: 2,400,000
Accepted rows: 2,372,814
Quarantined rows: 27,186
Row reconciliation: PASS
Zone join unmatched keys: 43
Collection boundary: daily_zone_summary (3,648 rows)
Output: mobility-clean.parquet
Manifest checksum: sha256:REDACTED_EXAMPLE

Add a validation table such as:

CHECK                         EXPECTED              OBSERVED             STATUS
trip_id dtype                 string                string               PASS
zone dimension uniqueness    duplicate keys = 0    0                    PASS
row reconciliation            2,400,000             2,400,000            PASS
post-join row count           2,372,814             2,372,814            PASS
collected summary rows        <= 5,000              3,648                PASS

This is illustrative output, not hard-coded expected data. Your fixture generator or supplied snapshot must define its own known truth.

3.5 Real World Outcome

A reviewer can open the notebook and answer, without inspecting hidden state:

What source snapshot was processed?
What schema did the reader infer, and where did it differ from the contract?
Why did each rejected row fail?
Did enrichment preserve the accepted trip population?
How many rows crossed the collection boundary?
Which exact artifacts were written, and how can their identity be verified?

The result is a small analytical data product rather than a one-off chart. Another operator can replay the run, verify the manifest, and use the Parquet outputs without trusting the original author’s memory.

4. Solution Architecture

4.1 High-Level Design

 Immutable trip snapshot                Versioned zone dimension
 +----------------------+               +------------------------+
 | CSV / Parquet        |               | zone_id, name, region  |
 | source id + checksum |               | unique-key contract    |
 +----------+-----------+               +------------+-----------+
            |                                            |
            v                                            v
 +----------------------+                    +-----------------------+
 | Inferred schema      |                    | Dimension validation  |
 | vs declared contract |                    | duplicates? null key? |
 +----------+-----------+                    +-----------+-----------+
            |                                            |
            v                                            |
 +----------------------+                                |
 | Parse + normalize    |                                |
 | timestamps / units   |                                |
 +----------+-----------+                                |
            |                                            |
            v                                            |
 +----------------------+     invalid     +--------------+---------+
 | Rule evaluation      +---------------->| Quarantine + reasons   |
 +----------+-----------+                 +--------------+---------+
            | accepted                                  |
            v                                           v
 +----------------------+                    mobility-quarantine.parquet
 | Many-to-one zone join|
 | pre/post count checks |
 +----------+-----------+
            |
            v
 +----------------------+      collect       +----------------------+
 | Lazy metric plan     +------------------->| daily_zone_summary   |
 | filter/group/summarize|   bounded rows     | evidence + preview   |
 +----------+-----------+                    +----------------------+
            |
            +-----------------------> mobility-clean.parquet
            |
            v
 +---------------------------------------------------------------+
 | Run manifest: source + policy + schemas + counts + checksums   |
 +---------------------------------------------------------------+

4.2 Key Components

Component	Responsibility	Evidence it emits
Source resolver	Select immutable trip and zone inputs	Logical name, path/reference, size, checksum
Contract registry	Define names, dtypes, nullability, units, timezone semantics	Contract version and schema table
Profiler	Measure nulls, distinctness, ranges, and parse failures	Before/after profile
Rule engine	Assign deterministic validation reasons	Counts by reason and sample identifiers
Quarantine splitter	Partition rows without loss	Accepted + quarantined reconciliation
Dimension guard	Enforce unique and non-null zone keys	Duplicate-key and null-key counts
Enricher	Add zone attributes	Pre/post count and unmatched-key evidence
Lazy metric planner	Build bounded aggregations	Plan description and intended boundary
Artifact writer	Write Parquet and manifest	Paths, sizes, checksums, write status
Evidence panel	Summarize invariants	PASS/FAIL table, never a cosmetic success banner

4.3 Data Structures

Use conceptual schemas rather than implementation-specific structs:

TripContract
  trip_id: string, required
  started_at: utc_datetime, required
  ended_at: utc_datetime, required
  pickup_zone_id: string, required
  dropoff_zone_id: string, optional
  vehicle_type: category, required
  distance_km: float, non-negative
  fare_minor_units: integer, non-negative

ValidationResult
  row_identity: string
  accepted: boolean
  reason_codes: ordered list<string>
  policy_version: string

RunManifest
  run_id, created_at, pipeline_version
  source identities and source checksums
  declared and observed schemas
  validation counts by reason
  source/accepted/quarantined reconciliation
  join uniqueness, unmatched, pre/post counts
  collection boundary name and row count
  output artifact checksums and sizes

If one row violates multiple rules, choose and document either all applicable reason codes or a stable precedence order. The same input and policy must produce the same reason representation.

4.4 Algorithm Overview

resolve immutable inputs
compare inferred schema with contract
normalize only according to explicit policies
evaluate validation predicates
partition rows into accepted and quarantine sets
assert source_count = accepted_count + quarantine_count
assert zone keys are unique
left-enrich accepted trips with zones
assert post_join_count = pre_join_count
measure unmatched keys and apply declared policy
build lazy daily-zone aggregation
collect only bounded summary
write clean and quarantine Parquet
write manifest after all prior checks pass
verify artifacts by reading metadata and recomputing checksums

For n trip rows and m zone rows, validation is approximately O(n). Dimension uniqueness is normally O(m) with hashing or O(m log m) with sorting. A hash join is expected near O(n + m) average time; grouped aggregation is near O(n) average time plus output ordering where required. Memory should be bounded by the execution backend and the deliberately collected summary, not by retaining every transformed row in rendered notebook state.

5. Implementation Guide

5.1 Development Environment Setup

Use a current Livebook release and versions of Explorer and KinoExplorer compatible with your Elixir/OTP runtime. Record the resolved versions in the notebook rather than copying a stale pin from this guide.

Setup sequence:

Create a fresh Livebook notebook with a reproducible dependency cell.
Attach the immutable trip and zone fixtures through notebook file references or documented local training paths.
Confirm the runtime can read and write Parquet in a temporary output directory.
Record runtime, dependency, operating-system, and source identities.
Run a small fixture first; only then run the 2.4-million-row snapshot.

Verification transcript:

runtime_ready: true
explorer_backend: recorded
trip_source_readable: true
zone_source_readable: true
parquet_round_trip_fixture: PASS
output_directory_writable: true

5.2 Project Structure

Organize the notebook as an executable report:

— Scope, safety, and success criteria
— Runtime and dependency versions
— Source resolution and identity
— Declared contracts
— Inferred schema and profiling
— Normalization policy
— Validation and quarantine
— Row reconciliation
— Zone dimension guard
— Enrichment and cardinality proof
— Lazy service-metric plan
— Bounded collection and visualization
— Parquet outputs
— Manifest and checksums
— Replay and failure tests

Keep source resolution, rule definitions, and reporting separate so that a UI change cannot silently modify data policy.

5.3 The Core Question You’re Answering

Where does an analytical pipeline actually execute, and what evidence proves no rows or meanings disappeared?

Annotate the notebook with execution boundaries. Identify which cells construct expressions, which trigger scans or aggregations, which materialize results, and which write persistent artifacts. Then associate each semantic risk—dtype, timezone, rejection, join cardinality—with an explicit piece of evidence.

5.4 Concepts You Must Understand First

Before implementing, be able to explain:

Why an identifier may need a string dtype even when every observed value is numeric.
The difference between a lazy dataframe plan and a collected dataframe.
Why a left join can multiply rows.
The difference between a rejected record and a missing record.
Why a byte checksum and a logical-content checksum answer different questions.
How Parquet projection and partitioning can reduce work without changing results.

Recommended reading: the Explorer DataFrame and introductory guides, plus the chapters on data representation and evaluation in Data Science for Business.

5.5 Questions to Guide Your Design

Which timestamp fields are UTC, localized, or timezone-naive at the source?
Is a negative duration always invalid, or can an upstream clock correction explain it?
Which errors are safe to normalize, which require quarantine, and which should abort the entire run?
Is an unmatched zone rejected, retained as unknown, or reported in a separate exception product?
What proves the zone dimension is valid for the trip snapshot’s effective date?
What is the maximum acceptable collected row count, and who sets it?
Should checksums cover file bytes, canonical sorted logical rows, or both?
Which manifest fields are operational metadata and therefore expected to differ across replays?

5.6 Thinking Exercise

Trace five adversarial rows through the system:

A valid trip crossing midnight UTC.
A trip with local timestamps around a daylight-saving transition.
A trip whose pickup zone occurs twice in the dimension.
A trip with a missing identifier and negative duration.
A valid trip whose zone was retired before the snapshot date.

For each row, write its dtype interpretation, normalization, validation reasons, final partition, join behavior, and manifest evidence. Then ask: could a reviewer distinguish “no matching zone” from “zone data failed to load”?

5.7 The Interview Questions They’ll Ask

What is lazy execution, and where would you place a collection boundary in this pipeline?
How do you detect and prevent accidental many-to-many joins?
Why is row-count reconciliation necessary but insufficient for data correctness?
How would you make a dataframe pipeline reproducible across dependency upgrades?
What information belongs in a data-product manifest?
How would you investigate a checksum mismatch when counts and schemas still match?

5.8 Hints in Layers

Hint 1 — Make the contract visible. Render declared and inferred schemas side by side before applying casts.

Hint 2 — Build reason expressions independently. Give every rule a stable identifier and count it before splitting the frame.

Hint 3 — Guard the dimension first. Abort enrichment if the intended unique key is duplicated; do not try to detect multiplication after publishing output.

Hint 4 — Aggregate before collection. Filter, group, and summarize while lazy; collect only the small daily-zone result.

Hint 5 — Test replayability. Use an immutable fixture, stable policy version, deterministic ordering, and a clearly defined checksum basis.

Pseudocode sketch:

observed = inspect_schema(source)
assert_compatible(observed, declared_contract)
classified = attach_reason_codes(normalize(source, policy))
accepted, quarantine = partition(classified)
assert_reconciled(source, accepted, quarantine)
assert_unique(zone_dimension, zone_id)
enriched = guarded_join(accepted, zone_dimension)
summary = enriched |> lazy_group_and_summarize |> collect_bounded
publish_only_if_all_assertions_pass()

5.9 Books That Will Help

Topic	Book	Suggested focus
Data contracts and analytical validity	Data Science for Business — Provost & Fawcett	Data, measurement, leakage, and evaluation
Reliable pipelines	Designing Data-Intensive Applications — Martin Kleppmann	Encoding, batch processing, derived data
Dimensional modeling	The Data Warehouse Toolkit — Ralph Kimball and Margy Ross	Fact grain, dimensions, slowly changing dimensions
Elixir data workflows	Elixir in Action — Saša Jurić	Immutable transformations and process/runtime fundamentals

5.10 Implementation Phases

Phase 1 — Contract and fixtures (2–3 hours)

Create a small deterministic fixture with known valid and invalid rows.
Declare trip and zone schemas, units, nullability, and timezone assumptions.
Define expected counts and known join behavior.

Phase 2 — Profiling and validation (3–4 hours)

Compare inferred and declared dtypes.
Add validation reason codes and the quarantine split.
Prove row reconciliation on fixtures and the full snapshot.

Phase 3 — Join and lazy metrics (3–5 hours)

Validate dimension uniqueness.
Add guarded enrichment and pre/post count checks.
Build lazy daily-zone metrics and enforce the bounded collection threshold.

Phase 4 — Artifacts and replay (3–5 hours)

Write Parquet outputs and manifest.
Re-read output metadata and compute checksums.
Replay from a fresh runtime and compare stable evidence.

Phase 5 — Failure demonstration (1–3 hours)

Introduce duplicate dimension keys, timezone defects, and premature collection.
Capture the expected failed checks and document the diagnosis path.

5.11 Key Implementation Decisions

Record these decisions in an Architecture Decision Record table inside the notebook:

Decision	Options	Required rationale
Timestamp policy	UTC-only, explicit source zone, reject naive	Prevent ambiguous duration calculations
Invalid-row policy	Reject, repair, abort run	Preserve meaning and traceability
Unmatched-zone policy	Quarantine, sentinel, separate exception	Avoid silent population changes
Checksum basis	Bytes, canonical logical rows, both	Define what “same output” means
Collection threshold	Fixed rows, fixed bytes, both	Protect runtime and browser
Output partitioning	Single file, date, region/date	Balance pruning and small-file costs

6. Testing Strategy

6.1 Test Categories

Use tests at four levels.

The suite combines contract fixtures, property and invariant checks, failure injection, and deterministic replay. Together they verify both the transformation result and the evidence needed to trust it.

6.2 Critical Test Cases

Contract fixtures

Case	Expected result
Numeric-looking string identifier	Remains string
Missing required trip ID	Quarantined with stable reason
End before start	Quarantined or repaired only by explicit policy
Coordinate outside valid bounds	Quarantined
Unknown optional category	Policy-defined sentinel or quarantine

Property and invariant tests

For every run, source = accepted + quarantined.
Every quarantined row has at least one recognized reason code.
Every accepted row satisfies every required predicate.
Zone uniqueness is proven before the join.
When dimension keys are unique, enrichment does not change accepted row count.
Collected summary rows never exceed the configured bound.

Failure-injection tests

Duplicate one zone key and verify publication stops before the join.
Remove a required column and verify a contract error, not a downstream null surprise.
Inject daylight-saving ambiguity and verify the timezone policy decides it explicitly.
Corrupt a Parquet output and verify its checksum fails.
Change rule ordering and confirm the chosen reason-code representation remains deterministic.
Force an unbounded collection request and verify the guard rejects it.

Replay test

Run from a fresh Livebook runtime twice with the same source snapshots and dependency lock. Compare declared/observed schemas, counts, reason distributions, join evidence, canonical logical checksums, and artifact metadata. Separate stable fields from expected run-specific fields such as wall-clock timestamps.

6.3 Test Data

Maintain a tiny hand-auditable fixture, a medium deterministic mobility snapshot, and deliberately malformed variants. Pin source checksums and include duplicate dimension keys, missing identifiers, invalid coordinates, timezone transitions, unknown categories, and reordered-but-logically-equivalent rows.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem: Trip count rises after zone enrichment

Likely cause: duplicate zone keys created a many-to-many join.
Evidence: dimension distinct-key count is lower than row count; duplicated key report is non-empty.
Fix: validate uniqueness before joining and resolve the dimension’s effective-date/grain problem.
Quick test: compare pre-join and post-join row counts on a fixture containing one deliberate duplicate.

Problem: The lazy pipeline consumes excessive memory

Likely cause: an early collect, rendered full dataframe, unbounded sort, or operation unsupported by the chosen lazy backend.
Evidence: memory rises at a specific action; plan boundary occurs earlier than documented.
Fix: project required columns, filter and aggregate first, and enforce a result bound before collection.
Quick test: run with increasing input sizes while the summary cardinality stays fixed.

Problem: Durations become negative near a timezone transition

Likely cause: naive local timestamps were interpreted as UTC or ambiguous wall times were normalized inconsistently.
Evidence: failures cluster around transition dates and a source region.
Fix: preserve source timezone metadata and define an ambiguity/nonexistent-time policy.
Quick test: include timestamps immediately before, during, and after the transition.

Problem: Replay checksum changes while metrics match

Likely cause: unstable row ordering, different file encoding metadata, floating-point formatting, or run timestamps included in the checksum.
Evidence: logical sorted rows match while raw file bytes differ.
Fix: distinguish byte identity from logical identity and canonicalize the latter.
Quick test: reorder equal input rows and compare both checksum types.

Problem: Reconciliation passes but results are still wrong

Likely cause: values were misparsed, repairs changed meaning, or accepted rows received incorrect dimensions.
Evidence: dtype/range/profile or referential-integrity checks fail despite equal counts.
Fix: treat reconciliation as one invariant among schema, domain, cardinality, and distribution checks.

7.2 Debugging Strategies

Debug from evidence boundaries: confirm source identity, inspect declared and inferred schemas, compare reason-code counts, prove dimension uniqueness, locate the collection boundary, and compare canonical logical rows before comparing physical file bytes. Change one policy or fixture condition at a time so the first diverging invariant remains visible.

7.3 Performance Traps

Watch for premature collection, rendering entire frames, unbounded sorts, full-dataset joins, and operations the lazy backend cannot push down. Measure row counts and memory at each boundary, and make maximum collected rows an executable assertion rather than a comment.

8. Extensions & Challenges

8.1 Beginner Extensions

Partition-aware ingestion: partition output by service date and show which partitions a seven-day summary reads.
Incremental run: process only new immutable partitions while preserving a global reconciliation ledger.

8.2 Intermediate Extensions

Data drift panel: compare dtype, null-rate, categorical, and quantile profiles against a baseline.
Slowly changing zones: resolve zone attributes by trip timestamp rather than current key alone.
Dual-engine validation: compute a small reference summary in SQL or Python and compare canonical results.

8.3 Advanced Extensions

Quality SLOs: fail or warn based on quarantined-row rate, unmatched-zone rate, and freshness.
Lineage graph: represent sources, transformations, and outputs as a small provenance DAG.
Artifact promotion: separate draft output from approved output and require all evidence gates before promotion.

For every extension, preserve the core invariants and add new acceptance tests. More features without stronger evidence are not an improvement.

9. Real-World Connections

9.1 Industry Applications

Transportation operations: demand planning, fleet balancing, service-zone performance, and regulatory reporting.
Financial data: transaction ledgers use the same reconciliation and quarantine patterns.
Data warehouses: fact/dimension grain and join cardinality are central to reliable marts.
Machine-learning features: dtype and temporal correctness prevent training/serving skew and leakage.
Data contracts: manifest and schema evidence form a lightweight producer-consumer agreement.
Incident response: deterministic replay and reason-coded rejects shorten diagnosis when a metric changes.

Explorer and Polars demonstrate dataframe planning and columnar execution; Apache Arrow and Parquet define interoperable in-memory and file representations; dbt and Great Expectations offer useful comparisons for contracts, tests, and lineage. Compare their guarantees without turning this notebook into a wrapper around them.

9.3 Interview Relevance

In production, this notebook would usually become a scheduled pipeline with artifact storage, catalog registration, alerting, access control, and lineage. The learning objective is to make those requirements visible before choosing an orchestration platform.

10. Resources

10.1 Essential Reading

Designing Data-Intensive Applications by Martin Kleppmann — data models, batch processing, and derived-data correctness.
Fundamentals of Data Engineering by Joe Reis and Matt Housley — lifecycle, orchestration, governance, and operational quality.

10.2 Video Resources

Prefer current Livebook and Explorer conference demonstrations that show plans, collection boundaries, and notebook artifacts. Reproduce the behavior against the pinned fixture instead of copying a presentation’s implementation.

10.3 Tools & Documentation

Official documentation

Standards and deeper reading

Verify package compatibility against current Hex documentation when you begin; versions evolve independently.

11. Self-Assessment Checklist

11.1 Understanding

I can distinguish inferred dtypes from a declared business contract.
I can point to the exact operations that trigger execution or materialization.
I can prove that every source row has a documented fate.
I can explain why the zone join is many-to-one and show evidence that it stayed that way.
I can identify the collection boundary and justify its maximum size.

11.2 Implementation

I can explain the timezone and invalid-row policies without reading implementation details.
I can reproduce the clean/quarantine counts from a fresh runtime.
I can distinguish raw-file and logical-content checksums.
I can diagnose duplicate-key, schema-drift, DST, and premature-collection failures.
I can hand the manifest to another operator and have them verify the run independently.

11.3 Growth

I can add one new validation rule without weakening reconciliation or replayability.
I can explain which parts belong in a production scheduler, catalog, or quality service.

12. Submission / Completion Criteria

Submit the Livebook notebook, immutable training fixtures or resolvable fixture references, clean Parquet, quarantine Parquet, manifest, and a short replay report.

The submission is complete when:

This expansion belongs to Livebook + Elixir Deep Dive. Return to the expanded project index for the rest of the sprint.