Project 8: Portable Cloud Data Lake Lab

Build one Livebook notebook that produces equivalent bounded summaries from a local snapshot and an S3-compatible object source while exposing resolution, caching, partition pruning, freshness, and secret use.

Quick Reference

Attribute	Value
Main language	Elixir
Comparison languages	SQL, Python
Primary tools	Livebook file references, Kino.FS, Explorer, S3-compatible storage
Difficulty	Level 3 — Advanced
Coolness	Level 3 — Very Cool
Business potential	Level 3 — Strong internal product
Estimated time	16–26 hours
Observable deliverables	Portable notebook, source manifests, equivalence report, offline/failure matrix
Core constraint	Same downstream contract for local and object-backed sources

1. Learning Objectives

By the end of this project, you will be able to:

Distinguish a logical notebook file reference from the runtime-specific path or object operation that resolves it.
Design one source adapter contract for local snapshots and S3-compatible data.
Use columnar layout and date partitioning to avoid reading an entire dataset for a bounded question.
Treat paths returned by notebook file resolution as read-only inputs.
Compare local and remote sources using schema, row counts, ranges, partitions, and canonical summary checksums.
Use Livebook secret names without exposing their values.
Explain what Livebook stamps approve, what invalidates them, and why they are not a sandbox.
Represent unavailable, unauthorized, stale-cache, corrupt, and partial-read states honestly.
Replay the notebook from a fresh runtime and a second environment without hard-coded machine paths.

Portability is not “the notebook opened elsewhere.” It is the ability to resolve equivalent data, apply the same contract, obtain equivalent evidence, and explain any deliberate environmental differences.

2. Theoretical Foundation

2.1 Core Concepts

Logical references and resolved resources. A notebook should refer to an input by a stable logical name such as mobility_snapshot, not by one author’s absolute path. Livebook file entries can resolve that logical name to an uploaded file, local file, URL-backed resource, or shared file storage. Kino.FS.file_path/1 may need to download and cache a resource before returning a local path, so resolution can take time. The returned path must be treated as read-only: depending on source and runtime, it may refer to an original file or a cache.

Source adapters. Local and S3-backed data should enter the same downstream analytical contract: selected partitions, logical schema, snapshot identity, read statistics, and a dataframe or lazy plan. Source-specific credential and resolution behavior belongs behind the adapter boundary. Analytical cells should not branch on path syntax or reconstruct cloud URLs.

Columnar formats and partition pruning. Parquet groups values by column and stores metadata that compatible readers can use for projection and predicate pushdown. Object datasets commonly add directory-style partitions such as date=2026-07-01/. If the selected window spans seven dates, the source layer should enumerate or resolve only those partitions and project only required columns. Pruning is evidence, not an assumption: report selected partitions, object/file count, and bytes read or estimated.

Caches and freshness. A cache changes availability and semantics. A cached snapshot may permit analysis during an outage, but it is not current merely because it is readable. Cache metadata must include source identity, object version/checksum, fetch time, coverage, and validation status. Fallback should be explicit and labeled; never silently substitute yesterday’s local file for today’s object source.

Secrets, file storages, and stamps. Shared secrets let notebooks refer to secret names while keeping values outside notebook source. Shared file storages can provide S3-compatible access across a Livebook team and its app servers; Git-backed shared storage is read-only. Livebook stamps can retain approved permissions for secrets and file systems. Modifying a stamped notebook or receiving it from another deployment can revoke those permissions. A stamp authorizes declared Livebook resources; it does not prove the notebook code is safe or sandbox arbitrary runtime authority.

2.2 Why This Matters

Notebook portability often fails at the boundary between code and environment. An analysis works on one Mac because it embeds an absolute path, relies on a preexisting download, or uses ambient cloud credentials. Another operator opens the same notebook and sees a missing file, a full-bucket scan, or—worse—a different snapshot with no warning.

This project turns the environment boundary into visible data. The source panel states which logical input was requested, how it resolved, which partitions were selected, which object version or checksum was observed, whether a cache was used, how old it is, how many bytes were read, and whether the source is read-only. The same analytical pipeline then computes a canonical bounded summary from local and object-backed variants.

These practices matter for reproducible research, incident notebooks, data science, regulated analytics, team handoff, and cloud-cost control.

2.3 Historical Context / Background

Early notebooks commonly assumed a local filesystem and a single user. As datasets moved into object stores, notebooks gained cloud SDKs and remote URLs, but this often scattered credentials and provider-specific logic through analysis cells. Object stores also changed performance assumptions: opening many small objects and downloading whole files can dominate computation.

Columnar formats such as Parquet and table layouts organized by partitions let analytical engines read less data. Filesystem abstraction libraries and notebook-managed file references make location less central to analysis code. Livebook’s file entries, shared file storages, secrets, and stamps address collaboration and deployment concerns, while Explorer supports tabular formats and remote paths through compatible filesystem specifications.

The modern goal is not location transparency at any cost. Location, version, authorization, caching, and bytes read remain operationally important; they should be abstracted from transformation logic but exposed as evidence.

2.4 Common Misconceptions

“A resolved file path is a permanent local file I can modify.” It may be the original or a temporary/cache path; treat it as read-only.
“S3-compatible means every provider behaves identically.” Authentication, addressing, consistency, versioning, and metadata support can differ.
“Parquet automatically prevents full scans.” Pruning requires appropriate filters, layout, metadata, and reader/backend support.
“A cache hit means current data.” It only means a previously fetched artifact is available.
“Matching row counts prove local and remote equivalence.” Schema, ranges, nulls, versions, partitions, and canonical summaries also matter.
“A secret name is safe to print because it is not the value.” Secret names are usually lower sensitivity, but still reveal system structure; display only the minimal approved names.
“A stamp makes arbitrary notebook code safe.” It records resource permission approval, not sandboxing or code trust.
“No hard-coded path means the notebook is portable.” Dependencies, credentials, source identity, and fallback semantics must also travel.

3. Project Specification

3.1 What You Will Build

Create a Livebook notebook that reads the same seven-day mobility dataset through two modes:

Local snapshot mode: a notebook-managed file/directory reference representing an immutable training snapshot.
S3-compatible mode: a training bucket or shared file storage containing equivalent date-partitioned Parquet objects.

Both modes must return one SourceResult contract and feed the same downstream transformation. The notebook will select the date range 2026-07-01 through 2026-07-07, project a fixed subset of columns, compute a bounded daily/zone summary, and compare the modes.

Provide a source panel containing logical reference, resolved source kind, version/checksum, cache status and age, selected partitions, object/file count, bytes read or best available estimate, and read-only status. Include an explicitly declared local-snapshot fallback for object-store unavailability; when used, label it CACHED_FALLBACK and never CURRENT.

3.2 Functional Requirements

Source contract

Accept a source selector: local_snapshot or s3_training.
Resolve by logical file/storage reference, never an author-specific absolute path.
Return source metadata plus a dataframe/lazy plan with the same logical schema.
Treat all resolved paths and object inputs as read-only.

Partition and projection behavior

Map the selected UTC date range to exact expected partitions.
Refuse unexpected gaps, duplicates, or out-of-range partitions unless a declared partial policy applies.
Read only required analytical columns.
Report selected vs available partitions and read-size evidence.

Equivalence

Compare normalized schemas and nullability expectations.
Compare selected partition set, row count, time range, key domain/profile, and canonical summary checksum.
Explain any metadata expected to differ, such as physical path or provider-specific object ETag.

Secrets and permissions

Refer to a training credential through a Livebook secret/shared-storage configuration.
Never render, log, export, or include secret values in an exception.
Show only an approved secret-name list in evidence.
Document stamp state and resource permissions without treating them as a security proof.

Fallback and failures

Distinguish unavailable, unauthorized, missing partition, stale cache, corrupt object, and partial read.
Allow fallback only when configured and validated against an immutable source identity.
Surface fallback source age and coverage.

3.3 Non-Functional Requirements

The same downstream cells must run unchanged for both source modes.
No absolute user-machine paths may appear in executable notebook configuration.
A seven-day query must not scan/download the full training dataset when pruning is supported.
Cache metadata and source evidence must be bounded and reproducible.
No secret values may appear in notebook source, output, logs, manifests, or screenshots.
Resolution and downloads must have explicit progress/loading and terminal failure states.
A fresh runtime must not depend on an undocumented warm cache.
A second environment must reproduce the summary using documented source configuration.
Every output must say whether it is current remote data, validated local snapshot, or cached fallback.

3.4 Example Usage / Output

The equivalence report should resemble:

PORTABILITY CHECK
Local source summary checksum:  8f0d...c21
S3 source summary checksum:     8f0d...c21
Schema equivalence: PASS
Row-count equivalence: PASS
Selected partitions: date=2026-07-01..2026-07-07
Rows collected: 4,218
Secret names used: [LB_TRAINING_S3_ACCESS] (values never displayed)

The source evidence panel might show:

logical_reference: mobility_training
source_mode: s3_training
source_state: CURRENT
object_version_manifest: sha256:example-redacted
selected_partitions: 7 / 365
projected_columns: 6 / 24
objects_read: 7
bytes_read_or_estimated: 41.8 MiB
cache_status: MISS
resolved_input_policy: READ_ONLY

When the object store is disabled:

requested_source: s3_training
remote_status: UNAVAILABLE
fallback: local_snapshot
fallback_status: CACHED_FALLBACK
snapshot_age: 26 hours
coverage: 2026-07-01..2026-07-07
current_claim: false

3.5 Real World Outcome

Another operator can open the notebook on a different machine, choose a configured source, and obtain the same bounded summary without editing analytical cells. They can tell whether the result came from current object storage, an immutable local snapshot, or a stale fallback.

A reviewer can independently verify:

logical and physical source identities;
selected partitions and projected columns;
schema, row-count, range, and checksum equivalence;
cache/fallback state and freshness;
absence of hard-coded paths and credential values;
reproducibility from a clean runtime and second environment.

The notebook becomes a portable analytical method with a visible data-access contract, not a machine-specific script.

4. Solution Architecture

4.1 High-Level Design

                  +--------------------------------+
                  | Source selector + date window  |
                  | local_snapshot | s3_training   |
                  +----------------+---------------+
                                   |
                                   v
                  +--------------------------------+
                  | Logical source adapter         |
                  | validate request + partitions  |
                  +-----------+--------------------+
                              |
             +----------------+----------------+
             |                                 |
             v                                 v
 +---------------------------+     +----------------------------+
 | Livebook file reference   |     | Shared S3-compatible store |
 | Kino.FS resolution        |     | secret/file storage config |
 | may use local cache       |     | object/version metadata    |
 +-------------+-------------+     +--------------+-------------+
               |                                  |
               +----------------+-----------------+
                                v
                  +--------------------------------+
                  | SourceResult contract          |
                  | schema + identity + freshness  |
                  | partitions + bytes + lazy data |
                  +----------------+---------------+
                                   |
                                   v
                  +--------------------------------+
                  | Shared downstream pipeline     |
                  | project -> filter -> summarize |
                  +----------------+---------------+
                                   |
                        bounded collect (4,218 rows)
                                   |
                 +-----------------+-----------------+
                 |                                   |
                 v                                   v
 +-------------------------------+   +-------------------------------+
 | Source evidence panel         |   | Canonical equivalence report  |
 | current/cache/fallback status |   | schema/count/range/checksum   |
 +-------------------------------+   +-------------------------------+

 Remote failure --> explicit state --> optional declared snapshot fallback
                                      (always labeled cached, never current)

4.2 Key Components

Component	Responsibility	Evidence
Source selector	Choose logical mode and validated date window	Normalized request
Local resolver	Resolve notebook file entry with Kino.FS	Logical name, resolved kind, checksum, cache status
Object resolver	Resolve S3-compatible partitions through approved configuration	Bucket/storage alias, object manifest/version, selected objects
Partition planner	Map range to partition keys and check completeness	Expected, selected, missing, duplicate partitions
Projection planner	Choose only required columns	Requested vs available columns
Source adapter	Produce one common `SourceResult`	Schema, identity, freshness, read statistics, lazy plan
Shared analyzer	Apply one pipeline to either source	Bounded canonical summary
Equivalence checker	Compare logical evidence	Schema/count/range/checksum report
Fallback controller	Decide whether validated snapshot may substitute	Explicit reason, age, coverage, non-current label
Secret/stamp panel	Report names and permission state safely	No values; permission status and limitations

4.3 Data Structures

SourceRequest
  mode: local_snapshot | s3_training
  logical_reference
  window_start_date, window_end_date
  required_columns
  allow_snapshot_fallback: boolean

SourceIdentity
  source_kind
  logical_reference
  immutable_snapshot_id or object_manifest_checksum
  coverage_start, coverage_end
  observed_at

SourceResult
  identity: SourceIdentity
  normalized_schema
  selected_partitions
  missing_partitions
  selected_object_count
  bytes_read_or_estimated + measurement_kind
  cache_state + cached_at
  freshness_state
  read_only: true
  data: common lazy dataframe contract

EquivalenceEvidence
  schema_equal
  row_count_equal
  timestamp_range_equal
  selected_partition_set_equal
  domain_profile_equal
  canonical_summary_checksum_equal
  physical_differences: documented list

The object ETag is not universally a content hash, especially for multipart uploads or provider-specific behavior. Prefer an explicit immutable version ID or a manifest checksum you control.

4.4 Algorithm Overview

normalize source request and date range
derive exact partition keys
resolve selected source through adapter
verify immutable identity, coverage, and partition completeness
project required columns and apply date predicate
normalize source schema to common contract
build shared lazy aggregate
collect bounded summary
compute canonical summary checksum
emit source evidence

for equivalence mode:
  execute above for local and object sources
  compare schema, partitions, counts, ranges, profiles, checksum
  report every physical difference separately

on object failure:
  classify unavailable / unauthorized / corrupt / partial
  if fallback allowed and matching snapshot validates:
    use it with CACHED_FALLBACK and age/coverage
  otherwise fail closed

For p selected partitions and n rows within them, partition planning is O(p), schema validation is proportional to columns, and aggregation is approximately O(n) plus grouping overhead. The central performance objective is for remote I/O to depend on selected partitions and projected columns rather than the full dataset. Equivalence checksum work is O(s log s) if canonical summary rows require sorting, where bounded s is the collected summary size.

5. Implementation Guide

5.1 Development Environment Setup

Use a current Livebook release and mutually compatible Kino and Explorer versions. Record resolved versions rather than relying on exact versions printed in this guide.

Produce a synthetic mobility dataset partitioned by UTC service date in Parquet.
Store an immutable local copy through a Livebook file reference.
Upload the same partition objects and an explicit manifest to a disposable S3-compatible training bucket or configured Livebook shared file storage.
Create a training-only credential with read access limited to the required prefix.
Add the credential through Livebook secrets or the shared file-storage configuration; never paste it into a notebook cell.
Verify a one-partition read, selected-column projection, and metadata capture.
Clear or isolate the cache and repeat to prove cold-start behavior.

Expected setup evidence:

local_logical_reference: configured
s3_training_storage: configured
training_credential_scope: read-only prefix
manifest_partitions: 365
single_partition_cold_read: PASS
secret_value_visible: false
resolved_path_write_attempt: not part of notebook behavior

5.2 Project Structure

— Portability contract and threat model
— Runtime and dependency versions
— Secret names, file storage, and stamp explanation
— Dataset schema and partition manifest
— SourceRequest and SourceResult contracts
— Local notebook-file adapter
— S3-compatible adapter
— Partition planner and projection policy
— Shared lazy analytical pipeline
— Bounded collection and summary checksum
— Source evidence panel
— Local-vs-object equivalence report
— Cache and fallback behavior
— Unauthorized/unavailable/corrupt/partial tests
— Fresh-runtime and second-environment replay

Do not let analytical cells call Kino.FS.file_path or construct object URLs directly. They should receive the common adapter result.

5.3 The Core Question You’re Answering

What must remain stable for an analytical method to move between a local file, object storage, and another operator’s machine?

Your answer should separate stable logical contracts—schema, selected snapshot, partitions, transformations, canonical result—from expected environmental differences—resolved path, cache location, network latency, and provider metadata.

5.4 Concepts You Must Understand First

Before implementation, explain:

The difference between a logical file reference and a resolved local path.
Why Kino.FS.file_path/1 can involve a download and why the result is read-only.
How column projection and partition pruning reduce remote work.
Why an ETag is not always a portable content checksum.
How a cache can be valid but stale.
What secret references and Livebook stamps do and do not guarantee.
Why identical row counts do not prove source equivalence.

5.5 Questions to Guide Your Design

What immutable identity binds the local snapshot to the object dataset?
Which timezone determines a date=... partition?
Is the end of the selected date range inclusive or exclusive?
How does the adapter report a missing partition versus an empty partition?
Which columns are required for the shared summary, and can the reader prove projection?
What evidence is available for actual bytes read versus only estimated bytes?
When is fallback permitted, and how old may it be?
What happens if a cache entry has the right logical name but the wrong manifest checksum?
How are provider differences isolated without erasing useful source evidence?
Which notebook modification invalidates a stamp, and how should the operator recover safely?
Can another environment use a different storage provider while preserving the same SourceResult contract?

5.6 Thinking Exercise

Trace the lifecycle of mobility_training through five situations:

A cold local-file resolution.
A warm local cache hit.
A successful seven-partition S3 read.
An unauthorized S3 request with a valid local snapshot available.
A modified notebook whose prior secret/file permissions are no longer approved.

For each, record logical reference, physical resolution, credential involvement, cache state, source identity, freshness, terminal state, and whether analysis may proceed. Then decide whether an unauthorized remote request should automatically fall back. Defend your answer in terms of fail-closed behavior and operator visibility.

5.7 The Interview Questions They’ll Ask

How would you make a notebook portable across local and object-backed data?
What is the difference between partition pruning and filtering after a full read?
Why should a resolved notebook file path be treated as read-only?
How do you prove two physical sources represent the same logical snapshot?
What metadata is required for a trustworthy cached fallback?
What security guarantees do secret stores and notebook permission stamps provide—and not provide?

5.8 Hints in Layers

Hint 1 — Define the adapter contract first. Make downstream code depend on SourceResult, not paths or buckets.

Hint 2 — Use a manifest. Bind partitions, sizes, schemas, and content checksums to one immutable snapshot identity.

Hint 3 — Derive partitions from the validated window. Do not list the whole bucket and filter filenames after download.

Hint 4 — Project early. Request only the six columns required by the summary.

Hint 5 — Compare canonical outputs. Sort the bounded summary by stable keys and normalize representation before hashing.

Hint 6 — Fail visibly. Never turn an authorization error into a quiet cache hit.

Pseudocode sketch:

request = validate_source_request(control_state)
partitions = partition_plan(request.window)
result = adapter_for(request.mode).resolve(request, partitions)
assert_identity_and_completeness(result)
summary = result.data |> shared_lazy_pipeline |> collect_bounded
evidence = summarize_source_and_checksum(result, summary)

if remote failure and explicit fallback policy permits:
  fallback = validate_local_snapshot_against_expected_manifest()
  mark_state(fallback, CACHED_FALLBACK, not_current=true)

5.9 Books That Will Help

Topic	Book	Suggested focus
Storage and data systems	Designing Data-Intensive Applications — Martin Kleppmann	Encoding, batch data, replication, derived data
Data engineering	Fundamentals of Data Engineering — Reis and Housley	Storage abstraction, data lifecycle, orchestration
Cloud architecture	Building Evolutionary Architectures — Ford, Parsons, Kua	Fitness functions and portability tradeoffs
Reproducible analysis	The Practice of Reproducible Research — Kitzes, Turek, Deniz	Provenance, environment, artifacts

5.10 Implementation Phases

Phase 1 — Dataset and immutable manifest (3–4 hours)

Build partitioned synthetic Parquet fixtures.
Define logical schema, coverage, partition keys, sizes, and checksums.
Create the local immutable snapshot and remote training copy.

Phase 2 — Common source contract (3–5 hours)

Define SourceRequest, SourceIdentity, and SourceResult.
Implement local file-reference resolution and evidence.
Implement S3-compatible resolution using approved secret/storage configuration.

Phase 3 — Pruned shared analysis (3–5 hours)

Derive exact partitions from the selected date window.
Project required columns and build one shared lazy pipeline.
Bound collection and emit read-size evidence.

Phase 4 — Equivalence and fallback (3–5 hours)

Compare schema, counts, range, profiles, partition set, and canonical checksum.
Add explicit cache/fallback states and freshness labels.
Disable remote access and verify fallback semantics.

Phase 5 — Portability and failure verification (2–4 hours)

Test unauthorized, unavailable, corrupt, missing-partition, and partial-read cases.
Clear caches and run from a fresh runtime.
Run in a second environment with only documented logical configuration.

5.11 Key Implementation Decisions

Decision	Options	Required rationale
Snapshot identity	Manifest hash, object version set, both	Bind local and remote content
Partition timezone	UTC, business locale	Prevent off-by-one-day reads
Remote access	Shared file storage, explicit FSS adapter	Isolate provider details
Cache policy	None, bounded validated cache	Balance offline use and freshness
Fallback trigger	Unavailable only, unavailable/unauthorized, manual	Avoid hiding auth failures
Equivalence basis	Bytes, logical rows, summary, combined	State what sameness means
Read evidence	Actual bytes, provider metric, estimate	Label measurement certainty
Missing partition	Fail, partial result	Preserve completeness truth

6. Testing Strategy

6.1 Test Categories

The suite separates adapter contract, cross-source equivalence, pruning, failure classification, secret leakage, and fresh-environment portability. No single successful checksum can substitute for all six forms of evidence.

6.2 Critical Test Cases

Contract tests

Local and remote adapters return the same normalized schema and required metadata fields.
Neither adapter exposes provider-specific path details to downstream analytical cells.
Returned local paths are never opened for mutation by notebook logic.
Required source identity and freshness fields are never optional in a successful result.

Equivalence tests

Check	Expected result
Normalized schema	Equal
Selected partition set	Equal
Row count	Equal
Event-time minimum/maximum	Equal
Key null/distinct profile	Equal within exact fixture contract
Canonical bounded summary checksum	Equal
Physical resolved path	Expected to differ; excluded from logical equality

Pruning tests

Select seven of 365 date partitions and verify only seven objects/files are requested.
Request six of 24 columns and verify the projection plan contains only those columns where supported.
Compare full-year and seven-day read-size evidence.
Fail the test if a source adapter lists/downloads the entire dataset before filtering.

Failure matrix

Injected condition	Expected state
Endpoint unreachable	`UNAVAILABLE`; optional explicit fallback
Invalid/expired credential	`UNAUTHORIZED`; never mislabeled unavailable
Missing selected partition	`INCOMPLETE` or declared `PARTIAL`
Manifest checksum mismatch	`CORRUPT_OR_WRONG_SNAPSHOT`
Cache older than threshold	`STALE_CACHE`
Truncated object/read	`PARTIAL_READ` or `CORRUPT`
Modified notebook loses resource approval	`PERMISSION_REQUIRED`

Secret leakage tests

Use a unique canary training secret.
Scan notebook source, rendered output, controlled errors, logs, manifests, and exported artifacts.
Verify the value never appears.
Verify only the approved secret name is shown in the evidence panel.

Fresh-environment tests

Clear runtime and any notebook-specific cache.
Resolve and analyze the local source.
Resolve and analyze the remote source.
Compare equivalence evidence.
Repeat in a second environment using different resolved paths.
Confirm analytical cells and canonical checksum remain unchanged.

6.3 Test Data

Use one immutable, versioned Parquet fixture split across 365 date partitions with a signed manifest and six required columns among a wider schema. Maintain corrupted, truncated, missing-partition, stale-cache, unauthorized, and alternate-path variants, plus a unique canary secret that must never appear in any artifact.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem: A seven-day request downloads the entire dataset

Why: partition pruning occurs after listing/downloading everything, the filter does not match partition keys, or the reader cannot push it down.
Evidence: selected objects/bytes resemble the full-year dataset.
Fix: derive exact partition references first, project columns, and verify the backend plan/read metrics.
Quick test: compare one-day, seven-day, and full-year object counts.

Problem: The notebook works only on the author’s Mac

Why: absolute paths, ambient credentials, or undocumented cache files leak into source resolution.
Evidence: fresh runtime or second environment cannot resolve the logical input.
Fix: use notebook file references/shared storage, documented secret names, and the common adapter.

Problem: Remote outage silently shows an old snapshot as current

Why: cache fallback reuses the ready-state label without source/freshness evidence.
Evidence: object access fails but CURRENT remains visible.
Fix: classify fallback explicitly, display snapshot age/coverage, and set current_claim=false.

Problem: Local and remote counts match but checksum differs

Why: schema coercion, timezone interpretation, null semantics, partition duplication, unstable ordering, or different source version.
Evidence: stepwise equivalence report identifies the first mismatch.
Fix: compare identity, schema, partitions, ranges, and canonicalized rows before the final checksum.

Problem: Writing to a resolved path corrupts a source or cache

Why: the path was treated as a workspace file rather than a read-only input reference.
Evidence: later reads vary or original input changes.
Fix: never mutate the resolved path; write outputs to a separate explicit destination.

Problem: Modified notebook unexpectedly loses access

Why: the prior stamp no longer authorizes changed content/resources.
Evidence: Livebook requests permission again or reports an unstamped/foreign state.
Fix: review changes and reapprove through the intended Livebook workflow; do not bypass permission checks or paste secrets into source.

Problem: ETag equality is treated as universal content equality

Why: multipart/provider behavior was ignored.
Evidence: documented checksum and ETag semantics disagree.
Fix: use a controlled manifest with explicit content hashes or immutable version IDs.

7.2 Debugging Strategies

Compare the source-result contract one field at a time: logical identity, manifest/version, selected partitions, projection, resolved read statistics, normalized schema, row range, and canonical summary. Preserve the original provider error class, and clear caches when testing portability so local residue cannot hide a missing configuration.

7.3 Performance Traps

Bucket-wide listings, download-before-filter behavior, absent projection, repeated remote resolution, copying large data into notebook state, and silent cache growth defeat portability and cost controls. Measure object count and bytes for one-day, seven-day, and full-year requests, and assert that read work scales with the selection.

8. Extensions & Challenges

8.1 Beginner Extensions

Third provider: add another S3-compatible implementation without changing downstream cells.
Git-backed small fixture: compare a read-only Git shared file storage for metadata/fixtures with object storage for large Parquet.
Incremental cache: validate and fetch only missing or changed partition objects.

8.2 Intermediate Extensions

Partition evolution: support a schema-version transition while preserving a normalized contract.
Cost estimator: report request count, expected bytes, and a provider-neutral cost model before execution.
Offline bundle: export notebook plus signed manifest and minimal selected snapshot for disconnected review.

8.3 Advanced Extensions

Object corruption drill: alter one training object and prove manifest validation identifies its partition.
Application deployment: deploy as a Livebook app and verify shared file storage/secrets on a separate app server.
Read-only policy test: instrument source paths and assert no write operation targets them.

Every extension must preserve explicit source identity, freshness, and equivalence evidence.

9. Real-World Connections

9.1 Industry Applications

Data lakes: partition layout, columnar formats, and object manifests govern cost and correctness.
Reproducible research: logical inputs and immutable identities allow another analyst to rerun a method.
Disaster recovery: validated local snapshots can support bounded offline analysis when labeled honestly.
Platform engineering: shared file storage and secrets separate environment configuration from notebook content.
Multi-cloud design: a stable adapter contract contains provider differences without pretending they do not exist.
Supply-chain and security review: stamps and manifests improve provenance but do not replace code review or runtime isolation.

Livebook and Kino provide logical file and secret integration; Explorer, Polars, Arrow, and Parquet provide columnar analysis; MinIO offers an S3-compatible training target; table formats such as Apache Iceberg illustrate the metadata and evolution layers beyond this lab. Compare contracts without assuming provider semantics are identical.

9.3 Interview Relevance

A production data platform would add table formats/catalogs, centralized identity, short-lived credentials, policy enforcement, observability, object versioning, lifecycle rules, and cost monitoring. The notebook establishes the behavioral contract those systems must preserve.

10. Resources

10.1 Essential Reading

Designing Data-Intensive Applications by Martin Kleppmann — storage, derived data, and distributed consistency.
Fundamentals of Data Engineering by Joe Reis and Matt Housley — storage lifecycle, governance, orchestration, and cost.

10.2 Video Resources

Prefer current Livebook Teams file-storage demonstrations and Apache data-format conference talks. Verify each portability claim in two fresh environments with different resolved paths.

10.3 Tools & Documentation

Official Livebook and Elixir data documentation

Formats and cloud interfaces

S3-compatible services can differ from Amazon S3. Validate the API, authentication, versioning, and metadata semantics of the training provider you actually use.

11. Self-Assessment Checklist

11.1 Understanding

I can explain the difference between a logical file reference and its resolved path/resource.
I treat every resolved input path as read-only.
I can show that both adapters satisfy the same SourceResult contract.
I can prove which partitions and columns were selected before collection.
I can compare local and remote schema, count, range, profiles, and canonical checksum.

11.2 Implementation

I can distinguish actual bytes read from an estimate and label it correctly.
I can explain cache identity, age, coverage, and validation state.
I can demonstrate unavailable, unauthorized, stale-cache, corrupt, and partial-read states.
I can explain what secrets and stamps protect and what they do not.
I can prove no secret value or hard-coded author path appears in the notebook or artifacts.
I can reproduce the result from a fresh runtime and second environment.

11.3 Growth

I can add a new storage adapter without changing downstream analytical cells.
I can explain which metadata and controls a production table catalog would add.

12. Submission / Completion Criteria

Submit the Livebook notebook, source-contract documentation, immutable partition manifest, local and S3-compatible training setup instructions, equivalence report, failure-matrix transcript, and second-environment replay evidence.

The submission is complete when:

This expansion belongs to Livebook + Elixir Deep Dive. Return to the expanded project index for the rest of the sprint.