Project 16: Duplicate File Quarantine Planner

Inventory files, prove byte identity, generate an immutable quarantine plan, apply it safely, and retain enough evidence to roll every move back.

Quick Reference

Attribute	Value
Difficulty	Level 1 - Beginner/Tinkerer
Suggested Seniority	Junior
Time Estimate	8-12 hours
Main Programming Language	Elixir
Alternative Programming Languages	Erlang, Gleam
Coolness Level	Level 3 - Genuinely Clever
Business Potential	Level 2 - Micro-SaaS/Pro Tool
Prerequisites	Functions, recursion, maps, result tuples, filesystem basics
Key Topics	File, Path, Stream, recursion, crypto hashing, immutable plans, safe side effects

1. Learning Objectives

By completing this project, you will:

Traverse a filesystem without assuming every directory entry is readable, stable, or a regular file.
Use cheap metadata grouping before content hashing, then verify duplicate candidates without confusing equal names with equal bytes.
Separate discovery and planning from mutation so dry-run output is the exact contract for apply.
Represent move preconditions, results, failures, and rollback steps as explicit Elixir data.
Produce deterministic manifests that remain useful when files disappear or change between scan and apply.

2. All Theory Needed (Per-Concept Breakdown)

Immutable Planning Around Filesystem Side Effects

Fundamentals

Elixir values are immutable, but filesystem operations are not. A safe tool therefore creates an immutable description of intended effects before performing them. Discovery reads directory entries and metadata; classification identifies regular files, directories, symlinks, and errors; hashing groups likely duplicates; planning chooses a keeper and destination for every quarantined copy; applying verifies recorded preconditions and executes moves; rollback consumes a manifest of completed moves. File and Path expose filesystem primitives, Stream can make traversal and hashing lazy, and “:crypto” provides content digests. None of these removes race conditions: files may change after metadata is read, disappear during hashing, or collide at the destination. The design must treat the filesystem as changing external state and return structured outcomes instead of assuming a scan remains true forever.

Deep Dive into the concept

The project’s most important boundary is between observation and mutation. A tempting implementation finds a duplicate and moves it immediately. That interleaves traversal, policy, and destructive effects, making behavior hard to preview or reverse. Instead, the scanner emits observations; the analyzer creates duplicate groups; the planner creates a complete proposed action list; only the applier mutates. A dry-run is then not a separate approximation—it is simply rendering the same plan the applier will consume.

Traversal must classify entries deliberately. Path operations build and normalize textual paths, while File operations query or mutate the filesystem. A recursive function can request entries for a directory and return a lazy stream of observations. Symlinks require an explicit policy. Following them can create cycles or escape the requested root; not following them means recording a skipped-symlink observation. The beginner-safe default is never follow symlinks. Permission errors and vanished paths become inventory diagnostics rather than crashes.

Hashing every byte of every file wastes I/O. First group regular files by byte size; singleton sizes cannot be duplicates. Hash only groups with two or more members. Hash content incrementally rather than reading entire files into memory. A cryptographic digest such as SHA-256 makes accidental collision extremely unlikely, but the tool can offer a paranoid final byte-for-byte comparison before moving. The report must state whether identity means same size plus digest or includes a final comparison.

Hash identity is not semantic identity. Two files with the same bytes but different metadata may both matter. The planner needs a keeper policy: oldest modification time, shortest relative path, preferred root, or explicit lexical order. Deterministic lexical order is simple and reproducible; a production extension can expose policy configuration. Never use the order returned by directory listing as policy because it is not a business guarantee.

A planned move needs more than source and destination. Record source relative path, expected size, expected digest, observed modification metadata, destination, duplicate-group id, keeper path, and collision strategy. At apply time, re-stat and preferably re-hash the source. If preconditions changed, skip the action and emit “source_changed”; do not move a file merely because it still has the same name. This is optimistic concurrency for a filesystem.

Destination design protects recoverability. Quarantine paths should preserve root-relative structure or encode a stable id so same-name files do not overwrite each other. Moves must not replace existing destination files silently. A plan can reserve every destination and reject internal collisions before apply. If the source and quarantine live on different filesystems, rename may fail because it is not a same-filesystem move. For beginner scope, require the quarantine directory to share a filesystem or stop with an actionable diagnostic; a copy-verify-delete fallback is an advanced extension because it has more failure states.

Application is a sequence of individually recorded effects. Before the first move, write a plan manifest atomically. After each successful move, append or update an apply journal using a crash-safe strategy. Rollback should act only on successful journal entries in reverse order, checking that the quarantined file still matches the expected digest and that the original path is free. If the original path is occupied, rollback reports a conflict instead of overwriting it.

Determinism matters for review. Sort roots, duplicate groups, keepers, and actions by stable relative path and digest. The same snapshot and policy should produce the same plan identifier and manifest content. Absolute machine-specific roots can live in run metadata while action identities use normalized relative paths.

The invariants are: each duplicate group contains files with equal size and verified digest; exactly one keeper is selected; no source is planned twice; no destination is reserved twice; apply never overwrites an unplanned file; changed preconditions prevent mutation; every completed move has a rollback record. Failure modes include symlink cycles, memory-heavy whole-file reads, digesting a changing file, permission failures, cross-device moves, destination collisions, partial apply, rollback conflicts, and a quarantine directory placed inside the scanned root.

How this fits in the project

Recursive discovery and lazy hashing teach core Elixir collection work, while immutable plan structs provide a clean boundary around dangerous File operations.

Definitions & key terms

Inventory observation: Structured fact or diagnostic found during scanning.
Digest: Fixed-size fingerprint computed from file bytes.
Keeper: One deterministic member retained in its original location.
Quarantine plan: Immutable ordered list of proposed moves and preconditions.
Apply journal: Durable record of actions that actually completed.
TOCTOU: Time-of-check/time-of-use race where external state changes between validation and action.

Mental model diagram

scan root -> classify entries -> group by size -> stream hashes
     |                                               |
     +-> skipped/errors                              v
                                       duplicate groups
                                               |
                                    deterministic keeper policy
                                               |
                                               v
                                      immutable move plan
                                  /             |             \
                              render        verify/apply      reject
                                               |
                                               v
                                         apply journal
                                               |
                                               v
                                           rollback

How it works

Reject quarantine roots nested under scan roots.
Traverse without following symlinks and record diagnostics.
Group regular files by size; hash only candidate groups.
Optionally byte-compare equal digests.
Select a keeper and build collision-free destinations.
Persist and render the deterministic dry-run plan.
Revalidate each source before moving and journal successes.
Roll back successful moves in reverse with conflict checks.

Minimal concrete example

duplicate group digest=abc size=4096:
  keeper: photos/2024/report.pdf
  candidate: downloads/report.pdf

planned action:
  source=downloads/report.pdf
  expected_digest=abc
  destination=quarantine/downloads/report.pdf

apply:
  if current digest equals expected and destination is absent
    move and journal success
  else
    record skipped precondition failure

Common misconceptions

Equal filenames do not imply duplicate content.
Equal size alone is insufficient proof.
A digest does not guarantee the file remained unchanged after hashing.
Dry-run is trustworthy only when apply consumes the same plan.
Rollback cannot safely overwrite whatever now occupies the original path.

Check-your-understanding questions

Why group by size before hashing?
Why revalidate at apply time?
Why keep a journal separate from the original plan?

Check-your-understanding answers

Files with different sizes cannot be byte-identical, so expensive I/O is avoided.
The filesystem can change after scanning; preconditions protect against moving new content.
The plan says what was intended, while the journal proves which mutations actually completed.

Real-world applications

Photo-library cleanup, migration planning, archive verification, build-artifact deduplication, backup inspection, and compliance retention tools use the same scan-plan-apply discipline.

Where you will apply it

Apply it to traversal, hashing, keeper selection, plan rendering, move execution, and rollback. Unlike Project 8, this is not an ETS cache; unlike Project 6, it is not synthetic fault injection; unlike Project 3, it is not a distributed store.

References

Key insight

Immutability becomes a safety tool when dangerous effects are represented as reviewable data and executed only after their preconditions are revalidated.

Summary

The project combines recursive enumeration, lazy I/O, maps, sorting, hashing, result tuples, and careful filesystem semantics in a tangible tool whose correctness is visible.

Homework/Exercises

Design a deterministic keeper policy and identify its surprising cases.
List the states an action can enter during apply and rollback.
Explain how the tool detects a file changed during hashing.

Solutions

Lexically smallest normalized relative path is reproducible but may not reflect user preference; document it.
Planned, applied, skipped-changed, skipped-conflict, failed, rolled-back, and rollback-conflict are useful states.
Compare metadata before and after hashing and re-hash at apply; any mismatch invalidates the observation.

3. Project Specification

3.1 What You Will Build

Build “dupeplan”, a CLI with “scan”, “apply”, and “rollback” modes. Scan emits a human summary and machine-readable plan. Apply consumes that exact plan. Rollback consumes the completed-action journal.

Scope boundary: Do not delete duplicates permanently, follow symlinks, build a GUI, monitor directories continuously, or distribute work. The product is a local safety planner, distinct from every existing P1-P13 service.

3.2 Functional Requirements

Traverse one or more roots while excluding quarantine and configured paths.
Record regular files, skipped symlinks, permission errors, and vanished entries.
Group by size then SHA-256; optionally confirm bytes.
Choose exactly one deterministic keeper per group.
Generate unique quarantine destinations preserving relative context.
Revalidate sources and refuse overwrites during apply.
Journal completed moves and support conflict-safe reverse rollback.

3.3 Non-Functional Requirements

Never delete files.
Keep hashing memory bounded by chunk size.
Make plans deterministic for the same snapshot and policy.
Expose bytes reclaimable, bytes hashed, skipped entries, and failure counts.
Redact roots in shareable reports if requested.

3.4 Example Usage / Output

$ mix dupeplan scan ~/Documents --quarantine ~/Documents/.quarantine --plan tmp/plan.json
files=1842 candidate_bytes=812441600 hashed_files=96
duplicate_groups=14 duplicate_files=19 reclaimable_bytes=32498112
skipped_symlinks=3 errors=1
plan_id=sha256:81be... actions=19 mode=dry_run

$ mix dupeplan apply tmp/plan.json
applied=18 skipped_changed=1 failed=0 journal=tmp/plan.applied.json

3.5 Data Formats / Schemas / Protocols

The plan contains schema version, roots, policy, hash algorithm, plan id, group summaries, ordered actions, and diagnostics. Each action records relative source, expected size/digest, destination, keeper, and preconditions. The journal adds status, safe error reason, and completion sequence.

3.6 Edge Cases

Symlink loops, unreadable directories, sparse files, empty files, hard links.
Files disappearing or changing during hashing.
Quarantine inside root, destination already present, same basename from different directories.
Cross-device rename and process interruption halfway through apply.
Original path occupied before rollback.

3.7 Real World Outcome

3.7.1 How to Run

Use a temporary fixture tree containing known duplicate and unique files. Scan, inspect the plan, apply, confirm keeper/content, then rollback and compare the full tree inventory with the original.

3.7.2 Golden Path Demo

The fixture has three duplicate groups, a same-name nonduplicate, a symlink, and one file modified after scan. Apply moves eligible duplicates, skips the changed file, and rollback restores all completed moves.

3.7.3 Exact Terminal Transcript

$ mix dupeplan scan test/fixtures/tree --quarantine tmp/q --plan tmp/p.json
files=9 duplicate_groups=3 actions=4 reclaimable_bytes=12288 errors=0
$ mix dupeplan apply tmp/p.json
applied=3 skipped_changed=1 failed=0
$ mix dupeplan rollback tmp/p.applied.json
restored=3 conflicts=0 failed=0 status=restored

4. Solution Architecture

4.1 High-Level Design

roots -> walker -> observations -> size index -> hash stream -> groups
                                                                |
                                                          plan builder
                                                                |
                                  renderer <---- immutable plan ----> applier
                                                                    |
                                                                 journal
                                                                    |
                                                                 rollback

4.2 Key Components

Root policy and exclusion validator
Non-following recursive walker
Chunked hasher and optional byte verifier
Duplicate analyzer and keeper policy
Plan validator/serializer
Preconditions-aware applier
Reverse-order rollback executor

4.3 Data Structures

Use entry observations, file fingerprints, duplicate groups, actions, diagnostics, plans, and journal entries as structs. Maps index sizes and digests; actions remain an ordered list.

4.4 Algorithm Overview

Validate roots, inventory, reduce by size, hash candidates, form deterministic groups, reserve destinations, serialize plan, revalidate and apply one action at a time, persist outcomes, and roll back successful actions in reverse.

5. Implementation Guide

5.1 Development Environment Setup

Use supported Elixir/OTP and a disposable fixture directory. Never test apply against personal data until every safety test passes and dry-run output has been reviewed.

5.2 Project Structure

lib/dupeplan/
  walker
  fingerprint
  analyzer
  plan
  apply
  rollback
  cli
test/fixtures/

5.3 The Core Question You’re Answering

How can an immutable Elixir plan make inherently mutable filesystem cleanup reviewable, deterministic, and reversible?

5.4 Concepts You Must Understand First

Understand recursion, Stream.resource or comparable lazy production, File versus Path, crypto hash state, maps of lists, stable sorting, tagged results, and atomic-write concepts.

5.5 Questions to Guide Your Design

What is proof of duplicate identity?
Which paths may traversal enter?
Which preconditions must still hold before a move?
How are destination collisions impossible by construction?
What can rollback promise after external changes?

5.6 Thinking Exercise

Draw a timeline where a file is scanned, hashed, modified, and then applied. Mark which observations are stale and which revalidation catches them. Repeat with a process crash after the second of four moves.

5.7 The Interview Questions They’ll Ask

How would you recursively traverse while handling errors as values?
Why use lazy streams for file content?
What is a TOCTOU race?
Why is size-plus-hash grouped in two phases?
How do you make dry-run and apply behavior consistent?
What makes rollback safe but necessarily conditional?

5.8 Hints in Layers

Hint 1: Build an inventory command before hashing anything.

Hint 2: Group by size and hash only nonsingletons.

Hint 3: Make a plan a value with no File mutation functions inside the planner.

Hint 4: At apply, compare current facts to recorded preconditions and journal after each successful move.

5.9 Books That Will Help

Topic	Book	Chapter
Elixir collections and files	“Programming Elixir >= 1.6” by Dave Thomas	Ch. 10, Processing Collections; Ch. 12, Control Flow
Failure-aware design	“Release It!, 2nd Edition” by Michael Nygard	Ch. 4, Stability Antipatterns; Ch. 5, Stability Patterns
Domain types and workflows	“Domain Modeling Made Functional” by Scott Wlaschin	Ch. 4, Understanding Types; Ch. 7, Modeling Workflows

5.10 Implementation Phases

Phase 1: Foundation (3-4 hours)

Build fixture-safe traversal, observation structs, exclusion policy, size grouping, and chunked hashes.

Phase 2: Core Functionality (3-5 hours)

Create duplicate groups, deterministic keepers, collision-free actions, plan serialization, and dry-run rendering.

Phase 3: Polish & Edge Cases (2-3 hours)

Add revalidation, journals, rollback, changed-file tests, interrupted apply simulation, and performance metrics.

5.11 Key Implementation Decisions

Document symlink, hard-link, digest, byte-verification, keeper, quarantine-root, cross-device, atomic-manifest, changed-file, and rollback-conflict policies.

6. Testing Strategy

6.1 Test Categories

Walker tests for regular, symlink, error, and vanished entries
Fingerprint tests for empty, large, and equal files
Planner invariant and deterministic-order tests
Apply precondition tests in temporary directories
Rollback conflict and reverse-order tests
End-to-end inventory-before/after tests

6.2 Critical Test Cases

Same name with different bytes never groups.
Same bytes with different names groups.
A file changed after scan is skipped.
Existing destination is never overwritten.
Quarantine nested under root is rejected.
Partial apply journal rolls back only completed moves.
Rollback refuses an occupied original path.

6.3 Test Data

Generate fixture bytes deterministically, including zero-byte files, large chunk-boundary files, same-size nonduplicates, duplicate trees, symlinks, and permission-restricted entries where portable.

6.4 Definition of Done

Traversal records errors without aborting unrelated roots.
Hashing is chunked and bounded.
Plans select one keeper and reserve unique destinations.
Dry-run and apply use the identical plan.
Changed sources and occupied destinations are not moved.
Completed actions can be rolled back conditionally.
No command deletes or overwrites user files.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Infinite traversal: Symlinks were followed. Classify links and skip by default.

Memory spike: Whole files were read at once. Stream fixed-size chunks into hash state.

Wrong file moved: Scan evidence was trusted during apply. Revalidate size/digest.

Lost quarantine file: Destination collision overwrote it. Reserve and reject collisions before mutation.

7.2 Debugging Strategies

Render every stage for one relative path: observation, fingerprint, group, keeper decision, action, revalidation, journal. Compare plan id and normalized roots when reproducing differences.

7.3 Performance Traps

Hashing all files, repeatedly reading metadata, list concatenation in recursion, sorting per directory, and byte-comparing every candidate waste I/O or CPU. Filter by size, accumulate efficiently, and sort once.

8. Extensions & Challenges

8.1 Beginner Extensions

Add minimum-size filters, extension summaries, ignored-path patterns, and Markdown plan reports.

8.2 Intermediate Extensions

Detect hard links, sign manifests, add copy-verify-delete for cross-device moves, and expose keeper-policy modules.

8.3 Advanced Extensions

Add bounded Task-based hashing with disk-aware concurrency, resumable journals, Merkle inventories, and property tests for plan invariants.

9. Real-World Connections

9.1 Industry Applications

Backup verification, content-addressed storage, media-library cleanup, archive migration, build cache analysis, and digital forensics all separate observation from controlled mutation.

Compare user-facing safety policies in tools such as rdfind or fdupes and content-addressed ideas in Git. Keep the implementation local and educational rather than cloning their features.

9.3 Interview Relevance

This project shows practical recursion, lazy I/O, crypto interoperability, explicit errors, deterministic algorithms, and mature handling of races and reversible side effects.