Project 34: Crash-Resilient Session Store: ETS Hot Path and DETS Journal

Build a production-shaped session and idempotency service that serves reads from ETS, acknowledges writes only after an explicit DETS durability boundary, survives process-owner failure through table transfer, and reconstructs correct state after the entire BEAM VM is killed.

Quick Reference

Attribute Value
Difficulty Level 3: Intermediate
Suggested Seniority Senior Elixir/OTP engineer
Time Estimate 24-36 hours
Main Programming Language Elixir
Alternative Programming Languages Erlang, Gleam with Erlang interop
Coolness Level Level 4: Very Cool
Business Potential Level 3: Useful Developer Infrastructure
Prerequisites Elixir terms and pattern matching, GenServer, Supervisor, monotonic/system time, basic filesystem behavior, ExUnit
Key Topics ETS ownership and heirs, DETS persistence, durable acknowledgements, replay, idempotency, TTL recovery, supervisor boundaries, crash testing

1. Learning Objectives

By completing this project, you will:

  1. Choose ETS table types, access modes, keys, and concurrency options from measured access patterns rather than habit.
  2. Explain why a named ETS table is discoverable but not durable, and why an ETS heir survives only a local owner-process failure.
  3. Distinguish an ETS snapshot made with tab2file from a continuously maintained disk authority.
  4. Use DETS within its real constraints: disk I/O, a 2 GB file limit, no ordered_set, explicit open/close/sync behavior, and potentially expensive repair after an unclean stop.
  5. Define one unambiguous source of truth between a hot in-memory projection and a durable event journal.
  6. Design idempotent writes for the failure window where a durable event exists but the caller did not receive the acknowledgement.
  7. Restore sessions, revocations, and expirations deterministically after an abrupt VM termination.
  8. Build a supervisor tree whose restart boundaries preserve healthy state and whose readiness signal stays false until recovery is complete.
  9. Prove recovery behavior with process kills, table-owner kills, kill -9, corrupt/truncated fixture files, and replay tests.

2. All Theory Needed (Per-Concept Breakdown)

ETS Ownership, Table Transfer, and Hot-State Design

Fundamentals

ETS is an in-memory term store implemented by the Erlang runtime. Processes can access a table directly, so a read does not have to serialize through the mailbox of a GenServer. Every ETS table still has exactly one owner process. Unless the table has an heir, the runtime deletes it when that owner terminates. Table type controls key semantics: set keeps one object per key, ordered_set keeps keys in Erlang term order, bag permits multiple distinct objects per key, and duplicate_bag also permits duplicate objects. Access can be private, protected, or public. Naming a table makes its identifier discoverable on the local node; it does not add persistence, distribution, or ownership independence. This project uses ETS as a rebuildable, low-latency projection—not as the durable record of accepted session changes.

Deep Dive

Begin with the access pattern. A session lookup is keyed by an opaque session identifier and returns at most one current record, so a set is the natural primary table. Expiration work needs a different access path. A second ordered_set can index {expires_at, session_id} so a sweeper can consume the earliest expirations without scanning every live session. That second index is derived state: if it is inconsistent, recovery must rebuild it from the same authority as the primary table. Trying to force lookup and expiry ordering into one clever key usually makes one operation expensive or ambiguous.

The table owner defines a failure boundary. If an API worker owns the table, an unrelated request bug can erase every cached session. A dedicated TableKeeper process is a better owner because its job is only lifecycle management. API workers may read from a protected table while mutations remain serialized through a storage coordinator. This balances direct reads with a single place that enforces version, TTL, and persistence invariants. public is tempting, but it lets every process bypass the durable write path. private prevents direct reads and can turn the owner mailbox into the bottleneck the project is meant to avoid.

An heir changes what happens when the owner exits. At table creation, the owner can nominate a local heir process. The runtime transfers the table to that process and can send an ETS-TRANSFER message containing identifying data. A restarted owner can later receive the table through give_away. This prevents a brief owner-process crash from forcing a full disk replay. It does not protect against a VM crash, host failure, or heir failure, and the heir must be on the same node. Treat inheritance as an availability optimization around one process boundary, not as persistence. The supervisor layout should avoid restarting the owner and heir together for every ordinary coordinator failure.

Table names and table identifiers have different trade-offs. A named table provides a stable local lookup name, but creating it while an inherited copy already exists can race. A robust startup handshake asks the heir whether it holds the table before creating a replacement. Anonymous table identifiers avoid global local names but must be published safely to readers. In either design, readiness remains false until ownership and both indexes are confirmed.

ETS concurrency options are tuning controls, not correctness controls. read_concurrency, write_concurrency, and decentralized counters change internal synchronization and memory behavior. They do not make a multi-table update transactional. Updating the session table and expiry index can still be interrupted between operations. The coordinator therefore treats ETS as a projection that can be repaired. It applies both derived mutations after the durable event is accepted, and it can rebuild the expiry index from the session table or replay the journal after a detected inconsistency. Benchmark the actual mixture of lookups, touches, revocations, and expiry sweeps before enabling options; a read optimization can make alternating reads and writes slower.

tab2file and file2tab are useful for exporting and restoring a point-in-time table image. A snapshot may shorten a later warm-up, support debugging, or seed a test fixture. It is not a write-ahead log and does not record every accepted mutation after the snapshot. A crash between snapshots can lose acknowledged state if the snapshot is treated as the authority. A snapshot also does not solve the relationship between two derived tables unless the cut is coordinated and versioned. In the core project, snapshots are optional accelerators whose version is checked against the durable DETS journal. If absent or stale, the system performs a full replay.

Memory accounting matters because ETS data is outside an individual Elixir process heap but still consumes VM memory. Large binaries may remain referenced, expired records can accumulate if the sweeper stalls, and table scans can create large result lists. Expose table size, memory words, oldest expiry, replay generation, and sweep lag as diagnostics. Use bounded select continuations for maintenance rather than converting the table to a list. The final invariant is simple: ETS must always be safe to delete and rebuild. If deleting it would lose accepted business truth, the authority boundary is wrong.

How this fits on the project

This concept defines the live session table, expiry index, direct-read path, mutation coordinator, owner/heir handshake, readiness state, table diagnostics, and the distinction between fast recovery from owner failure and full recovery from VM failure.

Definitions and key terms

  • Owner: The one local process responsible for an ETS table’s lifetime.
  • Heir: A nominated local process that inherits a table when its owner terminates.
  • Projection: Rebuildable representation optimized for a query or runtime access pattern.
  • Protected table: A table whose owner may read/write while other processes may read.
  • Derived index: Data computed from authoritative records, such as an expiry-ordered table.
  • Snapshot: A point-in-time image, not a continuous record of subsequent changes.
  • Readiness: A service state indicating that ownership and restoration invariants are satisfied.

Mental model diagram

                           local BEAM node

 readers ---------------------- lookup --------------------+
                                                          |
                                            +-------------v----------+
 mutations --> [Session Coordinator] ------>| ETS sessions (set)     |
                 |                          | id -> current envelope  |
                 |                          +-------------+----------+
                 |                                        |
                 |                                derived | update
                 |                                        v
                 |                          +------------------------+
                 |                          | ETS expiries           |
                 |                          | ordered {time, id}     |
                 |                          +------------------------+
                 |
                 +--> durable append before acknowledgement

 [Table Keeper: owner] --owner dies--> [Table Heir: temporary owner]
         ^                                      |
         +--------------- give_away ------------+

 VM dies: both ETS tables disappear; the disk journal remains.

How it works

  1. The table heir starts before the table keeper and publishes no readiness by itself.
  2. The table keeper asks whether a prior table was inherited; it accepts transfer or creates new tables.
  3. The coordinator waits for the journal and tables, then restores or verifies the current generation.
  4. Readers use the published session table identifier for direct protected lookups.
  5. Writers go through the coordinator so nobody can update ETS without the durable protocol.
  6. Expiry processing selects a bounded prefix of the ordered expiry index and submits expiry events.
  7. If the keeper dies, the heir receives the tables; the restarted keeper reclaims them.
  8. If the VM dies, startup creates empty tables and replays durable events.
  9. Invariant: an ETS record is never the only evidence for an acknowledged mutation.
  10. Failure modes include joint keeper/heir restart, stale published identifiers, index drift, memory growth, unbounded scans, and incorrectly treating a snapshot as a journal.

Minimal concrete example

TABLE SHAPES, NOT RUNNABLE CODE

sessions (set, protected):
  {session_id,
   version,
   status: active | revoked | expired,
   subject_id,
   issued_at_system,
   expires_at_system,
   last_request_id}

expiries (ordered_set, protected):
  {{expires_at_system, session_id}, version}

owner crash:
  TableKeeper exits -> ETS-TRANSFER to TableHeir
  restarted TableKeeper -> validates generation -> table give_away

Common misconceptions

  • A named ETS table survives because it is no longer tied to a PID.
  • An heir makes a table durable across VM or machine restarts.
  • write_concurrency: true makes updates across two ETS tables atomic.
  • tab2file continuously persists changes after the dump finishes.
  • ETS memory is free because it is not visible in a worker’s process heap.

Check-your-understanding questions

  1. Why should the API worker not own the session table?
  2. Which failure does an heir address, and which failure does it not address?
  3. Why is the expiry table allowed to be temporarily inconsistent during recovery?
  4. What correctness guarantee does write_concurrency add?

Check-your-understanding answers

  1. A request-processing bug would couple an ordinary worker failure to the lifetime of all hot state.
  2. It addresses local owner-process termination; it does not survive loss of the VM, node, or host.
  3. It is derived from authoritative session events and can be rebuilt before readiness is announced.
  4. None; it changes performance characteristics while preserving the same operation-level guarantees.

Real-world applications

  • Authentication and shopping-session caches
  • Idempotency-key lookup services
  • Feature/configuration projections
  • Rate-limit windows whose durable policy requires replay
  • Local indexes over an append-only business log

Where you will apply it

  • Project 34 table keeper, table heir, session projection, expiry projection, sweeper, and readiness gate

References

Key insight

ETS is most reliable when it is designed to be disposable: ownership preserves availability, while another authority preserves truth.

Summary

Use one-purpose ownership, protected access, derived indexes, bounded maintenance, and an heir for local transfer. Keep the durable boundary outside ETS and make readiness depend on successful restoration.

Homework/Exercises

  1. Draw the exact ownership sequence for a coordinator crash, table-keeper crash, heir crash, and whole-VM crash.
  2. Compare a single full-table TTL scan with an ordered expiry index for one million sessions.
  3. Define which metrics prove the expiry projection is keeping up.

Solutions

  1. The coordinator restart leaves tables intact; a keeper crash transfers to the heir; an heir crash is harmless while the keeper lives but removes transfer protection; VM loss requires disk replay.
  2. The scan touches all sessions per sweep, while the ordered index processes only the due prefix plus bounded continuation overhead.
  3. Track due-but-unprocessed count, oldest overdue timestamp, sweep batch duration, table sizes, repair count, and coordinator mailbox length.

DETS Durability, Replay, and Idempotent Acknowledgements

Fundamentals

DETS stores Erlang tuple objects in a disk file and exposes an API resembling part of ETS. It supports set, bag, and duplicate_bag, but not ordered_set; every lookup performs disk work and is far slower than ETS. A DETS file cannot exceed 2 GB. Tables must be opened before use and properly closed. After an abnormal VM termination, DETS can repair an improperly closed file when reopened, but repair may be expensive for a large or fragmented table. dets:sync forces updates to disk, while autosave controls periodic flushing; neither turns a sequence of separate objects into a transaction. This project uses one serialized journal append as the durable fact for a session command and builds the fast current-state view by replaying those facts.

Deep Dive

The first design decision is authority. If a request updates ETS, returns success, and only later writes DETS, a VM crash can erase an acknowledged session. Therefore the core strict mode validates the command, derives one durable event, inserts that event into DETS, synchronizes according to the declared durability policy, and only then updates ETS and acknowledges the caller. A crash after the durable write but before the ETS mutation is safe: recovery or a reconciliation pass applies the event. A crash after the durable write but before the response is uncertain from the caller’s perspective, so retries require a stable request identifier.

Model each accepted mutation as one self-contained event keyed by request_id. The event includes session_id, the expected prior version, resulting version, operation, subject where applicable, system timestamps, expiry, and a schema version. A DETS set makes repeating the same request identifier address the same durable object. The coordinator compares the canonical command fingerprint with the stored event. An exact retry returns the original result; reuse of the same request identifier for different parameters is rejected. This closes the classic timeout window without claiming multi-object transactions.

Concurrent changes to one session must be serialized or version-checked. One coordinator process is sufficient for the core project and makes version assignment deterministic. It is not a universal scalability solution: sharding later would require stable routing and per-session serialization. Because DETS has no transactions, avoid an event write plus a separate durable idempotency write. The single event carries both business transition and request identity. Avoid a separate durable head record in the core, because a crash between event and head would require a recovery protocol. The current session head is instead a rebuildable ETS projection.

DETS has no ordered table. Journal order must therefore be explicit in the objects, not inferred from traversal order. During startup, fold over all events, group by session, validate version chains, and sort each bounded chain by version or a monotonic logical sequence. Wall-clock timestamps are not safe ordering keys because clocks can repeat or move. If replay encounters two different events claiming the same next version, a missing predecessor, an unsupported schema version, or request-ID collision, the system fails closed and does not advertise readiness. A repair command can report the conflict, but it must not silently pick whichever event DETS returns first.

Time-to-live introduces two clocks. Store expiry as a system-time instant because it must retain meaning across process and VM restarts. Use monotonic time only for intervals within one running VM. On replay, compare stored expiry against current system time: an already expired active session becomes an expired projection or causes a durable expiry event according to policy. The system must never extend a session merely because it restarted. Record clock-skew diagnostics and test with an injected clock abstraction rather than modifying the host clock.

Durability must be named honestly. In strict mode, the response accepted_durable is emitted only after dets:sync succeeds. This costs more disk latency. A batch mode may acknowledge after insert and rely on autosave, but it must return a different durability class and document the possible loss window during abrupt termination. Do not call both modes durable without qualification. DETS is not a database transaction log with group commit, checksums exposed as an application contract, or automatic replication.

Opening and closing belong to one supervised journal owner. Normal shutdown explicitly syncs and closes. Abrupt termination can leave the file needing repair. Startup reports whether repair occurred and how long it took; readiness waits for open, repair, scan, validation, and ETS reconstruction. A repair that succeeds proves the file is readable, not that every business invariant holds, so replay validation still runs. A deliberately truncated copy belongs in destructive tests, never the only real data file.

The 2 GB limit and fragmentation make retention a core constraint. The project exposes file size, object count, replay time, oldest event, and estimated time to the limit. Sessions with finite retention allow compaction, but safe compaction is not “delete old events whenever the session expires.” You must retain enough evidence for idempotency and auditing. The core can enforce a bounded test dataset and document an operator threshold. An advanced extension creates a versioned snapshot plus a journal tail in new files, verifies both, atomically switches a manifest, and retains the previous generation for rollback.

tab2file can accelerate a cold start, but the system must tie the snapshot to a known journal generation or high-water mark. On startup, load the snapshot only if its metadata is valid, then replay later events. If validation fails, discard the snapshot and replay DETS from the beginning. This keeps optimization subordinate to correctness.

Finally, DETS is local durability. It does not replicate to another host. Disk loss, filesystem corruption, and machine loss remain possible. The project should state a backup policy or explicitly scope itself to one host. Project 35 moves to Mnesia to study replicated durable tables and distributed recovery rather than pretending DETS solves them.

How this fits on the project

This concept defines journal records, request idempotency, durability classes, recovery validation, TTL semantics across restarts, failure injection, retention limits, and the boundary between local recovery and distributed durability.

Definitions and key terms

  • Durable acknowledgement: A response sent only after the declared persistence boundary succeeds.
  • Journal event: Immutable fact describing one accepted command and its resulting version.
  • Idempotency key: Caller-provided identifier used to recognize a retry of the same command.
  • Command fingerprint: Canonical digest or representation used to reject conflicting reuse of an idempotency key.
  • Replay: Deterministic reconstruction of current state from durable facts.
  • Repair: DETS reconstruction after an unclean close; it is not business-level validation.
  • High-water mark: Journal position known to be represented by a snapshot.
  • Authority: The representation whose accepted facts define correct business state.

Mental model diagram

write request(request_id, expected_version)
                 |
                 v
       [validate + serialize]
                 |
        existing request_id?
          / yes          \ no
         v                v
 [return prior result] [append one DETS event]
                              |
                         dets:sync
                              |
                       +------v------+
                       | durable fact |
                       +------+------+
                              |
                    apply to ETS projections
                              |
                         acknowledge

 abrupt VM stop ------> ETS is lost
                         DETS may repair on open
                              |
                 validate versions + replay + TTL check
                              |
                         readiness=true

How it works

  1. Require a unique request ID and expected session version for every mutation.
  2. Look up an existing event by request ID and return it only if the command fingerprint matches.
  3. Validate the transition against the current reconstructed session.
  4. Construct one immutable event containing the idempotency and business result.
  5. Insert the event into the DETS set and perform the configured sync boundary.
  6. Apply the resulting current state and expiry key to ETS.
  7. Reply with result, version, request ID, and durability class.
  8. On startup, open/repair DETS, scan events, validate chains, replay, expire overdue sessions, and only then become ready.
  9. Invariant: no success response exists without exactly one matching durable event in strict mode.
  10. Failure modes include disk-full, 2 GB limit, slow repair, conflicting request-ID reuse, broken version chains, unsupported event schema, clock skew, and a crash between durability and response.

Minimal concrete example

EVENT SHAPE, NOT RUNNABLE CODE

key: request_id = "req-8c52"
value:
  session_id: "sess-f013"
  command_fingerprint: sha256(canonical command)
  expected_version: 6
  resulting_version: 7
  operation: touch
  expires_at_unix_ms: 1784115600000
  event_schema: 1
  result: accepted

ACKNOWLEDGEMENT RULE
  DETS insert -> DETS sync -> ETS apply -> accepted_durable

Common misconceptions

  • DETS is just ETS with transparent persistence and similar latency.
  • DETS traversal order can substitute for an event sequence.
  • Autosave and dets:sync make two separate inserts transactional.
  • Successful file repair proves every accepted session event survived and is consistent.
  • A local DETS file is a substitute for replication and off-host backup.

Check-your-understanding questions

  1. Why must the request ID and transition live in the same durable event?
  2. What happens if the VM dies after sync but before the caller receives the reply?
  3. Why should replay fail on a forked session version chain?
  4. Why is a system-time expiry stored even though monotonic time is preferred for timers?

Check-your-understanding answers

  1. Separate records would require atomic multi-object semantics DETS does not provide.
  2. The caller retries with the same ID, and the stored event returns the original result without applying the transition twice.
  3. Arbitrarily choosing a branch would conceal corruption or a concurrency bug and create nondeterministic state.
  4. Monotonic values are meaningful only within one VM instance; the expiry must be comparable after restart.

Real-world applications

  • Local agent state and idempotency journals
  • Edge-device session/token projections
  • Durable development tools without an external database
  • Single-host workflow checkpoints with explicit capacity limits
  • Read-through caches reconstructed from append-only facts

Where you will apply it

  • Project 34 journal owner, write coordinator, replay validator, TTL recovery, strict/batched durability modes, and crash harness

References

Key insight

Recovery is correct only when the durable acknowledgement rule and the retry rule describe every crash window explicitly.

Summary

Use DETS as a bounded local authority, write one idempotent event per accepted command, make order explicit, distinguish sync policies, validate before readiness, and treat ETS snapshots only as optional replay accelerators.

Homework/Exercises

  1. Enumerate every crash point from request receipt through response delivery and state the retry result.
  2. Design a safe retention policy for expired sessions whose request IDs must remain idempotent for seven days.
  3. Estimate when a journal reaches 2 GB from average event size and write rate.

Solutions

  1. Before durable insert, retry performs the command; after insert but before sync, outcome depends on declared mode; after strict sync, every retry returns the stored result; after ETS apply, the same result remains; after response, retry is still idempotent.
  2. Keep durable events until both business retention and the idempotency window expire, compact into a verified generation, and never delete the previous generation before the new manifest is durable.
  3. Divide the usable byte budget by average serialized event bytes, then divide the resulting event count by events per second; include headroom for fragmentation and repair operations.

3. Project Specification

3.1 What You Will Build

Build session-vault, a local HTTP/CLI-compatible service for creating, reading, touching, revoking, and expiring sessions. Reads come from ETS. Every mutation carries a request ID and expected version, is recorded as one DETS event, and updates ETS only after the configured persistence boundary. The service exposes readiness, storage diagnostics, replay reports, and a deterministic crash harness. It is deliberately single-node and bounded; the point is to learn exact local recovery semantics before adding replication.

3.2 Functional Requirements

  1. Create session: Create an opaque session with subject, metadata, issued time, expiry, version 1, and request ID.
  2. Get session: Return active, revoked, expired, or missing status from ETS without routing normal reads through the coordinator mailbox.
  3. Touch session: Extend expiry only when the expected version matches and the transition policy allows it.
  4. Revoke session: Persist a revocation that remains revoked after every restart.
  5. Idempotent mutation: Return the original result for an exact retry and reject conflicting request-ID reuse.
  6. TTL enforcement: Prevent expired sessions from becoming active again after restart; process due expirations in bounded batches.
  7. Owner transfer: Preserve ETS tables when the dedicated table keeper dies and the local heir remains alive.
  8. Cold recovery: Rebuild both ETS tables from DETS after abrupt whole-VM termination.
  9. Readiness gate: Reject traffic until DETS open/repair, version validation, replay, ownership, and index validation finish.
  10. Diagnostics: Report event count, DETS size, replay time, repair status, table memory, session counts, expiry lag, and durability mode.

3.3 Non-Functional Requirements

  • Correctness: Strict-mode acknowledgement implies a matching synced journal event.
  • Latency: Establish separate service objectives for direct ETS reads and synced writes; never hide the durability cost.
  • Capacity: Reject startup or writes at a documented threshold below the DETS 2 GB hard limit.
  • Memory: Maintenance work is bounded; no full table is converted into an Elixir list in the request path.
  • Operability: Every recovery phase and fail-closed decision emits structured diagnostics.
  • Security: Session IDs are opaque and logs redact credentials and user metadata.

3.4 Example Usage / Output

$ session-vault create --request req-001 --subject customer-42 --ttl 30m
accepted_durable session=sess-f013 version=1 expires_at=2026-07-15T18:30:00Z

$ session-vault touch sess-f013 --request req-002 --expect-version 1 --ttl 60m
accepted_durable session=sess-f013 version=2 expires_at=2026-07-15T19:00:00Z

$ session-vault touch sess-f013 --request req-002 --expect-version 1 --ttl 60m
idempotent_replay session=sess-f013 version=2 original_request=req-002

$ session-vault get sess-f013
active subject=customer-42 version=2 source=ets

3.5 Data Formats / Schemas / Protocols

Command:
  {request_id, operation, session_id?, expected_version?, payload, requested_at}

Durable event:
  {request_id, command_fingerprint, session_id, prior_version,
   resulting_version, operation, result_payload, expires_at, event_schema}

Current ETS envelope:
  {session_id, version, status, subject_id, metadata_digest,
   issued_at, expires_at, last_request_id}

Recovery report:
  {journal_generation, repaired, event_count, valid_chains,
   rejected_chains, replay_ms, expired_during_replay, ready}

3.6 Edge Cases

  • Duplicate request ID with different session or TTL
  • Touch arriving exactly as expiry becomes due
  • Revocation retried after the response was lost
  • Keeper and heir failure in different orders
  • Whole VM killed immediately before and after dets:sync
  • DETS open requiring repair after unclean termination
  • Disk full or permissions changed between boots
  • Journal approaching the configured capacity ceiling
  • Unsupported event schema after an application downgrade
  • System clock moves backward or forward between restarts
  • Snapshot high-water mark missing from the journal
  • One derived expiry entry missing or duplicated

3.7 Real World Outcome

The learner starts the service, creates a session, crashes an ordinary worker, crashes the table keeper, and observes uninterrupted reads because the table heir temporarily owns the ETS tables. The learner then kills the whole VM without a graceful stop. On restart, the terminal shows DETS repair status, event-chain validation, replay progress, expiry reconciliation, index validation, and the moment readiness becomes true. The same session returns with the same version. Retrying a command whose response was intentionally dropped returns the original result rather than applying it twice.

$ session-vault crash-drill --scenario vm-after-sync --request req-003
phase=journal_synced request=req-003 action=SIGKILL
process_exit=137

$ session-vault start --journal ./data/session-events.dets
recovery journal_opened repaired=true repair_ms=184
recovery events=3 valid_chains=1 conflicts=0
recovery ets_sessions=1 ets_expiries=1 replay_ms=22
readiness=true generation=17 durability=strict

$ session-vault retry --request req-003
idempotent_replay session=sess-f013 version=3 status=revoked

3.8 Scope Boundary Versus Existing Projects

  • Project 8 builds an ETS TTL cache; Project 34 makes disk authority, acknowledgement order, idempotent replay, owner transfer, and whole-VM recovery the primary outcome.
  • Project 3 builds a replicated distributed KV store; Project 34 is intentionally single-node and does not claim replication.
  • Project 24 uses Oban/PostgreSQL for durable jobs; Project 34 studies OTP-native local storage mechanics without an external database.
  • Project 35 adds Mnesia replication and partition policy after these local durability fundamentals are understood.

4. Solution Architecture

4.1 High-Level Design

                          SessionVault.Supervisor
                                   |
          +------------------------+-------------------------+
          |                        |                         |
          v                        v                         v
 [JournalServer]             [TableHeir]              [Telemetry]
 opens DETS                      |                         |
 append + sync                   | ETS-TRANSFER            |
          |                      v                         |
          |                [TableKeeper]                   |
          |                 owns/reclaims                  |
          |                    |      |                    |
          |             sessions ETS  expiry ETS           |
          |                    ^      ^                    |
          +---- durable ----> [Coordinator] <---- [TTL Sweeper]
                                    ^
                                    |
                            API / CLI mutations

 direct reads --------------------> sessions ETS
 readiness requires every storage component and replay validation

Keep the journal owner independent from request workers. Keep the table heir alive across table-keeper restarts. The coordinator serializes mutations, but direct reads access the protected session table. The readiness gate observes recovery state rather than assuming that a supervised process PID means the data is usable.

4.2 Key Components

Component Responsibility Key Decisions
JournalServer Open, repair, append, sync, fold, close, and report DETS One owner; strict durability is the default
TableHeir Temporarily inherit ETS tables Separate restart boundary from keeper
TableKeeper Create/reclaim tables and publish identifiers Protected tables; no business mutations
SessionCoordinator Validate versions, enforce idempotency, order DETS/ETS changes One writer in core project
SessionReader Perform direct ETS lookup and status interpretation No writes and no fallback to stale process state
TTLSweeper Consume due expiry keys in bounded batches Expiry becomes a durable transition
RecoveryManager Validate event schemas/chains and rebuild projections Fail closed on ambiguity
ReadinessGate Expose ready/not-ready with exact reason Ready only after ownership and replay complete
CrashHarness Trigger named failures at controlled protocol phases Enabled only in test/drill environments

4.3 Data Structures (No Full Code)

  • Journal event: One tuple per request ID containing complete transition evidence and a schema version.
  • Session envelope: Current version and state keyed by session ID.
  • Expiry key: Ordered composite key {system_expiry, session_id} plus the version it refers to.
  • Command fingerprint: Canonical digest excluding transport-only metadata.
  • Recovery accumulator: Per-session expected version, reconstructed state, conflicts, and metrics.
  • Ownership generation: Token used to reject stale table publication or transfer.

4.4 Algorithm Overview

Strict mutation protocol

  1. Validate command shape, request ID, session state, and expected version.
  2. If request ID exists, compare fingerprints and return or reject.
  3. Derive exactly one event and resulting session envelope.
  4. Append event to DETS and synchronize it.
  5. Update the session table and repair the corresponding expiry index entry.
  6. Return the durable acknowledgement.

Cold replay

  1. Open DETS and record repair diagnostics.
  2. Fold events under a bounded accumulator or staged external grouping appropriate to the configured capacity.
  3. Validate event schemas, request-ID uniqueness, fingerprints, and version chains.
  4. Reconstruct each session and apply current-time expiry policy.
  5. Populate a new generation of ETS tables.
  6. Validate counts/indexes, publish table identifiers, then enable readiness.

Complexity considerations

  • Session lookup: expected constant time in ETS set.
  • Due-expiry selection: logarithmic key positioning plus the bounded due batch in ordered_set.
  • Mutation: DETS disk cost plus expected constant-time ETS updates.
  • Full replay: proportional to journal events plus per-session ordering work; this is why retention and compaction matter.

5. Implementation Guide

5.1 Development Environment Setup

Use a current supported Elixir/OTP pair. Create a supervised Mix application, an isolated temporary data directory for each test, and a crash harness that launches the application as a separate OS process. Confirm that the filesystem used for drills permits sync and abrupt process termination. Do not run destructive repair tests against the only copy of a journal.

5.2 Project Structure

session_vault/
├── config/
│   ├── config.exs
│   └── runtime.exs
├── lib/session_vault/
│   ├── application.ex
│   ├── journal_server.ex
│   ├── table_heir.ex
│   ├── table_keeper.ex
│   ├── coordinator.ex
│   ├── reader.ex
│   ├── ttl_sweeper.ex
│   ├── recovery_manager.ex
│   ├── readiness_gate.ex
│   ├── event_schema.ex
│   └── diagnostics.ex
├── test/
│   ├── unit/
│   ├── integration/
│   ├── recovery/
│   └── fixtures/journals/
├── scripts/
│   └── crash_drill.sh
└── README.md

5.3 The Core Question You Are Answering

“When fast state disappears at any crash boundary, what exact durable fact lets the system reconstruct the same answer without applying a caller’s retry twice?”

Do not begin with storage calls. First write a timeline for every mutation and mark when the event is authoritative, when ETS changes, and when the caller can safely receive success.

5.4 Concepts You Must Understand First

  1. ETS ownership and protection
    • Who can read and write a protected table?
    • What exactly happens on owner exit?
    • Book Reference: Erlang and OTP in Action — ETS sections.
  2. Supervision versus data durability
    • What can a supervisor restart, and what data can it not recreate by itself?
    • Book Reference: Designing for Scalability with Erlang/OTP — supervision and state sections.
  3. DETS lifecycle
    • When are open, sync, close, auto-save, and repair observable?
    • What does the 2 GB limit imply for retention?
  4. Idempotency and uncertain outcomes
    • How does a client distinguish retry from a second command after a timeout?
  5. Time semantics
    • Which clock persists meaning across VM restarts, and which clock is safe for elapsed intervals?

5.5 Questions to Guide Your Design

  1. Which process owns each resource, and which failures restart them together?
  2. Can any module mutate ETS without creating a durable event first?
  3. Is acknowledgement tied to DETS insert, sync, or periodic autosave?
  4. How does the client learn the durability class?
  5. What exact data identifies conflicting reuse of a request ID?
  6. What makes a replay chain invalid, and does startup fail closed?
  7. How are expired sessions represented so a restart cannot resurrect them?
  8. How far below 2 GB does the operator threshold stop writes?
  9. How is a snapshot associated with a journal high-water mark?
  10. How will tests kill the VM at a reproducible protocol phase?

5.6 Thinking Exercise

Draw the seven failure windows

Trace one revoke command through validation, duplicate lookup, DETS insert, DETS sync, ETS update, response construction, and response delivery. At each boundary, kill the VM on paper. For every restart, write whether the event exists, what ETS contains after replay, and what an identical retry returns. If any cell says “unknown” or “probably,” the protocol is not specified enough to implement.

5.7 The Interview Questions They Will Ask

  1. “What happens to an ETS table when its owner terminates, and what can an heir actually protect?”
  2. “Why is tab2file not equivalent to a durable journal?”
  3. “How does DETS differ from ETS in performance, table types, concurrency, and capacity?”
  4. “How would you prevent a timeout retry from revoking or extending a session twice?”
  5. “Why must readiness wait after the supervisor has restarted every child?”
  6. “Where is the linearization or authority point in your mutation protocol?”

5.8 Hints in Layers

Hint 1: Make ownership boring

Give table lifecycle, journal lifecycle, and business mutation to different components. A process with fewer responsibilities is easier to restart safely.

Hint 2: Start with strict writes

Implement insert, sync, ETS apply, and acknowledgement in that order. Add a weaker batch mode only after its loss window can be demonstrated.

Hint 3: Store results with request identity

One event should be sufficient to decide whether a retry is identical and what original result it receives. Avoid a second idempotency table in the core.

Hint 4: Make crashes external

A test that calls a graceful stop does not prove abrupt recovery. Launch a child OS process, wait for a named phase marker, send an untrappable termination, then restart on the same copied data directory.

5.9 Books That Will Help

Topic Book Chapter/Section
ETS and shared runtime data Erlang and OTP in Action — Logan, Merritt, Carlsson ETS and data-storage sections
OTP ownership and failure boundaries Designing for Scalability with Erlang/OTP — Cesarini, Vinoski Supervision and state-management sections
Elixir process architecture Elixir in Action — Saša Jurić Server processes and fault tolerance chapters
Idempotent distributed operations Designing Data-Intensive Applications — Martin Kleppmann Replication/fault and stream-processing discussions

5.10 Implementation Phases

Phase 1: Hot projection and ownership (6-8 hours)

  • Build the session and expiry table shapes.
  • Implement protected direct reads and serialized mutations without persistence.
  • Add keeper/heir transfer and prove the table identifier remains valid or is republished safely.
  • Checkpoint: a table-keeper crash does not lose an active session, while a whole-VM restart deliberately does.

Phase 2: Durable protocol and replay (10-14 hours)

  • Define the versioned event and command fingerprint.
  • Implement strict DETS append/sync before ETS mutation.
  • Restore both projections and handle already-expired sessions.
  • Checkpoint: kill -9 after sync returns the original result on retry after restart.

Phase 3: Failure honesty and operations (8-14 hours)

  • Add readiness phases, repair diagnostics, capacity threshold, metrics, and redaction.
  • Build the crash matrix and corrupted-copy tests.
  • Benchmark reads, strict writes, replay, and repair fixtures.
  • Checkpoint: a written recovery report accounts for every fault and states any remaining loss window.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Durable authority ETS / snapshot / DETS event DETS event ETS and snapshots are rebuildable projections
Core write model many writers / one coordinator one coordinator Makes versions and DETS limitations explicit
ETS access public / protected / private protected Fast direct reads; controlled writes
Keeper failure rebuild / heir transfer heir transfer then validate Faster local recovery without claiming durability
Write acknowledgement autosave / sync sync in default mode Names the strict durability boundary
Expiry ordering full scan / ordered index derived ordered_set Bounded due work
Retry record separate table / event-contained event-contained Avoids missing multi-object atomicity
Corruption response best effort / fail closed fail closed with offline repair copy Prevents silent state invention

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Pure model tests Validate transitions and replay independently of storage create/touch/revoke/expire version chains
ETS lifecycle tests Validate ownership and projection integrity keeper death, heir transfer, index rebuild
DETS integration tests Validate disk lifecycle sync, close, reopen, repair fixture, capacity refusal
Idempotency tests Validate uncertain response handling exact retry and conflicting key reuse
Crash-process tests Validate whole-VM recovery kill before insert, after sync, after ETS apply
Time tests Validate TTL across restart already expired, future expiry, injected skew
Performance tests Establish honest service limits lookup latency, synced-write latency, replay duration

6.2 Critical Test Cases

  1. Worker crash: Killing an API worker changes neither ETS ownership nor stored sessions.
  2. Keeper crash: The heir receives both tables and the replacement keeper reclaims them without cold replay.
  3. Whole-VM crash before sync: Result matches the documented durability mode; strict mode never reports a success that lacks the event.
  4. Whole-VM crash after sync: Replay applies the event even if ETS never saw it before death.
  5. Lost response retry: Same request and fingerprint returns the same session/version; it creates no second event.
  6. Conflicting retry: Same request ID with different TTL or session is rejected and reported.
  7. Expired during downtime: Restart does not expose the session as active.
  8. Forked journal fixture: Two events claim the same next version; readiness remains false with a precise conflict.
  9. Repair fixture: An unclean copied DETS file is repaired or rejected, and replay validation still executes.
  10. Capacity threshold: Writes stop before the configured safe ceiling and reads/recovery diagnostics remain available.
  11. Snapshot mismatch: A stale or unverifiable ETS snapshot is ignored and full replay succeeds.
  12. Index drift: Missing expiry entry is detected and rebuilt from current session state.

6.3 Test Data

session A: create(v1) -> touch(v2) -> revoke(v3)
session B: create(v1, expiry during shutdown)
session C: create(v1) -> two competing touch(v2) events [invalid fixture]
request req-dup: same command twice [idempotent]
request req-conflict: same ID, different TTL [reject]
journal files: graceful-close, unclean-close copy, truncated copy, near-threshold fixture

6.4 Definition of Done

  • ETS sessions and ordered expiry projection have explicit owners, access modes, and invariants.
  • Keeper failure transfers tables through an heir and returns to a valid owner generation.
  • Whole-VM termination reconstructs the same session versions from DETS.
  • Strict success is emitted only after the event crosses the documented sync boundary.
  • Exact retries return the original result; conflicting request-ID reuse is rejected.
  • Expired and revoked sessions never resurrect after downtime.
  • Startup remains unready during open/repair/validation/replay/index checks.
  • DETS 2 GB limit, no ordered_set, repair cost, and local-only durability are documented and tested where practical.
  • tab2file is treated only as a validated optional snapshot accelerator.
  • Crash tests use a separate OS process and cover every named protocol phase.
  • No destructive test modifies the only copy of journal data.
  • The runbook states recovery, capacity, backup, and fail-closed procedures.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
API worker owns ETS Ordinary request crash empties cache Use a dedicated lifecycle owner
ETS updated before DETS Successful session disappears after VM crash Persist/sync before projection update
Heir called persistence Host restart loses inherited table Explain local transfer boundary and replay DETS
Named table recreated blindly Startup badarg or stale table reference Handshake with heir and generation token
Two durable records per command Event exists without idempotency record Put request identity and result in one event
DETS traversal treated as ordered Replay differs between runs Store explicit per-session version and validate
Autosave described as strict durability Last acknowledged updates vanish in kill test Use sync or name the weaker loss window
Repair accepted as correctness proof Readable file contains broken version chains Run business-level replay validation
Full expiry scan Latency spikes and scheduler pressure Ordered derived index and bounded batches
Unbounded journal Approaches 2 GB and replay becomes operationally unsafe Thresholds, retention, verified compaction plan

7.2 Debugging Strategies

  • Inspect ownership first: Report ETS owner, heir, table identifiers, generation, protection, sizes, and transfer messages.
  • Trace one request ID: Follow it through validation, journal lookup/append/sync, ETS apply, and response without logging secret payloads.
  • Separate file repair from replay validation: Record both outcomes and durations.
  • Reproduce on copied data: Preserve the original DETS file before force-repair or truncation experiments.
  • Compare journal and projection counts carefully: They need not match one-to-one because many events reduce to one current session.
  • Use phase markers in crash tests: Kill on an externally observable marker rather than a timing sleep.

7.3 Performance Traps

Strict sync per mutation can dominate latency, DETS scans make full replay grow with history, and ordered_set maintenance adds write cost. ETS concurrency flags can consume more memory or harm alternating workloads. Benchmark p50/p95/p99 for reads and strict writes separately. Measure replay and repair with realistic journal sizes. Do not improve benchmark numbers by weakening the acknowledgement guarantee without changing the reported durability class.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add session-list pagination using bounded ETS selection.
  • Add a diagnostic command explaining owner, heir, and generation.
  • Export a verified point-in-time ETS snapshot and prove it is optional.

8.2 Intermediate Extensions

  • Add a named batch durability mode with a measured and tested loss window.
  • Add versioned event upcasting and an unsupported-downgrade failure.
  • Build safe compaction into a new journal generation with manifest cutover and rollback.

8.3 Advanced Extensions

  • Shard coordinators by session ID while preserving per-session ordering and idempotency.
  • Encrypt journal payload fields with key rotation and recovery tests.
  • Stream verified off-host backups and conduct a disk-loss restoration drill.
  • Replace local authority with Mnesia or an external database and compare failure semantics rather than merely swapping APIs.

9. Real-World Connections

9.1 Industry Applications

  • Authentication gateways: Hot session validation with durable revocation evidence.
  • Payment/webhook receivers: Idempotency-key projections around uncertain responses.
  • Edge agents: Local durable configuration and command checkpoints when disconnected.
  • Developer tooling: Fast indexes rebuilt from local durable metadata.
  • Game/session servers: Read-heavy presence or session projections with explicit restart behavior.
  • Erlang/OTP :ets: The runtime storage primitive used for the hot projection.
  • Erlang/OTP :dets: The disk term store used for the bounded journal.
  • Ecto/Oban ecosystems: Useful production contrasts where an external transactional database supplies stronger multi-record guarantees.

9.3 Interview Relevance

This project prepares you to discuss process-owned state, supervisor/data boundaries, idempotency, write acknowledgement, replay, monotonic versus system time, crash consistency, and why a fast cache is not a database. A strong explanation names the exact failure each mechanism handles and refuses to claim durability from supervision alone.

10. Resources

10.1 Essential Official Reading

10.2 Books and Deeper Study

  • Erlang and OTP in Action by Martin Logan, Eric Merritt, and Richard Carlsson — OTP and ETS design.
  • Designing for Scalability with Erlang/OTP by Francesco Cesarini and Steve Vinoski — fault-tolerant architecture and state.
  • Elixir in Action by Saša Jurić — server processes and fault tolerance.
  • Designing Data-Intensive Applications by Martin Kleppmann — logs, durability, idempotency, and failure reasoning.

10.3 Verification Checklist Before Sharing the Project

  • Every persistence claim is tied to a tested acknowledgement boundary.
  • ETS heir behavior is described as local table transfer, not durable storage.
  • DETS limitations and repair behavior are visible in the guide and output.
  • Crash fixtures use copies and preserve the original data.
  • The project contains pseudocode, schemas, transcripts, and configuration ideas—but no complete runnable solution.
  • Official documentation links are current and primary.