Project 35: Distributed Maintenance Work-Order Ledger with Mnesia

Build a three-node maintenance ledger whose work-order transitions and audit events commit transactionally to replicated Mnesia disk tables, whose application stays unready until required tables are accessible, and whose operators practice node loss, partition policy, backup, restore, and disaster recovery instead of assuming replication makes data safe.

Quick Reference

Attribute Value
Difficulty Level 4: Advanced
Suggested Seniority Staff Elixir/BEAM or distributed-platform engineer
Time Estimate 35-55 hours
Main Programming Language Elixir with Erlang/OTP Mnesia APIs
Alternative Programming Languages Erlang, Gleam with Erlang interop
Coolness Level Level 5: Pure Magic
Business Potential Level 4: Open Core Operational Platform
Prerequisites OTP applications and supervision, distributed Erlang node naming/cookies/TLS awareness, transactions and locking, records/tuples, failure testing, basic backup operations
Key Topics Mnesia schema and tables, transactions, disc_copies, ram_copies, replication, majority writes, startup readiness, network partitions, backup/restore, recovery runbooks

1. Learning Objectives

By completing this project, you will:

  1. Explain the difference among Mnesia ram_copies, disc_copies, and disc_only_copies, including their memory, disk, startup, and performance implications.
  2. Create a stable multi-node schema and keep each node’s Mnesia directory unique and tied to a durable node identity.
  3. Model work-order state, immutable audit history, and a durable outbox with keys and table types that support real queries.
  4. Use Mnesia transactions for multi-table invariants and keep irreversible external side effects outside transaction functions.
  5. Distinguish transactional access from dirty operations and justify every dirty operation that remains.
  6. Gate application readiness with explicit table accessibility rather than merely checking that Mnesia or the supervisor has started.
  7. Add and remove table replicas while observing transfer, load, and failure behavior.
  8. Define a conservative network-partition policy using majority-protected critical tables and an operator-controlled recovery procedure.
  9. Prove that replication is not backup by deleting or corrupting data consistently, then restoring a verified backup into an isolated recovery cluster.
  10. Produce operational evidence for clean restart, abrupt node loss, replica catch-up, netsplit, backup, online restore, and full disaster recovery.

2. All Theory Needed (Per-Concept Breakdown)

Mnesia Tables, Transactions, and Durable Replication

Fundamentals

Mnesia is an Erlang/OTP distributed database for Erlang terms. A schema records table definitions and replica placement across named BEAM nodes. A table can keep ram_copies, which are memory-resident and not written to disk per transaction; disc_copies, which hold the full table in RAM while logging updates to disk; or disc_only_copies, which keep data only on disk and trade speed for lower memory use. Tables can be set, ordered_set, or bag. Mnesia transactions coordinate reads and writes with locks and return either an atomic result or an aborted reason. Replicas improve availability and local access, but every additional synchronous participant affects write cost. Mnesia operates over distributed Erlang; it does not replace node discovery, secure distribution, stable node naming, or network-partition policy.

Deep Dive

Start with domain invariants, not table APIs. A maintenance work order has one current version and a constrained state machine: for example, reported -> triaged -> assigned -> in_progress -> completed, with explicit cancellation paths. Every accepted transition must update the current work order and append an immutable audit event with the same resulting version. If a downstream notification is needed, a durable outbox item must be inserted in that same transaction. The invariant is that a visible current version has exactly one corresponding audit event and any promised side-effect intent—not that three independent writes usually succeed together.

The work_orders table is a set keyed by work-order ID. The work_order_events table can be an ordered_set keyed by {work_order_id, version} to support ordered history. The outbox is a set keyed by a stable event ID. All three critical tables use disc_copies on the three core database nodes so each node keeps a memory-resident replica backed by disk logging. A derived technician-presence or UI-count table can use ram_copies because it is deliberately ephemeral and reconstructable. A very large archive might motivate disc_only_copies, but it is slower and should not be introduced without benchmarks and query analysis.

Mnesia table types describe local data structure semantics, while copy types describe placement and persistence. These axes are independent: an ordered_set table may have disc_copies, and a set may have ram_copies. Keys and record attributes need a versioned schema. Mnesia records are tuples; table and record names, attribute order, and transforms matter during upgrades. Do not derive atoms from untrusted request fields. Keep external maps at the boundary, validate them, and convert only into known internal tuples.

Transactions are functions evaluated by Mnesia under a transaction context. Reads obtain appropriate locks; writes become visible atomically if the transaction commits. The result shape communicates {atomic, result} or {aborted, reason}. Abort reasons are part of the service contract: a stale expected version should become a business conflict, while node/table unavailability should become a temporary infrastructure failure. Do not flatten every abort to 500 or retry forever.

The transition transaction reads the work order with a write lock, checks expected version and allowed state, creates the next current record, event, and outbox entry, and commits them together. The external notification dispatcher runs after commit, reads outbox records, sends with an idempotency key, and marks completion in a later transaction. Never send email, call HTTP, or publish an irreversible message from inside the transaction function. Transaction work can abort and may be retried; an external effect is not rolled back by Mnesia.

Dirty operations bypass transaction coordination and locking. They can be faster, but a dirty multi-step update can leave related tables inconsistent. The project forbids dirty writes to work orders, audit history, and outbox. A dirty read might be acceptable for an approximate operations counter if the UI labels it approximate and no decision depends on it. This is a semantic choice, not a universal performance rule. Benchmark contention and redesign hot keys before discarding transactions that protect business truth.

Replication placement changes both availability and cost. With a replica on each core node, local reads avoid a network hop when the table is loaded locally. Writes involve the required replica/transaction protocol and can be affected by unavailable nodes. When a node returns, Mnesia can catch up inaccessible replicas, but operators must observe table loading and consistency events rather than assuming an immediately connected node is ready. Use table_info and system information to expose replica placement, storage type, load node, size, and accessibility.

The table option majority: true on critical tables rejects non-dirty updates unless a majority of replicas is available. In a three-replica design, a one-node minority should not accept authoritative work-order transitions. This favors consistency over availability. It is not a complete consensus system or an automatic conflict resolver. All critical tables participating in the invariant need compatible placement and policy; otherwise a transaction may face asymmetric availability. Dirty writes can bypass the protection and are therefore prohibited.

Mnesia stores data by node identity. Long node names, hostname resolution, distribution cookies/certificates, and persistent data directories must remain aligned. Reusing one directory from two nodes or renaming a node casually can make the schema unusable or dangerous. Each node owns a unique persistent directory. Adding a new node is an explicit schema/replica operation, not merely starting the same release with another node name.

Finally, Mnesia is suited to bounded, tightly operated BEAM clusters where its semantics and failure modes are understood. This project does not claim it replaces every external database. The learning value comes from seeing application processes, transactions, distribution, storage copies, and recovery in one OTP system—and from documenting where that coupling becomes an operational cost.

How this fits on the project

This concept defines the three critical ledger tables, ephemeral projections, transactional transition function, durable outbox, replica placement, majority protection, storage directories, and application-facing error classification.

Definitions and key terms

  • Schema: Distributed metadata describing tables, attributes, types, and replica placement.
  • Table type: Key/object semantics such as set, ordered_set, or bag.
  • Copy type: Storage placement such as ram_copies, disc_copies, or disc_only_copies.
  • Transaction: Coordinated activity that atomically commits or aborts database operations.
  • Dirty operation: Access that bypasses normal transaction management and locking.
  • Majority: Table policy requiring more than half of configured replicas for non-dirty updates.
  • Outbox: Durable intent for an external side effect committed with the business change.
  • Replica catch-up: Later propagation to a replica that was inaccessible during earlier activity.

Mental model diagram

                      one logical Mnesia schema

 client transition
       |
       v
 +---------------- transaction ----------------+
 | read work_order(version=N, write lock)       |
 | validate transition                          |
 | write work_order(version=N+1)                |
 | write audit_event(order_id, version=N+1)     |
 | write outbox(event_id, pending)               |
 +--------------------+-------------------------+
                      |
              atomic commit / abort
                      |
          +-----------+-----------+
          |                       |
   node maint-a              node maint-b              node maint-c
   disc_copies               disc_copies               disc_copies
   RAM + disk log            RAM + disk log            RAM + disk log
          |                       |                          |
          +---------------- replica protocol ---------------+

 after commit: OutboxDispatcher -> external provider(idempotency key)

How it works

  1. Bootstrap a schema for stable node names and start Mnesia on each configured database node.
  2. Create critical tables with compatible disc_copies placement and majority policy.
  3. Wait for every required table before accepting application traffic.
  4. Receive a transition with work-order ID, expected version, operation, actor, and request ID.
  5. In one transaction, lock/read current state, validate, write next state, append audit event, and create outbox intent.
  6. Return committed version only for an atomic result; classify aborts explicitly.
  7. Dispatch external effects after commit with the outbox ID as idempotency key.
  8. Add or restore replicas through controlled schema operations and observe table accessibility.
  9. Invariant: every committed current version has one matching event and durable side-effect intent.
  10. Failure modes include mismatched node identity, unavailable tables, incompatible replica placement, dirty-write invariant violations, hot locks, disk exhaustion, and external effects inside transactions.

Minimal concrete example

TRANSACTION PLAN, NOT RUNNABLE CODE

input:
  work_order=WO-1842
  expected_version=7
  transition=start_work
  actor=tech-19
  request_id=req-a81f

inside transaction:
  current = locked_read(WO-1842)
  require current.version == 7
  require transition_allowed(assigned, start_work)
  write current(state=in_progress, version=8)
  write event(key={WO-1842, 8}, request_id=req-a81f)
  write outbox(key=req-a81f, status=pending)

result: atomic({WO-1842, version=8, state=in_progress})

Common misconceptions

  • disc_copies means data lives only on disk and every read performs disk I/O.
  • Replicated tables remove the need for backups.
  • Mnesia automatically discovers and safely adds every connected BEAM node to the schema.
  • Dirty writes still honor transactions and majority protection.
  • External API calls inside a transaction are rolled back when the transaction aborts.
  • A connected node is ready before its required tables finish loading.

Check-your-understanding questions

  1. Why use disc_copies rather than ram_copies for work orders?
  2. Why are the current record, audit event, and outbox inserted in one transaction?
  3. What does majority: true protect, and what bypasses it?
  4. Why must each database node have a unique Mnesia directory?

Check-your-understanding answers

  1. Work orders must survive full node restarts; disc_copies keeps RAM access while logging updates to disk.
  2. They form one business invariant and must not become visible independently.
  3. It rejects non-dirty updates without a majority of replicas; dirty operations bypass normal transaction protections.
  4. The files contain node-specific schema and table state; sharing a directory creates unpredictable and unsafe behavior.

Real-world applications

  • Telecom/network configuration inside managed Erlang clusters
  • Industrial maintenance and dispatch ledgers
  • Durable control-plane metadata for bounded BEAM deployments
  • Workflow state with local low-latency reads and immutable audit history
  • Embedded/edge clusters where an external database is undesirable but operations are controlled

Where you will apply it

  • Project 35 schema bootstrap, work-order service, event history, outbox dispatcher, replica manager, and diagnostics

References

Key insight

Mnesia replication is useful only after the application’s atomic invariant, replica placement, node identity, and allowed failure behavior are all explicit.

Summary

Model current state, history, and side-effect intent transactionally; choose copy types by durability and access needs; prohibit dirty invariant writes; and treat replica topology as operated schema, not incidental node connectivity.

Homework/Exercises

  1. Classify work orders, audit events, presence, large attachments, and outbox items by table and copy type.
  2. Trace the result if the notification provider is called inside a transaction that later aborts.
  3. Decide whether a two-node cluster can satisfy the same majority availability goal as three nodes.

Solutions

  1. Work orders/events/outbox use durable critical tables; presence is rebuildable RAM state; attachments belong in object storage with durable references rather than large Mnesia records.
  2. The external notification can exist without committed business state, so the call must move to an idempotent post-commit dispatcher.
  3. With two replicas, loss of either leaves no majority for protected writes; three replicas tolerate one unavailable node while retaining a two-node majority.

Startup, Netsplit, Backup, and Disaster-Recovery Operations

Fundamentals

Starting the Mnesia application does not guarantee that every table needed by the service is accessible. Disk tables load during startup, replicas may wait for another node, and a communication failure may leave nodes with conflicting histories or an inconsistent_database event. Applications use mnesia:wait_for_tables/2 or equivalent readiness gates before serving dependent operations. Replication protects against some node failures, while backup protects against logical deletion, corruption, operator error, and loss of all replicas. Mnesia can create backups, restore tables online, and install a fallback for restart-oriented recovery. Network-partition recovery still requires a declared consistency policy and often an operator decision about which island is authoritative.

Deep Dive

Bootstrap and normal startup are different workflows. Schema creation is an administrative action performed for a known set of stable nodes, not something every application boot should repeat. Table creation and replica changes are migrations with their own idempotency and rollback plan. Normal startup starts distribution, connects to configured database peers, starts Mnesia, waits for required tables, validates table placement/schema version, and only then marks the work-order API ready. If a table wait times out, the supervisor can remain alive for diagnostics while readiness stays false; blindly restarting in a tight loop can make recovery harder.

Table loading has operational dependencies. A node with durable replicas can load them from its local directory under Mnesia’s rules, while RAM-only replicas need another source. Load order can prioritize small foundational tables. The readiness report should name each required table, local storage type, configured replica holders, load state, load source, and timeout. “Node connected” and “Mnesia running” are intermediate states, not proof that work-order transactions are safe.

During an ordinary one-node failure in a healthy three-node cluster, the remaining majority continues protected transactions. When the failed node returns with the same identity and data directory, Mnesia loads/catches up its replicas. The node must remain out of the application load balancer until required tables are accessible and the schema/replica audit passes. Test both graceful stop and abrupt OS-process loss. Also test disk loss separately: restarting with an empty directory is not the same recovery path as restarting an intact replica.

A network partition is harder because each side may remain alive. Mnesia can detect histories suggesting partitioned operation and emit inconsistent_database events. It does not decide which business history to preserve. Majority-protected critical tables prevent the one-node minority of a three-replica cluster from committing normal transactional updates. The two-node side can continue if it retains the majority. The application subscribes to Mnesia system events, marks any affected node unready, and records a recovery incident. It never automatically calls a force-load operation merely to make the error disappear.

After connectivity returns, the runbook identifies the authoritative island using the declared policy and evidence: majority membership, committed work-order versions, event chains, operator timeline, and backup checkpoints. Mnesia master-node controls can direct later table loading from chosen replicas. Force-loading a table is an emergency action because choosing the wrong copy can discard newer state. The project performs these steps only in an isolated drill and records commands, selected authority, before/after versions, and verification. If both islands accepted writes because majority protection was missing or dirty writes bypassed it, reconciliation becomes a domain-specific incident, not a transparent merge.

Replication and backup address different threats. Three replicas can faithfully replicate an accidental delete, a bad migration, or application corruption to every node. A backup is a separate recovery artifact with retention and integrity controls. mnesia:backup creates a backup of tables and schema through the backup mechanism. A backup is not trusted merely because the call returned success: store its checksum and metadata, copy it outside the cluster’s failure domain, and restore it regularly into an isolated cluster with different node identities or an explicitly planned mapping.

Online restore can clear, keep, skip, or recreate selected tables depending on options. The affected tables are write-locked, and the restore is performed as one transaction, so availability and capacity need planning. For very large recovery, Mnesia supports installing a fallback that takes effect on restart. Fallback operations and distributed schema restoration are advanced and potentially destructive; practice on disposable directories and verified copies. The core drill restores work-order tables into an isolated recovery environment, validates version/event invariants, and only then documents promotion steps.

Recovery objectives make the exercise measurable. Recovery point objective (RPO) is bounded by backup frequency for disasters that remove all current replicas. Recovery time objective (RTO) includes artifact retrieval, node/schema preparation, restore, table loading, validation, and traffic promotion. Replica failover may have a far shorter RTO and near-zero acknowledged-data RPO under the tested transaction policy, but do not transfer that claim to logical corruption or total-cluster loss.

Schema changes require the same caution. Changing record attributes or table placement during a partial outage can complicate recovery. Version migrations, backups, and rollback/fallback plans belong together. Before a migration, capture and verify a backup, ensure required nodes are healthy, transform under a documented procedure, and validate table metadata and sample business invariants. An application release and data schema are related but separate versioned concerns.

Observability should favor actionable state over raw volume. Track transaction abort categories, table wait duration, unavailable tables, replica placement drift, Mnesia system events, backup age, last restore drill, disk usage, log/dump activity, and outbox lag. Never expose the distribution cookie, sensitive work-order content, or raw internal files through a public endpoint. A local operator command can show the evidence needed for recovery without making unsafe repair automatic.

The final operational rule is to fail closed when authority is ambiguous. Keeping a process alive is not more important than preventing two different maintenance histories from being presented as one. The service may offer a clearly labeled read-only view during an incident only if the chosen data source and staleness are known. Writes remain unavailable until the consistency policy and table state are satisfied.

How this fits on the project

This concept defines bootstrap versus normal boot, table readiness, returning-node gates, partition event handling, majority-island policy, master-node/force-load caution, backup verification, isolated restore, RPO/RTO measurement, and recovery observability.

Definitions and key terms

  • Table accessibility: State in which a table can serve the required Mnesia activity.
  • Database node: Node included in the Mnesia schema and table replica topology.
  • Netsplit: Communication partition that separates live nodes into islands.
  • Inconsistent database event: Mnesia system evidence that partitioned histories may require intervention.
  • Master node: Configured authoritative source from which a table should load during recovery.
  • Force load: Emergency loading action that bypasses normal safety logic and can select stale data.
  • Backup: Separate recoverable representation of schema/table data outside current replicas.
  • Fallback: Backup installed for restoration during a subsequent Mnesia restart.
  • RPO/RTO: Maximum acceptable data-loss interval and target time to restore service.

Mental model diagram

normal state
  [maint-a] <----> [maint-b] <----> [maint-c]
       \______________|_______________/
              3 durable replicas

partition
  majority island                         minority island
  [maint-a] <----> [maint-b]              [maint-c]
       | protected writes continue             | majority write aborts
       |                                        | readiness=false
       +----------------- X --------------------+

heal is not “forget the incident”:
  detect event -> freeze unsafe node -> choose authority -> reload/catch up
              -> validate versions/events -> readiness -> traffic

separate failure domain:
  cluster --backup--> immutable artifact --restore drill--> isolated cluster

How it works

  1. Bootstrap known node identities, directories, schema, and table replicas through an administrative workflow.
  2. On normal boot, connect database peers, start Mnesia, wait for required tables, audit metadata, and then become ready.
  3. Continue majority-protected writes after one clean node loss; keep the returning node out of traffic until catch-up and validation finish.
  4. Subscribe to Mnesia system events and mark affected nodes unready on partition inconsistency.
  5. Allow writes only on the island satisfying the documented majority policy.
  6. After healing, select authoritative replicas using evidence and controlled recovery settings; never auto-force-load.
  7. Create versioned backups, checksum them, store them off-cluster, and test restore in isolation.
  8. Validate current records against ordered audit histories and pending outbox intents before promotion.
  9. Invariant: traffic reaches a node only when every required table and consistency check is ready.
  10. Failure modes include table wait timeout, hostname/node rename, shared data directory, replica drift, split-brain dirty writes, stale force-load, replicated logical deletion, untested backup, and restore that exceeds capacity.

Minimal concrete example

READINESS REPORT, NOT RUNNABLE CODE

node: maint-c@ops.internal
distribution: connected_to=[maint-a, maint-b]
mnesia: running
required_tables:
  work_orders: waiting_for_table
  work_order_events: loaded_from=maint-a
  outbox: loaded_from=maint-b
schema_version: 4
replica_policy: mismatch(work_orders expected=3 actual=2)
ready: false
reason: required_table_unavailable

Common misconceptions

  • mnesia:start() means all tables are ready.
  • A returning node can receive traffic as soon as Node.connect succeeds.
  • Three replicas protect against an accidental delete on all replicas.
  • Mnesia resolves partition conflicts automatically when connectivity returns.
  • force_load_table is a routine fix for slow startup.
  • A backup is valid without a successful isolated restore drill.

Check-your-understanding questions

  1. Why can a supervisor be healthy while the application remains unready?
  2. What failure does majority protection reduce in a three-node cluster?
  3. Why is replication not a defense against logical deletion?
  4. What evidence should be recorded before choosing master nodes after a partition?

Check-your-understanding answers

  1. Processes can be alive while required Mnesia tables are still loading, unavailable, or inconsistent.
  2. It prevents the one-node minority from committing normal protected updates without two replicas available.
  3. The delete is a valid database operation and is propagated to current replicas.
  4. Island membership, transaction/event versions, Mnesia events, operator timeline, table placement, and relevant backup checkpoints.

Real-world applications

  • Stateful BEAM control planes with strict readiness gates
  • Telecom systems using replicated local data
  • Industrial sites with small managed node clusters
  • Recovery automation that gathers evidence but leaves destructive authority decisions to operators
  • Compliance systems requiring tested backup and immutable audit history

Where you will apply it

  • Project 35 bootstrap task, readiness gate, cluster status command, partition event listener, backup manager, restore verifier, and disaster-recovery runbook

References

Key insight

Distributed durability is an operational protocol: table readiness, partition authority, and verified restore matter as much as successful transactions.

Summary

Separate bootstrap from boot, gate traffic on tables, choose consistency during partition, distrust automatic force recovery, and verify backups in another failure domain. Availability claims must name the exact failure tested.

Homework/Exercises

  1. Write the readiness state machine from process start through traffic admission.
  2. Draw outcomes for partitions of 2+1, node crash, disk loss, and logical deletion.
  3. Define RPO/RTO separately for replica failure and total-cluster loss.

Solutions

  1. Distribution/configured peers -> Mnesia running -> tables waiting -> metadata validated -> no unresolved inconsistency -> ready; any failed stage remains diagnosable but unready.
  2. A 2+1 partition permits majority writes only on two nodes; a node crash uses remaining replicas; disk loss requires a new/recovered copy; logical deletion needs backup or domain compensation.
  3. Replica failure can target near-zero acknowledged-data loss and short catch-up; total-cluster loss is bounded by backup interval plus artifact/schema restoration and validation time.

3. Project Specification

3.1 What You Will Build

Build maint-ledger, a three-node service used by a factory or field-service team to report, triage, assign, start, complete, and cancel maintenance work. Each transition records the current order, an immutable versioned event, and a durable notification outbox entry in one Mnesia transaction. Operators can inspect node/table readiness, remove a node from service, add or restore replicas, simulate a 2+1 network partition, create a backup, and restore it into an isolated recovery cluster.

3.2 Functional Requirements

  1. Report work: Create a work order with site, asset, severity, description digest, reporter, and version 1.
  2. Transition safely: Require expected version and allowed state transition.
  3. Audit completely: Append one immutable event for every committed current version.
  4. Dispatch reliably: Commit an outbox intent in the transition transaction; deliver after commit with idempotency.
  5. Read from any ready node: Query current state and ordered history from any node whose required tables are accessible.
  6. Expose topology: Show database nodes, running nodes, table replica holders/copy types, majority policy, and readiness.
  7. Survive one-node loss: Continue protected transitions on the two-node majority and recover the returning replica.
  8. Reject unsafe minority writes: A one-node island must not commit critical transitions.
  9. Handle inconsistency visibly: Subscribe to partition events, fail readiness, and require a controlled recovery decision.
  10. Back up and restore: Produce a versioned backup with checksum and prove restoration into an isolated cluster.
  11. Validate recovery: Compare work-order versions, event chains, outbox status, and counts before restored data is considered promotable.

3.3 Non-Functional Requirements

  • Consistency: Critical mutations use transactions and majority-protected durable replicas.
  • Availability: The healthy two-node majority remains usable after one node loss; minority writes fail closed.
  • Durability: Work orders, events, and outbox use disc_copies on three core nodes.
  • Readiness: No request is served before required table accessibility and metadata validation.
  • Performance: Publish transaction latency and abort rate under normal, node-loss, and catch-up conditions.
  • Security: Secure distributed Erlang in production design; never expose cookies or raw database files.
  • Recoverability: Backup age and last successful isolated restore are observable.

3.4 Example Usage / Output

$ maint-ledger report --site CWB-01 --asset press-7 --severity high --request req-101
committed work_order=WO-1842 state=reported version=1 node=maint-a

$ maint-ledger transition WO-1842 triage --expect-version 1 --request req-102
committed work_order=WO-1842 state=triaged version=2 audit_event=WO-1842/2

$ maint-ledger history WO-1842 --node maint-c
1 reported actor=operator-4 request=req-101
2 triaged actor=planner-2 request=req-102

$ maint-ledger cluster status
ready_nodes=3 database_nodes=3 critical_tables=ready majority_policy=enabled backup_age=04h12m

3.5 Data Formats / Schemas / Protocols

work_orders (set, disc_copies, majority=true):
  {work_order, id, version, state, site_id, asset_id, severity,
   assignee_id, created_at, updated_at, last_request_id}

work_order_events (ordered_set, disc_copies, majority=true):
  {work_order_event, {work_order_id, version}, transition, actor_id,
   request_id, occurred_at, prior_state, resulting_state}

outbox (set, disc_copies, majority=true):
  {outbox, event_id, work_order_id, version, destination,
   payload_digest, status, attempts, next_attempt_at}

readiness:
  {node, distribution_state, mnesia_state, required_tables,
   schema_version, replica_audit, inconsistency_state, ready}

3.6 Edge Cases

  • Two clients submit different transitions from the same expected version
  • Transaction coordinator node dies before or during commit
  • One table has different replica placement from the other invariant tables
  • Node restarts with the same name but an empty data directory
  • Node name/hostname changes while old replicas remain in schema
  • Required table waits indefinitely for a missing source
  • Minority island attempts a normal transaction and a dirty write
  • Network heals after a partition with an inconsistency event
  • Outbox dispatcher sends successfully but dies before marking delivered
  • Backup succeeds but artifact copy/checksum fails
  • Online restore conflicts with live writes and holds table locks
  • Schema migration is interrupted after some nodes change
  • All replicas contain a logically deleted work order

3.7 Real World Outcome

The learner opens three terminals or container logs showing maint-a, maint-b, and maint-c. A work order created on maint-a is immediately readable on maint-c with matching history. After maint-c is killed abruptly, two nodes continue committing transitions. When it returns, it remains outside traffic while tables load and then reports the caught-up version. During a 2+1 partition, the majority commits one transition while the minority returns a clear majority-unavailable error. Healing produces an explicit recovery event and checklist rather than an automatic “all good.” Finally, the learner restores a backup into an isolated cluster and validates every current version against its event history.

$ maint-ledger drill netsplit --isolate maint-c
partition majority=[maint-a,maint-b] minority=[maint-c]

$ maint-ledger transition WO-1842 start-work --expect-version 4 --node maint-a
committed work_order=WO-1842 state=in_progress version=5 replicas_available=2/3

$ maint-ledger transition WO-1842 cancel --expect-version 4 --node maint-c
aborted reason=no_majority table=work_orders ready=false

$ maint-ledger drill heal
event=inconsistent_database affected=maint-c action=traffic_blocked
authority_candidate=[maint-a,maint-b] evidence_version=5 operator_decision=required

$ maint-ledger restore verify ./backups/maint-2026-07-15.bup --cluster recovery
tables=3 work_orders=1842 event_chains_valid=1842 outbox_orphans=0 checksum=ok promotable=true

3.8 Scope Boundary Versus Existing Projects

  • Project 3 implements a custom sharded KV store; Project 35 studies Mnesia transactions, copy types, durable schema, backups, and recovery.
  • Project 11 teaches multi-network cluster formation; Project 35 assumes nodes can connect and focuses on database readiness and durable replicas.
  • Project 12 teaches a general WAN netsplit drill; Project 35 narrows the problem to authoritative Mnesia table histories, majority-protected writes, and recovery evidence.
  • Project 22 uses Ecto and a transactional external database for inventory reservations; Project 35 uses OTP-native distributed storage for maintenance workflow state.
  • Project 34 teaches local ETS/DETS authority and whole-VM replay; Project 35 adds distributed replicas and their operational costs.

4. Solution Architecture

4.1 High-Level Design

                    load balancer / CLI router
                   only sends to ready nodes
                              |
          +-------------------+-------------------+
          |                   |                   |
          v                   v                   v
     maint-a node        maint-b node        maint-c node
   +---------------+   +---------------+   +---------------+
   | WorkOrder API |   | WorkOrder API |   | WorkOrder API |
   | ReadinessGate |   | ReadinessGate |   | ReadinessGate |
   | EventListener |   | EventListener |   | EventListener |
   +-------+-------+   +-------+-------+   +-------+-------+
           |                   |                   |
           +----------- Mnesia schema ------------+
           | work_orders / events / outbox         |
           | disc_copies on all three nodes        |
           +---------------------------------------+
                              |
                         [Outbox workers]
                              |
                    external notification API

 separate path: BackupManager -> off-cluster artifact -> RecoveryVerifier

Application supervision manages APIs, readiness, event subscribers, and dispatchers. Mnesia data lifecycle is validated independently. A healthy process tree cannot override an inaccessible or inconsistent required table. The backup verifier operates against disposable recovery directories, not the production database directory.

4.2 Key Components

Component Responsibility Key Decisions
SchemaAdmin Bootstrap schema, create/change tables and replicas Administrative, explicit, idempotent migrations
MnesiaGate Start/observe Mnesia and wait for required tables Separate liveness from readiness
ReplicaAuditor Compare actual table metadata with policy Block traffic on critical drift
WorkOrderService Execute transactional state transitions Expected version and state machine
HistoryReader Return ordered immutable events No synthesized missing history
OutboxDispatcher Deliver post-commit notifications idempotently External effects outside transaction
MnesiaEventListener Subscribe to system/table events Partition events block readiness
PartitionCoordinator Gather authority evidence for operator Never auto-force-load
BackupManager Create/checksum/catalog backup artifacts Store outside cluster failure domain
RecoveryVerifier Restore into isolation and validate invariants Promotion is a separate decision
ClusterStatus Show node, table, replica, abort, and recovery state Redact secrets

4.3 Data Structures (No Full Code)

  • Work-order current tuple: Stable ID, state, version, assignment, severity, timestamps, and last request identity.
  • Event key: {work_order_id, resulting_version} for deterministic per-order history.
  • Outbox item: Stable event ID, destination, digest, attempt state, and next-attempt time.
  • Replica policy: Required tables, expected holders, copy types, majority flag, schema version.
  • Readiness report: Per-table accessibility and metadata plus unresolved inconsistency evidence.
  • Backup catalog entry: Backup version, source cluster, schema version, timestamp, checksum, size, storage URI, and last verified restore.

4.4 Algorithm Overview

Work-order transition

  1. Validate request and known transition name before entering Mnesia.
  2. Transactionally read current order with write intent.
  3. Compare expected version and state-machine rule.
  4. Write next current version, immutable event, and pending outbox item.
  5. Commit and return; map abort reason precisely.
  6. Let the supervised dispatcher process outbox after commit.

Node admission

  1. Confirm stable node identity, secure distribution, and unique persistent directory.
  2. Connect configured database peers and start Mnesia.
  3. Wait for required tables with a bounded timeout.
  4. Audit schema version, copy type, replica holders, majority, and unresolved events.
  5. Add node to traffic only if every condition is satisfied.

Recovery verification

  1. Copy/checksum the backup artifact into an isolated environment.
  2. Prepare recovery schema/directories under the drill plan.
  3. Restore selected tables using the documented operation.
  4. Validate every current version against event chains and outbox references.
  5. Record RPO/RTO and a promotable/not-promotable result.

Complexity considerations

  • Current lookup: key-based table access, plus distribution/load state.
  • History: ordered range proportional to events returned.
  • Transition: transactional work across three tables and configured replicas.
  • Full restore verification: proportional to all current rows and audit events.

5. Implementation Guide

5.1 Development Environment Setup

Use a current supported Elixir/OTP pair and three isolated node data directories. Local development can use three OS processes or containers with stable resolvable names. Production design must replace a shared plaintext cookie assumption with a secure distribution plan and restricted network access. Build scripts for partitions and abrupt termination must target only the disposable drill environment.

5.2 Project Structure

maint_ledger/
├── config/
│   ├── config.exs
│   └── runtime.exs
├── lib/maint_ledger/
│   ├── application.ex
│   ├── schema_admin.ex
│   ├── mnesia_gate.ex
│   ├── replica_auditor.ex
│   ├── work_order_service.ex
│   ├── transition_policy.ex
│   ├── history_reader.ex
│   ├── outbox_dispatcher.ex
│   ├── mnesia_event_listener.ex
│   ├── partition_coordinator.ex
│   ├── backup_manager.ex
│   ├── recovery_verifier.ex
│   └── cluster_status.ex
├── test/
│   ├── model/
│   ├── transaction/
│   ├── cluster/
│   ├── partition/
│   └── recovery/
├── scripts/
│   ├── start_three_nodes.sh
│   ├── partition_drill.sh
│   └── restore_drill.sh
└── docs/runbooks/
    ├── bootstrap.md
    ├── node-rejoin.md
    ├── netsplit.md
    └── disaster-recovery.md

5.3 The Core Question You Are Answering

“When three BEAM nodes hold durable replicas, what proves that one committed work-order history remains authoritative through node loss, a network split, and restoration from backup?”

Answer with invariants, table-copy policy, readiness evidence, and operator decisions—not with “Mnesia handles it.”

5.4 Concepts You Must Understand First

  1. Distributed Erlang identity and connectivity
    • Why are node names, hostname resolution, cookies/TLS, and ports part of database operation?
    • Book Reference: Designing for Scalability with Erlang/OTP — distribution sections.
  2. Mnesia schema and copy types
    • Which metadata is distributed, and where is each table held?
    • What persists for each copy type?
  3. Transactions and locks
    • Which work-order invariants require one atomic activity?
    • Why are dirty writes unsafe here?
  4. Supervision versus readiness
    • Why can every child be alive while tables remain inaccessible?
  5. Partition consistency policy
    • Which island may write, and what evidence chooses authority after healing?
  6. Backup versus replication
    • Which failures are copied to every replica?
    • How is a backup proven restorable?

5.5 Questions to Guide Your Design

  1. Which node names and data directories remain stable for the cluster lifetime?
  2. Which tables form one invariant and therefore need compatible replica/majority policy?
  3. Which state is ephemeral enough for ram_copies?
  4. Can any application path issue a dirty critical write?
  5. What abort reasons are retryable, conflicting, or operationally fatal?
  6. What exact checks make a node ready for reads and writes?
  7. What happens when a returning node has an empty versus intact directory?
  8. Which island may write during a 2+1 split?
  9. Who authorizes master-node or force-load recovery?
  10. Where are backups stored, how are they checksummed, and when was the last successful restore?
  11. How are schema transformations coordinated with backup and rollback?

5.6 Thinking Exercise

Four failures, four different recoveries

Draw one timeline each for: application-process crash, one BEAM node crash with intact disk, one node’s disk loss, and a 2+1 network partition. For each, mark surviving processes, table replicas, majority, allowed writes, readiness, required operator action, and whether backup is needed. If two diagrams look identical, revisit the failure assumptions.

5.7 The Interview Questions They Will Ask

  1. “What is the difference between ram_copies, disc_copies, and disc_only_copies?”
  2. “How do Mnesia transactions differ from dirty operations?”
  3. “Why does replication not eliminate the need for backup?”
  4. “How do you prevent a minority partition from accepting conflicting work-order updates?”
  5. “Why must an application wait for tables after Mnesia starts?”
  6. “What would you inspect before calling force_load_table or selecting master nodes?”

5.8 Hints in Layers

Hint 1: Make one invariant visible

Start with one current table and one history table. Prove they never diverge before adding outbox or clustering.

Hint 2: Add replicas deliberately

Begin on one disposable node, then bootstrap three stable names and add copies through an explicit administrative flow. Print table metadata after every change.

Hint 3: Readiness is a report

Represent readiness as evidence for each required table and policy. A boolean alone hides the reason an operator needs.

Hint 4: Treat recovery as destructive

Copy directories and backup artifacts before force-load, fallback, truncation, or restore experiments. Automate evidence collection, not the authority decision.

5.9 Books That Will Help

Topic Book/Paper Chapter/Section
Mnesia architecture Mnesia — A Distributed Robust DBMS for Telecommunications Applications — Wikström, Nilsson Full paper
OTP distribution and recovery Designing for Scalability with Erlang/OTP — Cesarini, Vinoski Distribution and operations sections
Erlang databases and OTP Erlang Programming — Cesarini, Thompson Mnesia and distributed programming sections
Partition trade-offs Designing Data-Intensive Applications — Kleppmann Replication, transactions, and distributed-system trouble chapters

5.10 Implementation Phases

Phase 1: Transactional single-node ledger (10-14 hours)

  • Define state machine, current/event/outbox records, and transaction result mapping.
  • Prove expected-version conflicts and event/current invariants.
  • Add dispatcher idempotency without external effects inside transactions.
  • Checkpoint: forced transaction abort leaves no partial current/event/outbox state.

Phase 2: Three durable replicas and readiness (12-18 hours)

  • Bootstrap stable nodes/directories and critical disc_copies.
  • Add table waits, replica audit, cluster status, and traffic admission.
  • Kill/rejoin one node and measure continuation/catch-up.
  • Checkpoint: two ready nodes continue while a returning node remains unready until tables validate.

Phase 3: Partition and disaster recovery (13-23 hours)

  • Enable/test majority policy and Mnesia event subscription.
  • Run a 2+1 partition and controlled heal procedure.
  • Create/checksum/offload backup and restore it into isolation.
  • Measure RPO/RTO and complete operator runbooks.
  • Checkpoint: recovery verifier proves current/event/outbox invariants before promotion.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Critical copy type RAM / disk+RAM / disk-only disc_copies on three nodes Durable restart plus local RAM reads
Critical access transaction / dirty transaction Protects multi-table invariant and majority policy
Side effects inside transaction / outbox outbox Keeps irreversible I/O after commit and retryable
Cluster size two / three three core replicas One-node loss retains majority
Admission process alive / table-ready policy table-ready policy Prevents serving incomplete state
Partition policy both islands write / majority writes majority writes Explicit consistency preference
Heal behavior automatic force load / operator authority operator authority with evidence Avoids silently choosing stale history
Backup proof file exists / isolated restore isolated restore Tests actual recoverability

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
State-machine tests Prove domain transitions allowed/forbidden transitions and version increments
Transaction tests Prove atomic invariant abort after each planned write, concurrent expected versions
Replica tests Prove placement and durability table-copy audit, node stop/rejoin, empty data directory
Readiness tests Prove traffic gating table timeout, schema drift, unresolved inconsistency
Partition tests Prove majority policy 2+1 split, minority transaction, dirty-write guard
Backup tests Prove recoverability checksum, isolated restore, invariant scan
Performance tests Measure operating envelope transaction latency, abort rate, catch-up, restore duration

6.2 Critical Test Cases

  1. Atomic transition: Current row, audit event, and outbox item appear together or not at all.
  2. Optimistic conflict: Two callers use version 4; exactly one reaches version 5 and the other receives a conflict.
  3. Coordinator failure: Node/process loss during a transaction never exposes partial invariant tables.
  4. Clean node stop: Two-node majority remains ready and commits protected writes.
  5. Abrupt node loss: Same availability goal, with explicit event/catch-up evidence.
  6. Returning intact replica: Node stays unready until required tables load and latest version is visible.
  7. Lost data directory: Node is not treated as an intact replica; controlled replica restoration is required.
  8. Minority partition: Normal critical transaction aborts for no majority.
  9. Dirty-write prohibition: Application API contains no route that bypasses the transactional policy; a drill demonstrates why.
  10. Heal event: Inconsistent-database evidence blocks readiness until the runbook resolves authority.
  11. Logical delete: Delete reaches every replica, proving replica count is not backup.
  12. Isolated restore: Backup restores and every current order has a contiguous event chain and valid outbox references.
  13. Corrupt backup: Checksum or restore failure produces not-promotable status.
  14. Schema mismatch: Application refuses traffic when table attributes/version do not match expected metadata.

6.3 Test Data

WO-100: reported -> triaged -> assigned -> in_progress -> completed
WO-101: reported -> cancelled
WO-102: competing assigned and cancelled transitions from same version
WO-103: pending outbox delivery with provider timeout
cluster fixtures: healthy-3, node-down-1, empty-rejoin, split-2-plus-1
backup fixtures: valid, checksum-mismatch, missing event, orphan outbox

6.4 Definition of Done

  • Three stable database nodes use unique persistent Mnesia directories.
  • Critical tables have documented type, attributes, disc_copies placement, and majority policy.
  • Current work order, audit event, and outbox intent commit atomically.
  • No external network side effect occurs inside a Mnesia transaction.
  • Dirty critical writes are absent and guarded by design/tests.
  • Every node remains unready until required tables and replica policy validate.
  • One-node clean and abrupt loss preserve tested majority operation.
  • Returning replicas catch up before traffic admission.
  • The one-node minority cannot commit protected transitions during a 2+1 partition.
  • Healing a partition produces evidence and an operator-controlled authority decision.
  • Backup artifacts are checksummed, stored outside the cluster failure domain, and restored in isolation.
  • Restore verification checks current versions, contiguous event chains, and outbox references.
  • Runbooks cover bootstrap, node rejoin, netsplit, backup, online restore, and disaster recovery.
  • RPO/RTO results distinguish replica failure from total-cluster loss.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Schema bootstrap on every boot Conflicting create/change operations Separate administrative bootstrap/migrations
Shared Mnesia directory Unpredictable files and corruption risk One unique persistent directory per node
Renamed node with old data Tables wait or schema references ghost node Treat node identity as durable configuration
API ready after mnesia:start Requests fail with table unavailable Wait/audit every required table
Mixed replica policy Multi-table transaction loses expected availability Align critical table holders/copy/majority settings
Dirty invariant update Current state without matching event Prohibit dirty writes to critical tables
External call in transaction Duplicate notification after transaction retry/abort Durable outbox after commit
Majority assumed universal Dirty or differently configured table writes on minority Test every critical access path
Automatic force load Stale island becomes authoritative Require evidence and operator decision
Replicas called backups Logical delete exists everywhere Off-cluster backup and restore drills
Untested backup Artifact exists but cannot recreate valid schema/state Isolated restore plus invariant verification

7.2 Debugging Strategies

  • Inspect table metadata: Compare configured and actual replica holders, copy types, majority, load source, size, and accessibility.
  • Separate distribution from database state: Test node ping/RPC, then Mnesia running nodes, then table readiness.
  • Classify aborts: Count conflicts, no-majority, unavailable-table, lock/contention, and application validation separately.
  • Subscribe to system events: Preserve the first partition/inconsistency evidence before attempting recovery.
  • Trace one work-order version: Compare current record, ordered event key, and outbox event ID on each ready node.
  • Recover only from copies: Duplicate directories and backup artifacts before destructive investigation.

7.3 Performance Traps

Replicated transactions pay network and disk costs, hot work orders contend on locks, long transaction functions retain locks, and large records increase replication overhead. disc_only_copies can reduce memory but slow access substantially. Backup/restore and replica catch-up compete for I/O. Measure normal and degraded modes separately. Do not substitute dirty writes for a hot design without first partitioning contention or changing the domain workflow.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add read-only work-order filtering by site and state with justified indexes.
  • Show a per-table readiness explanation in a small operations UI.
  • Add a local-only ephemeral technician-presence ram_copies table and document its restart behavior.

8.2 Intermediate Extensions

  • Add a controlled new-node join and old-node retirement procedure.
  • Add versioned table transformations with pre-migration backup and rollback drill.
  • Add backup retention, off-cluster upload, and scheduled isolated verification.

8.3 Advanced Extensions

  • Implement encrypted Erlang distribution and certificate rotation without changing node identity.
  • Fragment a high-volume event table and measure routing/rebalance trade-offs.
  • Design a multi-site read-only replica strategy and explicitly reject unsafe WAN write assumptions.
  • Compare Mnesia recovery semantics against an external consensus database using the same work-order invariant and fault matrix.

9. Real-World Connections

9.1 Industry Applications

  • Industrial maintenance: Durable work state and immutable transition evidence close to factory systems.
  • Telecommunications: Small managed Erlang clusters historically motivated Mnesia’s integrated distribution model.
  • Control planes: Replicated configuration/state with strict node readiness.
  • Edge sites: Local cluster operation during upstream disconnection, under a defined authority policy.
  • Audit workflows: Current state linked to ordered immutable business events.
  • Erlang/OTP Mnesia: The database and recovery system used directly by the project.
  • Phoenix/Elixir clustering libraries: Possible discovery layer; they do not create Mnesia schema or choose data authority.
  • Oban/Ecto/PostgreSQL: A useful external-database comparison for durable jobs and transactions.

9.3 Interview Relevance

This project prepares you to reason about OTP-native persistence, transaction boundaries, replica placement, majority availability, split-brain policy, table readiness, schema identity, outbox patterns, and backup validation. Strong answers distinguish process supervision, database replication, and disaster recovery instead of treating them as synonyms.

10. Resources

10.1 Essential Official Reading

10.2 Books and Deeper Study

  • Mnesia — A Distributed Robust DBMS for Telecommunications Applications by Claes Wikström and Hans Nilsson — foundational architecture paper.
  • Designing for Scalability with Erlang/OTP by Francesco Cesarini and Steve Vinoski — distributed OTP design and operations.
  • Erlang Programming by Francesco Cesarini and Simon Thompson — distributed Erlang and Mnesia.
  • Designing Data-Intensive Applications by Martin Kleppmann — replication, transactions, partitions, and recovery framing.

10.3 Verification Checklist Before Sharing the Project

  • Table-copy and majority claims match current official Mnesia documentation.
  • The guide never equates node connection with table readiness.
  • The netsplit exercise requires an explicit authority policy and avoids automatic force-load advice.
  • Replication and backup protect different failures in the project narrative and tests.
  • Restore drills use isolated directories and verified artifacts.
  • The project contains pseudocode, schemas, transcripts, and commands—but no complete runnable solution.
  • Every destructive operation is scoped to disposable drill data.