Project 30: :gen_statem Escrow and Approval Workflow Engine

Build an auditable money-movement workflow whose legal transitions, deadlines, duplicate-command behavior, and crash recovery are explicit rather than hidden inside a large GenServer callback.

Quick Reference

Attribute Value
Difficulty Level 4: Expert
Suggested Seniority Staff Elixir engineer
Time Estimate 24-36 hours
Main Programming Language Elixir using Erlang/OTP :gen_statem
Alternative Programming Languages Erlang
Coolness Level Level 4: Hardcore Tech Flex
Business Potential Level 3: Service & Support Model
Prerequisites OTP behaviours, supervision, messages, timeouts, persistence fundamentals, idempotency
Key Topics :gen_statem, transition invariants, event timeouts, postponed events, command deduplication, journal recovery

1. Learning Objectives

By completing this project, you will:

  1. Choose :gen_statem when behavior depends on both current state and event type.
  2. Model legal and illegal escrow transitions as a reviewable transition contract.
  3. Use calls, casts, internal events, state-enter actions, timeouts, and postponed events deliberately.
  4. Separate an accepted command from an external side effect and its eventual confirmation.
  5. Make duplicate commands, late provider callbacks, and deadline races deterministic.
  6. Recover a workflow from a durable journal without repeating completed money movement.
  7. Test state-machine traces and invariants rather than only individual callback examples.

2. All Theory Needed (Per-Concept Breakdown)

:gen_statem Event Semantics and Temporal State Machines

Fundamentals

:gen_statem is the OTP behaviour for processes whose response depends explicitly on a finite state and an incoming event. It supports state-functions and handle-event callback modes, synchronous calls, asynchronous casts and information messages, internal events, state-enter calls, event timeouts, state timeouts, generic timeouts, postponed events, and transition actions. The callback returns a next state, next data, and actions; :gen_statem supplies system-message handling, tracing, termination behavior, and supervision compatibility. Unlike a map stored inside a generic GenServer, the state name is part of the callback contract. This makes temporal rules visible, but it does not make them correct automatically. The design must define legal transitions, side effects, reply timing, and what each timeout means.

Deep Dive

A workflow state is not merely a status label. It is a claim about which events are legal, which invariants hold, which obligations remain, and which deadlines are active. In an escrow engine, submitted might mean the command was validated and journaled but risk review has not begun. risk_review means a review request exists and a response or deadline is expected. approved means authorization to capture exists but funds are not yet captured. capture_pending means an external provider request was issued with an idempotency key. captured, cancelled, and expired are terminal for the ordinary path, though a later dispute can create a separate branch if the domain permits it.

Choose callback mode based on readability. State-functions make each state’s accepted events visually obvious and work well when state-specific behavior dominates. handle_event_function centralizes patterns and can reduce duplication for events accepted everywhere, such as inspection. Either mode is legitimate; mixing mental models is not. The project should write the transition table before callbacks and generate tests from it where practical.

Event type matters. A synchronous call carries a caller waiting for a reply; use it for commands that need a clear acceptance decision, not for waiting on an unpredictable payment provider. Reply after validating and durably recording acceptance, then represent provider completion as a later asynchronous event. A cast is fire-and-forget and cannot prove acceptance to its sender. An info event arrives from arbitrary process messages and requires careful validation. Internal events let one transition schedule immediate follow-up work without waiting for the mailbox, while avoiding nested callback calls that bypass the behaviour.

Timeout categories have different semantics. A state timeout is associated with residence in a state and is cancelled by a state change. An event timeout is tied to event activity and can be cancelled by another event. A named generic timeout can coexist with others. For an approval deadline, use a timeout whose cancellation rules match the domain, and store the absolute deadline durably. BEAM monotonic timers are correct for in-process elapsed time but do not survive restart. Recovery computes remaining duration from persisted wall-clock deadline using a documented policy for clock movement and already-expired cases.

Postponing an event tells :gen_statem to retry it after a state change. This is useful when an event is valid later, but dangerous if no transition can ever make it valid. A postponed cancel command during a short internal initialization step may be reasonable; indefinitely postponing an invalid capture request creates memory growth and mystery. Define a bounded postponement policy and reject events that cannot become valid.

State-enter calls centralize entry actions such as installing a timeout or emitting an internal event. They should not repeat non-idempotent external side effects on every re-entry. Treat entry actions as orchestration around already-journaled intent. The callback data carries workflow identifiers, command deduplication information, deadline metadata, provider references, and journal sequence—not a hidden alternate state name.

Transition actions also control replies and hibernation. Reply actions must be emitted exactly once for a call. Long computation must not run in the state-machine process; dispatch it and await a correlated result. The machine should remain responsive to cancellation, inspection, provider results, and deadlines. A powerful invariant is: every accepted domain transition is journaled before the machine reports acceptance, and every external completion is correlated to a known pending intent.

How this fits on the project

This concept defines the workflow callback mode, transition table, event taxonomy, deadlines, postponed-event policy, and responsiveness guarantees.

Definitions and key terms

  • State-functions: Callback mode where the state name selects the callback function.
  • Event type: Call, cast, info, internal, state-enter, or timeout context.
  • Action: A reply, timeout, next event, postpone directive, or runtime instruction returned with a transition.
  • State timeout: A timer whose meaning is tied to remaining in one state.
  • Postponed event: An event retained for retry after a state change.
  • Transition invariant: A property that must hold before and after a legal state change.

Mental model diagram

                 risk_rejected
               +---------------> [cancelled]
               |
[submitted] -> [risk_review] -> [approved] -> [capture_pending] -> [captured]
     |              |               |                |
     | deadline     | deadline      | cancel         | provider_timeout
     v              v               v                v
 [expired]       [expired]       [cancelled]      [manual_review]

Every arrow = event + guard + journal entry + next-state actions
No arrow     = explicit rejection, not an accidental catch-all transition

How it works

  1. Define named states and domain invariants.
  2. Enumerate command, provider, timeout, internal, and inspection events.
  3. Create a transition table with guard, journal event, next state, reply, and actions.
  4. Select a callback mode and implement only table-approved transitions.
  5. Install deadlines on state entry from persisted absolute timestamps.
  6. Dispatch external work outside the machine and correlate results.
  7. Reject impossible events; postpone only events with a bounded future-valid path.
  8. Invariant: terminal state never returns to a non-terminal state without a separately modeled domain process.
  9. Failure modes include duplicate replies, stale timeout delivery, unbounded postponed events, blocking callbacks, and side effects repeated on state entry.

Minimal concrete example

TRANSITION RECORD, NOT RUNNABLE CODE
current: approved
event:   command capture(command_id=C9)
guard:   C9 not seen; amount equals approved amount
journal: capture_requested(C9, provider_idempotency_key=escrow-42-capture)
next:    capture_pending
reply:   accepted(sequence=17)
action:  dispatch provider request asynchronously

Common misconceptions

  • :gen_statem automatically persists state across process restarts.
  • A state timeout is a durable scheduler.
  • Postponing is a safe default for any currently invalid event.
  • Entering a state is always safe to use for non-idempotent side effects.

Check-your-understanding questions

  1. Why should a capture call not block until the payment provider replies?
  2. When is a state timeout preferable to a generic named timeout?
  3. What must be true before postponing an event?

Check-your-understanding answers

  1. Provider latency would block the machine and caller; accept durable intent and process the correlated result asynchronously.
  2. When the deadline is meaningful only while the machine remains in that state and should be cancelled by leaving it.
  3. A known bounded transition can make it valid, and retention cannot grow without limit.

Real-world applications

  • Payment authorization and capture
  • Device provisioning protocols
  • Order fulfillment and approval flows
  • Connection handshakes and session lifecycles

Where you will apply it

  • Project 30 transition engine, deadline behavior, provider correlations, and trace tests

References

Key insight

A state machine is valuable because it makes valid temporal behavior explicit, not because it gives a process a more sophisticated name.

Summary

Use :gen_statem to encode event-sensitive transitions, reply and timeout semantics, and bounded postponement while keeping slow or non-idempotent work outside the callback process.

Homework/Exercises

  1. Add a disputed branch without allowing an uncaptured escrow to be disputed.
  2. Decide the correct timeout type for a provider response and explain restart behavior.

Solutions

  1. Permit open_dispute only from captured with required settlement reference; create explicit dispute states rather than reusing risk review.
  2. Persist an absolute provider deadline, install a suitable named timeout while pending, and recompute remaining time during recovery.

Durable Intent, Idempotency, and Recovery

Fundamentals

A supervised process restart restores code, not business history. Durable workflow recovery requires a journal or transactional state store that records accepted facts in order. Idempotency means a retried command or external callback produces the same durable outcome as the original, without duplicating money movement. Each command needs a stable identifier, each external side effect needs a stable provider idempotency key, and each response needs correlation to a known pending intent. Recovery loads a snapshot if available, replays later journal events through a pure reducer, reconstructs deadlines, and resumes only unfinished intentions. The state-machine process owns in-memory sequencing for one escrow, while the durable store enforces uniqueness across restarts and races.

Deep Dive

The journal should contain domain facts, not arbitrary serialized callback data. Events such as escrow_submitted, risk_approved, capture_requested, capture_confirmed, and escrow_expired explain what happened and can rebuild state. Storing a raw internal map couples recovery to every implementation detail and makes upgrades dangerous. Define a versioned event envelope with escrow ID, monotonically increasing sequence, event type, schema version, command or provider correlation, occurrence time, and payload.

Command handling follows a disciplined order. Validate command shape and current-state legality. Check whether its command ID already has an outcome. If duplicate, return the stored acceptance or rejection without appending a second event. If new and legal, append the event with an expected current sequence so competing writers cannot both succeed. Only after append confirmation does the machine transition and reply accepted. This optimistic concurrency contract protects against accidental multiple owners even though the runtime design intends one process per escrow.

External side effects require an outbox-like intention. Appending capture_requested records that the system owes a provider request with a stable idempotency key. A dispatcher observes or is notified of the intent and calls the provider. If it crashes after the provider accepts but before the local confirmation is stored, retry uses the same provider key. The provider either returns the original outcome or declines the duplicate according to its contract. Never generate a new key on retry. A correlated callback then appends capture_confirmed or capture_failed only if the expected intent exists.

Exactly-once execution is not a realistic blanket promise across independent systems. The achievable contract is durable intent, at-least-once delivery, idempotent provider interaction, and exactly-once local state transition under a uniqueness constraint. State these semantics plainly. A callback arriving twice is harmless. A callback for an unknown provider reference is quarantined for investigation, not guessed into the current workflow.

Recovery must be pure before it becomes operational. Load a snapshot with sequence S, verify its schema and digest, then replay events S+1...N through a reducer that performs no external work. Confirm sequences are contiguous and every event is legal for reconstructed state. After state is rebuilt, derive pending obligations: deadlines not reached, capture requested without confirmation, or notification owed. Reinstall timers from persisted absolute deadlines and hand pending side effects to the idempotent dispatcher. Do not replay historical side effects merely because their events were replayed.

Snapshots are performance optimizations, not the source of truth. Create them at defined sequence intervals or terminal states, write them atomically, and retain enough journal to audit. A corrupted or incompatible snapshot should fall back to replay where policy permits. Event schema evolution needs upcasters or a version-aware reducer; silently interpreting old payload under new assumptions is unacceptable for financial state.

Time races need a domain rule. If risk approval and expiry arrive near the deadline, process ordering alone is insufficient unless the rule says mailbox order decides. Prefer comparing the event’s trusted occurrence/correlation against the persisted deadline and journal sequence policy. Document whether an approval produced before but delivered after deadline is accepted. Build deterministic tests around both orderings.

How this fits on the project

This concept drives journal schemas, duplicate-command results, provider dispatch, snapshot/replay, deadline reconstruction, and recovery testing.

Definitions and key terms

  • Domain event: A durable fact describing an accepted business transition.
  • Expected sequence: Optimistic concurrency value used during append.
  • Idempotency key: Stable identifier that makes retries return or preserve one effect.
  • Outbox intent: Durable record that an external action is owed.
  • Snapshot: Rebuild optimization containing state at a known journal sequence.
  • Upcaster: Transformation from an older event schema to the current reducer input.

Mental model diagram

command --> [state machine]
               | validate + deduplicate
               v
        [append with expected sequence] ---- conflict ---> reload/retry decision
               |
               v
       journaled domain event ---> transition + reply
               |
       pending external intent
               v
        [idempotent dispatcher] <---- retry same key
               |
        provider result/callback
               v
       correlated journal event

restart: snapshot + later events -> pure reducer -> timers + pending intents

How it works

  1. Route one escrow identity to one active machine.
  2. Recover from snapshot and contiguous journal before accepting commands.
  3. Deduplicate by command ID and return the recorded outcome.
  4. Append legal transitions with expected sequence.
  5. Transition only after durable append succeeds.
  6. Dispatch side effects from durable intent using stable keys.
  7. Correlate provider outcomes and append them idempotently.
  8. Invariant: journal sequence and in-memory sequence agree after every accepted transition.
  9. Failure modes include split ownership, missing sequence, repeated provider effect, corrupted snapshot, schema drift, and deadline races.

Minimal concrete example

RECOVERY TRANSCRIPT
snapshot sequence=12 state=approved
replay 13 capture_requested key=escrow-42-capture
replay complete sequence=13 state=capture_pending
derived obligation: provider capture has no terminal result
action: retry provider with SAME key escrow-42-capture

Common misconceptions

  • Supervision alone gives durable workflow recovery.
  • Retrying with a new provider key is harmless.
  • Replaying events should repeat their historical side effects.
  • A snapshot can replace the audit journal.

Check-your-understanding questions

  1. Why transition only after append confirmation?
  2. What does optimistic expected-sequence append protect against?
  3. Which operations are allowed during pure replay?

Check-your-understanding answers

  1. Otherwise a reported acceptance can disappear after a crash.
  2. Concurrent or split owners both attempting to advance the same escrow.
  3. Deterministic state reduction and validation; no provider calls, notifications, or new journal writes.

Real-world applications

  • Financial workflows and payout orchestration
  • Durable approvals and compliance review
  • Subscription lifecycle state
  • Long-running provisioning with external providers

Where you will apply it

  • Project 30 journal adapter, command path, provider dispatcher, snapshots, and recovery harness

References

Key insight

Reliable workflows turn side effects into durable, idempotently fulfilled obligations rather than pretending crashes cannot happen between systems.

Summary

Journal accepted facts, deduplicate commands, persist external intent before dispatch, recover through pure replay, and reconstruct only unfinished obligations.

Homework/Exercises

  1. Trace a crash immediately after the provider captures funds but before local confirmation.
  2. Define the outcome for a duplicate rejected command.

Solutions

  1. Recovery finds capture_requested without confirmation and retries with the same provider key; the provider returns the existing result, which is then journaled.
  2. Store or deterministically reproduce the original rejection under the command ID so the retry cannot become accepted merely because time passed.

3. Project Specification

3.1 What You Will Build

Build an escrow workflow service with one supervised :gen_statem process per active escrow, a Registry for routing, a versioned journal adapter, a provider-dispatch adapter, and an operator CLI. The reference workflow covers submission, risk review, approval, capture, cancellation, expiry, and manual review after uncertain provider outcomes.

3.2 Functional Requirements

  1. Create an escrow with amount, currency, parties, and explicit deadlines.
  2. Accept commands with stable command IDs and return stored outcomes for duplicates.
  3. Enforce a documented legal transition table.
  4. Install and recover risk, approval, and provider deadlines.
  5. Journal every accepted transition with a contiguous sequence.
  6. Dispatch capture from durable intent using a stable provider idempotency key.
  7. Correlate duplicate, late, and out-of-order provider callbacks.
  8. Recover state and pending obligations after process termination.
  9. Expose state, journal sequence, deadline, and pending obligation to operators.

3.3 Non-Functional Requirements

  • Illegal transitions never mutate durable or in-memory state.
  • Each accepted command has one durable outcome.
  • State-machine callbacks remain responsive and do not perform provider network calls.
  • Recovery is deterministic from the same snapshot and event sequence.
  • Sensitive payment data is excluded from logs and journal payloads.
  • Transition and recovery traces are explainable to an operator.

3.4 Example Usage / Output

$ escrowctl submit --id ESC-42 --amount 12500 --currency BRL --command C1
accepted escrow=ESC-42 sequence=1 state=risk_review deadline=2026-07-15T18:00:00Z

$ escrowctl approve ESC-42 --command C2
accepted sequence=2 state=approved

$ escrowctl capture ESC-42 --command C3
accepted sequence=3 state=capture_pending provider_key=ESC-42-capture-v1

3.5 Data Formats / Schemas / Protocols

COMMAND ENVELOPE
{escrow_id, command_id, command_type, expected_sequence?, actor, issued_at, payload}

JOURNAL EVENT
{schema_version, escrow_id, sequence, event_type, command_id?,
 provider_correlation?, occurred_at, payload_digest, payload}

PROVIDER RESULT
{provider_key, provider_reference, outcome, occurred_at, signed_metadata}

3.6 Edge Cases

  • Approval and expiry cross in flight
  • Duplicate capture command after a restart
  • Provider callback arrives before dispatcher receives its local response
  • Unknown provider reference
  • Journal append conflict indicating two owners
  • Snapshot behind a compacted or missing journal sequence
  • Cancellation arrives while capture outcome is unknown
  • Old event schema requires upcasting
  • Timer fires after state already changed

3.7 Real World Outcome

3.7.1 How to Run

$ mix deps.get
$ mix test
$ mix escrow.demo --scenario crash-after-provider-success

3.7.2 Golden Path Demo

Submit, approve, and capture one escrow. Kill its machine after capture_requested is durable. Restart it, replay the journal, retry the provider with the same idempotency key, receive the original provider outcome, and reach captured with exactly one provider effect.

3.7.3 Exact Terminal Transcript

$ mix escrow.demo --scenario crash-after-provider-success
ESC-42 sequence=3 state=capture_pending action=provider_capture
provider key=ESC-42-capture-v1 result=success reference=PAY-9001
FAULT injected=kill_before_local_confirmation
RECOVERY snapshot=none replayed=3 state=capture_pending
RETRY provider key=ESC-42-capture-v1 result=already_succeeded reference=PAY-9001
JOURNAL sequence=4 event=capture_confirmed
FINAL state=captured provider_effect_count=1 invariant=ok

3.8 Scope Boundary Versus Projects 1-13

Projects 1 and 6 introduce supervision and failure isolation, but this project is not another crash-and-restart demonstration. Its focus is explicit state/event semantics, durable intent, idempotent side effects, temporal races, and recovery of business truth. It does not add distribution, ETS caching, or LiveView.

4. Solution Architecture

4.1 High-Level Design

client/operator
      |
 [command API] --> [Registry: escrow_id -> machine]
                         |
                [DynamicSupervisor]
                         |
                   [:gen_statem]
                    /    |     \
             [journal] [timer] [intent dispatcher] ---> provider
                 ^                    |                   |
                 |                    +--- correlated ----+
                 |
           snapshot/replay

4.2 Key Components

Component Responsibility Key decision
Router Locate or recover one escrow machine Never create two owners knowingly
State Machine Enforce transition and timeout semantics No provider calls in callbacks
Journal Adapter Append with expected sequence and replay Events are source of durable truth
Snapshot Store Accelerate recovery Snapshot is optional and verifiable
Intent Dispatcher Fulfill external obligations Stable idempotency key across retries
Provider Adapter Normalize provider outcomes Explicit unknown/uncertain result
Operator View Show state and pending obligation Readable without inspecting raw process state

4.3 Data Structures (No Full Code)

  • MachineData: escrow identity, amount, parties, journal sequence, deadlines, provider intent, deduplication window.
  • TransitionSpec: current state, event class, guard, journal event, next state, actions.
  • CommandOutcome: accepted/rejected result keyed by command ID.
  • PendingIntent: external action, stable key, attempt metadata, expected result.
  • RecoveryPlan: reconstructed state, timers to install, intents to resume.

4.4 Algorithm Overview

Route a command to the machine, validate against the transition table, deduplicate the command, append the domain event using expected sequence, then transition and reply. For external work, the appended event creates an intent consumed idempotently by the dispatcher. Recovery replays through a pure reducer, verifies sequence and legality, then schedules only derived unfinished work.

Per command transition work is O(1) aside from journal latency and deduplication lookup. Full replay is O(E) for E events; snapshots reduce typical replay to O(E-S). Active memory is O(A + C), where A is active machines and C is bounded remembered command outcomes per machine.

5. Implementation Guide

5.1 Development Environment Setup

Use a current supported Elixir/OTP pair. Begin with an in-memory deterministic journal adapter that supports expected sequence, then add an Ecto/PostgreSQL adapter after transition and recovery tests are stable.

5.2 Project Structure

escrow_workflow/
├── lib/escrow_workflow/
│   ├── application.ex
│   ├── router.ex
│   ├── machine.ex
│   ├── transition_table.ex
│   ├── event_reducer.ex
│   ├── journal.ex
│   ├── snapshot_store.ex
│   ├── intent_dispatcher.ex
│   ├── provider.ex
│   └── operator_view.ex
├── test/fixtures/
└── test/

5.3 The Core Question You Are Answering

“How can a BEAM process coordinate a long-running money workflow when commands, deadlines, provider outcomes, and crashes may arrive in any order?”

5.4 Concepts You Must Understand First

  1. OTP process lifecycle, links, and supervision.
  2. :gen_statem event types, callback modes, actions, and timeouts.
  3. Domain-event journals and optimistic sequence checks.
  4. Idempotency keys and at-least-once delivery.
  5. Monotonic elapsed time versus persisted wall-clock deadlines.

5.5 Questions to Guide Your Design

  1. Which states and transitions are legally meaningful to the domain?
  2. At what durable point may a client receive “accepted”?
  3. Which side effects require stable external idempotency keys?
  4. What happens when a result arrives after its deadline or state change?
  5. How will recovery distinguish historical effects from pending obligations?

5.6 Thinking Exercise

Draw two timelines for the same capture: provider succeeds before local crash, and provider succeeds after local retry. Demonstrate why a stable provider key and journaled intent converge both timelines to one captured outcome.

5.7 The Interview Questions They Will Ask

  1. “When should you use :gen_statem instead of GenServer?”
  2. “What is the difference between state timeout, event timeout, and generic timeout?”
  3. “How do postponed events become a memory or correctness problem?”
  4. “Why can’t supervision provide durable business recovery?”
  5. “What exactly-once guarantee can you honestly make across a payment provider?”
  6. “How do you recover timers and side effects after replay?”

5.8 Hints in Layers

Hint 1: Write the table first — Define events, guards, journal fact, next state, reply, and actions before callbacks.

Hint 2: Journal before acceptance — A transition that is only in process memory is not an accepted financial fact.

Hint 3: Side effects are obligations — Record provider intent, then fulfill it with a stable key outside the machine.

Hint 4: Replay is pure — Rebuild state first; derive timers and unfinished work only after replay ends.

5.9 Books That Will Help

Topic Book Precise reading
Stateful OTP Elixir in Action, 3rd ed. — Saša Jurić Chapters 6 “Generic Server Processes” and 7 “Building a Concurrent System”
Failure boundaries Elixir in Action, 3rd ed. Chapter 9 “Isolating Error Effects”
OTP behaviours Designing for Scalability with Erlang/OTP — Cesarini and Vinoski Chapters covering generic behaviours and finite-state machines
State-machine contrast Learn You Some Erlang for Great Good! — Fred Hébert “Rage Against the Finite-State Machines,” read with current gen_statem docs because its legacy behaviour differs

5.10 Implementation Phases

Phase 1: Domain and Pure Reducer (5-7 hours)

  • Define states, event schemas, transition invariants, and deadlines.
  • Build a pure event reducer and transition-table tests.
  • Add command IDs and deterministic duplicate outcomes.

Phase 2: :gen_statem Runtime (6-9 hours)

  • Implement chosen callback mode, replies, internal events, and timeouts.
  • Add Registry, DynamicSupervisor, and operator inspection.
  • Prove callbacks remain responsive while provider work is pending.

Phase 3: Durable Intent and Recovery (7-11 hours)

  • Add expected-sequence journal and snapshots.
  • Add provider intent dispatcher with stable keys.
  • Recover state, deadlines, and unfinished obligations after injected crashes.

Phase 4: Race and Operations Hardening (6-9 hours)

  • Test expiry/approval and cancellation/provider races.
  • Add unknown-callback quarantine and schema versioning.
  • Document operator actions for uncertain provider outcomes.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Callback mode state functions / handle-event function State functions for core domain Makes legal events per state visible
Durable truth raw state snapshots / event journal Versioned event journal + optional snapshots Preserves audit and rebuild logic
Provider work callback performs call / dispatcher Idempotent dispatcher Keeps machine responsive and retryable
Timer persistence timer reference / absolute deadline Absolute deadline + reconstructed timer Survives restart
Invalid event postpone / ignore / explicit reject Explicit reject unless bounded future validity Prevents hidden queues and ambiguity

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Transition Verify legal table every state/event pair
Trace/Property Check invariants over event sequences terminal monotonicity, non-negative sequence
Timeout Verify temporal policy before, at, and after deadline
Idempotency Prevent duplicate effects repeated commands and callbacks
Recovery Rebuild deterministic state every crash point around append/dispatch/result
Concurrency Expose ownership races two starters, append conflict
Security Protect financial metadata redaction and actor authorization boundary

6.2 Critical Test Cases

  1. Every unspecified state/event pair returns an explicit stable rejection.
  2. Duplicate accepted and rejected command IDs return their original outcomes.
  3. Crash before journal append leaves no accepted transition.
  4. Crash after append but before reply recovers the accepted command result.
  5. Crash after provider success but before confirmation creates one provider effect.
  6. Duplicate and out-of-order provider callbacks are harmless or quarantined.
  7. Expiry and approval at the same boundary follow the documented rule in both arrival orders.
  8. Replay rejects missing or duplicate journal sequence numbers.
  9. Old event schema is upcast or rejected explicitly.
  10. A split-owner append conflict causes one owner to stop and reload.

6.3 Test Data

Use a deterministic fake clock, journal, and provider. Create trace fixtures for happy capture, risk rejection, cancellation, expiry, uncertain provider response, duplicate callbacks, corrupt snapshot, and old event schemas. The provider records keys and effect count so exactly-one external effect is observable.

6.4 Definition of Done

  • A reviewed transition table covers every state/event pair.
  • Illegal commands never append journal events.
  • Accepted commands are journaled before acceptance is reported.
  • Duplicate commands and provider callbacks are deterministic.
  • All external actions originate from durable intent with stable keys.
  • Recovery performs pure replay before installing timers or dispatching work.
  • Crash injection around every durability boundary converges correctly.
  • Deadline race policy is documented and tested in both event orders.
  • Operator output exposes pending obligations and journal sequence.
  • Sensitive values are redacted from logs, test snapshots, and errors.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem Why it happens Fix Quick test
Machine blocks during capture Provider called inside callback Journal intent and dispatch asynchronously Delay fake provider while inspecting state
Money movement repeats Retry generates new provider key Derive stable key from escrow and operation Crash after provider success
Restart loses deadline Only timer reference was stored Persist absolute deadline and reconstruct Restart halfway to expiry
Invalid commands accumulate Everything is postponed Postpone only bounded future-valid events Send repeated impossible captures
Replay repeats notifications Event reducer performs side effects Keep replay pure; derive unfinished work afterward Count fake side effects during replay
Two processes advance one escrow Runtime registry assumed infallible Expected-sequence append and ownership conflict policy Race two starts

7.2 Debugging Strategies

  • Render the transition table row selected for each event.
  • Include escrow ID, state, journal sequence, event class, and correlation in structured logs.
  • Compare in-memory sequence with durable tail after any suspicious transition.
  • Use :sys tracing and :gen_statem debugging options in controlled tests.
  • Re-run a production-like journal through the pure reducer without starting a machine.

7.3 Performance Traps

  • Unbounded command-deduplication data held forever in every process
  • Replaying an entire long journal on every activation without verified snapshots
  • One global dispatcher serialized across unrelated escrows
  • Large journal payloads copied into process messages
  • Keeping terminal machines active indefinitely instead of passivating them safely

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add an ASCII transition-table renderer.
  • Add an operator command that explains why one event is illegal.
  • Snapshot terminal workflows with a digest.

8.2 Intermediate Extensions

  • Add a dispute sub-workflow with separate deadlines.
  • Add bounded terminal-process passivation and on-demand recovery.
  • Add signed provider callback verification.

8.3 Advanced Extensions

  • Model saga-style compensation across multiple provider actions.
  • Run model-based property tests generated from the transition table.
  • Add an event-schema upcaster registry and migration report.
  • Formally specify core invariants in a state-machine modeling tool and compare traces.

9. Real-World Connections

9.1 Industry Applications

Payments, approvals, identity verification, device provisioning, and order fulfillment all combine long waits, duplicate messages, deadlines, external systems, and operator intervention. :gen_statem makes legal temporal behavior visible, while journaling and idempotent intent bridge runtime reliability to business durability.

  • Erlang/OTP :gen_statem, the core behaviour
  • Ecto for a PostgreSQL journal adapter
  • StreamData or PropCheck-style model testing for event traces
  • Oban as an optional durable dispatcher implementation, while keeping workflow truth in the journal

9.3 Interview Relevance

A strong staff-level explanation distinguishes process restart from durable recovery, timeout categories from persisted deadlines, and at-least-once delivery plus idempotency from mythical global exactly-once execution.

10. Resources

10.1 Essential Official Reading

10.2 Books and Deeper Study

  • Elixir in Action, 3rd edition, by Saša Jurić — Chapters 6, 7, and 9.
  • Designing for Scalability with Erlang/OTP by Francesco Cesarini and Steve Vinoski — generic behaviours, state machines, and supervision sections.
  • Learn You Some Erlang for Great Good! by Fred Hébert — “Rage Against the Finite-State Machines,” contrasted with current gen_statem documentation.

10.3 Verification Checklist Before Sharing the Project

  • Publish the complete transition table and deadline race rule.
  • Demonstrate the crash-after-provider-success scenario with one external effect.
  • Show one unknown callback quarantined rather than guessed.
  • Verify replay invokes no side-effect adapter.
  • Keep examples at the level of transition records, event schemas, pseudocode, and CLI transcripts.