Project 24: Durable Document Processing Workflow with Oban

Build a database-backed document workflow whose validation, preview conversion, metadata extraction, and notification stages survive crashes, retry safely, and remain operable after failure.

Quick Reference

Attribute Value
Difficulty Level 3
Suggested Seniority Senior Elixir developer
Time Estimate 18-26 hours
Main Programming Language Elixir
Alternative Programming Languages Erlang
Coolness Level Level 3
Business Potential Level 4
Prerequisites OTP basics, Ecto changesets and transactions, SQL, external-process awareness
Key Topics Oban.Worker, durable jobs, retries, uniqueness, idempotency, cancellation, operator history

1. Learning Objectives

By completing this project, you will be able to:

  1. Distinguish durable database-backed jobs from transient BEAM messages and demand-driven streams.
  2. Model a multi-stage document workflow as explicit durable transitions rather than one opaque long-running job.
  3. Return success, retryable errors, discard decisions, snooze decisions, and cancellation outcomes intentionally from workers.
  4. Make every external side effect idempotent under retries and unknown process outcomes.
  5. Use uniqueness as an admission policy without mistaking it for business idempotency or a concurrency limit.
  6. Provide operators with searchable attempts, failure reasons, cancellation, safe retry, and artifact lineage.

2. All Theory Needed (Per-Concept Breakdown)

Durable Work, Retries, and Idempotent Stage Transitions

Fundamentals

An Oban job is a durable database record describing work that a supervised Elixir process may execute. Durability changes the failure model: killing the process or restarting the node does not erase the intent, but it also means the same logical work may run more than once. A worker must classify outcomes rather than treating every failure alike. Retryable infrastructure errors should return an error and use bounded backoff; permanently invalid input should be discarded with evidence; deliberate delay may be snoozed; successful work must record enough state that a retry cannot duplicate side effects. Uniqueness controls whether similar job rows may be inserted within a period or state set. It is not a substitute for a domain idempotency key, and concurrency configuration limits simultaneous execution rather than proving one business effect.

Deep Dive into the concept

A document pipeline has several failure domains. Ingested bytes may be corrupt or unsupported. Preview conversion may depend on an external tool that exits, hangs, or produces malformed output. Metadata extraction may be CPU-heavy. Notification may time out after the remote service accepted it. If these operations live in one process and that process crashes, the system loses both progress and the reason. One enormous retryable job is not much better: successful early steps repeat, temporary and permanent failures share a policy, and operators cannot see which transition is stuck.

Model the workflow with a durable document record and separate stage jobs. The record owns stable input identity, current workflow status, input digest, artifact references, and a version. Each job argument contains stable identifiers and a schema version, not a huge document or arbitrary serialized term. A stage loads current state, determines whether its transition is still applicable, performs bounded work, and commits the resulting artifact/state plus insertion of the next stage. The transition must tolerate a duplicate attempt. If validation is already complete for the same input digest, a repeated validation job returns success without creating a second transition.

Oban’s persistence answers “will the system remember to try?” It does not answer “is my side effect exactly once?” The worker can crash after an external service accepts a request but before the job is acknowledged. The database later retries. Exactly-once execution across independent systems is generally unavailable, so design at-least-once attempts with idempotent effects. Use stable artifact keys derived from document/stage/version, idempotency keys accepted by notification providers, conditional creation, or a durable effect record that distinguishes prepared, attempted, and confirmed outcomes. If the remote system cannot deduplicate, explicitly document the residual duplicate risk.

Classify failures at the boundary. Validation errors such as “encrypted PDF unsupported” are permanent for these bytes and should mark the document rejected, retain a safe reason, and discard downstream work. A converter process exiting because of malformed input is likely permanent; exiting because the host is temporarily out of disk may be retryable. An HTTP 429 should retry after policy delay. A coding exception should normally fail and retry under bounded attempts while alerting operators; swallowing it as success loses work. Define a closed error taxonomy whose categories map to job outcomes and workflow state.

Backoff is a load-control and recovery policy. Immediate retries can amplify an outage. Use increasing delay with jitter and a maximum attempt count appropriate to the dependency. The final exhausted attempt should transition the document to an operator-visible failed state rather than merely leave an opaque job. Keep time budgets per attempt: conversion has a deadline, output-size limit, and process kill/cleanup path. A stuck external command must not occupy a worker forever.

Uniqueness has precise semantics. A unique configuration may prevent inserting another job with the same selected argument keys while matching jobs are available, scheduled, executing, retryable, or completed within a period. That is useful for suppressing duplicate scheduling, but races and replacement behavior must be understood. It does not protect a side effect after the uniqueness period, and it does not replace a unique domain transition such as {document_id, stage, input_version}. Put that identity in the workflow/artifact schema as a database constraint.

Concurrency is separate again. Queues determine worker pools and can isolate expensive conversion from lightweight notification. A queue limit of two converters protects CPU and external-tool capacity; ten metadata workers may be safe; notifications may have a provider-specific rate policy. Reducing concurrency does not prevent the same logical job from being retried later, and uniqueness does not bound how many different documents convert simultaneously. Operability improves when the queue topology mirrors resource constraints.

Cancellation is cooperative. Cancelling an available or scheduled job is straightforward, but an executing worker may already own an external process or be inside a non-interruptible operation. The workflow needs cancellation intent on the document, workers need checkpoints, and external resources need cleanup. After expensive work but before committing an artifact, check whether cancellation still applies. Define whether already-created artifacts are retained for audit or removed by a cleanup job. A cancelled parent must prevent insertion or useful execution of later stages.

Workflow orchestration can be explicit without importing a magical workflow engine. After a stage transaction commits its output, insert the next job in the same Ecto.Multi where possible. This prevents “state says validated but preview job was never scheduled.” Alternatively, a reconciler periodically finds transitions missing their next job. Fan-out and fan-in need counts or stage records: metadata and preview can run in parallel, while notification waits until required stage results are successful. A unique finalization transition should be safe when both branches try to trigger it.

Observability should connect document, stage, job, attempt, and artifact. Operators need queries such as “documents stuck in preview for 20 minutes,” “top permanent validation reasons,” and “all attempts for document D.” Store compact structured failure classifications, not raw documents or unbounded stack traces. Attach telemetry for queue delay, execution duration, outcome, attempt number, and external tool exit class. High-cardinality IDs belong in logs/traces and operator queries, not metric labels.

Testing must prove recovery, not simply perform/1 in isolation. Insert jobs, execute with the testing mode appropriate to the scenario, inject failure after artifact creation and before acknowledgement, restart the supervision tree, and show the repeated attempt reuses the artifact. Verify time with a controlled clock where possible. Test a real transaction rollback around state plus next-job insertion. The final demonstration should kill a worker mid-stage and later show the durable workflow converging to the same single result.

How this fits on projects

The document record is the workflow source of truth, Oban holds durable execution intent, workers implement idempotent transitions, Ecto transactions bind state and next-job scheduling, and operator views explain every attempt.

Definitions & key terms

  • Durable job: Persisted work intent that survives process or node loss.
  • Attempt: One execution of a job; a job may have several attempts.
  • Idempotent stage: Repetition produces the same committed business effect.
  • Unique job: Admission policy that suppresses matching Oban job insertion for configured states/period.
  • Snooze: Intentionally reschedule work without classifying it as ordinary failure.
  • Discard: Stop retrying work considered permanently invalid or exhausted.
  • Artifact lineage: Link between input version, stage, attempt, and produced output.

Mental model diagram

[Upload/API] -> [Document + input digest] -> enqueue VALIDATE
                                             |
                                             v
       +---------------- durable DB boundary ----------------+
       | Oban jobs | stage states | artifacts | effect keys  |
       +------------------------------------------------------+
                 |                    |
                 v                    v
            [VALIDATE] ------> [PREVIEW] ----+
                 |             external tool |
                 +-----------> [METADATA] ----+--> [FINALIZE] --> [NOTIFY]
                                                        |
                                        operator retry/cancel/history

Every arrow is an idempotent, versioned, database-visible transition.

How it works (step-by-step, with invariants and failure modes)

  1. Persist document identity and input reference in a transaction with validation job insertion.
  2. Worker loads current document version and checks cancellation/current-stage applicability.
  3. Perform bounded work with stable artifact/effect identity.
  4. Classify result as success, retry, snooze, permanent rejection, or cancellation.
  5. Commit stage result and next job(s) atomically.
  6. Finalization waits for required branch outcomes and inserts notification once.
  7. Invariants: one committed artifact per document-stage-version; no later stage for rejected/cancelled input; notification effect uses one stable idempotency key.
  8. Failure modes: crash after external effect, duplicate scheduling, poisoned input, exhausted retries, output explosion, cancellation during work, and missing next-stage insertion.

Minimal concrete example

PSEUDOCODE — stage decision

perform(job(document_id, input_version, stage)):
  document <- load_current(document_id)
  if document.cancelled: return cancelled
  if stage_result_exists(document_id, input_version, stage): return ok
  if input_version != document.current_version: return discard(stale_input)

  result <- execute_bounded_stage(document, stage)
  case classify(result):
    success(artifact) -> transaction(record_once(artifact), enqueue_next_once())
    temporary(reason) -> retry(reason)
    wait(seconds) -> snooze(seconds)
    permanent(reason) -> transaction(mark_rejected(reason), stop_descendants())

Common misconceptions

  • “Oban guarantees exactly once.” It durably retries; business effects still need idempotency.
  • “Unique jobs are a lock.” Uniqueness is a configurable insertion policy, not arbitrary mutual exclusion.
  • “One job per document is simpler.” It obscures stage progress and repeats successful expensive work.
  • “Cancellation immediately stops an external tool.” Running work needs cooperative checkpoints and resource cleanup.

Check-your-understanding questions

  1. Why can a successful notification be attempted again?
  2. How do uniqueness and concurrency limits differ?
  3. What transaction prevents a successful stage from losing its successor job?

Check-your-understanding answers

  1. The worker may crash after the remote accepts it but before the job is acknowledged.
  2. Uniqueness controls matching job insertion; concurrency controls simultaneous workers.
  3. Commit stage result and successor insertion together with Ecto.Multi.

Real-world applications

  • Media and document ingestion.
  • Invoicing, export generation, and report pipelines.
  • Durable integrations with unreliable external dependencies.

Where you’ll apply it

  • Workflow records and queues in Section 4.
  • Worker result classification and transition identity in Section 5.
  • crash/retry/cancel evidence in Section 6.

References

  • Oban: https://hexdocs.pm/oban/Oban.html
  • Oban.Worker: https://hexdocs.pm/oban/Oban.Worker.html
  • Unique jobs: https://hexdocs.pm/oban/unique_jobs.html
  • Ecto.Multi: https://hexdocs.pm/ecto/Ecto.Multi.html

Key insight

Durability remembers that work must run; idempotent transitions make repeated attempts safe.

Summary

Break the workflow into observable durable stages, give each transition stable identity, classify failures intentionally, couple state with successor scheduling, and treat uniqueness, concurrency, retries, and idempotency as separate controls.

Homework/Exercises to practice the concept

  1. Classify converter exit, HTTP 429, unsupported encryption, and node crash outcomes.
  2. Design an artifact key that survives retries but changes when input changes.
  3. Trace cancellation arriving before execution, during conversion, and after preview commit.

Solutions to the homework/exercises

  1. Converter exit depends on reason; 429 retries/snoozes; unsupported encryption is permanent; node crash leaves the job retryable.
  2. Use document ID, input version/digest, stage, and transformation version—not attempt number.
  3. Prevent execution if early, cooperatively kill/clean if running, and suppress descendants if the stage already committed.

3. Project Specification

3.1 What You Will Build

Build an API/CLI workflow with validate, preview, metadata, finalize, and notify stages. A local fixture converter simulates an external tool. Operators can inspect stage history, retry permitted failures, cancel active workflows, and verify artifact lineage.

3.2 Functional Requirements

  1. Persist upload identity and first job atomically.
  2. Use separate queues for conversion, metadata, and notification.
  3. Make stage transitions unique by document, input version, and stage.
  4. Distinguish retryable, snoozed, permanent, exhausted, and cancelled outcomes.
  5. Persist artifacts under deterministic keys and validate output limits.
  6. Insert successor jobs atomically with stage completion.
  7. Provide status, attempt history, cancel, and safe retry operations.

3.3 Non-Functional Requirements

  • Worker or node restart loses no accepted document.
  • External work has time, memory/output, and cleanup budgets.
  • Queue configuration isolates expensive conversion from notification.
  • Logs and metrics expose stage/outcome without document contents.

3.4 Example Usage / Output

$ docflow submit contract.pdf --key upload-2026-41
document=doc_41 input_version=1 state=queued validation_job=901

$ docflow status doc_41
validate=succeeded preview=retrying(attempt=2) metadata=succeeded
finalize=waiting notify=waiting

$ docflow attempts doc_41 --stage preview
attempt=1 outcome=temporary_failure reason=converter_unavailable
attempt=2 outcome=succeeded artifact=preview/doc_41/v1/transform-v2.png

3.5 Data Formats / Schemas / Protocols

Document state stores current input version and cancellation intent. Stage results store unique transition identity, outcome, reason class, artifact reference/digest, and completion time. Job args contain IDs, versions, and schema version only.

3.6 Edge Cases

  • Duplicate submission with one upload idempotency key.
  • Worker killed after artifact write but before stage commit.
  • Input replaced while old stage jobs remain queued.
  • Preview output exceeds configured bytes.
  • Metadata and preview complete simultaneously and both try to finalize.
  • Cancellation while converter owns an OS process.
  • Notification accepted remotely before worker crash.

3.7 Real World Outcome

3.7.1 How to Run (Copy/Paste)

$ mix ecto.create
$ mix ecto.migrate
$ mix test
$ mix run priv/demo_docflow.exs
$ docflow status demo-doc --history

3.7.2 Golden Path Demo (Deterministic)

The demo injects one converter failure, kills the next preview worker after artifact creation, restarts processing, and proves one artifact and one notification exist when the workflow completes.

3.7.3 Exact terminal transcript

$ mix run priv/demo_docflow.exs
submitted document=demo-doc validate_job=1001
validate attempt=1 outcome=succeeded
preview attempt=1 outcome=retry reason=converter_unavailable
preview attempt=2 fault=injected_after_artifact worker_killed=true
processor restarted durable_job_recovered=true
preview attempt=3 outcome=succeeded artifact_reused=true
metadata attempt=1 outcome=succeeded
finalize outcome=succeeded notify attempt=1 outcome=succeeded
artifacts preview=1 metadata=1 notifications=1 duplicate_effects=0
DEMO PASS

4. Solution Architecture

4.1 High-Level Design

[API] -> [Documents Context] -> [PostgreSQL: documents/stages/artifacts/oban_jobs]
                                      |
           +--------------------------+--------------------------+
           v                          v                          v
 [Validation Queue]          [Conversion Queue]          [Metadata Queue]
                                      | external tool
           +--------------------------+--------------------------+
                                      v
                              [Finalize Transition]
                                      |
                              [Notification Queue]
                                      |
                             [Operator API/Telemetry]

4.2 Key Components

Component Responsibility Key Decision
Document Context Own workflow state and commands Job modules do not become domain API
Stage Workers Execute one versioned transition Closed outcome taxonomy
Artifact Store Deterministic output identity Validate digest and size before commit
External Tool Port Bound conversion process Deadline, exit status, output limit
Finalizer Join required stage outcomes Unique transition handles races
Operator Service Search/retry/cancel/history Preserve evidence and authorization boundary

4.3 Data Structures (No Full Code)

  • Document: ID, upload key, current input version/digest, workflow state, cancellation intent.
  • Stage result: document/version/stage unique identity, status, reason, artifact, attempts summary.
  • Artifact: stable key, content digest, bytes, transform version, retention state.
  • Notification effect: stable idempotency key and external receipt state.

4.4 Algorithm Overview

Each stage transition uses indexed O(1) state operations plus its domain work. Queue polling and retry scheduling are delegated to Oban. Finalization reads a bounded set of required stage records.

5. Implementation Guide

5.1 Development Environment Setup

Use an actively supported Elixir/OTP pair, Ecto, PostgreSQL, Oban, and a deterministic fixture executable or script representing the converter. Use Oban test modes intentionally: inline for small unit scenarios and manual/sandbox-compatible execution for durability tests.

5.2 Project Structure

document_flow/
├── lib/document_flow/documents/{document,stage_result,artifact,context}.ex
├── lib/document_flow/workers/{validate,preview,metadata,finalize,notify}.ex
├── lib/document_flow/converter_port.ex
├── lib/document_flow/operator.ex
├── priv/repo/migrations/
├── priv/demo_docflow.exs
└── test/{workers,workflow,recovery,operator}/

5.3 The Core Question You’re Answering

“How can a multi-stage Elixir workflow survive arbitrary worker failure without losing intent or duplicating external effects?”

5.4 Concepts You Must Understand First

  1. Oban worker lifecycle — Oban documentation, “Workers” and “Error Handling”.
  2. Ecto transactions — Programming Ecto, Chapter 6, “Transactions and Multi”.
  3. Idempotent receivers — Enterprise Integration Patterns, “Idempotent Receiver”.
  4. Stability and timeouts — Release It!, 2nd Edition, Chapters 4 and 5.

5.5 Questions to Guide Your Design

  1. What is the durable identity of each stage result and side effect?
  2. Which errors should retry, snooze, reject, or alert after exhaustion?
  3. How are state completion and successor insertion kept atomic?
  4. What happens when cancellation races stage completion?
  5. Which queue protects each constrained resource?

5.6 Thinking Exercise

Choose one stage and mark every point at which the process can die. For each point, identify durable records, temporary files, remote effects, and what the next attempt observes. Revise the design until every point converges safely.

5.7 The Interview Questions They’ll Ask

  1. What durability does Oban provide, and what does it not guarantee?
  2. How do uniqueness, concurrency, and idempotency differ?
  3. When should a worker return an error, discard, or snooze?
  4. How would you prevent a successful stage from losing its next job?
  5. How do you cancel a job that owns an external process?
  6. What evidence lets an operator safely retry a failed workflow?

5.8 Hints in Layers

Hint 1: One durable transition — Implement validation with a unique stage row before modeling the whole graph.

Hint 2: Crash after success — Force a kill after artifact creation and design the repeated attempt around that fact.

Hint 3: Separate policies — Put conversion and notification in different queues and error classifiers.

Hint 4: Reconcile missing edges — Add a query that finds successful stages lacking their required successor.

5.9 Books That Will Help

Topic Book Chapter
Ecto transactions Programming Ecto Ch. 6
Idempotency Enterprise Integration Patterns Idempotent Receiver
Failure boundaries Release It!, 2nd Edition Ch. 4–5
Data workflows Designing Data-Intensive Applications, 2nd Edition Stream/batch processing chapters

5.10 Implementation Phases

Phase 1: Durable State and One Worker (5-7 hours)

  • Create document/stage/artifact schemas, validation worker, outcome taxonomy, and operator query.

Phase 2: Workflow Graph and Failure Control (7-10 hours)

  • Add parallel preview/metadata, finalization, external tool bounds, retries, and idempotency.

Phase 3: Operability and Recovery Proof (6-9 hours)

  • Add notification, cancellation, history, reconciliation, telemetry, and kill/restart demo.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Stage size One giant job, explicit stages Explicit stages Independent policy and evidence
Artifact key Attempt-based, input/stage/version-based Stable transition identity Reuse under retry
Next scheduling After transaction, same transaction Same transaction No missing edge
Invalid input Retry, permanent rejection Permanent rejection Bytes will not heal
Queue layout One pool, resource-oriented queues Resource-oriented Fault and capacity isolation

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Worker Result classification permanent vs retryable converter failures
Transition Idempotent domain effects duplicate stage job
Transaction State plus successor atomicity injected rollback
Recovery Survive process/node loss kill after artifact write
Cancellation Prevent descendants and clean resources cancel during conversion
Operator Safe history/retry behavior exhausted and rejected jobs

6.2 Critical Test Cases

  1. Duplicate submit key creates one document/input version.
  2. Duplicate stage jobs create one committed stage result and artifact.
  3. Worker killed after artifact write reuses or validates the artifact on retry.
  4. Permanent invalid input is not retried and schedules no descendants.
  5. Temporary failures follow bounded backoff and eventually exhaust visibly.
  6. Parallel preview/metadata completion triggers finalization once.
  7. Cancellation during conversion cleans the external process and prevents notification.
  8. Notification timeout after remote acceptance produces one effect via stable idempotency key.

6.3 Test Data

Include valid, corrupt, encrypted-unsupported, huge-output, slow-converter, metadata-edge, and remote-rate-limit fixtures. Use fixed input digests and deterministic failure injection points.

6.4 Definition of Done

  • Accepted document and first durable job are inserted atomically.
  • Every stage has stable versioned identity and idempotent repetition behavior.
  • Retryable, snoozed, permanent, exhausted, and cancelled outcomes are distinct.
  • Conversion has deadline, output limit, exit handling, and cleanup.
  • Stage completion and successor insertion cannot split.
  • Operators can inspect, cancel, and safely retry authorized failures.
  • Kill/restart demo completes with one artifact per stage and one notification.
  • Metrics expose queue delay, runtime, attempt, and low-cardinality outcome.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem: “Retries duplicate previews or notifications.”

  • Why: Output identity includes attempt number or remote calls lack idempotency keys.
  • Fix: Derive effects from document/stage/input version and persist unique transition identity.
  • Quick test: Kill after effect creation and force another attempt.

Problem: “Document says validated but never progresses.”

  • Why: State committed before successor job insertion.
  • Fix: Insert both in one transaction and add a missing-edge reconciler.
  • Quick test: Inject failure between the two intended operations.

Problem: “Poison documents retry forever.”

  • Why: All failures are returned as generic errors.
  • Fix: Classify permanent input failures and discard with a domain-state transition.
  • Quick test: Observe attempt count for a known unsupported fixture.

7.2 Debugging Strategies

  • Query by document ID across stage results and Oban jobs to build one timeline.
  • Record converter exit class, duration, and output size, not document bytes.
  • Inspect whether a job is waiting on schedule, queue capacity, dependency state, or retry delay.
  • Run the missing-successor and orphan-artifact reconciliation queries.

7.3 Performance Traps

  • Storing entire documents in job arguments.
  • Running CPU-heavy conversion in the same high-concurrency queue as notifications.
  • Retrying immediately during dependency outages.
  • Holding database transactions open during external conversion.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add per-stage progress summaries without storing noisy updates as separate jobs.
  • Add retention status and cleanup preview for artifacts.

8.2 Intermediate Extensions

  • Add a compensating cleanup job for cancelled workflows.
  • Add tenant-aware queue limits and operator authorization.
  • Add property tests over generated retry/cancel/complete sequences.

8.3 Advanced Extensions

  • Fan out page-level processing and implement durable fan-in.
  • Add an outbox for downstream workflow events.
  • Perform a controlled database failover drill and measure recovery backlog.

9. Real-World Connections

9.1 Industry Applications

Durable workflows power document ingestion, media transcoding, financial exports, notification delivery, and integration synchronization. The same patterns apply whenever work lasts longer than one request and external dependencies can fail ambiguously.

  • Oban for PostgreSQL-backed job execution.
  • Ecto.Multi for atomic domain transition and successor scheduling.
  • Broadway/GenStage for transient event-flow workloads with demand, a different abstraction.

9.3 Interview Relevance and Scope Boundary

P24 teaches durable database-backed work, retries, and effect idempotency. Project 5 teaches in-memory GenStage demand and backpressure; it is not a durable workflow engine. Project 6 injects OTP failures; P24 proves business convergence after those failures through persisted intent and stage identity.

10. Resources

10.1 Essential Reading

  • Oban: https://hexdocs.pm/oban/Oban.html
  • Oban.Worker: https://hexdocs.pm/oban/Oban.Worker.html
  • Oban unique jobs: https://hexdocs.pm/oban/unique_jobs.html
  • Ecto.Multi: https://hexdocs.pm/ecto/Ecto.Multi.html
  • Oban testing: https://hexdocs.pm/oban/Oban.Testing.html

10.2 Standards and Further Study

  • Enterprise Integration Patterns, Idempotent Receiver, Message Store, and Competing Consumers.
  • Release It!, 2nd Edition, Chapters 4–5 on stability and timeouts.
  • Designing Data-Intensive Applications, 2nd Edition, processing and delivery-semantics chapters.