BEAM (Erlang/Elixir) Mastery - Real World Projects

Goal: Build a first-principles mental model of how BEAM languages (Erlang, Elixir, Gleam) combine functional data transformation with fault-tolerant concurrency. You will internalize Elixir’s pattern matching, binaries, protocols, macros, testing tools, Ecto data guarantees, OTP behaviors, distribution semantics, scheduling, native boundaries, numerical computing, embedded deployment, and release engineering. Across 37 production-shaped projects, you will learn when BEAM is the right tool, how to make failures observable and recoverable, and how to extend the ecosystem without hiding its operational trade-offs.

Introduction

BEAM is the virtual machine and runtime system behind Erlang and Elixir. Its design assumes that failures are normal and that concurrency should be cheap, isolated, and easy to supervise. This guide teaches you to reason about BEAM systems as networks of small processes coordinated by supervisors, not as monolithic servers protected by locks.

What problem does it solve today?

It makes it practical to build fault-tolerant systems by isolating failures and restarting components automatically.
It enables huge numbers of concurrent activities without shared-memory locks, using message passing as the default.
It supports distribution (multiple nodes) with transparent message passing and links/monitors across nodes.

What you will build across the projects:

A supervised real-time chat system
A rate limiter and circuit breaker service
A distributed key-value store
A Phoenix LiveView dashboard
A GenStage backpressure pipeline
A fault-injection harness to test supervision strategies
A hot-code-upgrade drill with release handling artifacts
A presence and notification service (Discord-style)
An ETS-powered session/cache service
A telemetry and observability pipeline
A multi-network cluster formation lab (nodes across isolated networks)
A WAN netsplit recovery and reconciliation drill
A federated edge event bus with backpressure across sites
Three foundation tools for financial reconciliation, calendar conflict detection, and safe duplicate-file quarantine
A concurrent link/TLS auditor, a property-tested settlement parser, and a Mix SBOM/license auditor
A compile-time pricing DSL, webhook authenticity gateway, pluggable object-storage library, and compiler-aware migration linter
Transactional inventory, durable Oban workflows, dynamic tenant repositories, and a custom Ecto adapter
A framed TCP device gateway, a Nerves cold-chain appliance, and a scheduler-safe Rustler NIF
An Nx forecasting engine, an explicit :gen_statem escrow workflow, and a BEAM artifact reproducibility auditor
An ETS/DETS session service, a replicated Mnesia work-order ledger, and deterministic crash/restart recovery drills
A self-contained Mix release deployed to an air-gapped clean host with no preinstalled Erlang or Elixir
A distributed service directory that compares local registration, :pg groups, and controlled global naming

In scope:

BEAM processes, message passing, and isolation
OTP behaviors (GenServer, Supervisor, Application)
Supervision strategies and fault recovery
Scheduling, reductions, and per-process GC
Distribution, nodes, links, and monitors
ETS, DETS, and Mnesia storage, persistence, backup, and recovery
Hot code upgrades, self-contained Mix releases, and release handling
Real-time web with Phoenix LiveView and backpressure with GenStage
Functional transformation with pattern matching, guards, binaries, Enum, and Stream
Protocols, behaviours, Mix tasks, quoted ASTs, macros, compiler tracers, and Hex packaging
Ecto changesets, constraints, transactions, durable jobs, and dynamic repositories
Plug security, TCP framing, Erlang interoperability, Nerves firmware, Rustler/NIF safety, and Nx tensors
Local process registration, distributed process discovery, and stale-route recovery

Out of scope:

Low-level VM internals beyond behavioral guarantees
Full SIP/Telco protocol implementations
Handwritten C NIFs, custom ERTS builds, and VM implementation internals

Big-Picture ASCII Diagram

INPUTS -> ELIXIR DATA LAYER -> BOUNDARIES -> BEAM RUNTIME -> DURABLE SYSTEMS
   |             |                 |             |                |
   v             v                 v             v                v
 files        patterns          HTTP/TCP       processes       Ecto/Oban
 calendars    streams           NIF/Nerves     supervisors     releases
 tensors      protocols         Erlang APIs    schedulers      clusters

bad data -> tagged error | local crash -> supervisor restart | durable failure -> retry/audit

How to Use This Guide

Read the Theory Primer first to build the mental model (do not skip it).
Build projects in order for your first pass; later, jump by learning path.
After each project, verify behavior using the exact outputs in the Definition of Done.
Keep a failure log: for each crash you cause, write what recovered it and why.

Prerequisites & Background Knowledge

Essential Prerequisites (Must Have)

Programming fundamentals: functions, recursion, pattern matching or conditional logic
Basic concurrency concepts: what a process/thread is, what a message is
Command-line basics: running commands, reading logs
Recommended Reading: “Operating Systems: Three Easy Pieces” - Concurrency chapters

Helpful But Not Required

Basic networking (TCP/HTTP) for real-time projects
Database basics for distributed KV and dashboards
UI basics for LiveView (HTML/CSS)

Self-Assessment Questions

Can you explain the difference between a process and a thread?
Can you describe a queue and why ordering matters?
Can you reason about what happens if one component in a system crashes?
Can you read and interpret a simple log stream?

Development Environment Setup Required Tools:

Erlang/OTP (latest stable)
Elixir (latest stable)
Mix (Elixir build tool)

Recommended Tools:

observer or :observer for runtime inspection
recon or telemetry libraries for instrumentation
PostgreSQL for the LiveView dashboard project

Testing Your Setup:

$ elixir --version
Erlang/OTP: <version>
Elixir: <version>

$ erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell
"<otp_release>"

Time Investment

Simple projects: 4-8 hours each
Moderate projects: 10-20 hours each
Complex projects: 20-40 hours each
Total sprint: 2-4 months

Important Reality Check BEAM mastery is about system behavior under failure, not just syntax. If your system never crashes during these projects, you are not pushing it hard enough. The goal is to learn how to design for recovery, not how to avoid every error.

Big Picture / Mental Model

Think of BEAM systems as a forest of tiny, isolated processes supervised by a hierarchy that enforces recovery policies. Message passing is the only coordination mechanism, and the scheduler guarantees fairness by preempting work.

          +-------------------+           +-------------------+
          |   Supervisor A    |           |   Supervisor B    |
          +---------+---------+           +---------+---------+
                    |                               |
          +---------+---------+           +---------+---------+
          |  Worker 1 (GS)    |           |  Worker 3 (GS)    |
          +---------+---------+           +---------+---------+
                    |                               |
          +---------+---------+           +---------+---------+
          |  Worker 2 (ETS)   |           |  Worker 4 (Stage) |
          +-------------------+           +-------------------+

If Worker 2 crashes -> Supervisor A restarts only Worker 2 (one_for_one).

Theory Primer

This primer is a mini-book. Each concept below maps directly to multiple projects.

Concept 1: BEAM Processes and Actor Isolation

Fundamentals BEAM processes are lightweight, isolated units of execution with their own heap and mailbox. They are not OS threads; they are managed by the BEAM runtime and are designed to be created in very large numbers. Messages between processes are copied by default, which eliminates shared-memory data races and makes isolation a first-class property. This isolation allows failures to be contained: if one process crashes, others remain unaffected unless they are explicitly linked or monitored. The actor model in BEAM is therefore not just a design pattern but a runtime guarantee. Process identity, mailbox ownership, links, monitors, and exit signals create an explicit vocabulary for lifecycle and failure. This vocabulary matters because concurrency correctness depends not only on what message is sent, but also on who owns state, how replies are correlated, and what happens when either participant disappears.

Deep Dive In BEAM, every process is a self-contained actor with three core pieces: mailbox, state (heap), and behavior (message handlers). Unlike thread-based concurrency models that rely on shared memory and locks, BEAM uses message passing, which drastically reduces the complexity of reasoning about concurrency. The price you pay is message copying, but the reward is deterministic isolation: no process can mutate another process’s memory, and no data race is possible by construction.

This is not a theoretical statement. The efficiency guide documents that a newly spawned Erlang process uses a small, fixed amount of memory, with a conservative initial heap size so that systems can run hundreds of thousands or millions of processes. The runtime expands and shrinks heaps as needed, which means memory use is proportional to work, not to worst-case allocation. This supports a system design style where you spawn a new process per connection, per task, or per workflow step without worrying about OS thread exhaustion.

Message passing itself has important semantics. Messages are copied between heaps; this creates a clear ownership boundary and avoids aliasing bugs. When messages cross nodes, they are encoded into the external term format and transported over TCP, then decoded on the receiving node. This means distribution is conceptually the same as local message passing, but with explicit latency, serialization, and security considerations.

The mailbox is FIFO, but selective receive can reorder processing because the runtime scans the mailbox for a matching pattern. This is a subtle performance and correctness issue: if you write a receive clause that matches a rare pattern, the runtime may scan a long mailbox for every receive, adding overhead. Message copying and mailbox scanning together influence the architecture: you typically use tagged messages and short queues, or you split responsibilities across processes to keep mailboxes small.

A BEAM process can be linked or monitored. Links are bidirectional and propagate exits; monitors are unidirectional and deliver a DOWN message. These primitives allow you to build fault detection and cascading recovery policies. Supervisors use these to restart crashed workers. When you design your own process tree, you must decide which failures should be isolated and which should propagate upward.

The key design mindset is that processes are cheap and disposable. Instead of writing complex defensive code for every edge case, you let the process crash and rely on supervision to recover. This is not recklessness; it is a deliberate architecture that trades local complexity for system-level stability.

Process topology should follow failure and concurrency boundaries, not nouns in a domain diagram. A process per request may be ideal for short independent work, while a process per database row can create needless lifecycle and routing overhead. Long-running state owners need bounded mailboxes, call timeouts, and explicit reply correlation. Synchronous calls introduce waiting dependencies and possible cycles; asynchronous messages reduce coupling but require acknowledgement, ordering, and overload policies. Process dictionaries and registered names can be useful, yet ambient state or global naming makes ownership harder to see. A well-designed process API therefore documents message shapes, maximum expected queue depth, timeout semantics, and exit behavior. It also separates pure transformation from process orchestration so business rules can be tested without scheduling. The runtime guarantees isolation, but the architecture must still guarantee that producers cannot create unlimited work, replies cannot be confused, and dead recipients do not leave callers waiting forever.

How this fit on projects You will use this concept in every project, especially the chat system, rate limiter, and distributed store.

Definitions & key terms

Process: A BEAM-managed actor with its own heap and mailbox.
Mailbox: FIFO queue of incoming messages.
Message passing: Communication by sending immutable messages between processes.
Link/Monitor: Failure propagation and observation primitives.

Mental model diagram

[Process A] --send--> [Mailbox B] -> [Receive Loop] -> [State Update]
           (copy)          (queue)         (pattern match)

How it works (step-by-step, with invariants and failure modes)

A process sends a message to another process.
The message is copied into the receiver’s mailbox.
The receiver scans for a matching receive clause.
On match, the process updates its local state.
Invariant: no process can mutate another’s memory.
Failure modes: mailbox buildup, selective receive overhead, unhandled messages.

Minimal concrete example

Process Counter:
- State: count
- On message {inc}: count = count + 1
- On message {get, reply_to}: send {count, value} to reply_to

Common misconceptions

“Processes are threads.” They are lighter and runtime-managed.
“Messages are references.” They are copied (with refc binary exceptions).

Check-your-understanding questions

Why does message copying prevent data races?
What happens when a mailbox grows very large?
How does selective receive affect performance?

Check-your-understanding answers

Each process owns its memory; no shared mutation is possible.
Receive scans become expensive; latency grows.
The runtime may scan many messages to find a match.

Real-world applications

Connection-per-process servers
Fault-isolated background jobs
Concurrent pipelines with message passing

Where you’ll apply it

Project 1 (Chat System)
Project 2 (Rate Limiter)
Project 3 (Distributed KV)
Project 6 (Fault Injection Harness)

References

Key insights Isolation + message passing is the foundation of BEAM reliability.

Summary BEAM processes are lightweight, isolated actors with mailbox-driven concurrency. This model avoids shared-memory hazards and makes failure recovery a system design concern rather than a local coding burden.

Homework/Exercises to practice the concept

Draw a message flow between three processes for a request/response cycle.
Explain why selective receive can slow a busy process.

Solutions to the homework/exercises

The sender posts a message; the receiver replies; the sender processes the reply.
The mailbox must be scanned for the matching pattern each time.

Concept 2: OTP Behaviors and Supervision Trees

Fundamentals OTP behaviors are standardized patterns for long-running processes (GenServer, Supervisor, Application). They provide uniform lifecycle, error handling, and integration with supervision. A supervision tree is a hierarchy of supervisors and workers where supervisors monitor and restart children according to a defined strategy. This structure is the practical implementation of the “let it crash” philosophy: local failures are expected and recovered by a supervising process rather than by complex defensive code. Behaviours also make system messages, shutdown, debugging, and code-change hooks consistent. Child specifications record start, restart, shutdown, type, and module information, turning lifecycle policy into inspectable data. Applications group components into startable units and define the root of their supervision structure.

Deep Dive OTP behaviors exist because many processes in BEAM systems follow the same lifecycle: initialize state, receive messages, handle calls/casts, and terminate cleanly. GenServer formalizes this cycle, providing callbacks for initialization, synchronous calls, asynchronous casts, and miscellaneous messages. The benefit is not convenience alone; it is interoperability. A GenServer can be supervised, introspected, and upgraded consistently across the system.

Supervision trees are the backbone of fault tolerance. The design principles describe supervisors as processes that monitor workers and restart them when they fail. Supervisors apply strategies such as one_for_one (restart only the failed child), one_for_all (restart all children), or rest_for_one (restart the failed child and those started after it). The strategy is a semantic decision about dependency: if workers are independent, one_for_one is safer; if they depend on shared state, one_for_all may be appropriate.

The restart intensity and period parameters provide circuit-breaker behavior: too many restarts in a short time can force a supervisor to give up, which then escalates the failure up the tree. This is the mechanism that prevents infinite restart loops and signals that a deeper issue exists. The tree is not a simple retry mechanism; it is a controlled failure policy.

OTP behaviors also encode system messages: a GenServer automatically handles system-level calls such as code upgrades or state inspection. This is why you should not write your own manual receive loops for long-lived services unless you need custom semantics. By using behaviors, you gain the runtime’s built-in tooling, introspection, and fault handling.

Another critical aspect is naming and registration. GenServers and supervisors can be registered locally or globally, which affects discoverability and distribution. Naming semantics are shared between GenServer and Supervisor in Elixir. If you register globally, you must handle network partitions and name conflicts; if you register locally, you need a discovery mechanism. These are architectural trade-offs that become explicit in distributed projects.

The “let it crash” philosophy only works when supervision trees are designed with intent. You must classify which failures are recoverable locally and which should cascade. For example, a crashed cache worker should be restarted; a corrupted database connection might need escalation to shut down the service cleanly. Supervision is therefore not just restart logic; it is policy design.

Shutdown is as important as restart. Supervisors terminate children in reverse start order so dependents can stop before the resources they use. A child may need a finite shutdown budget to flush a bounded queue or return a lease, but infinite shutdown can make deployments hang. Restart type carries meaning: permanent children return after any exit, transient children return after abnormal exits, and temporary children are never restarted. Dynamic supervisors manage children discovered at runtime, while registries solve naming; neither replaces domain routing or persistence. The most common architectural error is using one GenServer as a universal serialization point. OTP makes that server reliable, but not scalable. Partitioning, separate pure work, and explicit resource ownership are still required. A supervision tree should be reviewable as a failure map: for every child, the designer can explain what its crash loses, which siblings depend on it, how restart state is reconstructed, and when repeated failure should escalate instead of looping.

How this fit on projects Projects 1, 2, 3, 5, and 6 are rooted in OTP behaviors and supervision decisions.

Definitions & key terms

OTP behavior: A standardized process pattern (GenServer, Supervisor).
Supervisor: A process that starts, monitors, and restarts children.
Supervision tree: Hierarchical structure of an application into workers and supervisors.
Restart strategy: Policy for handling child failures.

Mental model diagram

Supervisor
  |-- Worker A (GenServer)
  |-- Worker B (GenServer)
  |-- Supervisor C
        |-- Worker C1

How it works (step-by-step, with invariants and failure modes)

Supervisor starts children in defined order.
Children run their workloads.
On failure, supervisor applies its strategy.
If restart intensity is exceeded, supervisor terminates and escalates.
Invariant: supervisors remain responsible for child lifecycle.
Failure modes: incorrect strategy choice, restart loops, missing cleanup.

Minimal concrete example

Service Tree:
- Top supervisor
  - DB worker (restart: transient)
  - Cache worker (restart: permanent)

If DB worker fails repeatedly, supervisor escalates; cache restarts alone.

Common misconceptions

“Supervision trees prevent all outages.” They limit failure blast radius, not eliminate outages.
“One_for_all is always safer.” It is only safer when children are tightly coupled.

Check-your-understanding questions

When would you choose one_for_one vs one_for_all?
Why do supervisors restart children in reverse order on shutdown?
What happens if restart intensity is exceeded?

Check-your-understanding answers

One_for_one for independent children; one_for_all for tightly dependent ones.
Supervisors terminate children in reverse start order.
The supervisor terminates and failure escalates up the tree.

Real-world applications

Web servers with resilient worker pools
Background job systems
Fault-tolerant caches and queues

Where you’ll apply it

Project 1 (Chat System)
Project 2 (Rate Limiter)
Project 5 (GenStage Pipeline)
Project 6 (Fault Injection Harness)

References

Key insights Supervision is policy, not just restart logic.

Summary OTP behaviors standardize process structure and supervision trees enforce fault recovery policies. This is the core of BEAM reliability.

Homework/Exercises to practice the concept

Sketch a tree for a web app with DB, cache, and worker pool.
Decide which children should be permanent vs transient.

Solutions to the homework/exercises

Separate supervisors for DB and workers; cache as a child of app supervisor.
DB connection often transient; worker pool permanent.

Concept 3: Scheduling, Reductions, and Per-Process GC

Fundamentals BEAM uses preemptive scheduling based on reductions to ensure no single process can monopolize CPU time. This creates fairness across thousands of processes and enables soft real-time behavior. Each process has its own heap and garbage collector, so GC pauses are localized rather than global. These design choices make latency more predictable than in stop-the-world GC systems. Scheduler fairness is not the same as unlimited throughput: runnable queues, mailbox pressure, copying, allocation, and native work still consume finite resources. Normal schedulers execute BEAM code, while dirty CPU and dirty I/O schedulers provide separate lanes for appropriately classified native work. Measurement must distinguish CPU saturation, run-queue imbalance, garbage collection, and waiting on external resources.

Deep Dive The BEAM scheduler is designed for fairness and responsiveness. Instead of allowing a process to run indefinitely, the scheduler counts reductions (units of work) and yields execution after a quota. This means CPU-bound tasks cannot starve I/O-bound tasks. The exact reduction count is an implementation detail, but the design guarantee is that preemption occurs regularly and cannot be disabled by user code.

Per-process garbage collection is equally important. Each process has a private heap; GC runs on that heap only. The efficiency guide documents that the heap grows as needed and can shrink under certain conditions. This prevents global pauses. If one process allocates too much memory and triggers GC, only that process is paused. The rest of the system keeps running. This is a key property for soft real-time systems where latency spikes must be bounded.

The efficiency guide also notes that you can control minimum heap size to reduce GC overhead for short-lived processes, but warns this is an optimization that requires careful measurement. This highlights a broader lesson: BEAM’s defaults are conservative and safe, but you can tune them when you understand your workload.

Scheduler fairness interacts with mailbox patterns. If a process is constantly handling messages, it might be scheduled frequently, while a CPU-bound process will be preempted. This is why BEAM systems prefer to split heavy computation into separate processes or use ports/NIFs for compute-intensive tasks. The scheduler model encourages concurrency-friendly, reactive workloads rather than long-running CPU loops.

Understanding these mechanics matters because they shape how you design services. For example, a GenStage pipeline assumes that work is distributed across many processes so backpressure can be applied. If you put all work in a single process, scheduler fairness cannot help you; the bottleneck remains. The right architecture is one that aligns with BEAM’s scheduling and GC model.

Memory behavior crosses process boundaries through messages. Small terms are generally copied to the receiver heap, while large reference-counted binaries may be shared with off-heap references. A tiny slice can therefore retain a large binary longer than expected. Processes that accumulate messages also retain the terms referenced by those messages, so mailbox length and message size must be considered together. Explicit garbage collection is rarely a first choice; reducing allocation, shortening process lifetime, avoiding accidental binary retention, and partitioning work usually provide clearer wins.

Latency investigations should observe scheduler utilization, run queues, process reductions, garbage-collection counts, heap size, mailbox length, and long-scheduling events under a repeatable workload. Average CPU alone cannot explain a single overloaded scheduler or a blocked normal NIF. Work should be moved to more processes only when it can be partitioned without creating coordination overhead. CPU-heavy algorithms may belong in Nx, a Port, or a dirty NIF, but each choice changes failure isolation and data-transfer cost. The scheduler provides preemption and fairness; engineers remain responsible for bounded admission, workload placement, and evidence-based tuning.

Backpressure completes this model by preventing runnable work and mailbox growth from becoming unbounded. Fair scheduling shares finite CPU; it does not decide which incoming work the system should refuse, defer, or shed.

How this fit on projects Projects 2, 5, and 10 rely on predictable scheduling and GC behavior.

Definitions & key terms

Reduction: A unit of work used for scheduling fairness.
Per-process GC: Garbage collection scoped to a single process.
Soft real-time: Systems with bounded but not hard deterministic latency.

Mental model diagram

Scheduler
  -> run P1 for N reductions
  -> run P2 for N reductions
  -> run P3 for N reductions

GC runs inside each process heap, not globally.

How it works (step-by-step, with invariants and failure modes)

Scheduler picks runnable processes.
Runs each for a fixed reduction budget.
Preempts and moves to the next.
GC occurs when a process heap threshold is reached.
Invariant: no global stop-the-world pauses.
Failure modes: CPU-bound single process, excessive allocations, mailbox overflow.

Minimal concrete example

Pipeline:
- Producer process emits events
- Multiple worker processes handle tasks
- Each worker yields to scheduler periodically

Common misconceptions

“BEAM is always fast.” It is fair and responsive, but CPU-heavy tasks can still bottleneck.
“GC is global.” It is per-process.

Check-your-understanding questions

Why does per-process GC reduce latency spikes?
How does reduction-based scheduling prevent starvation?
What happens if one process receives all work?

Check-your-understanding answers

Only the allocating process pauses; others continue.
Processes are preempted after a fixed budget.
It becomes the bottleneck regardless of scheduler fairness.

Real-world applications

High-concurrency web services
Stream processing pipelines
Real-time dashboards

Where you’ll apply it

Project 2 (Rate Limiter)
Project 5 (GenStage Pipeline)
Project 10 (Telemetry Pipeline)

References

Key insights Fair scheduling and localized GC are core to BEAM responsiveness.

Summary BEAM schedules processes fairly and performs GC locally, enabling predictable latency under heavy concurrency when workloads are well-distributed.

Homework/Exercises to practice the concept

Explain why a single CPU-bound process can hurt system responsiveness.
Draw a process graph that avoids a single bottleneck.

Solutions to the homework/exercises

The scheduler can preempt but the workload remains centralized.
Use multiple workers with a supervisor and load distribution.

Concept 4: Distribution, Nodes, and Fault Boundaries

Fundamentals A distributed Erlang system consists of multiple runtime nodes that communicate over TCP/IP. Message passing between processes at different nodes, as well as links and monitors, are transparent when pids are used. Registered names, however, are local to each node. This means distribution feels like local message passing but introduces network latency, partial failure, and security concerns. Distributed nodes must be explicitly named, and secure distribution requires TLS configuration. Node identity, discovery, authorization, topology, and recovery are separate concerns. A connection proves reachability at one moment; it does not provide consensus, durable delivery, or a conflict-resolution policy. Remote messages should therefore be treated as network operations even when the API resembles local send.

Deep Dive Distribution is one of BEAM’s strongest differentiators. When you connect nodes, you gain transparent messaging, links, and monitors across machines. The same primitives used for local fault detection can therefore be used across a cluster. This simplifies distributed design because you do not need a separate protocol for failure detection; the runtime already provides it.

However, transparency does not remove the realities of distributed systems. Messages must be serialized into an external term format and transmitted over TCP; this adds latency and increases the cost of large messages. The system can still experience partitions or node failures. When a node goes down, linked or monitored processes receive exit or DOWN messages, which you must handle in supervision logic. This is how BEAM represents network failure in the same language as process failure.

Naming is another critical issue. Pids are globally unique within a distributed system, but ordinary registered names and Elixir Registry entries are node-local. If you want to address a named process on another node, you must include the node name or add a discovery/routing layer. Registry supports unique keys, duplicate-key use cases such as local pub/sub, and :via tuples, but it does not become cluster-wide merely because nodes are connected. Distributed designs can use :pg for process groups whose membership is visible across connected nodes, a carefully bounded :global name when singleton semantics are truly required, or an application-specific directory. Each option has different behavior during churn and partitions. Lookup results are observations, not leases: a pid can disappear immediately after discovery, so callers must monitor, retry, and make work idempotent where necessary.

Security is also explicit. The distribution documentation warns that starting a node without TLS (inet_tls) exposes it to attacks that may give complete access to the node and the cluster. This means secure distribution is not optional in production. You must choose cookies, TLS, and network isolation deliberately.

The distributed model lends itself to certain architectures: sharded state per node, data locality via process placement, and supervision trees that span nodes. But you must also design for partition tolerance. If you choose to register a single global name, what happens during a split-brain? Your projects will force you to answer these questions by building a distributed KV store and a presence service that handles node loss gracefully.

Delivery semantics remain intentionally modest. Sending a message does not prove that the remote process received, processed, or durably recorded it. A monitor can report that a process or node became unreachable, but during a partition the observer cannot know whether the remote side is dead or merely disconnected. Application protocols need correlation IDs, acknowledgements, deadlines, retry rules, and idempotency where those guarantees matter. Large cross-node terms incur encoding, copying, and network cost, so message design should favor bounded payloads and data locality.

Cluster topology is also an operational choice. A full mesh is simple for small trusted clusters but connection count grows rapidly. WAN links introduce variable latency and partitions that can amplify global registries or synchronous calls. Gateway or federated topologies can contain that blast radius at the expense of explicit routing. Secure distribution requires more than a cookie: network isolation, TLS identity, certificate rotation, restricted port ranges, and a deliberate EPMD strategy all belong to production design. Distribution is most successful when the application marks boundaries rather than pretending every pid is equally local. The learner should be able to state what happens during asymmetric connectivity, simultaneous ownership, rolling versions, and delayed messages after reconnection.

How this fit on projects Projects 3, 7, 11, 12, 13, and 37 move from remote messaging and cluster formation to explicit service discovery, routing, and recovery from stale membership.

Definitions & key terms

Node: A named Erlang runtime in a distributed system.
Distribution: Transparent messaging across nodes.
Registered name: A local alias for a pid on a node.
Process discovery: Resolving a logical service or capability to one or more currently reachable pids.
Process group: A named membership set such as an Erlang :pg group.
Partition: Loss of connectivity between nodes.

Mental model diagram

Node A (name@host) <--- TCP ---> Node B (name@host)
   |                                 |
  Pid A                             Pid B
send(Pid B, Msg) works like local message send

How it works (step-by-step, with invariants and failure modes)

Start nodes with explicit names.
Connect nodes; establish distribution channel.
Send messages using pids across nodes.
Monitor remote pids for failure.
Invariant: messaging semantics are the same locally and remotely.
Failure modes: network partitions, node crashes, name collisions, insecure distribution.

Minimal concrete example

Cluster messaging:
- Node A sends {ping, t} to Pid on Node B
- Node B replies {pong, t}
- If Node B disconnects, Node A receives DOWN

Common misconceptions

“Distributed Erlang hides all network issues.” It hides APIs, not failures.
“Registered names are global.” They are local per node.

Check-your-understanding questions

What must be included when sending to a registered name on another node?
Why is TLS important for distribution?
How do you detect that a remote node is down?

Check-your-understanding answers

You must include the node name because registrations are local.
Without TLS the node may be fully compromised.
Links or monitors deliver exit/DOWN messages.

Real-world applications

Distributed caches
Presence systems and chat backends
Clustered job processors

Where you’ll apply it

Project 3 (Distributed KV Store)
Project 7 (Presence and Notification Service)
Project 11 (Multi-Network Cluster Formation Lab)
Project 12 (WAN Netsplit Recovery Drill)
Project 37 (Distributed Service Directory and Worker Fleet Router)

References

Key insights Distribution is transparent in API, not in failure semantics.

Summary Distributed Erlang makes cross-node messaging feel local while preserving the realities of latency and failure. You must design supervision and naming with these realities in mind.

Homework/Exercises to practice the concept

Sketch a two-node cluster and describe how to detect node failure.
Describe a naming strategy for a distributed service.

Solutions to the homework/exercises

Use monitors on remote pids and handle DOWN messages.
Use a local registry plus a routing process per node.

Concept 5: State Management with ETS, DETS, and Mnesia

Fundamentals BEAM systems often store mutable state inside processes, but ETS (Erlang Term Storage) provides in-memory tables with efficient access for larger shared datasets. ETS tables are dynamic tables created by a process, and they support multiple table types (set, ordered_set, bag, duplicate_bag). DETS stores Erlang terms on disk through a table-shaped API, while Mnesia adds transactional, replicated tables with RAM and disk copy types. Access protection (private, protected, or public), read/write concurrency options, key position, ownership transfer, heir processes, file-open ordering, and copy placement affect performance and lifecycle. None of these tools removes the need to define authority, persistence, consistency, backup, and recovery; they provide different primitives for implementing those policies.

Deep Dive In BEAM, mutable state is typically owned by a single process, which serializes access. This is ideal for small, strongly encapsulated state. For large datasets or shared lookups, ETS is the standard tool. ETS provides constant-time access for sets and logarithmic access for ordered sets, making it useful for caches, registries, and session tables.

ETS tables are created by a process and are destroyed when that process terminates unless ownership is transferred to an heir. This is an important lifecycle property: availability depends on ownership, and an heir only preserves the table inside the same VM lifetime. tab2file can create a snapshot, but a snapshot is not continuous durability. If you put ETS in a supervisor-managed owner process, the table can be recreated on restart, but the design must name the durable authority and replay boundary.

DETS is useful when one node needs a disk-backed term table without a separate database. It persists across VM restarts, supports set, bag, and duplicate_bag, and deliberately lacks ETS ordered_set. It also has operational constraints, including a 2 GB per-table limit and repair behavior after an unclean close. A common learning architecture uses DETS as the authoritative journal or snapshot and ETS as a reconstructed read-optimized hot path. The write order, acknowledgement point, idempotency key, and compaction policy determine whether a crash loses, duplicates, or resurrects data.

ETS is not a database; it is a high-performance in-memory store optimized for BEAM use. It supports match and select operations, but these can be expensive and may require scanning the whole table. The tables-and-databases guide explicitly warns that select/match can become expensive and recommends structuring data to minimize full scans. This is a key design constraint: ETS is fast for key-based access, but you must design your keys and queries intentionally.

Mnesia adds transactions and distribution. Its API resembles ETS for basic operations, but it supports replication and transactions across nodes. This makes it attractive for distributed KV stores, but it also introduces complexity: partitions, schema management, and recovery. For learning projects, you can simulate a Mnesia-like log with ETS plus append-only persistence to understand the trade-offs.

In OTP systems, ETS often sits behind a GenServer or GenStage pipeline. The process serializes writes and defines a clear API, while ETS provides fast reads. This pattern gives you the speed of shared memory with the safety of controlled access. It is not magic; you still must handle concurrency, but you do so at the process boundary rather than with locks.

Choosing between process state and ETS starts with access shape. A small state machine with ordered updates belongs naturally in one process. A read-heavy lookup table shared by many callers may benefit from direct protected ETS reads plus a narrow write owner. Serializing every read through a GenServer can become a bottleneck; allowing every caller to write a public table can make invariants impossible to enforce. Atomic ETS operations such as counters can be powerful, but compound business invariants still need one authority or a transactional store.

Lifecycle design must answer what happens when the owner exits. A stable supervised owner can recreate derived tables, an heir can receive ownership, or persistent data can be reloaded from an authoritative source. None of these is automatic recovery of lost state. Table scans, match specifications, and continuation-based selects need workload tests because a convenient query can block a scheduler or create large intermediate results. Keys should be designed around the dominant reads instead of recreating relational queries in memory.

Mnesia adds location-aware tables, transactions, and replication, but its behavior under membership changes and partitions must be planned. Replication is not consensus, and transactions cannot make an unavailable partition disappear. Schema operations, copy types, startup order, and recovery procedures become part of operations. Use Mnesia when its consistency and deployment model match the system; do not choose it merely because it ships with OTP. Projects should state the source of truth, acceptable data loss, conflict policy, and how state is verified after restart.

How this fit on projects Projects 2, 3, and 8 introduce ETS-based storage. Project 34 makes DETS the durable recovery authority behind an ETS hot path, while Project 35 uses Mnesia transactions, disk copies, replication, backup, and restore for a clustered operational ledger.

Definitions & key terms

ETS: Built-in term storage with fast access.
DETS: Disk-based term storage with a table API and explicit file lifecycle.
Table owner: The process that created an ETS table.
Select/match: Query operations that may scan tables.
Mnesia: Distributed DB built on Erlang concepts.

Mental model diagram

[Client] -> [GenServer] -> [ETS Table]
                |            (fast lookup)
                v
             [State]

How it works (step-by-step, with invariants and failure modes)

A process creates an ETS table.
Clients access via API or direct table reads.
Data is inserted, updated, looked up by key.
If owner dies, the table is destroyed.
Invariant: ETS access is efficient for key-based reads.
Failure modes: table loss on crash, expensive scans, inconsistent writes.

Minimal concrete example

Session Cache:
- Key: session_id
- Value: user_id, expiry
- Read path: lookup by session_id

Common misconceptions

“ETS is durable.” It is in-memory; the table is destroyed on owner crash.
“ETS is a full database.” It is optimized for key access, not complex queries.

Check-your-understanding questions

What happens to an ETS table when its owner crashes?
Why are select/match operations potentially expensive?
When would you choose Mnesia over ETS?

Check-your-understanding answers

The table is destroyed with its owner process.
They may scan the entire table.
When you need replication or transactions across nodes.

Real-world applications

Session stores and caches
Local registries
Distributed KV stores

Where you’ll apply it

Project 2 (Rate Limiter)
Project 3 (Distributed KV Store)
Project 8 (ETS Cache Service)
Project 34 (ETS/DETS Session Store)
Project 35 (Mnesia Work-Order Ledger)

References

Key insights ETS gives you fast shared data when you respect its lifecycle and query limits.

Summary ETS is a high-performance in-memory table system; Mnesia adds distribution and transactions. Both require careful lifecycle and query design.

Homework/Exercises to practice the concept

Design a table schema for a rate limiter.
Identify which operations require full-table scans and avoid them.

Solutions to the homework/exercises

Key by {user_id, window} with count as value.
Use key lookups; avoid scan-based queries.

Concept 6: Real-Time Web and Backpressure (LiveView + GenStage)

Fundamentals Phoenix LiveView enables rich, real-time user experiences with server-rendered HTML, diff tracking, and WebSocket-based updates. A persistent connection is established between client and server, which reduces work per request and allows faster reactions to user events. GenStage provides backpressure-aware pipelines where consumers explicitly demand events and producers never send more than requested. Together, they demonstrate how BEAM’s process model supports real-time systems without heavy client-side code or external queues. The browser connection, LiveView process, PubSub subscriptions, producers, consumers, and external systems still have independent lifecycles. Real-time correctness therefore requires reconnect behavior, authorization, bounded assigns, demand budgets, acknowledgement semantics, and clear ownership of durable state.

Deep Dive LiveView works by rendering HTML on the server, sending it to the client initially as a static page, then maintaining a persistent connection that streams updates. This design means the client does not need to own application state; instead, the server is authoritative. The runtime diffs state changes and sends only the minimal updates. This is powerful for dashboards, collaborative tools, and real-time monitoring where consistency matters more than offline capability.

GenStage addresses the other side of real-time systems: throughput and backpressure. In a naive system, producers can overwhelm consumers, leading to mailbox growth and latency spikes. GenStage lets consumers explicitly demand a specific number of events, ensuring that producers never outpace the system’s capacity. This is not a library convenience; it is a concurrency contract that aligns with BEAM’s message-passing model.

The Discord engineering blog describes how Elixir was used for a highly concurrent real-time system, reporting nearly five million concurrent users and millions of events per second by July 2017. This scale is only achievable when flow control is a first-class concern. You cannot rely on infinite queues or manual throttling. The runtime must treat demand as a signal that shapes how work moves through the system.

LiveView and GenStage are often combined. A real-time dashboard can subscribe to a GenStage pipeline that feeds events; the LiveView process receives messages and updates its state, and the runtime only sends changed parts of the UI. This is a BEAM-native approach to real-time systems: concurrency, backpressure, and UI updates all in one runtime.

LiveView processes are per connected client, so every assign has a multiplicative cost. Storing a large history or high-frequency raw stream in each socket process creates memory and diff work proportional to connected users. Aggregation should happen in dedicated processes or storage, while each LiveView holds only the viewport and controls needed to render. Temporary assigns, streams, throttled updates, and pagination can reduce retention, but their semantics should be tested across reconnects. Authentication at HTTP mount time is not always enough; connected authorization and resource ownership must remain valid for events received later.

GenStage demand bounds the relationship between adjacent stages, not every upstream or downstream resource automatically. A producer connected to an unbounded external broker still needs an acknowledgement and prefetch policy. A consumer that writes to a database may request more work than its pool can sustain. Buffer size, max_demand, min_demand, batch size, stage concurrency, and failure policy interact. The correct tuning target is not maximum throughput at any cost, but stable queue depth and latency under expected bursts and slow dependencies.

Delivery semantics must be explicit. If a consumer crashes after an external effect but before acknowledgement, an event may be retried. If a LiveView disconnects, transient messages may be missed. Durable facts belong in a durable store; PubSub and process messages are delivery mechanisms, not databases. The project should distinguish ephemeral UI freshness from business completion. Instrument demand, buffer occupancy, processing time, dropped/compacted events, and socket update rate so the system exposes overload before mailboxes become the only signal.

How this fit on projects Projects 4, 5, and 10 are built on these concepts.

Definitions & key terms

LiveView: Server-rendered, real-time UI with diff updates.
Persistent connection: Long-lived channel between client and server.
Backpressure: Consumer-controlled demand to prevent overload.
Producer/Consumer: Components in a data pipeline.

Mental model diagram

Events -> GenStage Producer -> GenStage Consumer -> LiveView -> Browser
         (demand-based flow)        (bounded)       (diff updates)

How it works (step-by-step, with invariants and failure modes)

Producers emit events only when consumers demand them.
Consumers process events and update state.
LiveView diffs state and pushes UI updates.
Invariant: demand controls flow; UI updates are incremental.
Failure modes: unbounded demand, heavy LiveView processes, burst storms.

Minimal concrete example

Real-time chart:
- Producer emits metrics
- Consumer aggregates
- LiveView updates chart every second

Common misconceptions

“LiveView is just websockets.” It is server-rendered with diff updates.
“Backpressure is optional.” Without it, queues grow until failure.

Check-your-understanding questions

Why is demand-driven flow safer than unbounded queues?
What does LiveView send after the initial render?
How would you prevent a slow consumer from collapsing a pipeline?

Check-your-understanding answers

It bounds work and prevents overload.
Only the diffs (changed parts) are sent.
Limit demand and use backpressure-aware stages.

Real-world applications

Monitoring dashboards
Chat systems and activity feeds
Burst-resistant pipelines

Where you’ll apply it

Project 4 (LiveView Dashboard)
Project 5 (GenStage Pipeline)
Project 10 (Telemetry Pipeline)

References

Key insights Real-time systems need both UI diffing and backpressure to stay stable.

Summary LiveView delivers server-rendered realtime UI; GenStage delivers safe flow control. Combined, they are a BEAM-native real-time stack.

Homework/Exercises to practice the concept

Design a pipeline with bounded demand and describe how to tune it.
Sketch a LiveView state update flow for a dashboard.

Solutions to the homework/exercises

Set demand limits and buffer sizes at each stage.
Server state changes trigger diff updates to the client.

Concept 7: Code Loading, Mix Releases, and Release Handling

Fundamentals Erlang/OTP supports runtime code replacement and release handling through SASL. The release handler installs upgrades based on appup and relup instructions, and can reload, restart, or replace applications as needed. Core applications (ERTS, kernel, stdlib, sasl) require runtime restarts during upgrades via restart_new_emulator, while other upgrades may use restart_emulator or in-place changes. Elixir’s Mix release task also assembles an immutable operational artifact containing applications, boot scripts, configuration providers, commands, and—by default—the Erlang Runtime System (ERTS). A target host therefore does not need Elixir, Mix, or Erlang preinstalled when the compatible runtime is included. Upgrade and packaging safety include state compatibility, target OS/architecture/ABI, native/system-library dependencies, protocol compatibility, artifact provenance, runtime configuration validation, and rollback—not merely loading a changed module.

The operational unit is the whole release, not an isolated source file. A safe change must preserve startup ordering, dependency versions, secrets and environment expectations, health checks, and a tested path back to the previous artifact.

Deep Dive Runtime code loading is a distinguishing feature of BEAM. The system can load a new version of a module while the system runs, and processes can transition to the new code when they next call into that module. This is not magic; it relies on careful design of process state and upgrade callbacks. The release handling framework in OTP formalizes this for full releases. It uses appup files to describe application upgrade steps and relup files to describe release-level steps.

Release handling is explicit about restart boundaries. The documentation describes restart_new_emulator for upgrades that change the runtime system or core applications. This instruction reboots the runtime and is required when ERTS or core apps are upgraded. For other upgrades, restart_emulator can be used at the end of a relup to reboot after upgrade instructions are executed. This makes it clear that “hot upgrade” has constraints: some upgrades require a controlled reboot, not a pure in-place swap.

A key challenge in hot upgrades is state migration. If the internal state structure of a process changes, you must define a code_change step to transform state. This is a design commitment; if you do not plan for it, hot upgrades will be painful. Projects in this guide include a controlled upgrade drill so you can practice the mechanics of a safe state transition.

Release handling also interacts with distribution. Each node can have its own release version, and upgrades can be coordinated across nodes using synchronization instructions. This enables rolling upgrades or staged rollouts, but only if you design your upgrade plan and compatibility boundaries.

The goal of learning this concept is not to make every project hot-upgradeable. It is to understand the constraints and the tooling so that when uptime requirements demand it, you can design for upgrade safety rather than retrofitting under pressure.

The BEAM code server can keep current and old versions of a module. A fully qualified external call reaches current code, while a process continuing through local recursion may remain in old code until it makes an external call or is explicitly migrated. Lingering processes can prevent old code from being purged, and forced purge can terminate them. This makes upgrade-aware loops and state-transition callbacks part of the design. Message and storage formats also need compatibility while nodes run mixed versions during a rolling deployment.

Modern operational practice often favors rolling restarts over complex hot upgrades, but the same compatibility questions remain. Can the new release read old persisted state? Can old and new nodes exchange messages? Are environment variables validated before traffic? Does rollback require reversing a database migration? Release handling cannot solve an irreversible external schema change. Expand-contract migrations, feature flags, and versioned protocols may be safer than an in-place transformation.

Artifact inspection strengthens the feedback loop. BEAM files contain code and metadata chunks whose paths, compile information, documentation, and debug data can change reproducibility or reveal internals without changing runtime behavior. Native libraries and Nerves firmware add platform-specific assets. A production release process should record checksums, toolchain versions, application versions, configuration requirements, and provenance; it should verify the assembled artifact rather than assuming compilation output is complete. Upgrade and rollback drills need health checks that observe real capability, not just a running node.

Mix release assembly and deployment are separate from hot upgrade mechanics. The default include_erts behavior bundles the runtime used for the build target. After assembly, the :tar step creates a distributable archive. That convenience does not make the artifact universally portable: it must be built for a compatible operating system, CPU architecture, C library/ABI, and native dependency set. runtime.exs and release environment variables defer secrets and node-specific values until boot, while release commands such as start, stop, remote, and eval provide an operational interface. The decisive verification is a clean-host test where elixir, mix, and erl are absent, the tarball is unpacked, runtime configuration is injected, and a real health check passes.

How this fit on projects Project 9 teaches hot upgrade handling, Project 32 audits compiled release artifacts, and Project 36 assembles, archives, deploys, operates, and rolls back a self-contained Mix release on a clean compatible host.

Definitions & key terms

Release handler: OTP component that installs upgrades.
appup/relup: Upgrade instruction files for apps and releases.
restart_new_emulator: Required for core runtime upgrades.
restart_emulator: Reboot instruction for non-core upgrades.
Mix release: A versioned bundle of applications, boot metadata, configuration machinery, commands, and optionally ERTS.
ERTS: The Erlang Runtime System that executes BEAM bytecode.

Mental model diagram

Old Release -> appup/relup -> release_handler -> Upgrade Steps -> New Release
      |                                                |
      +---- code_change(state) ------------------------+

How it works (step-by-step, with invariants and failure modes)

Build release with appup/relup instructions.
Install release package on running node.
Release handler executes upgrade steps.
Processes transition state via code_change.
Invariant: state transitions must be explicit and safe.
Failure modes: incompatible state, missing appup, core upgrade without restart.

Minimal concrete example

State migration:
- v1 state: {user_id, count}
- v2 state: {user_id, count, last_seen}
- code_change adds last_seen with default

Common misconceptions

“All upgrades are hot.” Core runtime upgrades require restart.
“State changes are automatic.” You must define migration steps.

Check-your-understanding questions

Why do core OTP applications require runtime restart?
What does an appup file describe?
Why is state migration central to hot upgrades?

Check-your-understanding answers

The runtime itself cannot hot-swap its core components.
Upgrade instructions between application versions.
Processes carry state across versions; you must transform it.

Real-world applications

Zero-downtime upgrades in telecom systems
Rolling upgrades in distributed services

Where you’ll apply it

Project 9 (Hot Code Upgrade Drill)
Project 1 (Chat System) as optional enhancement
Project 32 (BEAM Artifact Reproducibility Auditor)
Project 36 (Self-Contained Mix Release and Air-Gapped Deployment)

References

Key insights Hot upgrades are possible only when state and upgrade steps are explicit.

Summary Release handling makes runtime upgrades possible but requires careful design of state, versioning, and upgrade steps.

Homework/Exercises to practice the concept

Describe a state change that requires migration.
Sketch an upgrade plan with a rollback strategy.

Solutions to the homework/exercises

Add a new field with a default value during upgrade.
Deploy appup, verify health, and maintain rollback relup.

Concept 8: Elixir Data Transformation, Pattern Matching, and Binary Parsing

Fundamentals Elixir programs are built by transforming immutable values through functions whose clauses describe the shapes they accept. Pattern matching is therefore both a data-extraction mechanism and a lightweight way to state preconditions. Guards refine those shapes with predictable checks, while tagged tuples such as {:ok, value} and {:error, reason} make expected failure part of the returned data. Lists, maps, structs, dates, binaries, Enum, and lazy Stream pipelines form the working vocabulary for ingestion and transformation. The important mental shift is that a pipeline does not mutate a record in place: each stage returns a new value or an explicit error. Binary patterns extend the same idea to protocols and fixed-width records, allowing field boundaries and type expectations to be expressed declaratively rather than through offset-heavy indexing.

Deep Dive Data-oriented Elixir begins with function clauses. Instead of one function containing a long conditional tree, several clauses can identify valid shapes, special cases, and malformed inputs. The ordering of clauses matters: specific patterns should precede general fallbacks. Guards are intentionally restricted to side-effect-free operations so clause selection stays predictable and optimizable. When a value cannot be classified by shape alone, a guard can check a length, numeric range, map size, or type. A final catch-all clause can turn everything else into a structured rejection record rather than raising unexpectedly.

Immutability changes how workflows are designed. A reconciliation tool can parse a row into a raw struct, normalize it into a canonical struct, validate domain invariants, index it by a composite match key, and finally produce a report. Each transformation is independently testable because it receives a value and returns a value. The pipeline operator makes this flow readable, but it does not provide error handling by itself. Functions should still use stable return contracts. with is useful when several dependent operations return tagged tuples; it short-circuits on the first non-matching result and allows one explicit else boundary to classify errors. It should not be used to hide unrelated branches or to collapse errors that callers need to distinguish.

Enum and Stream answer different questions. Enum consumes an enumerable immediately and returns a materialized result. Stream composes lazy transformations and performs work only when consumed. Lazy streams are valuable for large files because parsing and validation can occur one record at a time. They do not make memory bounded automatically: a later Enum.group_by, global sort, or accumulation can still retain the full dataset. Good designs state exactly where laziness ends and why. Resource-backed streams must also close file descriptors when consumption stops early.

Binary pattern matching applies these ideas below the line-oriented level. A binary can be split into typed segments, fixed-size fields, or a prefix plus remainder. The parser’s job is to maintain an invariant: every successful step consumes a known amount of input and returns the unconsumed tail; every failure reports the record position and expected shape. Network streams add a complication because one receive operation is not one logical message. A frame may arrive in pieces or several frames may arrive together, so the parser must retain incomplete bytes and repeatedly extract complete frames. The same principle appears in fixed-width settlement files, certificate decoding, and BEAM artifact chunks.

Calendars and numerical data add semantic normalization. Date-time comparison is valid only after a time zone and ambiguity policy are explicit. Money should use integer minor units or a decimal representation rather than floating-point equality. File identity should be established by bytes or a documented digest workflow rather than names. The language makes these policies visible through structs, custom guards, and result types, but correctness still depends on choosing the right domain representation.

Expected data problems should usually be returned; programmer errors and violated internal invariants may raise. This boundary is crucial. If every malformed CSV row crashes the process, a useful batch tool becomes fragile. If every impossible internal state is converted into {:error, :unknown}, bugs become invisible. Projects should define an error taxonomy with enough context for the operator to act: source, record number, category, and safe details. Exceptions can then remain reserved for genuinely exceptional or invariant-breaking conditions.

How this fit on projects Projects 14-18 make transformation correctness visible through reports, schedules, file plans, concurrent audit rows, and parser invariants. Project 25 applies binary parsing to TCP frames, and Project 29 carries immutable transformation into tensors.

Definitions & key terms

Pattern: A structural description used to match and extract data.
Guard: A restricted predicate that refines a matching clause.
Tagged tuple: A return value whose first element identifies success or failure.
Enumerable: Data that can be reduced; lists, maps, streams, and many custom structures qualify.
Lazy stream: A suspended transformation pipeline evaluated on demand.
Binary segment: A typed or sized portion of a binary pattern.

Mental model diagram

raw input -> parse by shape -> normalize -> validate -> index/aggregate -> report
    |              |              |          |              |
    v              v              v          v              v
 source line   pattern/guard   canonical   tagged error   deterministic
                              representation             observable output

binary buffer -> [complete frame][complete frame][partial...]
                  consume          consume       retain

How it works (step-by-step, with invariants and failure modes)

Preserve source context before parsing.
Match the most specific valid shape and use guards for bounded refinements.
Normalize into a domain struct with stable types.
Return tagged errors for expected invalid input.
Use lazy stages until a business operation genuinely requires materialization.
Invariant: a successful transform never silently loses input meaning.
Failure modes: broad catch-all clauses, unbounded collection, float-based money, partial binary frames, and ambiguous dates.

Minimal concrete example

RECONCILIATION PSEUDOCODE
for each source row lazily:
  parse shape
  when valid -> normalize amount/date/reference
  when invalid -> rejection(source, line, reason)
index valid rows by {reference, date_window, amount_minor_units}
emit matched, unmatched, duplicate, and rejected groups

Common misconceptions

“The pipe operator handles errors.” It only passes values; functions still need compatible contracts.
“A Stream is always memory efficient.” A downstream global grouping can materialize everything.
“One TCP receive equals one packet.” TCP exposes an ordered byte stream, not application message boundaries.
“Pattern matching replaces validation.” Patterns validate shape; domain rules still require explicit checks.

Check-your-understanding questions

When is with clearer than nested case, and when does it hide too much?
Why can a lazy file pipeline still exhaust memory?
What state must a framed binary parser keep between socket messages?

Check-your-understanding answers

Use it for a linear chain of tagged results; use explicit branching when failures need different recovery paths.
A terminal operation such as sorting or grouping may retain the entire input.
The unconsumed partial frame plus enough metadata to know the next required length.

Real-world applications

Financial and settlement-file reconciliation
Calendar conflict analysis and scheduling
File inventory, integrity, and quarantine planning
Binary device protocols and artifact inspection

Where you’ll apply it

Projects 14, 15, 16, 17, 18, 25, and 29

References

Key insights Model the valid shapes and failure shapes explicitly, then make every transformation observable.

Summary Elixir’s functional core makes data pipelines auditable: immutable representations, pattern-driven clauses, tagged errors, lazy enumeration, and binary segments turn messy external data into controlled transformations.

Homework/Exercises to practice the concept

Design a tagged error taxonomy for a three-source reconciliation job.
Trace how two complete frames plus half a third frame move through a TCP parser.

Solutions to the homework/exercises

Include source, position, category, safe original value, and remediation hint; keep parser and domain errors distinct.
Emit the first two frames and retain the incomplete tail until the next socket message arrives.

Concept 9: Protocols, Behaviours, Mix, and Compile-Time Tooling

Fundamentals Elixir has several extension mechanisms, and choosing the right one is an architectural decision. A behaviour defines callbacks that modules implement, which is ideal for adapters and test seams. A protocol dispatches by the data type of its first argument, which is ideal when operations vary by struct or built-in type. Mix provides project metadata, dependency management, compilation, tasks, and packaging; custom tasks turn architectural rules into repeatable developer workflows. Quoted ASTs, macros, module attributes, compiler callbacks, and tracers operate at compile time, where they can generate APIs or produce diagnostics. These mechanisms are powerful because they integrate with the compiler and ecosystem, but they also create coupling. Stable contracts, precise diagnostics, and a strict boundary between generated structure and ordinary runtime functions are essential.

Deep Dive Behaviours and protocols solve different forms of polymorphism. A storage adapter behaviour says, “any module that provides these callbacks can satisfy this capability.” The caller selects the adapter through configuration or dependency injection. A protocol says, “perform this operation differently depending on the type of the value.” Protocol consolidation can optimize dispatch in releases, while behaviour callback checking can reveal missing functions during compilation. Using a protocol merely because it feels object-oriented is a mistake; if dispatch is based on configuration rather than a value’s type, a behaviour is usually clearer.

A public library needs more than callbacks. Typespecs describe contracts for tools and readers, @impl documents intentional callback implementations, doctests protect public examples, and a shared contract suite prevents adapters from drifting. Optional callbacks should be rare and their fallback semantics explicit. Error values belong to the public API, so changing their shapes can be a breaking change even when function arity stays constant. Hex packaging forces another useful discipline: only intended files should enter the artifact, documentation must build, licenses must be discoverable, and semantic versions must describe compatibility honestly.

Mix tasks turn this same contract thinking inward. A dependency policy auditor can inspect the dependency graph, normalize package metadata, apply license rules, emit a deterministic SBOM, and exit non-zero in CI. A task should be idempotent, composable, and explicit about environment. It should not assume that development dependencies or umbrella children look like a single production release. Project metadata can vary by environment, so a report must state the environment and lockfile it analyzed.

Macros operate one level above runtime values: their inputs and outputs are quoted expressions. They are appropriate when a library must introduce a declarative construct, generate repetitive clauses with source metadata, or participate in compilation. A pricing-policy DSL is a legitimate example because invalid declarations should fail at compile time and generated functions should retain file-and-line attribution. A macro is not a substitute for a normal function. Most business logic should remain in ordinary functions so it can be called, tested, profiled, and understood without expanding ASTs.

Hygiene prevents generated variables and aliases from accidentally capturing caller context. quote, unquote, and carefully selected module attributes allow a DSL to accumulate declarations and emit code in @before_compile. The design invariant is that compile-time data is finite, deterministic, and safe to embed. Generating atoms or evaluating untrusted source can create security and memory risks. Diagnostics should point to the declaration that caused the problem, not to an opaque generated function.

Compiler-aware analysis adds a different constraint. AST traversal can find syntactic calls, while compiler tracers can observe resolved aliases, imports, references, and environments. Tracer callbacks must remain fast because they run during compilation; heavier analysis belongs in a later task consuming captured facts. Safe refactoring previews should preserve comments and token metadata and should never rewrite a file unless a rule can prove its transformation preconditions. BEAM artifact inspection sits after compilation: :beam_lib reads chunks without loading modules, which is safer and more reproducible than executing unknown code to discover metadata.

The deepest lesson is that extension points become ecosystem contracts. A custom Ecto adapter must translate between Ecto’s query structures and a remote document store without pretending unsupported joins or transactions exist. Returning an explicit unsupported error is more correct than emulating a guarantee badly. Whether designing a behaviour, macro, task, tracer, or adapter, the same rule applies: make capabilities and limitations mechanically visible.

How this fit on projects Projects 19, 20, 23, 28, 32, and 33 progress from custom Mix tooling to publishable libraries, compile-time DSLs, compiler analysis, artifact inspection, and framework extension.

Definitions & key terms

Behaviour: A module-level callback contract.
Protocol: Type-based dispatch on a value.
Quoted AST: Elixir code represented as nested terms.
Macro hygiene: Isolation of generated variables, imports, and aliases from the caller.
Compiler tracer: A callback that observes compiler events and environments.
Adapter: A component translating one contract into another system’s capabilities.

Mental model diagram

source declarations -> parser/AST -> compile-time validation -> generated runtime API
       |                  |                 |                         |
       v                  v                 v                         v
   file + line       hygienic terms    diagnostics             ordinary functions

caller -> behaviour contract -> selected adapter -> external capability
value  -> protocol dispatch  -> type implementation

How it works (step-by-step, with invariants and failure modes)

Choose behaviour dispatch, protocol dispatch, or ordinary functions based on the actual axis of variation.
Specify callbacks, types, errors, and capability limits before implementation.
Keep macros thin and delegate runtime behavior to normal functions.
Preserve source metadata for actionable diagnostics.
Keep compiler tracer callbacks bounded; analyze captured facts afterward.
Invariant: every extension advertises what it guarantees and rejects what it cannot support.
Failure modes: macro overuse, protocol/behaviour confusion, unstable adapter errors, non-deterministic tasks, and unsafe code loading.

Minimal concrete example

PRICING DSL PSEUDOCODE
at compile time:
  collect rule declarations with file and line
  reject overlapping exclusive rules
  generate small dispatch clauses
at runtime:
  evaluate normalized facts
  return {price, explanation_trace}

Common misconceptions

“Protocols are interfaces.” They dispatch on data type; behaviours define module callbacks.
“Macros make code faster.” Their primary effect is compile-time transformation, not automatic runtime speed.
“If it compiles, a custom adapter is compatible.” Semantic guarantees matter more than callback presence.
“A linter can safely rewrite any matched AST.” Syntactic similarity is not proof of semantic equivalence.

Check-your-understanding questions

Why is a configurable storage backend usually a behaviour rather than a protocol?
What logic should remain outside a macro?
Why should a compiler tracer avoid performing full-project analysis inline?

Check-your-understanding answers

Dispatch depends on selected capability/module, not the type of the stored value.
Runtime calculation, I/O, and testable business rules should live in ordinary functions.
It runs on the compilation path and can make builds slow or deadlock-prone.

Real-world applications

Hex libraries with pluggable adapters
Policy DSLs and schema declarations
CI dependency and license enforcement
Linters, migration assistants, and build reproducibility tools

Where you’ll apply it

Projects 19, 20, 23, 28, 32, and 33

References

Key insights Extension mechanisms are contracts first and conveniences second.

Summary Behaviours, protocols, Mix, macros, tracers, and adapters let Elixir developers extend both applications and tooling, but sustainable extensions keep runtime logic ordinary, diagnostics precise, and unsupported semantics explicit.

Homework/Exercises to practice the concept

Decide whether three output renderers should use a behaviour, protocol, or functions and justify the dispatch axis.
Write a design checklist for a macro that emits source-located diagnostics.

Solutions to the homework/exercises

Use a protocol when rendering varies by value type, a behaviour when a configured module provides the renderer, and functions for a closed internal set.
Preserve caller metadata, validate declarations before generation, keep generated code small, avoid dynamic atoms, and test expansion plus runtime behavior separately.

Concept 10: Ecto Integrity, Transactions, and Durable Workflows

Fundamentals Ecto separates application data modeling from database guarantees. Schemas describe persisted shapes, changesets cast and validate external input, constraints translate database-enforced invariants into domain errors, queries compose data access, and repositories execute work. Transactions define atomic boundaries, while Ecto.Multi can name a sequence of dependent operations and report the exact failed step. Durable job systems such as Oban persist work in the same database and apply retry, uniqueness, scheduling, and cancellation semantics. These tools do not create exactly-once side effects. Correct systems combine database constraints, idempotency keys, explicit state transitions, and observable recovery. Dynamic repositories add another dimension: each tenant’s connection pool becomes a supervised resource whose identity, capacity, migration state, and failure boundary must be managed deliberately.

Deep Dive Validation and constraints answer different questions. A changeset validation can reject a malformed quantity without touching the database. It cannot safely prove that a reservation is unique or that concurrent stock updates will never go negative. Those guarantees belong to database constraints, locks, or transaction isolation. Ecto converts named constraint violations into changeset errors, allowing the API to return a domain result without ignoring the database as the final authority. The correct design uses cheap validations for immediate feedback and constraints for race-safe integrity.

An inventory ledger illustrates the difference. Stock movements are append-only facts; a reservation changes the available quantity and records a unique operation identifier. A naive read-then-write sequence can oversell under concurrency because two transactions observe the same balance. The solution must choose and document a concurrency strategy: an atomic conditional update, row lock, serializable transaction, or optimistic lock with retry. The invariant is stronger than “tests usually pass”: for every committed state, the sum of movements and reservations must reconcile and available stock must not be negative.

Ecto.Multi helps name transactional work and inspect it before execution. It is valuable when multiple changesets and dependent functions must commit or roll back together. It does not make external HTTP calls transactional. If a transaction sends an email and then rolls back, the email cannot be unsent. External effects should be represented as durable intent—often an outbox row or Oban job—written in the same transaction, then performed afterward by an idempotent worker.

Oban provides durable execution, but its guarantees must be read precisely. Job uniqueness is evaluated at insertion time and is distinct from concurrency control. Retries use recorded attempts and backoff, yet an external service may receive a request before the worker crashes and still receive it again on retry. Idempotency therefore belongs at the business boundary. A document-processing workflow can store a stable document/version key, record stage completion, and make every stage safe to repeat. Operator-visible errors, cancellation rules, and repair actions are part of the product, not secondary logging.

Long-running workflows and explicit state machines need persistence boundaries. An escrow process can use :gen_statem for clear state/event semantics, but an in-memory state machine alone loses progress on node failure. The project must decide which transitions are durable, how state is reconstructed, which commands are idempotent, and how stale timeouts are recognized. Exactly-once delivery is generally an illusion across independent systems; the practical goal is at-least-once processing plus idempotent state transitions and traceable causation.

Multi-tenant repositories expose resource ownership. Starting one Ecto repository per tenant provides isolation but can exhaust connections and memory. A control plane should use a DynamicSupervisor and Registry, cap active pools, serialize startup and migrations, propagate tenant identity explicitly, and evict only idle tenants with no checked-out work. Process-local dynamic repo selection is convenient, but losing tenant context when spawning a Task can route a query to the wrong database. Tenant identity must be part of request and job data, not an ambient assumption.

Extending Ecto with a custom adapter reveals the semantic core. The adapter must translate schemas, dumped values, query structures, and result rows into a remote document API. If the backend lacks joins or multi-document transactions, the adapter must reject them. Pretending to support an operation with weaker behavior corrupts caller expectations. A compliance suite should test supported filters, ordering, pagination, type loading/dumping, error mapping, and initialization under supervision.

How this fit on projects Projects 22, 24, 30, 31, and 33 move from application-level database invariants through durable workflows and tenant resource control to an ecosystem-level adapter.

Definitions & key terms

Changeset: Data structure for casting, validation, change tracking, and constraint declarations.
Constraint: Database-enforced invariant translated into an application error.
Transaction: Atomic database boundary in which all included changes commit or roll back.
Idempotency key: Stable identifier that lets a repeated command return the original logical result.
Durable job: Persisted work that survives process and node restarts.
Dynamic repository: Repository process selected or started at runtime.

Mental model diagram

request -> changeset -> transaction -> durable facts + outbox/job -> worker -> external side effect
             |              |                    |                  |
             v              v                    v                  v
         validation     constraints          retryable intent   idempotency check

tenant request -> registry -> supervised tenant Repo -> tenant database
                         capacity + migration + eviction policy

How it works (step-by-step, with invariants and failure modes)

Cast and validate external data without treating validation as a concurrency guarantee.
Encode race-sensitive invariants in the database.
Keep transactions focused on database work and durable intent.
Make workers repeatable and record stable idempotency keys.
Persist workflow transitions that must survive crashes.
Bound dynamic repository resources and propagate tenant identity explicitly.
Invariant: every committed state satisfies database and domain constraints.
Failure modes: unsafe uniqueness checks, external effects inside transactions, duplicate job effects, leaked tenant context, and adapters that overpromise.

Minimal concrete example

RESERVATION PSEUDOCODE
transaction operation_id:
  reject if operation_id already committed
  atomically reserve only when available >= requested
  append ledger movement
  enqueue durable confirmation intent
commit -> worker sends confirmation using operation_id as idempotency key

Common misconceptions

“Changeset validation prevents races.” Only a database guarantee can arbitrate concurrent writes safely.
“A unique Oban job runs alone.” Uniqueness controls insertion, not execution concurrency.
“A transaction can roll back an HTTP call.” It can only roll back work owned by the transactional resource.
“Tenant routing in process state is inherited everywhere.” Spawned processes need explicit context propagation.

Check-your-understanding questions

Why should a reservation API use both changeset validation and a database constraint?
What makes a retried document stage safe?
When must a custom adapter return unsupported rather than emulate a feature?

Check-your-understanding answers

Validation improves feedback; the constraint preserves the invariant under races.
A stable operation key, persisted stage state, and side effects that can detect repeats.
Whenever the backend cannot provide the semantic guarantee callers associate with the Ecto operation.

Real-world applications

Reservation, payment, and inventory ledgers
Durable document and notification workflows
Database-per-tenant SaaS systems
Framework adapters for non-relational services

Where you’ll apply it

Projects 22, 24, 30, 31, and 33

References

Key insights Durability comes from explicit invariants and repeatable effects, not from optimistic naming such as “exactly once.”

Summary Ecto and Oban support strong production workflows when validations, constraints, transactions, jobs, state transitions, and tenant resources each carry a precise responsibility.

Homework/Exercises to practice the concept

Identify the transaction boundary and idempotency key for a reservation followed by an email.
List the resources that must be capped in a database-per-tenant design.

Solutions to the homework/exercises

Commit reservation plus durable email intent atomically; use the reservation operation ID when sending.
Active Repo processes, database connections, concurrent migrations, startup attempts, idle timers, and queued requests.

Concept 11: Boundary Engineering for HTTP, TCP, Native Code, Embedded Devices, and Nx

Fundamentals BEAM systems are strongest when external boundaries are explicit. Plug turns an HTTP request into a controlled connection pipeline; :gen_tcp exposes socket ownership and byte-stream semantics; Erlang libraries such as :ssl, :crypto, and :beam_lib extend Elixir through ordinary module interoperation. Nerves packages an OTP release with a minimal Linux system for embedded targets. Rustler exposes native functions inside the VM, where scheduler blocking or a native crash can affect the entire node. Nx introduces tensors and defn computation graphs that may execute on specialized backends. Each boundary has different trust, latency, failure, memory, and upgrade properties. Correct designs normalize inputs, bound work, preserve backpressure or capacity signals, and choose isolation based on blast radius rather than convenience.

Deep Dive HTTP authenticity begins before JSON decoding. Many webhook providers sign the exact raw request bytes plus a timestamp. If middleware parses and re-encodes the body first, verification may fail or—worse—verify different bytes from those the application processes. A gateway should read the body with an explicit size limit, retain the raw binary, parse provider headers, reject stale timestamps, compute the expected MAC, and compare equal-length values in constant time. Provider-specific logic belongs behind behaviours so the Plug pipeline owns common limits, audit shape, and secret-handling policy. Authentication proves origin under the provider’s scheme; it does not make event delivery unique, so replay identifiers and idempotency remain necessary.

TCP is a lower-level boundary. It delivers an ordered stream, not records. A device gateway must handle partial headers, partial payloads, several frames in one receive, invalid lengths, checksum failures, and clients that stop reading responses. Socket ownership determines which process receives active-mode messages. Transferring the controlling process must be coordinated so messages do not remain in the acceptor’s mailbox. active: :once is often useful because the connection process explicitly rearms the socket after processing one message, creating a local flow-control point. Iodata avoids flattening large response buffers unnecessarily.

Native code changes the failure model. A Port runs an external OS process and preserves VM isolation at the cost of serialization and process-management overhead. A NIF executes inside the VM and can offer low call overhead, but a crash can terminate the entire runtime and a long normal-scheduler call can stall scheduling. Rustler improves type conversion and memory safety, yet it cannot make an algorithm’s scheduling choice correct. CPU-heavy work must be short, chunked, or placed on an appropriate dirty scheduler. The project should benchmark not only operation latency but scheduler responsiveness while concurrent Elixir processes run.

Nerves moves the boundary to hardware. Firmware contains the application, runtime, kernel, and filesystem layout. The root filesystem is normally read-only; persistent state belongs in the data partition. A cold-chain gateway must treat sensors and networks as unreliable: validate readings, timestamp them monotonically where possible, buffer offline data within storage limits, and make alarms locally even when the cloud is unreachable. Firmware updates need a known-good path, validation window, and rollback behavior. A device that boots new firmware but loses networking should not mark it healthy merely because the application process started.

Nx changes data representation and execution. Tensors have shapes, axes, dtypes, and backend placement. defn builds numerical computations that can be compiled, so not every ordinary Elixir operation is available inside it. Forecasting correctness depends on window construction, train/test separation, scaling, loss functions, and numerical stability before acceleration matters. Benchmarking eager and compiled paths should include warm-up and compilation time separately. A model that is fast but leaks future observations into training is not correct.

All these boundaries share capacity questions. HTTP bodies need limits, concurrent TLS audits need bounded tasks, TCP connections need admission and idle policies, NIFs need scheduler budgets, devices need storage budgets, and tensor batches need shape and memory budgets. Supervision can restart a failed component, but it cannot recover bytes that were never persisted or protect the VM from unsafe native memory. Boundary design therefore starts with a failure matrix: what can fail, what remains trustworthy, what can be retried, and which resource must remain bounded.

How this fit on projects Projects 17, 21, 25, 26, 27, and 29 explore progressively lower or more specialized boundaries: TLS services, signed HTTP, raw TCP, hardware firmware, in-VM native code, and compiled tensor workloads.

Definitions & key terms

Raw body: Exact request bytes received before decoding or normalization.
Constant-time comparison: Comparison whose timing does not reveal matching prefixes.
Socket owner: Process that receives active socket messages and controls ownership transfer.
NIF: Native function loaded into and invoked inside the BEAM runtime.
Dirty scheduler: Scheduler pool intended for longer blocking native work.
Firmware validation: Marking an embedded update healthy only after defined checks pass.
Tensor: Multidimensional typed array with a defined shape and backend.

Mental model diagram

untrusted world
  HTTP bytes -> size/replay/MAC gate -> normalized event
  TCP bytes  -> framed parser -> connection process -> domain command
  sensor     -> validated reading -> offline buffer -> uplink
  tensor data -> shape/type checks -> defn graph -> backend

BEAM process -> Rustler NIF -> dirty scheduler/native memory
             blast radius: entire node if native code crashes

How it works (step-by-step, with invariants and failure modes)

Preserve and bound raw input before decoding.
Authenticate or frame data using the exact documented bytes.
Normalize external errors into stable domain categories.
Assign one explicit owner to sockets, devices, native resources, or serving processes.
Bound concurrency, buffers, execution time, and persistent storage.
Select isolation: supervised BEAM process, external Port, dirty NIF, or separate device.
Invariant: failure at the boundary cannot silently corrupt accepted domain data.
Failure modes: body rewriting, replay, partial frames, ownership races, scheduler-blocking NIFs, invalid firmware, and tensor shape drift.

Minimal concrete example

WEBHOOK GATE PSEUDOCODE
read raw body up to configured limit
reject timestamp outside replay window
expected = MAC(secret, timestamp + delimiter + raw_body)
constant-time compare expected and supplied signature
decode only after authenticity succeeds
record accepted event id for idempotent downstream handling

Common misconceptions

“Valid JSON is safe input.” Authenticity, size, replay, and semantic validation are separate concerns.
“TCP preserves my writes.” It preserves byte order, not write boundaries.
“Rust makes a NIF unable to crash BEAM.” Memory safety reduces risks but panics, native libraries, and blocking remain operational hazards.
“A successful firmware boot is a successful update.” Health validation must include the device’s critical capabilities.
“Nx makes ordinary Elixir code run on a GPU.” Only supported tensor computations inside the numerical boundary are compiled.

Check-your-understanding questions

Why must a webhook signature usually be checked before body decoding?
What does active: :once let a TCP connection process control?
What measurements distinguish a fast NIF from a safe NIF?

Check-your-understanding answers

The provider signs exact bytes, and decoding may change their representation.
When it is ready to receive the next socket message, limiting mailbox pressure.
Operation latency plus scheduler responsiveness, reductions/latency of unrelated processes, crash behavior, and memory/resource cleanup.

Real-world applications

Secure webhook ingestion and compliance gateways
IoT/device protocol concentrators
Embedded monitoring appliances with resilient updates
Native media or cryptographic acceleration
Numerical forecasting and anomaly detection services

Where you’ll apply it

Projects 17, 21, 25, 26, 27, and 29

References

Key insights Choose a boundary by its failure and capacity semantics, not merely by the shortest API.

Summary HTTP, TCP, native code, embedded firmware, and numerical backends all extend Elixir’s reach. They remain reliable only when raw inputs, ownership, resource budgets, isolation, and recovery evidence are designed explicitly.

Homework/Exercises to practice the concept

Compare the blast radius and observability of a Port, normal NIF, and dirty-scheduler NIF.
Define firmware health checks for a temperature gateway after an update.

Solutions to the homework/exercises

A Port can crash independently and is supervised through exit status; a normal NIF shares VM fate and must finish quickly; a dirty NIF still shares VM fate but avoids blocking normal schedulers.
Confirm application boot, sensor read, persistent storage, network authentication, queued-data upload, alarm output, and watchdog stability before validation.

Glossary

BEAM: The Erlang virtual machine and runtime system.
OTP: Open Telecom Platform; a set of libraries and design principles.
GenServer: OTP behavior for server processes.
Supervisor: OTP behavior that monitors and restarts children.
Reduction: Scheduling unit for fairness.
ETS: In-memory term storage.
DETS: Disk-based term storage for single-node persistent tables.
Mnesia: OTP’s distributed database with transactional RAM and disk table copies.
Process registration: Binding a logical local name or key to a live process.
Process discovery: Resolving a service or capability to reachable process identities at runtime.
Distributed Erlang: Nodes connected for transparent message passing.
LiveView: Server-rendered real-time UI with diffs.
GenStage: Backpressure-aware pipeline.
Release handling: OTP upgrade framework.
Self-contained release: A deployable release that includes a compatible ERTS, so the target does not need Erlang or Elixir installed.
Tagged tuple: Return value whose first element names success or a failure category.
Behaviour: Module callback contract used for adapters and test seams.
Protocol: Polymorphic dispatch based on the type of a value.
Changeset: Ecto value for casting, validation, change tracking, and database constraints.
Idempotency: Property that makes repeating an operation produce one logical effect.
NIF: Native function loaded into the BEAM runtime.
Port: BEAM-owned connection to an isolated external OS process.
Tensor: Typed multidimensional array used by Nx numerical computations.

Why BEAM Matters

BEAM’s process model and supervision trees are explicitly designed for fault tolerance at scale.
Discord reports scaling Elixir to nearly five million concurrent users and millions of events per second (July 6, 2017), demonstrating BEAM’s relevance for real-time systems.
LiveView and GenStage provide a BEAM-native approach to real-time UI and backpressure-driven pipelines.
Current Elixir includes first-party concurrency and partitioning tools such as Task, DynamicSupervisor, Registry, and PartitionSupervisor, while its quoted AST and Mix APIs support compiler-aware tooling (PartitionSupervisor documentation, Macro documentation).
The ecosystem now spans relational integrity and durable jobs through Ecto and Oban, embedded firmware through Nerves, safe Rust NIF ergonomics through Rustler, and compiled numerical workloads through Nx (Ecto.Multi, Oban, Nerves, Rustler, Nx).
The Erlang ecosystem reported in December 2024 that internet scans observed 85,000+ publicly exposed EPMD instances, reinforcing why secure distribution boundaries are non-negotiable in production (source).

ASCII diagram: old vs new concurrency model

OLD (Threads + Locks)             NEW (Processes + Messages)
Shared state                      Isolated state
Locks and contention              Message passing
One crash can corrupt state       Crash is isolated
Complex recovery                  Supervisor restarts

Concept Summary Table

Concept Cluster	What You Need to Internalize
Process Model	Lightweight isolated processes, message copying, mailbox semantics.
OTP + Supervision	GenServer/Supervisor lifecycle and restart strategies.
Scheduling + GC	Reduction-based fairness and per-process GC.
Distribution	Node naming, transparent messaging, process discovery, and failure semantics.
State + ETS/DETS/Mnesia	In-memory and disk tables, ownership, transactions, replication, backup, and recovery.
Real-Time + Backpressure	LiveView diffing and GenStage demand control.
Release Handling	Mix release assembly, ERTS packaging, runtime configuration, appup/relup, code_change, and upgrade boundaries.
Data Transformation + Binaries	Pattern clauses, guards, tagged errors, lazy streams, calendars, money, and framed binary parsing.
Protocols + Compile-Time Tooling	Behaviours, protocols, Mix tasks, Hex packages, quoted ASTs, macros, compiler tracers, and adapters.
Ecto + Durable Workflows	Changesets, constraints, transactions, idempotent jobs, explicit workflow state, and dynamic repositories.
Boundary Engineering	Raw HTTP authenticity, TCP ownership/framing, NIF scheduler safety, embedded firmware validation, and Nx backend boundaries.

Project-to-Concept Map

Project	Concepts Applied
Project 1	Process Model, OTP + Supervision
Project 2	OTP + Supervision, State + ETS/DETS/Mnesia
Project 3	Distribution, State + ETS/DETS/Mnesia
Project 4	Real-Time + Backpressure
Project 5	Real-Time + Backpressure, Scheduling + GC
Project 6	OTP + Supervision, Scheduling + GC
Project 7	Distribution, Process Model
Project 8	State + ETS/DETS/Mnesia
Project 9	Release Handling
Project 10	Scheduling + GC, Real-Time + Backpressure
Project 11	Distribution, OTP + Supervision
Project 12	Distribution, Process Model, OTP + Supervision
Project 13	Distribution, Real-Time + Backpressure
Project 14	Data Transformation + Binaries
Project 15	Data Transformation + Binaries
Project 16	Data Transformation + Binaries
Project 17	Data Transformation + Binaries, Boundary Engineering, OTP + Supervision
Project 18	Data Transformation + Binaries
Project 19	Protocols + Compile-Time Tooling
Project 20	Protocols + Compile-Time Tooling
Project 21	Boundary Engineering, Protocols + Compile-Time Tooling
Project 22	Ecto + Durable Workflows
Project 23	Protocols + Compile-Time Tooling
Project 24	Ecto + Durable Workflows, OTP + Supervision
Project 25	Boundary Engineering, Process Model, OTP + Supervision
Project 26	Boundary Engineering, OTP + Supervision, Release Handling
Project 27	Boundary Engineering, Scheduling + GC
Project 28	Protocols + Compile-Time Tooling
Project 29	Boundary Engineering, Data Transformation + Binaries, Scheduling + GC
Project 30	Ecto + Durable Workflows, OTP + Supervision
Project 31	Ecto + Durable Workflows, OTP + Supervision
Project 32	Protocols + Compile-Time Tooling, Release Handling
Project 33	Protocols + Compile-Time Tooling, Ecto + Durable Workflows
Project 34	State + ETS/DETS/Mnesia, OTP + Supervision
Project 35	State + ETS/DETS/Mnesia, Distribution, OTP + Supervision
Project 36	Release Handling, Boundary Engineering
Project 37	Distribution, Process Model, OTP + Supervision

Requested Topic Coverage Audit

This matrix answers the ten requested learning outcomes directly. “Expanded” means the topic existed but the new project adds the missing end-to-end operational proof.

#	Requested Topic	Coverage	Projects That Teach It
1	Data structures	Covered	P14 uses lists, maps, `MapSet`, structs, tagged tuples, and composite indexes; P16, P18, and P29 add trees, binaries, parser state, and tensors.
2	Data abstraction: modules, hierarchical data, polymorphism with protocols	Covered	P23 is the primary modules/behaviours/protocols project; P20 and P33 add hierarchical AST/query data.
3	Registering processes	Covered and expanded	P30 and P31 use local `Registry`; P37 compares local names, `Registry`, `:via`, `:pg`, and bounded `:global` use.
4	Processes connected over a network	Covered	P3, P7, and P11 exercise remote pids, cross-node messages, links, monitors, and disconnects.
5	Processes in network clusters	Covered	P11, P12, and P13 cover cluster formation, partitions, healing, and federation.
6	Supervised recovery from persisted state	Covered and expanded	P30 replays durable events and deadlines; P24 resumes persisted jobs; P34 and P35 add table recovery drills.
7	Supervisor trees	Covered	P1 and P6 teach restart strategies and fault injection; P25 and P31 apply nested and dynamic trees.
8	ETS and persistent alternatives	Expanded to full coverage	P8 teaches ephemeral ETS; P34 adds DETS-backed restart recovery; P35 adds Mnesia transactions, disk copies, replication, backup, and restore.
9	Release tarball that runs without Elixir/Erlang installed	New full coverage	P36 builds with ERTS included, creates the tarball, and proves it on a clean compatible target. P9 remains the hot-upgrade drill and P32 the artifact auditor.
10	Process discovery	Expanded to full coverage	P11 discovers nodes; P30/P31 discover local processes; P37 makes distributed service membership, routing, monitoring, and stale-pid recovery the main outcome.

Deep Dive Reading by Concept

Concept	Book and Chapter	Why This Matters
Process Model	“Programming Erlang” by Joe Armstrong - Processes & Message Passing	Actor model and concurrency fundamentals
OTP + Supervision	“Designing for Scalability with Erlang/OTP” - Supervision	Practical fault-tolerance design
Scheduling + GC	“The BEAM Book” - Scheduler & Memory	Understand runtime behavior
Distribution	“Programming Erlang” - Distributed Erlang	Cluster semantics and failure handling
State + ETS/DETS/Mnesia	“Erlang and OTP in Action” - ETS/Mnesia	Data structures, local durability, transactions, and replicated storage
Real-Time + Backpressure	“Elixir in Action” - Processes and GenStage	Flow control in pipelines
Release Handling	“Programming Erlang” - Code upgrades	Safe runtime upgrades
Data Transformation + Binaries	“Elixir in Action” - Language Basics, Data Abstractions, and Concurrent Tasks	Build reliable pipelines from matching, streams, tagged results, and binaries
Protocols + Compile-Time Tooling	“Metaprogramming Elixir” - Quoting, Macros, and Extending Elixir	Use compile-time extension points without hiding runtime behavior
Ecto + Durable Workflows	“Programming Ecto” - Changesets, Constraints, and Multi	Put integrity in the database and make background effects repeatable
Boundary Engineering	“Designing for Scalability with Erlang/OTP” - External Resources and Runtime Boundaries	Choose isolation, ownership, and capacity policies for non-BEAM systems

Quick Start: Your First 48 Hours

Day 1:

Read Concept 8, then start Project 14 and produce a deterministic reconciliation report from two small fixtures.
Read Concept 1 and compare pure transformation failures with process crashes.

Day 2:

Validate Project 14 against the Definition of Done, including malformed and duplicate rows.
Read Concept 2 and start Project 1 only after you can explain when data should return an error versus when a supervised process should crash.

Recommended Learning Paths

Path 1: The Elixir Language Builder

Project 14 -> Project 18 -> Project 23 -> Project 20 -> Project 28

Path 2: The Production Application Engineer

Project 17 -> Project 21 -> Project 22 -> Project 24 -> Project 31

Path 3: The OTP and Network Engineer

Project 1 -> Project 5 -> Project 25 -> Project 30 -> Project 3 -> Project 37 -> Project 12

Path 4: The Runtime and Edge Specialist

Project 16 -> Project 26 -> Project 27 -> Project 29 -> Project 32

Path 5: The Principal Ecosystem Builder

Project 19 -> Project 23 -> Project 28 -> Project 32 -> Project 33

Path 6: The Stateful BEAM Operator

Project 8 -> Project 34 -> Project 35 -> Project 36 -> Project 37

Success Metrics

You can explain supervision strategies and justify your choices.
You can design a process tree for a real service.
You can identify and fix mailbox backlogs.
You can run a controlled upgrade with a state migration step.
You can bootstrap and operate BEAM clusters spread across multiple network segments.
You can model malformed input with specific tagged errors and prove a binary parser handles arbitrary chunking.
You can explain when to use a function, protocol, behaviour, macro, Mix task, or adapter.
You can defend database invariants under concurrency and make background effects idempotent.
You can choose among BEAM processes, TCP owners, Nerves devices, and NIFs by blast radius and capacity.
You can inspect compiler and BEAM artifacts without executing untrusted modules.
You can classify state as ephemeral, reconstructable, or durable and prove its recovery after owner, VM, and node failures.
You can package and run a release on a compatible host that has no Erlang or Elixir toolchain installed.
You can choose among local registration, Registry, :pg, and :global and recover safely from stale discovery results.

Optional Appendix: BEAM Tooling Cheat Sheet

observer: visualize processes, memory, and message queues
:sys: inspect GenServer state and system messages
:dbg: trace function calls and message flow
telemetry: instrument and export metrics
mix xref: inspect compile-time dependencies and call relationships
mix hex.build --unpack: inspect a package before publication
:beam_lib: inspect BEAM chunks without loading modules
:erlang.trace / trace: capture bounded process, message, scheduling, and call evidence
StreamData: generate and shrink adversarial inputs

Project Overview Table

#	Project Name	Main Language	Difficulty	Seniority	Time Estimate	Core Concepts	Coolness
1	Supervised Chat System	Elixir	Level 2	Mid-level	10-15 hrs	OTP + Supervision	Level 3
2	Rate Limiter + Circuit Breaker	Elixir	Level 2	Mid-level	12-18 hrs	ETS + Supervision	Level 3
3	Distributed KV Store	Erlang/Elixir	Level 3	Senior	20-30 hrs	Distribution + ETS	Level 4
4	LiveView Real-Time Dashboard	Elixir	Level 2	Mid-level	12-20 hrs	LiveView	Level 3
5	GenStage Backpressure Pipeline	Elixir	Level 3	Senior	18-25 hrs	Backpressure	Level 4
6	Fault Injection Harness	Elixir	Level 2	Mid-level	10-15 hrs	Supervision	Level 3
7	Presence and Notification Service	Erlang/Elixir	Level 3	Senior	20-30 hrs	Distribution	Level 4
8	ETS Cache Service	Erlang/Elixir	Level 2	Mid-level	10-15 hrs	ETS	Level 3
9	Hot Code Upgrade Drill	Erlang	Level 3	Senior	15-25 hrs	Release Handling	Level 4
10	Telemetry Pipeline + Live Metrics	Elixir	Level 3	Senior	15-25 hrs	Scheduling + LiveView	Level 4
11	Multi-Network Cluster Formation Lab	Elixir/Erlang	Level 3	Senior	18-28 hrs	Distribution + Security	Level 5
12	WAN Netsplit Recovery Drill	Elixir/Erlang	Level 4	Staff	24-36 hrs	Partitions + Reconciliation	Level 5
13	Federated Edge Event Bus	Elixir	Level 4	Staff	24-40 hrs	Distributed Backpressure	Level 5
14	Bank Statement Reconciliation CLI	Elixir	Level 1	Junior	6-10 hrs	Matching + Streams	Level 2
15	iCalendar Conflict Detector	Elixir	Level 1	Junior	8-12 hrs	Calendar + Intervals	Level 3
16	Duplicate File Quarantine Planner	Elixir	Level 1	Junior	8-12 hrs	Files + Lazy Streams	Level 2
17	Concurrent Link and TLS Expiry Auditor	Elixir/Erlang	Level 2	Mid-level	10-16 hrs	Task + SSL	Level 3
18	Fixed-Width Settlement Parser	Elixir	Level 2	Mid-level	12-18 hrs	Binaries + StreamData	Level 3
19	Mix SBOM and License Auditor	Elixir	Level 2	Mid-level	12-18 hrs	Mix + Dependency Graphs	Level 3
20	Compile-Time Pricing Policy DSL	Elixir	Level 3	Senior	16-24 hrs	Macros + AST	Level 4
21	Webhook Authenticity Gateway	Elixir	Level 2	Mid-level	12-18 hrs	Plug + Cryptography	Level 3
22	Inventory Ledger and Reservation API	Elixir	Level 3	Senior	18-28 hrs	Ecto + Transactions	Level 3
23	Pluggable Object Storage Library	Elixir	Level 2	Mid-level	14-22 hrs	Behaviours + Protocols	Level 3
24	Oban Document Workflow	Elixir	Level 3	Senior	18-26 hrs	Durable Jobs + Idempotency	Level 4
25	TCP Device Telemetry Gateway	Elixir/Erlang	Level 3	Senior	20-30 hrs	gen_tcp + Binaries	Level 4
26	Nerves Cold-Chain Sensor Gateway	Elixir	Level 3	Senior	20-32 hrs	Nerves + Firmware	Level 5
27	Rustler Perceptual-Hash NIF	Elixir/Rust	Level 4	Staff	24-36 hrs	NIFs + Dirty Schedulers	Level 5
28	Compiler-Aware Migration Linter	Elixir	Level 3	Senior	20-30 hrs	Compiler Tracers + AST	Level 4
29	Nx Demand Forecasting Engine	Elixir	Level 3	Senior	20-32 hrs	Nx + defn	Level 4
30	:gen_statem Escrow Workflow	Elixir/Erlang	Level 4	Staff	24-36 hrs	State Machines + Timeouts	Level 4
31	Multi-Tenant Dynamic Repo Control Plane	Elixir	Level 4	Staff	24-40 hrs	DynamicSupervisor + Ecto	Level 4
32	BEAM Artifact Reproducibility Auditor	Elixir/Erlang	Level 4	Staff	22-34 hrs	beam_lib + Build Metadata	Level 4
33	Custom Ecto HTTP Document Adapter	Elixir	Level 5	Principal	35-60 hrs	Ecto Internals + Query Translation	Level 5
34	Crash-Resilient ETS/DETS Session Store	Elixir	Level 3	Senior	24-36 hrs	ETS + DETS + Recovery	Level 4
35	Distributed Mnesia Work-Order Ledger	Elixir/Erlang	Level 4	Staff	35-55 hrs	Mnesia + Transactions + Recovery	Level 5
36	Self-Contained Mix Release Deployment	Elixir/Erlang	Level 3	Senior	22-34 hrs	Mix Release + ERTS + Operations	Level 4
37	Distributed Service Directory and Fleet Router	Elixir	Level 4	Staff	28-42 hrs	Registry + pg + Process Discovery	Level 5

Project List

The following projects form three interleaved tracks: Projects 1-13 teach runtime fault tolerance and distribution, Projects 14-33 broaden the journey from junior-level transformation to principal-level ecosystem extension, and Projects 34-37 close the operational loop around durable BEAM tables, self-contained releases, and distributed process discovery.

Project 1: Supervised Real-Time Chat System

View Detailed Guide

File: P01-supervised-chat-system.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Gleam
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Knowledge Area: Concurrency, Fault Tolerance
Software or Tool: OTP/GenServer
Main Book: “Programming Erlang”

What you will build: A multi-room chat service where each room is a supervised process and failures are isolated.

Why it teaches BEAM: It forces you to model state as processes and to recover from crashes using supervisors.

Core challenges you will face:

Process isolation -> Process Model
Room supervision -> OTP + Supervision
Message routing -> Process Model

Real World Outcome

You can run a CLI client for two rooms, kill a room process, and watch it restart without losing other rooms.

$ chatctl create room general
room general: pid=<0.215.0>

$ chatctl send general "hello"
[general] user42: hello

$ chatctl crash room general
room general crashed; supervisor restarted it

$ chatctl send general "we are back"
[general] user42: we are back

The Core Question You Are Answering

“How do I build a service where one failing chat room does not bring down the whole system?”

Concepts You Must Understand First

BEAM process isolation
- Why are messages copied and state isolated?
- Book Reference: “Programming Erlang” - Processes chapter
Supervision strategy
- Which restart strategy fits independent rooms?
- Book Reference: “Designing for Scalability with Erlang/OTP” - Supervision
GenServer lifecycle
- How are calls and casts handled?
- Book Reference: “Elixir in Action” - OTP section

Questions to Guide Your Design

State model
- Will each room be one process or multiple?
- How will you store room history safely?
Failure model
- What happens when a room crashes mid-message?
- How do you inform clients that the room restarted?

Thinking Exercise

Room Isolation Sketch

Draw a supervision tree for 3 rooms. Decide where to place a router process that handles room discovery.

Questions to answer:

What should happen if the router dies?
Which processes should be linked vs monitored?

The Interview Questions They Will Ask

“Why use one process per room instead of shared state?”
“How does supervision improve reliability?”
“How do GenServer calls differ from casts?”
“What happens to messages during a crash?”
“How would you scale this across nodes?”

Hints in Layers

Hint 1: Starting Point Start with one room process and a simple send/receive API.

Hint 2: Next Level Add a supervisor that restarts the room on crash.

Hint 3: Technical Details Pseudocode:

Room process:
- state: list of users
- on {join, user}: add to list
- on {message, user, text}: broadcast

Hint 4: Tools/Debugging Use observer to confirm the room process restarts after a crash.

Books That Will Help

Topic	Book	Chapter
Processes	“Programming Erlang”	Processes chapter
Supervision	“Designing for Scalability with Erlang/OTP”	Supervision chapter

Common Pitfalls and Debugging

Problem 1: “Room crashes kill the whole app”

Why: You started rooms without a supervisor.
Fix: Put each room under a one_for_one supervisor.
Quick test: Crash a room and verify the supervisor restarts only that room.

Definition of Done

Each room is its own process
Rooms restart on crash without affecting others
Messages still flow after a restart
Process tree visible in observer

Project 2: Rate Limiter and Circuit Breaker Service

View Detailed Guide

File: P02-rate-limiter-circuit-breaker.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Gleam
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Knowledge Area: State, Fault Tolerance
Software or Tool: ETS + GenServer
Main Book: “Erlang and OTP in Action”

What you will build: A rate limiter backed by ETS and a circuit breaker that trips on repeated failures.

Why it teaches BEAM: You must manage state safely and choose supervision policies.

Core challenges you will face:

ETS design -> State + ETS
Supervisor strategy -> OTP + Supervision
Timeout handling -> Scheduling + GC

Real World Outcome

$ ratelimit check user42
allowed (remaining=9)

$ ratelimit check user42
blocked (retry_in=12s)

$ breaker status serviceA
state: open (cooldown=30s)

The Core Question You Are Answering

“How do I enforce limits and protect dependencies without locks or shared memory races?”

Concepts You Must Understand First

ETS tables
- What happens when the owner crashes?
- Book Reference: “Erlang and OTP in Action” - ETS chapter
Supervision strategy
- How do you restart a limiter safely?
- Book Reference: “Designing for Scalability with Erlang/OTP”
GenServer state
- How do you serialize updates?
- Book Reference: “Elixir in Action”

Questions to Guide Your Design

Rate limiting algorithm
- Fixed window, sliding window, or token bucket?
- How will you store timestamps efficiently?
Circuit breaker
- What error threshold trips the breaker?
- How long before a half-open state?

Thinking Exercise

State Table Design

Design the ETS key for {user_id, window} and decide how to prune expired windows.

Questions to answer:

How do you keep reads constant time?
What is your cleanup strategy?

The Interview Questions They Will Ask

“Why choose ETS for a rate limiter?”
“How do you avoid race conditions without locks?”
“What supervision strategy is best for a limiter?”
“How do you prevent restart storms?”

Hints in Layers

Hint 1: Starting Point Model the limiter as one GenServer that owns ETS.

Hint 2: Next Level Store counters per time window and expire old keys.

Hint 3: Technical Details Pseudocode:

on check(user):
  window = floor(now / window_size)
  count = lookup({user, window})
  if count < limit -> increment and allow
  else -> deny

Hint 4: Tools/Debugging Use observer to confirm ETS table size over time.

Books That Will Help

Topic	Book	Chapter
ETS design	“Erlang and OTP in Action”	ETS chapter
Fault tolerance	“Designing for Scalability with Erlang/OTP”	Supervision chapter

Common Pitfalls and Debugging

Problem 1: “Limiter forgets counts after crash”

Why: ETS table is destroyed when owner dies.
Fix: Rebuild from logs or accept reset as design choice.
Quick test: Crash owner and verify expected behavior.

Definition of Done

Limits enforced per user and window
ETS table lifecycle documented
Circuit breaker transitions documented
Supervisor restarts without global failure

Project 3: Distributed Key-Value Store

View Detailed Guide

File: P03-distributed-kv-store.md
Main Programming Language: Erlang or Elixir
Alternative Programming Languages: Gleam
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Knowledge Area: Distribution, Data Stores
Software or Tool: Distributed Erlang + ETS
Main Book: “Programming Erlang”

What you will build: A sharded KV store that runs on multiple nodes and replicates keys.

Why it teaches BEAM: It forces you to design with node failure, message passing, and ETS-backed state.

Core challenges you will face:

Node discovery -> Distribution
Shard mapping -> Process Model
Replication -> Distribution + ETS

Real World Outcome

$ kv put user:42 "active" --node nodeA
ok

$ kv get user:42 --node nodeB
"active" (replica)

$ kv cluster status
nodes: [nodeA,nodeB,nodeC]
shards: 64
replication: 2

The Core Question You Are Answering

“How do I maintain state across nodes while handling node failures gracefully?”

Concepts You Must Understand First

Distributed Erlang nodes
- How are nodes named and connected?
- Book Reference: “Programming Erlang” - Distribution chapter
ETS lifecycle
- What happens to tables when a node restarts?
- Book Reference: “Erlang and OTP in Action” - ETS chapter
Supervisor policies
- How do you restart shard processes?
- Book Reference: “Designing for Scalability with Erlang/OTP”

Questions to Guide Your Design

Sharding
- How will you assign keys to shards?
Replication
- How many replicas, and how do you keep them consistent?

Thinking Exercise

Failure Drill

Simulate node loss: what happens to keys on that node and how do clients recover?

Questions to answer:

How do you detect that a node is down?
How do you reassign shards?

The Interview Questions They Will Ask

“How does distributed Erlang handle messaging between nodes?”
“What happens to ETS state on node failure?”
“How do you handle split-brain?”
“Why use supervision for shard processes?”

Hints in Layers

Hint 1: Starting Point Start with two nodes and a single shard process on each.

Hint 2: Next Level Use consistent hashing to map keys to shard owners.

Hint 3: Technical Details Pseudocode:

shard = hash(key) mod shard_count
primary = shard_owner(shard)
replica = next_owner(shard)

Hint 4: Tools/Debugging Use node monitors to detect failures and log shard reassignment.

Books That Will Help

Topic	Book	Chapter
Distribution	“Programming Erlang”	Distribution chapter
ETS	“Erlang and OTP in Action”	ETS chapter

Common Pitfalls and Debugging

Problem 1: “Keys disappear after node crash”

Why: State was only on the crashed node.
Fix: Add replication or persistence.
Quick test: Kill a node and verify replicas still answer.

Definition of Done

Keys retrievable from replica node
Shard ownership rebalances after node loss
Node discovery documented
Cluster health command works

Project 4: LiveView Real-Time Operations Dashboard

View Detailed Guide

File: P04-liveview-ops-dashboard.md
Main Programming Language: Elixir
Alternative Programming Languages: None
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Knowledge Area: Real-Time Web
Software or Tool: Phoenix LiveView
Main Book: “Programming Phoenix”

What you will build: A real-time dashboard that streams metrics to the browser with LiveView diff updates.

Why it teaches BEAM: LiveView uses server-rendered HTML with diff tracking and a persistent connection for real-time updates.

Core challenges you will face:

State streaming -> LiveView
Efficient updates -> LiveView diffing
Process isolation -> Process Model

Real World Outcome

The dashboard shows:

A top bar with system status (green/yellow/red)
A grid of cards for CPU, memory, message queue sizes
A live chart that updates once per second

Behavior:

When metrics spike, only the affected card updates (no full page reload).

The Core Question You Are Answering

“How do I deliver live UI updates without building a heavy client app?”

Concepts You Must Understand First

LiveView lifecycle
- How does the initial render differ from updates?
- Book Reference: “Programming Phoenix” - LiveView chapters
Process model
- What process owns the dashboard state?
- Book Reference: “Programming Erlang” - Processes

Questions to Guide Your Design

Update frequency
- How often do you push updates without overwhelming clients?
Data pipeline
- Where do metrics originate and how are they buffered?

Thinking Exercise

Diff Strategy

Decide which UI components can update independently and which must update together.

Questions to answer:

What data should be computed per client?
What can be shared across clients?

The Interview Questions They Will Ask

“Why use LiveView instead of client-side JS?”
“How does LiveView minimize network traffic?”
“How do you handle slow clients?”
“What happens if a LiveView process crashes?”

Hints in Layers

Hint 1: Starting Point Build a static LiveView page with placeholders.

Hint 2: Next Level Add a timer process that sends metric updates.

Hint 3: Technical Details Pseudocode:

Every 1s:
  read metrics
  update assigns
  LiveView pushes diff

Hint 4: Tools/Debugging Use browser devtools to confirm small diff payloads.

Books That Will Help

Topic	Book	Chapter
LiveView	“Programming Phoenix”	LiveView chapters

Common Pitfalls and Debugging

Problem 1: “Whole page re-renders”

Why: You are reassigning the entire state on each tick.
Fix: Update only the changed assigns.
Quick test: Track diff size in logs.

Definition of Done

Live metrics update without page refresh
Diff payloads remain small
UI degrades gracefully under load
Crash of dashboard process recovers cleanly

Project 5: GenStage Backpressure Pipeline

View Detailed Guide

File: P05-genstage-backpressure-pipeline.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Knowledge Area: Stream Processing
Software or Tool: GenStage
Main Book: “Elixir in Action”

What you will build: A producer-consumer pipeline that processes bursty events with explicit demand control.

Why it teaches BEAM: GenStage demand is a concrete, backpressure mechanism.

Core challenges you will face:

Demand control -> Backpressure
Work distribution -> Scheduler/GC
Failure recovery -> Supervision

Real World Outcome

$ pipeline send 10000
accepted

$ pipeline status
producer_buffer=500
consumer_demand=200
processed=9800

The Core Question You Are Answering

“How do I prevent a burst of events from collapsing my system?”

Concepts You Must Understand First

Backpressure
- Why demand should be explicit?
- Book Reference: “Elixir in Action” - GenStage section
Scheduler fairness
- Why distribute work across processes?
- Book Reference: “The BEAM Book” - Scheduler chapter

Questions to Guide Your Design

Demand size
- How many events should a consumer request at a time?
Failure policy
- What happens if a consumer crashes mid-batch?

Thinking Exercise

Burst Simulation

Design a scenario where input spikes 10x and decide how the pipeline should respond.

Questions to answer:

Where do events buffer?
How do you detect lag?

The Interview Questions They Will Ask

“What is backpressure and why do we need it?”
“How do GenStage producers and consumers coordinate?”
“How do you measure pipeline lag?”
“What happens if a consumer crashes?”

Hints in Layers

Hint 1: Starting Point Start with one producer and one consumer.

Hint 2: Next Level Add multiple consumers and split demand.

Hint 3: Technical Details Pseudocode:

Consumer requests N events
Producer sends N events
Consumer processes and requests more

Hint 4: Tools/Debugging Log demand and buffer size on each stage.

Books That Will Help

Topic	Book	Chapter
Backpressure	“Elixir in Action”	GenStage chapter

Common Pitfalls and Debugging

Problem 1: “Mailbox growth without bound”

Why: You are emitting without honoring demand.
Fix: Enforce demand-based flow control.
Quick test: Trigger a large burst and verify buffers stay bounded.

Definition of Done

Demand-based flow implemented
Burst input does not crash the pipeline
Consumer failure is recovered via supervision
Metrics show bounded queues

Project 6: Fault Injection and Supervision Harness

View Detailed Guide

File: P06-fault-injection-harness.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Knowledge Area: Fault Tolerance
Software or Tool: OTP Supervision
Main Book: “Designing for Scalability with Erlang/OTP”

What you will build: A harness that deliberately crashes workers under different strategies and records recovery behavior.

Why it teaches BEAM: You will see supervision strategies in action, not just read about them.

Core challenges you will face:

Strategy selection -> OTP + Supervision
Crash scenarios -> Process Model
Observability -> Scheduling/GC

Real World Outcome

$ faultlab run --strategy one_for_one
worker A crashed -> restarted
worker B unaffected

$ faultlab run --strategy one_for_all
worker A crashed -> all workers restarted

The Core Question You Are Answering

“What actually happens when a process crashes under different supervision strategies?”

Concepts You Must Understand First

Supervision strategies
Process isolation

Questions to Guide Your Design

Crash injection
- How will you trigger controlled failures?
Measurement
- What metrics show recovery speed and impact?

Thinking Exercise

Failure Tree

Draw a tree with 3 workers and decide which should restart together.

The Interview Questions They Will Ask

“How do one_for_one and one_for_all differ?”
“How would you test your supervision tree?”
“What is a restart storm and how do you prevent it?”

Hints in Layers

Hint 1: Starting Point Create workers that crash on demand.

Hint 2: Next Level Swap the supervisor strategy and log the outcomes.

Hint 3: Technical Details Pseudocode:

if trigger == crash:
  worker exits with error
supervisor applies strategy

Hint 4: Tools/Debugging Use observer to watch process restarts.

Books That Will Help

Topic	Book	Chapter
Supervision	“Designing for Scalability with Erlang/OTP”	Supervision chapter

Common Pitfalls and Debugging

Problem 1: “All workers restart unexpectedly”

Why: You selected one_for_all for independent workers.
Fix: Use one_for_one when workers are independent.
Quick test: Crash one worker and verify only that worker restarts.

Definition of Done

Supports multiple strategies
Logs restart behavior
Demonstrates escalation when intensity exceeded
Clear report of recovery time

Project 7: Presence and Notification Service

View Detailed Guide

File: P07-presence-notifications.md
Main Programming Language: Elixir or Erlang
Alternative Programming Languages: Gleam
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Knowledge Area: Distribution, Real-Time
Software or Tool: Distributed Erlang
Main Book: “Programming Erlang”

What you will build: A multi-node presence service that tracks online users and pushes notifications.

Why it teaches BEAM: It forces you to use node naming, message passing, and failure detection across nodes.

Core challenges you will face:

Node awareness -> Distribution
Presence consistency -> Process Model
Failure handling -> Supervision

Real World Outcome

$ presence login user42 --node nodeA
ok

$ presence status user42 --node nodeB
online (via nodeA)

$ presence notify user42 "ping"
queued

The Core Question You Are Answering

“How do I maintain a consistent view of online users across nodes?”

Concepts You Must Understand First

Distributed nodes
Message passing semantics

Questions to Guide Your Design

Source of truth
- Is presence authoritative on one node or replicated?
Failure handling
- What happens when a node goes down?

Thinking Exercise

Partition Scenario

Simulate a network split. How do you reconcile presence when nodes reconnect?

The Interview Questions They Will Ask

“How does distributed Erlang handle messaging?”
“What are the risks of global registries?”
“How do you detect node failures?”

Hints in Layers

Hint 1: Starting Point Start with local presence per node.

Hint 2: Next Level Add a router that forwards queries across nodes.

Hint 3: Technical Details Pseudocode:

if user not found locally:
  query other nodes

Hint 4: Tools/Debugging Kill a node and verify presence cleans up.

Books That Will Help

Topic	Book	Chapter
Distribution	“Programming Erlang”	Distribution chapter

Common Pitfalls and Debugging

Problem 1: “Presence shows users online after node crash”

Why: You did not remove presence on DOWN events.
Fix: Monitor node and purge presence on disconnect.
Quick test: Stop a node and re-check status.

Definition of Done

Presence queries work across nodes
Notifications routed to correct node
Node failure cleans up state
Basic partition handling documented

Project 8: ETS Cache and Session Service

View Detailed Guide

File: P08-ets-cache-service.md
Main Programming Language: Erlang or Elixir
Alternative Programming Languages: Gleam
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Knowledge Area: Data Storage
Software or Tool: ETS
Main Book: “Erlang and OTP in Action”

What you will build: An ETS-backed cache with TTL eviction and stats reporting.

Why it teaches BEAM: It forces you to manage ETS lifecycle and cleanup safely.

Core challenges you will face:

Table lifecycle -> ETS
TTL expiration -> Scheduling
Concurrency -> Process Model

Real World Outcome

$ cache put key1 value1 --ttl 30s
ok

$ cache get key1
value1

$ cache stats
entries=1200 evictions=45

The Core Question You Are Answering

“How do I build a fast in-memory cache without corrupting shared state?”

Concepts You Must Understand First

ETS basics
Table lifecycle

Questions to Guide Your Design

Eviction
- Time-based vs size-based eviction?
Ownership
- Which process owns the table?

Thinking Exercise

TTL Sweep

Design a cleanup process that removes expired keys without scanning all entries every time.

The Interview Questions They Will Ask

“What happens if the ETS owner dies?”
“How do you avoid full-table scans?”
“Why use ETS instead of a GenServer map?”

Hints in Layers

Hint 1: Starting Point Store expiry timestamps with each key.

Hint 2: Next Level Use a periodic sweep process.

Hint 3: Technical Details Pseudocode:

Every 5s:
  scan a slice of keys
  delete expired

Hint 4: Tools/Debugging Track table size and eviction counts over time.

Books That Will Help

Topic	Book	Chapter
ETS	“Erlang and OTP in Action”	ETS chapter

Common Pitfalls and Debugging

Problem 1: “Eviction freezes system”

Why: Full table scan in one process.
Fix: Incremental sweeps and batching.
Quick test: Load with 1M keys and verify stable latency.

Definition of Done

TTL eviction works
Table survives normal load
Stats report accurate counts
Owner process supervised

Project 9: Hot Code Upgrade Drill

View Detailed Guide

File: P09-hot-code-upgrade-drill.md
Main Programming Language: Erlang
Alternative Programming Languages: Elixir
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Knowledge Area: Release Engineering
Software or Tool: Release handling
Main Book: “Programming Erlang”

What you will build: A controlled upgrade scenario with state migration and rollback.

Why it teaches BEAM: Release handling is unique to OTP and critical for zero-downtime systems.

Core challenges you will face:

appup design -> Release Handling
State migration -> Code Change
Rollback -> Release Handling

Real World Outcome

$ reltool build
release built: v1 -> v2

$ reltool upgrade --apply
upgrade applied
state migrated: ok

$ reltool rollback
rollback complete

The Core Question You Are Answering

“How do I upgrade running systems without corrupting state?”

Concepts You Must Understand First

Release handler
appup/relup instructions

Questions to Guide Your Design

Migration
- What state changes need transformation?
Rollback
- How do you verify upgrade success before committing?

Thinking Exercise

State Evolution

Design a version change that adds a new field to state and requires a default.

The Interview Questions They Will Ask

“What is an appup file?”
“Why do core apps require restart?”
“How do you handle rollback?”

Hints in Layers

Hint 1: Starting Point Create a minimal appup with restart_application.

Hint 2: Next Level Add a code_change step for state migration.

Hint 3: Technical Details Pseudocode:

code_change:
  add default field to state

Hint 4: Tools/Debugging Use release_handler logs to verify upgrade steps.

Books That Will Help

Topic	Book	Chapter
Upgrades	“Programming Erlang”	Code upgrade chapter

Common Pitfalls and Debugging

Problem 1: “Upgrade fails at runtime”

Why: Missing appup instructions.
Fix: Add explicit upgrade steps and test on a staging node.
Quick test: Run upgrade in a controlled environment.

Definition of Done

Upgrade runs without downtime
State migration works
Rollback works
Logs capture each step

Project 10: Telemetry Pipeline + Live Metrics

View Detailed Guide

File: P10-telemetry-live-metrics.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Knowledge Area: Observability
Software or Tool: Telemetry + LiveView
Main Book: “Elixir in Action”

What you will build: A telemetry pipeline that collects metrics and streams them to a LiveView dashboard.

Why it teaches BEAM: It combines scheduling, backpressure, and LiveView updates in one system.

Core challenges you will face:

Event aggregation -> Scheduler + GC
Backpressure -> GenStage
Live UI updates -> LiveView

Real World Outcome

The UI shows:

A rolling chart of request latency
A table of top message queue sizes
A live count of process restarts

CLI validation:

$ metricsctl status
events/sec=1200 queue=low live_clients=8

The Core Question You Are Answering

“How do I observe a BEAM system in real time without overwhelming it?”

Concepts You Must Understand First

Backpressure
LiveView rendering

Questions to Guide Your Design

Sampling
- How often do you emit events?
Aggregation
- Do you batch or stream raw events?

Thinking Exercise

Noise Control

Decide which metrics are critical and which can be sampled.

The Interview Questions They Will Ask

“Why is backpressure necessary for telemetry?”
“How does LiveView reduce bandwidth?”
“What metrics reveal mailbox pressure?”

Hints in Layers

Hint 1: Starting Point Emit simple counters and render them.

Hint 2: Next Level Add a GenStage pipeline for aggregation.

Hint 3: Technical Details Pseudocode:

collect -> aggregate -> publish -> LiveView

Hint 4: Tools/Debugging Check event rates and backlog in your logs.

Books That Will Help

Topic	Book	Chapter
Pipelines	“Elixir in Action”	Processes/GenStage

Common Pitfalls and Debugging

Problem 1: “Dashboard lags under load”

Why: Too many events pushed to LiveView.
Fix: Aggregate before publishing.
Quick test: Increase load and observe latency trend.

Definition of Done

Metrics update in real time
Pipeline handles burst without backlog
Dashboard remains responsive
Metrics are accurate and documented

Project 11: Multi-Network Cluster Formation Lab

View Detailed Guide

File: P11-multi-network-cluster-formation.md
Main Programming Language: Elixir or Erlang
Alternative Programming Languages: Gleam
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Knowledge Area: Distribution, Cluster Operations
Software or Tool: Distributed Erlang + TLS Distribution
Main Book: “Programming Erlang”

What you will build: A cluster bootstrap lab that forms one logical BEAM system from nodes spread across multiple private networks.

Why it teaches BEAM: It forces you to deal with node naming, connectivity, secure distribution, and cross-network observability.

Core challenges you will face:

Node identity and discovery -> Distribution
Secure node-to-node transport -> Distribution security
Cross-network health checks -> OTP + Supervision

Real World Outcome

$ clusterctl topology
site=us-east nodes=[edge-us-1@10.10.1.11,edge-us-2@10.10.1.12]
site=eu-west nodes=[edge-eu-1@10.20.1.11,edge-eu-2@10.20.1.12]
gateways=[gw-us@10.10.9.10,gw-eu@10.20.9.10]
links=8 secure=true

$ clusterctl ping edge-us-1@10.10.1.11 edge-eu-1@10.20.1.11
pong latency_ms=84 route=gw-us->gw-eu

$ clusterctl members --service presence
[presence@edge-us-1,presence@edge-us-2,presence@edge-eu-1,presence@edge-eu-2]

The Core Question You Are Answering

“How do I make BEAM nodes in different networks behave like one system without ignoring security boundaries?”

Concepts You Must Understand First

Distributed node naming
- How do longnames and shortnames affect connectivity?
- Book Reference: “Programming Erlang” - Distributed Erlang chapter
Secure Erlang distribution
- What changes when you use TLS distribution instead of plain TCP?
- Book Reference: “Designing for Scalability with Erlang/OTP” - Operations sections
Cluster supervision
- Which processes should restart locally vs escalate cluster-wide alerts?
- Book Reference: “Elixir in Action” - OTP supervision chapters

Questions to Guide Your Design

Network topology
- Which nodes are edge nodes, and which are gateways?
- How do you prevent accidental full-mesh cross-site links?
Failure model
- What happens when one gateway is unreachable?
- How do you degrade gracefully while keeping local traffic alive?

Thinking Exercise

Topology Contract

Draw two sites and one DR site. Mark exactly which node pairs are allowed to connect and which are explicitly blocked.

Questions to answer:

Which links are mandatory for quorum-critical services?
Which links can be lazy or on-demand?

The Interview Questions They Will Ask

“How do you connect Erlang nodes across different private networks?”
“Why is TLS distribution important in BEAM clusters?”
“How do you debug node reachability without guessing?”
“What are common causes of cross-site node flapping?”

Hints in Layers

Hint 1: Starting Point Start with two nodes in different subnets and verify bidirectional ping.

Hint 2: Next Level Add gateway nodes and route inter-site traffic through them.

Hint 3: Technical Details Pseudocode:

for each site:
  connect local mesh
connect site gateways
monitor remote gateway status

Hint 4: Tools/Debugging Track nodeup/nodedown events and correlate with network ACL changes.

Books That Will Help

Topic	Book	Chapter
Distributed Erlang	“Programming Erlang”	Distributed Erlang chapter
Cluster reliability	“Designing for Scalability with Erlang/OTP”	Reliability and operations chapters

Common Pitfalls and Debugging

Problem 1: “Nodes connect locally but never connect across networks”

Why: Name resolution, cookie mismatch, or blocked distribution ports.
Fix: Verify names, cookies, and firewall rules on both ends.
Quick test: Run a node-to-node ping plus a remote rpc call.

Definition of Done

Two or more network segments form one logical BEAM cluster
Cross-site ping and remote calls work consistently
TLS distribution path is documented and validated
Cluster topology command exposes node/link health

Project 12: WAN Netsplit Recovery Drill

View Detailed Guide

File: P12-wan-netsplit-recovery-drill.md
Main Programming Language: Erlang or Elixir
Alternative Programming Languages: Gleam
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Knowledge Area: Distributed Systems, Failure Recovery
Software or Tool: Distributed Erlang + Supervision
Main Book: “Designing for Scalability with Erlang/OTP”

What you will build: A controlled lab that injects WAN partitions between sites, then reconciles presence/session data after healing.

Why it teaches BEAM: You learn to model partitions as first-class failures and design deterministic recovery workflows.

Core challenges you will face:

Partition detection -> Distribution
Conflict reconciliation -> Process Model
Recovery orchestration -> OTP + Supervision

Real World Outcome

$ netsplitctl isolate us-east eu-west
partition active: us-east <-> eu-west

$ presencectl status user42 --site us-east
online source=us-east version=7

$ presencectl status user42 --site eu-west
offline source=eu-west version=6 conflict=true

$ netsplitctl heal us-east eu-west
healed; reconciliation_job=running

$ presencectl status user42 --all-sites
online source=us-east reconciled=true version=8

The Core Question You Are Answering

“When networks split and later heal, how do I recover consistent state without losing service availability?”

Concepts You Must Understand First

Node monitors and failure signals
- What events tell you a remote node is unreachable?
- Book Reference: “Programming Erlang” - Distribution + monitoring sections
Conflict-resolution policy
- Do you use last-write-wins, version vectors, or authoritative-site rules?
- Book Reference: “Designing for Scalability with Erlang/OTP” - Distributed reliability sections
Supervisor restart intensity
- How do you avoid restart storms during reconnect flapping?
- Book Reference: “Elixir in Action” - Supervisor strategy sections

Questions to Guide Your Design

Consistency semantics
- Which states can be eventually consistent?
- Which states require strict ordering?
Recovery workflow
- Which process starts reconciliation?
- How do you guarantee idempotent replays?

Thinking Exercise

Partition Timeline

Build a timeline for t0 partition start, t1 conflicting writes, t2 reconnect, t3 reconciliation complete.

Questions to answer:

Which events are irreversible?
Which events can be replayed safely?

The Interview Questions They Will Ask

“What is a netsplit in Erlang clusters?”
“How do you reconcile conflicting updates after partition healing?”
“How do supervisors behave during repeated connect/disconnect cycles?”
“How do you test partition recovery without production risk?”

Hints in Layers

Hint 1: Starting Point Start with deterministic partition scripts and fixed test users.

Hint 2: Next Level Add version metadata to every mutable record.

Hint 3: Technical Details Pseudocode:

on reconnect:
  fetch divergent keys
  apply merge policy
  emit reconciliation audit events

Hint 4: Tools/Debugging Persist reconciliation decisions to an audit log for replay and verification.

Books That Will Help

Topic	Book	Chapter
Distributed failure models	“Designing for Scalability with Erlang/OTP”	Fault-tolerance chapters
Process communication	“Programming Erlang”	Messaging and distributed chapters

Common Pitfalls and Debugging

Problem 1: “State keeps oscillating after reconnect”

Why: Merge logic is non-idempotent or order-dependent.
Fix: Use monotonic versioning and idempotent merge rules.
Quick test: Replay the same reconciliation batch twice and confirm identical final state.

Definition of Done

Partition and heal events are reproducible from scripts
Reconciliation policy is explicit and documented
Final state converges after heal in repeated test runs
Recovery metrics/logs expose merge decisions

Project 13: Federated Edge Event Bus

View Detailed Guide

File: P13-federated-edge-event-bus.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Knowledge Area: Distribution, Stream Processing
Software or Tool: GenStage + Distributed Erlang
Main Book: “Elixir in Action”

What you will build: A federated event bus where edge clusters publish local events and gateway processes replicate only selected streams to remote sites.

Why it teaches BEAM: It combines node distribution, flow control, and failure isolation across multiple network domains.

Core challenges you will face:

Cross-site routing -> Distribution
Demand-aware flow control -> Real-Time + Backpressure
Gateway isolation -> OTP + Supervision

Real World Outcome

$ busctl publish sensor.us-east temp=71
acked route=edge-us->gw-us->core

$ busctl subscribe alerts.eu-west --demand 50
subscription=ok consumer=alerts-eu-1

$ busctl stats
local_queue=12 wan_queue=3 dropped=0 backpressure=healthy
replication_filters=[alerts.*,ops.*]

$ busctl failover gw-us
gateway gw-us down; traffic rerouted via gw-us-backup

The Core Question You Are Answering

“How do I replicate events across geographically separated BEAM clusters without overwhelming slower links?”

Concepts You Must Understand First

Backpressure contracts
- How does consumer demand shape producer throughput?
- Book Reference: “Elixir in Action” - GenStage sections
Distributed routing
- How do gateway nodes isolate WAN failure from local producers?
- Book Reference: “Programming Erlang” - Distributed messaging sections
Supervision boundaries
- Which failures should restart only a local gateway pipeline?
- Book Reference: “Designing for Scalability with Erlang/OTP” - Supervision design sections

Questions to Guide Your Design

Routing policy
- Which topics remain local and which replicate globally?
- How do you handle topic ownership changes?
Capacity policy
- What is the maximum WAN queue depth per route?
- When do you drop, defer, or compact events?

Thinking Exercise

Demand Mismatch Scenario

Your US site can produce 1000 events/sec but EU can consume only 200 events/sec for one topic.

Questions to answer:

Where should throttling happen?
Which metrics reveal hidden queue growth early?

The Interview Questions They Will Ask

“How does GenStage help across network boundaries?”
“How do you prevent one slow region from stalling everyone?”
“What do you monitor to catch WAN backpressure failures?”
“How do you design failover for gateway processes?”

Hints in Layers

Hint 1: Starting Point Implement one topic replicated between two sites.

Hint 2: Next Level Add per-topic demand windows and bounded queues.

Hint 3: Technical Details Pseudocode:

producer -> local buffer -> gateway stage -> remote gateway -> consumer
if remote demand=0:
  hold or compact events by policy

Hint 4: Tools/Debugging Plot per-route queue depth and demand over time to detect chronic imbalance.

Books That Will Help

Topic	Book	Chapter
Backpressure	“Elixir in Action”	GenStage and process chapters
Distribution	“Programming Erlang”	Distributed systems chapter

Common Pitfalls and Debugging

Problem 1: “Remote site latency spikes and queues explode”

Why: Producers ignore remote demand and treat WAN as infinite.
Fix: Enforce bounded queue policies and topic-level demand gating.
Quick test: Reduce remote demand to near-zero and verify queue growth remains bounded.

Definition of Done

Cross-site publish/subscribe works for multiple topics
Demand limits are enforced on WAN routes
Gateway failover preserves service continuity
Queue, drop, and latency metrics are visible and explainable

Project 14: Bank Statement Reconciliation CLI

File: P14-bank-statement-reconciliation-cli.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Gleam
Coolness Level: Level 2 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 1 (See REFERENCE.md)
Suggested Seniority: Junior
Knowledge Area: Data Transformation, Financial Reconciliation
Software or Tool: Elixir, OptionParser, File/Stream
Main Book: “Elixir in Action”

What you will build: A CLI that normalizes two bank or processor exports, matches transactions, and produces matched, unmatched, duplicate, and rejected-row reports.

Why it teaches Elixir/BEAM: It makes pattern matching, immutable transformations, tagged errors, structs, Enum, and lazy Stream processing solve an immediately useful problem before concurrency is introduced.

Core challenges you will face:

Messy row shapes -> Data Transformation + Binaries
Money/date normalization -> Data Transformation + Binaries
Deterministic matching -> Data Transformation + Binaries

Real World Outcome

$ mix reconcile bank.csv processor.csv --date-window 2 --out reports/
loaded bank=1,248 processor=1,251 rejected=3
matched=1,219 exact=1,173 tolerant=46
unmatched_bank=26 unmatched_processor=29 duplicates=4
invariant bank_total=R$184,923.11 processor_total=R$184,923.11 delta=R$0.00
reports written: matched.csv unmatched.csv duplicates.csv rejected.csv summary.json

The learner can open each report and trace every result back to source file and row number. Malformed rows remain visible as rejection records; no input disappears silently.

The Core Question You Are Answering

“How do immutable transformations and pattern-matched results turn unreliable external data into an auditable business decision?”

Concepts You Must Understand First

Patterns, guards, and tagged tuples
- Which failures are expected data and which indicate a programmer bug?
- Book Reference: “Elixir in Action” - Language Basics and Data Abstractions
Money and date normalization
- Why should matching use integer minor units and explicit date windows?
- Book Reference: “Designing Data-Intensive Applications” - Encoding and Data Evolution

Questions to Guide Your Design

Which fields form a stable exact-match key, and what evidence permits a tolerant match?
How will duplicate candidates be reported without choosing one arbitrarily?
Where does lazy processing end, and what operation requires materializing data?

Thinking Exercise

Trace two bank rows and three processor rows where two candidates share the same amount. Define the evidence needed to call one exact, tolerant, ambiguous, or unmatched.

The Interview Questions They Will Ask

“How is pattern matching different from assignment?”
“When would you choose Stream over Enum?”
“How do you model recoverable parsing errors in Elixir?”
“How would you prove a reconciliation did not drop money?”

Hints in Layers

Hint 1: Starting Point Parse each source into a source-specific raw struct before sharing normalization logic.

Hint 2: Next Level Index canonical transactions by reference and amount; never use floating-point equality for money.

Hint 3: Technical Details Pseudocode: parse -> normalize -> validate -> index -> classify -> verify totals -> render.

Hint 4: Tools/Debugging Keep tiny golden fixtures where every source row has a known final classification.

Books That Will Help

Topic	Book	Chapter
Elixir transformations	“Elixir in Action”	Language Basics; Data Abstractions
Audit invariants	“Designing Data-Intensive Applications”	Encoding and Data Evolution

Common Pitfalls and Debugging

Problem 1: “Totals match but individual rows are wrong”

Why: Amount-only matching permits accidental pairs.
Fix: Require an evidence hierarchy and report ambiguity instead of guessing.
Quick test: Add two same-amount transactions and verify neither is auto-matched without sufficient evidence.

Definition of Done

Four classification reports preserve source and row identity
Money uses an exact representation and totals reconcile
Ambiguous candidates are never silently selected
Malformed rows produce actionable tagged errors
Golden fixtures make output deterministic

Project 15: iCalendar Availability and Conflict Detector

File: P15-icalendar-conflict-detector.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Gleam
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 1 (See REFERENCE.md)
Suggested Seniority: Junior
Knowledge Area: Calendars, Interval Algorithms
Software or Tool: Elixir Calendar, DateTime, RFC 5545 subset
Main Book: “Elixir in Action”

What you will build: A CLI that reads a documented subset of .ics calendars, normalizes events into one time zone, detects overlaps and travel-time violations, and proposes free meeting windows.

Why it teaches Elixir/BEAM: It teaches structs, binary/string parsing, comprehensions, sorting, interval logic, tagged errors, and the standard Calendar types through a problem where representation mistakes are visible.

Core challenges you will face:

Calendar parsing -> Data Transformation + Binaries
Time-zone normalization -> Data Transformation + Binaries
Interval merging -> Data Transformation + Binaries

Real World Outcome

$ mix calendar.audit work.ics personal.ics --zone America/Sao_Paulo --duration 45
events=37 accepted=35 unsupported=2
conflicts=2 travel_violations=1
2026-08-04 available: 09:00-10:15, 14:30-16:00
2026-08-05 available: 08:30-09:30, 13:15-15:45
details written: calendar-audit.json

The Core Question You Are Answering

“What must be explicit before two date-time values can be compared safely?”

Concepts You Must Understand First

Calendar types and zones
- What is lost when a zoned event becomes a NaiveDateTime?
- Book Reference: “Elixir in Action” - Data Types and Modules
Intervals
- When do sorted intervals overlap, touch, or remain separate?
- Book Reference: “Algorithms” by Sedgewick and Wayne - Sorting

Questions to Guide Your Design

Which RFC 5545 fields are supported, rejected, or preserved as warnings?
What is your policy for ambiguous or nonexistent local times?
Does travel buffer merge into an event or remain a separate constraint?

Thinking Exercise

Draw three events spanning a daylight-saving transition and predict their order in UTC and the selected display zone.

The Interview Questions They Will Ask

“Why is a NaiveDateTime not a timestamp?”
“How do you merge overlapping intervals efficiently?”
“How would you represent unsupported recurrence rules?”
“Where does pattern matching simplify the parser?”

Hints in Layers

Hint 1: Starting Point Support single, non-recurring events with explicit start and end values.

Hint 2: Next Level Sort normalized intervals once, then scan while carrying the current merged interval.

Hint 3: Technical Details Keep original text and normalized time together so diagnostics remain traceable.

Hint 4: Tools/Debugging Test UTC, fixed-offset, ambiguous, nonexistent, all-day, and cross-midnight fixtures.

Books That Will Help

Topic	Book	Chapter
Elixir date modeling	“Elixir in Action”	Data Types and Modules
Interval processing	“Algorithms”	Elementary Sorts and Applications

Common Pitfalls and Debugging

Problem 1: “Events shift by one hour”

Why: Local times were compared without a time-zone database and ambiguity policy.
Fix: Normalize to instants, then format in the selected display zone.
Quick test: Include an event on a zone transition and compare expected UTC instants.

Definition of Done

Supported RFC subset is documented and enforced
Unsupported constructs are reported rather than ignored
Conflicts and free windows are correct across zones
Interval algorithm has boundary-case tests
Output preserves source calendar and event identity

Project 16: Duplicate File Quarantine Planner

File: P16-duplicate-file-quarantine-planner.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 2 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 1 (See REFERENCE.md)
Suggested Seniority: Junior
Knowledge Area: Filesystems, Safety, Integrity
Software or Tool: File, Path, Stream, :crypto
Main Book: “The Pragmatic Programmer”

What you will build: A safe duplicate-file inventory that computes content identity, writes a dry-run quarantine plan, applies only verified moves, and generates a rollback manifest.

Why it teaches Elixir/BEAM: It combines recursion, lazy enumeration, pattern matching over filesystem failures, hashing through Erlang interop, immutable planning, and carefully separated side effects.

Core challenges you will face:

Recursive inventory -> Data Transformation + Binaries
Content identity -> Boundary Engineering
Plan-before-mutate safety -> Data Transformation + Binaries

Real World Outcome

$ mix dedupe.plan ~/Downloads --quarantine ~/Quarantine --out plan.json
visited=18,442 regular=17,901 unreadable=7 symlinks_skipped=534
duplicate_groups=286 reclaimable=8.4 GiB
plan_id=sha256:85d4... dry_run=true

$ mix dedupe.apply plan.json
preflight verified=612 changed_since_scan=2 missing=1
moved=609 skipped=3 rollback=rollback-85d4.json

The Core Question You Are Answering

“How do I separate discovery, decision, and mutation so a useful filesystem tool is reversible?”

Concepts You Must Understand First

Lazy resource traversal
- Where can a recursive traversal still retain too much state?
- Book Reference: “Elixir in Action” - Enumerables and Streams
Identity versus names
- Why are size, path, and modification time insufficient proof of equality?
- Book Reference: “The Pragmatic Programmer” - Pragmatic Paranoia

Questions to Guide Your Design

How will you treat symlinks, hard links, unreadable files, and files that disappear?
Which file becomes the keeper, and can the policy be explained?
What must be revalidated immediately before a move?

Thinking Exercise

Trace a duplicate group containing two hard links, one symlink, and one file modified after hashing. Decide which entries are movable.

The Interview Questions They Will Ask

“Why group by size before hashing?”
“How would you bound memory during a huge scan?”
“Why is a dry-run plan safer than moving during discovery?”
“What races remain between verification and rename?”

Hints in Layers

Hint 1: Starting Point Inventory metadata first and hash only size groups with more than one candidate.

Hint 2: Next Level Represent every planned move as data with expected digest, size, destination, and rollback path.

Hint 3: Technical Details Pseudocode: scan -> group by size -> hash -> verify equality -> plan -> preflight -> apply.

Hint 4: Tools/Debugging Use a temporary fixture tree and compare complete directory manifests before apply, after apply, and after rollback.

Books That Will Help

Topic	Book	Chapter
Safe automation	“The Pragmatic Programmer”	Pragmatic Paranoia
Hash-based grouping	“The Joys of Hashing”	Hash Tables and Collisions

Common Pitfalls and Debugging

Problem 1: “Rollback overwrites a new file”

Why: The rollback path was treated as guaranteed free.
Fix: Preflight rollback destinations and stop on conflicts.
Quick test: Create a conflicting path after apply and verify rollback refuses destructive overwrite.

Definition of Done

Discovery never mutates the filesystem
Hashing and hard-link policy are documented
Apply revalidates every source before moving
Rollback manifest restores a verified fixture tree
Permission, disappearance, and symlink cases are explicit

Project 17: Concurrent Link and TLS Expiry Auditor

File: P17-link-tls-expiry-auditor.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Suggested Seniority: Mid-level
Knowledge Area: Bounded Concurrency, HTTP, TLS
Software or Tool: Task.Supervisor, Finch, :ssl, :public_key
Main Book: “Elixir in Action”

What you will build: A finite URL auditor that reports status, redirect chain, latency, certificate identity and expiry, with bounded concurrency and reproducible output ordering.

Why it teaches Elixir/BEAM: It shows when supervised Tasks are simpler than a long-lived pipeline and forces explicit timeout, exit, cancellation, and Erlang-library interop handling.

Core challenges you will face:

Finite fan-out/fan-in -> OTP + Supervision
TLS inspection -> Boundary Engineering
Stable failure taxonomy -> Data Transformation + Binaries

Real World Outcome

$ mix site.audit urls.txt --max-concurrency 20 --warn-expiry 30
checked=250 ok=231 redirects=8 failed=11 elapsed=4.82s
WARN api.example.test tls_expires_in=12d issuer="Let's Encrypt"
FAIL old.example.test category=tls hostname_mismatch
FAIL slow.example.test category=timeout phase=response_headers
report written: audit-2026-07-15.json exit_status=2

The Core Question You Are Answering

“How do I run many independent operations concurrently without losing timeout, exit, or ordering semantics?”

Concepts You Must Understand First

Tasks and ownership
- Who receives exits, and how are timed-out operations terminated?
- Book Reference: “Elixir in Action” - Working with Tasks
Certificate identity
- Why are chain validity, hostname identity, and expiry separate checks?
- Book Reference: “Computer Networks” - Application Security

Questions to Guide Your Design

Should results preserve input order or completion order, and why?
Which timeout phases must be distinguished?
How do connection-pool limits interact with Task concurrency?

Thinking Exercise

Predict the output for five URLs where the fastest task belongs to the last input and two tasks exit rather than return errors.

The Interview Questions They Will Ask

“What is the difference between a Task return, exit, and timeout?”
“How does Task.async_stream bound concurrency?”
“Why can HTTP pool size become the real concurrency limit?”
“How do Elixir and Erlang modules interoperate?”

Hints in Layers

Hint 1: Starting Point Audit one URL and normalize every outcome into one result struct.

Hint 2: Next Level Introduce bounded Tasks only after single-target timeouts and TLS parsing are deterministic.

Hint 3: Technical Details Keep input index in each task so final reports can be sorted reproducibly.

Hint 4: Tools/Debugging Run against local endpoints that delay connect, headers, and body independently.

Books That Will Help

Topic	Book	Chapter
Task concurrency	“Elixir in Action”	Working with Concurrent Systems
TLS model	“Computer Networks”	Network Security

Common Pitfalls and Debugging

Problem 1: “Increasing concurrency makes the audit slower”

Why: HTTP pool, DNS, file descriptors, or remote rate limits are saturated.
Fix: Measure queueing at each boundary and tune one limit at a time.
Quick test: Compare throughput and p95 latency across concurrency 5, 20, and 100.

Definition of Done

Concurrency and connection pools are bounded
Task returns, exits, and timeouts become distinct result categories
Certificate expiry and hostname failures are reported separately
Output is reproducible despite out-of-order completion
Local fault fixtures cover every timeout phase

Project 18: Fixed-Width Settlement Parser and Property-Test Lab

File: P18-fixed-width-settlement-parser.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Suggested Seniority: Mid-level
Knowledge Area: Binary Parsing, Verification
Software or Tool: Bitstrings, ExUnit, StreamData
Main Book: “Elixir in Action”

What you will build: A parser for a realistic header/detail/trailer settlement format plus property tests proving it never crashes on arbitrary input and never accepts inconsistent trailers.

Why it teaches Elixir/BEAM: Binary patterns, guards, recursive parsing, structs, typespecs, ExUnit, generators, and shrinking are exercised against a concrete financial protocol.

Core challenges you will face:

Fixed-width binary layout -> Data Transformation + Binaries
Cross-record invariants -> Data Transformation + Binaries
Generated adversarial evidence -> Data Transformation + Binaries

Real World Outcome

$ mix settlement.validate fixtures/batch-20260715.dat
header merchant=47291 currency=BRL declared_records=120
details accepted=120 rejected=0 amount=R$83,421.77
trailer count=120 amount=R$83,421.77 status=valid

$ mix test --only property
property parser_never_crashes: 1,000 cases, 0 failures
property valid_round_trip: 500 cases, 0 failures
property altered_trailer_rejected: 500 cases, 0 failures

The Core Question You Are Answering

“How can patterns and generated counterexamples make a parser’s accepted language precise?”

Concepts You Must Understand First

Binary segment patterns
- How do size, type, and remainder segments define consumption?
- Book Reference: “Elixir in Action” - Data Types and Pattern Matching
Property-based testing
- What invariant matters across thousands of generated records?
- Book Reference: “Test Driven Development: By Example” - Test Patterns

Questions to Guide Your Design

Does a parser return the unconsumed tail, a line number, or both?
Which field errors are local, and which invalidate the whole batch?
How will generators create valid data before mutating one invariant?

Thinking Exercise

Design the smallest settlement file that can expose a count mismatch and the smallest one that can expose an amount mismatch. Predict how shrinking should reduce each failure.

The Interview Questions They Will Ask

“How does binary pattern matching differ from string slicing?”
“What makes a useful property rather than an example?”
“What is shrinking and why is it valuable?”
“How do you avoid creating atoms from untrusted fields?”

Hints in Layers

Hint 1: Starting Point Write a format table with offsets, lengths, encodings, and valid ranges before any parser clauses.

Hint 2: Next Level Keep local parsing pure; perform trailer invariants only after accumulating accepted details.

Hint 3: Technical Details Generate valid records from typed fields, serialize them, then mutate exactly one dimension for rejection properties.

Hint 4: Tools/Debugging Print failing binaries as bounded hex plus a field map, not as unstructured text.

Books That Will Help

Topic	Book	Chapter
Pattern-driven parsing	“Elixir in Action”	Pattern Matching and Binaries
Test design	“Test Driven Development: By Example”	Test Patterns

Common Pitfalls and Debugging

Problem 1: “The property passes but malformed trailers are accepted”

Why: The generator rarely creates structurally valid files with one semantic defect.
Fix: Generate valid batches first, then targeted mutations.
Quick test: Force a one-cent trailer difference and require a trailer-specific error.

Definition of Done

Format layout and accepted encodings are documented
Errors include record position and field category
Parser never raises on arbitrary binary input
Round-trip and altered-invariant properties shrink usefully
Declared and computed count/amount invariants are enforced

Project 19: Mix Dependency SBOM and License Auditor

File: P19-mix-sbom-license-auditor.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Suggested Seniority: Mid-level
Knowledge Area: Build Tooling, Supply Chain
Software or Tool: Mix.Task, Mix.Project, Hex, CycloneDX
Main Book: “The Pragmatic Programmer”

What you will build: A reusable Mix task that walks direct and transitive dependencies, emits a deterministic CycloneDX-style SBOM, and fails CI on configurable source, checksum, version, or license violations.

Why it teaches Elixir/BEAM: The project exposes Mix task lifecycle, project configuration, dependency graphs, umbrella traversal, Version, behaviours, maps/sets, deterministic serialization, and shell exit contracts.

Core challenges you will face:

Mix integration -> Protocols + Compile-Time Tooling
Dependency graph traversal -> Protocols + Compile-Time Tooling
Reproducible CI policy -> Protocols + Compile-Time Tooling

Real World Outcome

$ mix deps.audit --format cyclonedx --policy .dependency-policy.exs
components=74 direct=18 transitive=56
licenses allowed=70 review=3 denied=1 unknown=0
DENY package=legacy_xml version=0.8.1 license=GPL-3.0-only path=my_app->reporter->legacy_xml
written=artifacts/sbom.cdx.json exit_status=3

The Core Question You Are Answering

“How can project metadata become a deterministic, enforceable engineering contract?”

Concepts You Must Understand First

Mix task and project lifecycle — Which environment, lockfile, umbrella children, and dependency states are being inspected? Book Reference: “The Pragmatic Programmer” - Automation.
Graph traversal and identity — How do you avoid duplicating shared transitive dependencies while preserving paths? Book Reference: “Graph Algorithms the Fun Way” - Traversal.

Questions to Guide Your Design

What is the stable component identity for Hex, git, path, and umbrella dependencies?
How are multiple paths to one component represented?
Which findings warn and which fail CI?

Thinking Exercise

Draw a diamond dependency graph and determine component count, paths, and policy result when one branch applies an override.

The Interview Questions They Will Ask

“How do custom Mix tasks participate in project configuration?”
“How do you make serialized output reproducible?”
“How should umbrella dependencies be represented?”
“Why is an SBOM not a vulnerability scanner?”

Hints in Layers

Hint 1: Starting Point Inspect locked direct dependencies and emit one stable table. Hint 2: Next Level Traverse transitive children while recording every parent path. Hint 3: Technical Details Normalize then sort components before serialization; never depend on map iteration order. Hint 4: Tools/Debugging Run the task twice in clean builds and compare artifact hashes.

Books That Will Help

Topic	Book	Chapter
Automation contracts	“The Pragmatic Programmer”	Automation
Dependency graphs	“Graph Algorithms the Fun Way”	Graph Traversal

Common Pitfalls and Debugging

Problem 1: “SBOM hash changes without dependency changes”

Why: Timestamps, unordered maps, or absolute paths leak into output.
Fix: Define canonical ordering and isolate optional build metadata.
Quick test: Generate twice from the same lockfile and require identical hashes.

Definition of Done

Direct/transitive and umbrella dependencies are represented
Hex, git, and path sources have stable identities
Policies produce documented exit statuses
Output is deterministic for an unchanged lockfile
A CI fixture proves one allowed, warning, and denied case

Project 20: Compile-Time Pricing Policy DSL

File: P20-pricing-policy-dsl.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Metaprogramming, Domain Modeling
Software or Tool: Macro, Module, quote/unquote
Main Book: “Domain Modeling Made Functional”

What you will build: A restricted pricing DSL for tiers, segments, date windows, and exclusive discounts that compiles into normal functions, rejects contradictory declarations with source-located diagnostics, and returns an explanation trace with every price.

Why it teaches Elixir/BEAM: It requires quoted ASTs, hygienic macros, module attributes, @before_compile, generated clauses, protocols, compiler metadata, and a disciplined compile-time/runtime boundary.

Core challenges you will face:

Declarative AST capture -> Protocols + Compile-Time Tooling
Compile-time conflict detection -> Protocols + Compile-Time Tooling
Explainable runtime dispatch -> Data Transformation + Binaries

Real World Outcome

$ mix compile
Compiling 8 files (.ex)
error: overlapping exclusive pricing rules :vip_weekend and :clearance
  lib/pricing/policies.ex:42 conflicts with lib/pricing/policies.ex:57

$ mix price.explain --sku EPS-42 --customer vip --at 2026-08-15
base=120.00 BRL rules=[vip_10,weekend_5] final=102.60 BRL
trace="120.00 -> vip_10 (-12.00) -> weekend_5 (-5.40)"

The Core Question You Are Answering

“When does moving validation to compile time improve a domain API, and when does it merely hide logic in macros?”

Concepts You Must Understand First

Quoted code and hygiene — What does a macro receive and what must it return? Book Reference: “Metaprogramming Elixir” - Quote and Unquote.
Domain invariants — Which pricing contradictions can be proven from declarations alone? Book Reference: “Domain Modeling Made Functional” - Types and Domain Modeling.

Questions to Guide Your Design

What is declarative configuration versus runtime customer data?
How are file and line preserved through generation?
Can users inspect the generated rule plan without reading expanded AST?

Thinking Exercise

Classify ten pricing rules into compile-time conflicts, runtime non-matches, and valid compositions before designing syntax.

The Interview Questions They Will Ask

“What is macro hygiene?”
“Why keep generated code small?”
“How do module attributes participate in compilation?”
“What makes a DSL error actionable?”

Hints in Layers

Hint 1: Starting Point Model policies as plain structs and functions before adding syntax. Hint 2: Next Level Let macros only collect validated declarations with caller metadata. Hint 3: Technical Details Generate small dispatch clauses that call ordinary runtime evaluators. Hint 4: Tools/Debugging Inspect expansion and assert diagnostic file/line in compilation tests.

Books That Will Help

Topic	Book	Chapter
Elixir AST	“Metaprogramming Elixir”	Quote, Unquote, and Macros
Domain rules	“Domain Modeling Made Functional”	Domain Modeling with Types

Common Pitfalls and Debugging

Problem 1: “Stacktraces point into generated code”

Why: Caller source metadata was discarded.
Fix: Preserve locations and delegate runtime work to named functions.
Quick test: Trigger one invalid declaration and one runtime error and inspect both locations.

Definition of Done

Plain-function domain model exists beneath the DSL
Invalid declarations fail with correct file and line
Generated variables and aliases are hygienic
Every price includes a deterministic explanation trace
Expansion and runtime behavior are tested separately

Project 21: Multi-Provider Webhook Authenticity Gateway

File: P21-webhook-authenticity-gateway.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Suggested Seniority: Mid-level
Knowledge Area: HTTP Security, API Integration
Software or Tool: Plug, Plug.Crypto, :crypto
Main Book: “Serious Cryptography”

What you will build: A Plug gateway that preserves raw bodies, verifies provider-specific HMAC/timestamp schemes, enforces replay windows, normalizes accepted events, and records safe rejection audits.

Why it teaches Elixir/BEAM: Plug pipelines, body streaming, behaviours, with, binaries, Erlang :crypto interop, constant-time comparison, and explicit error taxonomies all meet at a security-sensitive boundary.

Core challenges you will face:

Raw-body authenticity -> Boundary Engineering
Provider adapter contracts -> Protocols + Compile-Time Tooling
Replay/idempotency separation -> Ecto + Durable Workflows

Real World Outcome

$ curl -i -X POST localhost:4000/webhooks/acme -H 'x-signature: ...' --data-binary @event.json
HTTP/1.1 202 Accepted
{"event_id":"evt_92f","provider":"acme","status":"verified"}

gateway rejection provider=acme category=stale_timestamp age_s=611 request_id=req_41
gateway rejection provider=contoso category=signature_mismatch request_id=req_42

The Core Question You Are Answering

“Which exact bytes are trusted, and what evidence proves a webhook is authentic but not necessarily unique?”

Concepts You Must Understand First

HMAC and constant-time comparison — Why is ordinary equality inappropriate for secrets? Book Reference: “Serious Cryptography” - Message Authentication.
Plug connection lifecycle — When is the raw body consumed? Book Reference: “Programming Phoenix” - Plugs and the Request Pipeline.

Questions to Guide Your Design

What are the body, header, timestamp, and clock-skew limits?
How are provider errors normalized without losing safe details?
Which identifiers support replay rejection and downstream idempotency?

Thinking Exercise

Trace a valid signed event delivered twice, then a stale but correctly signed event. Separate authenticity, freshness, replay, and business handling decisions.

The Interview Questions They Will Ask

“Why verify before JSON decoding?”
“What does constant-time comparison prevent?”
“How do behaviours help provider integrations?”
“Why doesn’t a valid signature prevent duplicate processing?”

Hints in Layers

Hint 1: Starting Point Implement one provider with fixed raw-body fixtures. Hint 2: Next Level Extract a verifier behaviour only after common and provider-specific responsibilities are clear. Hint 3: Technical Details Bound body reads and prune secrets from logs and stacktraces. Hint 4: Tools/Debugging Preserve safe digests of fixture bytes so accidental middleware rewriting is visible.

Books That Will Help

Topic	Book	Chapter
HMAC	“Serious Cryptography”	Message Authentication
Plug pipeline	“Programming Phoenix”	Plugs

Common Pitfalls and Debugging

Problem 1: “Known-good signatures fail”

Why: Middleware decoded or normalized the body first.
Fix: Capture bounded raw bytes before parsing and sign the provider’s exact canonical input.
Quick test: Assert the verified bytes hash equals the provider fixture hash.

Definition of Done

Raw body is bounded and preserved before decoding
At least two provider behaviours share one gateway contract
Timestamp and constant-time signature checks are separate
Replay and downstream idempotency policies are documented
Logs and errors contain no secret material

Project 22: Transactional Inventory Ledger and Reservation API

File: P22-inventory-ledger-reservation-api.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Relational Integrity, Concurrency
Software or Tool: Ecto, PostgreSQL, Ecto.Multi
Main Book: “Designing Data-Intensive Applications”

What you will build: An API with append-only stock movements and atomic reserve, release, and commit operations that cannot oversell or duplicate a retried operation.

Why it teaches Elixir/BEAM: Ecto schemas, changesets, constraints, queries, Ecto.Multi, optimistic or pessimistic concurrency, tagged transaction errors, and concurrent tests become concrete domain guarantees.

Core challenges you will face:

Database-enforced invariants -> Ecto + Durable Workflows
Concurrent reservations -> Ecto + Durable Workflows
Idempotent commands -> Ecto + Durable Workflows

Real World Outcome

$ curl -s -X POST localhost:4000/skus/A1/reservations -d '{"operation_id":"op-77","quantity":3}'
{"reservation_id":"r-901","available":7,"status":"held"}

$ mix test test/inventory/concurrency_test.exs
100 concurrent requests for stock=40
committed=40 rejected=60 final_available=0 duplicate_ledger_entries=0
1 test, 0 failures

The Core Question You Are Answering

“Which invariants must the database arbitrate when many BEAM processes act concurrently?”

Concepts You Must Understand First

Validation versus constraints — Which checks are safe without database arbitration? Book Reference: “Programming Ecto” - Changesets and Constraints.
Transaction isolation — What can two reservations observe and update? Book Reference: “Designing Data-Intensive Applications” - Transactions.

Questions to Guide Your Design

Is availability derived from a ledger, stored counter, or both?
Which operation ID uniqueness is enforced in the database?
How are serialization/deadlock retries bounded and observed?

Thinking Exercise

Interleave two transactions that each request the final unit. Show how a naive read/write oversells and how your chosen database operation prevents it.

The Interview Questions They Will Ask

“Why is unsafe_validate_unique insufficient for integrity?”
“What does Ecto.Multi add to a transaction?”
“How do optimistic and pessimistic locking differ?”
“How do idempotency keys interact with retries?”

Hints in Layers

Hint 1: Starting Point Define ledger and reservation invariants before schemas. Hint 2: Next Level Express race-sensitive rules as conditional writes or database constraints. Hint 3: Technical Details Return named failed operations from the transaction boundary. Hint 4: Tools/Debugging Use barriers so concurrent tests reach the critical section together.

Books That Will Help

Topic	Book	Chapter
Ecto integrity	“Programming Ecto”	Changesets; Constraints; Multi
Transactions	“Designing Data-Intensive Applications”	Transactions

Common Pitfalls and Debugging

Problem 1: “Unit tests pass, production oversells”

Why: Tests run sequentially and integrity lives only in application validation.
Fix: Move arbitration to the database and run synchronized concurrent tests.
Quick test: Race more requests than available stock and verify the invariant repeatedly.

Definition of Done

Ledger and availability invariants are documented
Database constraints arbitrate concurrent writes
Duplicate operation IDs return one logical result
Transaction failures identify the named failed step
Repeated concurrency tests never oversell

Project 23: Publishable Pluggable Object Storage Library

File: P23-pluggable-object-storage-library.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Gleam
Coolness Level: Level 3 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 2 (See REFERENCE.md)
Suggested Seniority: Mid-level
Knowledge Area: Library Design, Packaging
Software or Tool: Behaviours, Protocols, ExUnit, Hex
Main Book: “The Pragmatic Programmer”

What you will build: A Hex-ready object-storage library with local-disk, in-memory, and HTTP-compatible adapters, one stable API, and a shared adapter contract suite.

Why it teaches Elixir/BEAM: Behaviours, protocols, typespecs, @impl, doctests, configuration boundaries, semantic versioning, package contents, and test seams are learned as a real reusable library.

Core challenges you will face:

Adapter callbacks -> Protocols + Compile-Time Tooling
Stable public errors/types -> Protocols + Compile-Time Tooling
Package discipline -> Protocols + Compile-Time Tooling

Real World Outcome

$ mix test
adapter contract: memory .... 12 tests, 0 failures
adapter contract: local ..... 12 tests, 0 failures
adapter contract: http ...... 12 tests, 0 failures

$ mix hex.build --unpack
package=my_object_store-0.1.0.tar files=18 docs=true license=Apache-2.0
forbidden_runtime_dependencies=0

The Core Question You Are Answering

“How do behaviours, protocols, typespecs, and contract tests turn replaceable implementations into a trustworthy public API?”

Concepts You Must Understand First

Behaviour versus protocol dispatch — Is variability selected by module configuration or data type? Book Reference: “Elixir in Action” - Data Abstractions.
Compatibility — Which error or option changes break callers? Book Reference: “Building Evolutionary Architectures” - Fitness Functions.

Questions to Guide Your Design

What minimum semantics can every adapter honestly guarantee?
How are streaming bodies, metadata, not-found, and conflict represented?
Which dependency belongs in core versus an optional adapter?

Thinking Exercise

Compare local atomic rename with remote eventual visibility. Define the shared contract without claiming guarantees the remote backend cannot provide.

The Interview Questions They Will Ask

“When do you choose a behaviour over a protocol?”
“What belongs in an adapter contract suite?”
“How do optional dependencies affect library users?”
“What is a breaking change in an Elixir library?”

Hints in Layers

Hint 1: Starting Point Write the public result and error types before adapters. Hint 2: Next Level Make the in-memory adapter the fast executable specification. Hint 3: Technical Details Run identical contract cases against every adapter factory. Hint 4: Tools/Debugging Inspect the unpacked Hex artifact and build docs with warnings treated seriously.

Books That Will Help

Topic	Book	Chapter
Elixir abstractions	“Elixir in Action”	Data Abstractions
Evolution	“Building Evolutionary Architectures”	Fitness Functions

Common Pitfalls and Debugging

Problem 1: “Adapters pass unit tests but disagree on not-found”

Why: Public semantics were never encoded as shared contract cases.
Fix: Make error shape and metadata expectations part of the adapter suite.
Quick test: Run the same missing-key scenario against all adapters.

Definition of Done

Public callbacks, typespecs, and errors are documented
Three adapters pass one shared contract suite
Optional backend dependencies do not leak into core
Doctests and generated documentation pass
Unpacked Hex package contains only intended files

Project 24: Durable Document Processing Workflow with Oban

File: P24-oban-document-workflow.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Durable Jobs, Workflow Reliability
Software or Tool: Oban, Ecto, PostgreSQL
Main Book: “Release It!”

What you will build: A durable validate, preview, metadata-extraction, and notification workflow with retries, uniqueness, cancellation, idempotent side effects, and operator-visible repair history.

Why it teaches Elixir/BEAM: Oban worker callbacks, changeset-built jobs, pattern-matched results, Ecto transactions, retry/backoff, supervision, telemetry, and the limits of exactly-once claims become operationally visible.

Core challenges you will face:

Durable orchestration -> Ecto + Durable Workflows
Retry-safe effects -> Ecto + Durable Workflows
Operator repair paths -> OTP + Supervision

Real World Outcome

$ mix documents.submit contract.pdf --notify ops@example.test
workflow=doc_7f2 stages=validate,preview,metadata,notify status=queued

$ mix documents.status doc_7f2
validate=completed preview=retryable(attempt=2,next=14:32:08)
metadata=blocked notify=blocked last_error="renderer exited 75"

$ mix documents.retry doc_7f2 preview
retry accepted; duplicate_notification_risk=none idempotency_key=doc_7f2:v3:notify

The Core Question You Are Answering

“What must be persisted so a multi-stage workflow can resume safely after any process or node disappears?”

Concepts You Must Understand First

Durable jobs and uniqueness — Why is insertion uniqueness different from execution concurrency? Book Reference: “Release It!” - Stability Patterns.
Idempotent side effects — How does a repeated stage detect prior completion? Book Reference: “Enterprise Integration Patterns” - Idempotent Receiver.

Questions to Guide Your Design

Is each stage one job, a state transition, or both?
Which retries are automatic, operator-triggered, or terminal?
How are cancellation and already-running work reconciled?

Thinking Exercise

Crash immediately after the notification provider accepts a request but before the worker records success. Explain the next attempt’s evidence and behavior.

The Interview Questions They Will Ask

“What guarantees does a durable job queue actually provide?”
“How are Oban uniqueness and queue concurrency different?”
“How do you design an idempotent external effect?”
“When should a job be discarded rather than retried?”

Hints in Layers

Hint 1: Starting Point Make one stage repeatable before composing a workflow. Hint 2: Next Level Persist stage version and outcome; do not infer completion from missing jobs. Hint 3: Technical Details Write workflow state plus next-stage intent in one database transaction. Hint 4: Tools/Debugging Test crashes before effect, after effect, and after persistence separately.

Books That Will Help

Topic	Book	Chapter
Stability	“Release It!”	Stability Patterns
Idempotency	“Enterprise Integration Patterns”	Idempotent Receiver

Common Pitfalls and Debugging

Problem 1: “Users receive duplicate notifications”

Why: A crash occurred after the provider accepted the first request.
Fix: Use a stable provider idempotency key and persist the attempt identity.
Quick test: Inject that exact crash window and verify one logical delivery.

Definition of Done

Workflow resumes after node restart
Each external effect is idempotent under retry
Uniqueness and concurrency policies are documented separately
Cancellation and manual repair have auditable outcomes
Failure injection covers every stage boundary

Project 25: TCP Device Telemetry Gateway

File: P25-tcp-device-telemetry-gateway.md
Main Programming Language: Elixir and Erlang
Alternative Programming Languages: Gleam
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: TCP, Binary Protocols, Connection Lifecycle
Software or Tool: :gen_tcp, DynamicSupervisor, bitstrings
Main Book: “TCP/IP Illustrated, Volume 1”

What you will build: A single-node gateway for thousands of simulated sensors using a framed binary protocol, partial-frame reconstruction, checksums, acknowledgements, safe socket ownership, and slow-client eviction.

Why it teaches Elixir/BEAM: It applies process-per-connection design, Erlang :gen_tcp, active: :once, controlling-process transfer, binary patterns, iodata, supervision, and mailbox pressure to a real transport.

Core challenges you will face:

TCP stream framing -> Boundary Engineering
Socket ownership -> Process Model
Connection admission and cleanup -> OTP + Supervision

Real World Outcome

$ mix gateway.demo --devices 5000 --fragment-rate 0.35 --duration 60
accepted=5000 authenticated=4988 rejected=12
frames=3,704,119 checksum_errors=31 incomplete_buffers=0
p50_ack=8ms p99_ack=43ms max_mailbox=37 disconnected_slow=6
invariant accepted_frames=acked+rejected status=ok

The Core Question You Are Answering

“How do byte-stream framing and process ownership interact when thousands of devices send at different speeds?”

Concepts You Must Understand First

TCP byte streams — Why can one frame arrive across many receives? Book Reference: “TCP/IP Illustrated” - TCP Data Flow.
Socket ownership and mailboxes — Which process receives active messages? Book Reference: “Programming Erlang” - Concurrent Servers.

Questions to Guide Your Design

How are header length, payload maximum, checksum, and version encoded?
When is the socket moved from acceptor to connection process?
What makes a client slow or abusive enough to disconnect?

Thinking Exercise

Feed half a header, then two frames plus half a third. Draw the retained buffer after each receive and every acknowledgement.

The Interview Questions They Will Ask

“Does TCP preserve message boundaries?”
“What does active: :once control?”
“Why does socket ownership transfer race?”
“How do iodata and binaries affect copying?”

Hints in Layers

Hint 1: Starting Point Make a pure parser accept a binary buffer and return frames plus remainder. Hint 2: Next Level Add one supervised connection process and explicit ownership transfer. Hint 3: Technical Details Rearm active-once only after processing or deliberately queueing the frame. Hint 4: Tools/Debugging Randomize fragmentation and coalescing in the simulator.

Books That Will Help

Topic	Book	Chapter
TCP semantics	“TCP/IP Illustrated, Volume 1”	TCP Data Flow
BEAM servers	“Programming Erlang”	Concurrent Programming

Common Pitfalls and Debugging

Problem 1: “Frames vanish during connection startup”

Why: Active messages reached the acceptor while ownership was transferring.
Fix: Coordinate passive acceptance, transfer ownership, then activate in the connection process.
Quick test: Send immediately on connect thousands of times and account for every frame.

Definition of Done

Parser handles arbitrary split and coalesced frames
Socket ownership has no message-loss window
Connection count, mailbox, payload, and idle time are bounded
Malformed clients cannot crash acceptors or peers
Load test accounts for every accepted frame

Project 26: Nerves Cold-Chain Sensor Gateway

File: P26-nerves-cold-chain-gateway.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Embedded Systems, Fleet Reliability
Software or Tool: Nerves, Circuits, fwup, OTP releases
Main Book: “Making Embedded Systems”

What you will build: A Nerves appliance that reads temperature sensors, emits local alarms, buffers readings while offline, uploads in order after reconnect, and validates or rolls back tentative firmware.

Why it teaches Elixir/BEAM: OTP ownership and supervision move onto hardware, while Nerves targets, read-only firmware, /data persistence, update slots, watchdogs, and health validation expose operational constraints absent from server apps.

Core challenges you will face:

Hardware process ownership -> Boundary Engineering
Offline durability -> Ecto + Durable Workflows
Firmware validation/rollback -> Release Handling

Real World Outcome

nerves.local> cold_chain status
sensor=probe_1 temp=3.8C range=2.0..8.0 alarm=clear
uplink=offline buffered=18,442 oldest=2026-07-14T22:11:03Z
firmware=0.4.0 slot=b tentative=true validation=waiting_for_uplink

nerves.local> cold_chain simulate reconnect
uploaded=18,442 duplicates=0 order=preserved
firmware validation=passed slot=b marked_good=true

The Core Question You Are Answering

“What must an embedded BEAM system keep doing correctly when sensors, storage, power, or the network become unreliable?”

Concepts You Must Understand First

OTP resource ownership — Which process owns the sensor bus and alarm output? Book Reference: “Designing for Scalability with Erlang/OTP” - Supervision.
Embedded update lifecycle — What evidence marks firmware good? Book Reference: “Making Embedded Systems” - Reliability and Watchdogs.

Questions to Guide Your Design

How are invalid, missing, and physically impossible readings classified?
What happens when /data reaches its storage budget?
Which critical checks must pass before firmware validation?

Thinking Exercise

Trace power loss during buffered-write, firmware update, first boot, and replay. Identify the durable boundary and recovery evidence for each.

The Interview Questions They Will Ask

“How does Nerves package an OTP application?”
“Why is persistent data outside the firmware filesystem?”
“What makes a firmware update safe to validate?”
“How do supervisors help—and not help—with hardware faults?”

Hints in Layers

Hint 1: Starting Point Use a simulated sensor and one supervised sampling loop. Hint 2: Next Level Add bounded append-only offline storage with replay checkpoints. Hint 3: Technical Details Validate firmware only after sensor, persistence, alarm, network auth, and replay checks. Hint 4: Tools/Debugging Pull power at documented checkpoints and preserve serial logs.

Books That Will Help

Topic	Book	Chapter
Embedded reliability	“Making Embedded Systems”	Watchdogs and Fault Handling
OTP recovery	“Designing for Scalability with Erlang/OTP”	Supervision Trees

Common Pitfalls and Debugging

Problem 1: “Bad firmware becomes permanent”

Why: Validation occurred immediately after application start.
Fix: Require a timed health contract covering every critical capability.
Quick test: Ship firmware with broken networking and verify automatic reversion.

Definition of Done

Sensor, storage, uplink, and alarm processes have explicit ownership
Offline buffer is bounded and replay is ordered/idempotent
Local alarm works without network access
Tentative firmware validates only after the full health contract
Power-loss and bad-firmware drills demonstrate recovery

Project 27: Rustler Perceptual-Hash NIF

File: P27-rustler-perceptual-hash-nif.md
Main Programming Language: Elixir and Rust
Alternative Programming Languages: Erlang and C
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 2 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Suggested Seniority: Staff
Knowledge Area: Native Integration, Runtime Safety
Software or Tool: Rustler, NIFs, dirty schedulers, Benchee
Main Book: “Rust for Rustaceans”

What you will build: A Rust-backed perceptual image hash with a pure-Elixir reference, scheduler-responsiveness benchmark, dirty-scheduler policy, bounded inputs, error mapping, and a safe fallback when the native library is unavailable.

Why it teaches Elixir/BEAM: The learner must reason about term conversion, binary lifetime, NIF loading, Rust errors, dirty CPU scheduling, node-wide crash risk, release artifacts, and performance evidence beyond raw throughput.

Core challenges you will face:

Native term boundary -> Boundary Engineering
Scheduler safety -> Scheduling + GC
Fallback and release packaging -> Release Handling

Real World Outcome

$ mix phash.bench fixtures/images --concurrency 200
implementation pure_elixir throughput=41/s p99=188ms scheduler_probe_p99=7ms
implementation rustler_dirty throughput=612/s p99=19ms scheduler_probe_p99=8ms
implementation rustler_normal throughput=640/s p99=18ms scheduler_probe_p99=921ms UNSAFE
hash_equivalence=1000/1000 fallback_test=passed

The Core Question You Are Answering

“When is lower native-call overhead worth sharing the BEAM node’s fate with foreign code?”

Concepts You Must Understand First

Normal and dirty schedulers — What happens when native work exceeds a normal scheduler budget? Book Reference: “The BEAM Book” - Scheduling.
Rust/BEAM ownership — Which binaries are borrowed, copied, or retained? Book Reference: “Rust for Rustaceans” - Ownership and FFI.

Questions to Guide Your Design

What input and time budgets are enforced before entering native code?
What errors cross as tagged results rather than panics?
How does the release behave on an unsupported platform?

Thinking Exercise

Compare failure and latency for pure Elixir, an external Port, a normal NIF, and a dirty NIF while 1,000 unrelated probe processes run.

The Interview Questions They Will Ask

“How can a NIF affect the whole VM?”
“What problem do dirty schedulers solve?”
“How does Rustler map terms and errors?”
“What benchmark proves scheduler safety?”

Hints in Layers

Hint 1: Starting Point Build and freeze a pure-Elixir reference corpus. Hint 2: Next Level Add a tiny bounded native function and map every error explicitly. Hint 3: Technical Details Run unbounded CPU work only on a dirty CPU scheduler and cap input dimensions. Hint 4: Tools/Debugging Measure unrelated process latency during the benchmark.

Books That Will Help

Topic	Book	Chapter
BEAM scheduling	“The BEAM Book”	Scheduler and NIFs
Rust FFI discipline	“Rust for Rustaceans”	Ownership; Unsafe and FFI

Common Pitfalls and Debugging

Problem 1: “NIF is fast but the service freezes”

Why: Long CPU work runs on normal schedulers.
Fix: Bound or chunk work and schedule appropriate operations as dirty CPU.
Quick test: Track a 10 ms probe process’s p99 latency during maximum native load.

Definition of Done

Pure and native results match a fixed corpus
Inputs, errors, and native resource lifetimes are bounded
Scheduler probe remains responsive under load
Unsupported/native-load failure uses a documented fallback
Release artifact includes and verifies the correct native library

Project 28: Compiler-Aware Deprecation and Migration Linter

File: P28-compiler-aware-migration-linter.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Compiler Tooling, Static Analysis
Software or Tool: Code, Macro, compiler tracers, Mix.Task
Main Book: “Engineering a Compiler”

What you will build: A Mix task and compiler tracer that reports deprecated or organization-banned calls with resolved module/function/arity, source locations, CI JSON, and safe rewrite previews.

Why it teaches Elixir/BEAM: Quoted ASTs, token/comment metadata, Macro.prewalk, compiler tracer events, Macro.Env, alias/import resolution, asynchronous analysis, diagnostics, and cautious transformations become a real developer tool.

Core challenges you will face:

Resolved compiler facts -> Protocols + Compile-Time Tooling
Source-preserving AST analysis -> Protocols + Compile-Time Tooling
Safe migration previews -> Protocols + Compile-Time Tooling

Real World Outcome

$ mix migrate.audit --rules config/migration_rules.exs --format text
lib/billing/retry.ex:88:12 warning Legacy.retry/2 -> Resilience.retry/3
  resolution=remote_call confidence=exact autofix=preview
lib/api/parser.ex:31:5 error banned :erlang.binary_to_term/1 on untrusted input
findings=14 errors=1 warnings=13 elapsed=182ms exit_status=4

The Core Question You Are Answering

“What can be proven from syntax alone, and what requires compiler context before a migration is safe?”

Concepts You Must Understand First

AST and source metadata — How are calls, aliases, comments, and locations represented? Book Reference: “Metaprogramming Elixir” - AST Traversal.
Compiler environments — How do imports and aliases change name resolution? Book Reference: “Engineering a Compiler” - Static Semantics.

Questions to Guide Your Design

Which facts are captured synchronously and analyzed later?
What confidence is required before previewing a rewrite?
How are generated/vendor files and suppressions represented?

Thinking Exercise

Classify local, imported, aliased, captured, quoted, and dynamically applied calls to the same function. Decide which rules can resolve each.

The Interview Questions They Will Ask

“What does Elixir’s AST look like?”
“Why must compiler tracers stay fast?”
“How are imported calls resolved?”
“Why is a syntactic rewrite not automatically semantics-preserving?”

Hints in Layers

Hint 1: Starting Point Report exact remote calls from parsed source with locations. Hint 2: Next Level Capture compiler events to resolve aliases and imports. Hint 3: Technical Details Store small facts in tracer callbacks; perform policy analysis after compilation. Hint 4: Tools/Debugging Golden-test diagnostics and rewrite previews without modifying files.

Books That Will Help

Topic	Book	Chapter
AST tooling	“Metaprogramming Elixir”	Traversal and Compilation
Static analysis	“Engineering a Compiler”	Static Semantics

Common Pitfalls and Debugging

Problem 1: “The linter doubles compile time”

Why: Heavy analysis runs inside tracer callbacks.
Fix: Capture bounded compiler facts and analyze asynchronously afterward.
Quick test: Compare clean compilation time with tracer disabled and enabled.

Definition of Done

Remote, imported, and aliased calls resolve correctly
Diagnostics preserve exact file, line, and column
Tracer overhead has a measured budget
Machine-readable output is deterministic
Rewrites remain previews unless preconditions are proven

Project 29: Nx Demand Forecasting Engine

File: P29-nx-demand-forecasting-engine.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Python for result comparison
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Numerical Computing, Forecasting
Software or Tool: Nx, Nx.Defn, EXLA, Livebook
Main Book: “AI Engineering”

What you will build: A forecasting engine that turns historical demand into tensors, creates leakage-safe windows, trains a regression baseline, reports error metrics, and benchmarks eager versus compiled execution.

Why it teaches Elixir/BEAM: It extends immutable transformation into shapes, axes, dtypes, containers, defn computation graphs, JIT warm-up, backend placement, batching, and supervised serving.

Core challenges you will face:

Tensor representation -> Boundary Engineering
Numerical transformations -> Data Transformation + Binaries
Compiled workload scheduling -> Scheduling + GC

Real World Outcome

$ mix forecast.train data/demand.csv --horizon 7 --window 28
rows=1095 train=876 validation=109 test=110 leakage_check=passed
model=linear_windowed mae=8.42 rmse=11.06 mape=6.8%
eager_predict=4.8ms compiled_warm=0.31ms compile_once=287ms
artifact=artifacts/demand-v1.nx metadata=artifacts/demand-v1.json

The Core Question You Are Answering

“How do tensor shapes and data-split invariants determine correctness before acceleration matters?”

Concepts You Must Understand First

Tensors, shapes, axes, dtypes — Which dimension means samples, history, and features? Book Reference: “Hands-On Machine Learning” - End-to-End ML Project.
Forecast validation — Why is random splitting usually wrong for time series? Book Reference: “AI Engineering” - Evaluation.

Questions to Guide Your Design

How are missing intervals and future leakage prevented?
Which metrics fit near-zero demand?
Are compilation time, warm execution, and batch latency reported separately?

Thinking Exercise

Draw tensor shapes through windowing, batching, prediction, and metric calculation for 100 observations, window 14, horizon 7.

The Interview Questions They Will Ask

“What is a tensor axis and why does shape matter?”
“What does defn change about execution?”
“How do you avoid time-series leakage?”
“How do JIT warm-up and serving affect latency claims?”

Hints in Layers

Hint 1: Starting Point Beat a naive last-value or seasonal baseline before adding complexity. Hint 2: Next Level Represent window creation as an independently verified transformation. Hint 3: Technical Details Separate compilation, warm execution, and batch-size benchmarks. Hint 4: Tools/Debugging Print shape/dtype contracts at each boundary and fail early.

Books That Will Help

Topic	Book	Chapter
ML evaluation	“AI Engineering”	Evaluation and Serving
Forecast pipeline	“Hands-On Machine Learning”	End-to-End ML Project

Common Pitfalls and Debugging

Problem 1: “Excellent test error cannot be reproduced live”

Why: Future observations leaked through scaling, windows, or random split.
Fix: Fit preprocessing on training history and split chronologically.
Quick test: Assert every feature timestamp precedes its target timestamp.

Definition of Done

Tensor shapes and dtypes are documented and asserted
Chronological split and leakage checks pass
Forecast beats a stated baseline on held-out data
Eager, compile, warm, and batch timings are separate
Model artifact includes preprocessing and evaluation metadata

Project 30: `:gen_statem` Escrow and Approval Workflow Engine

File: P30-gen-statem-escrow-workflow.md
Main Programming Language: Elixir and Erlang
Alternative Programming Languages: Gleam
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Suggested Seniority: Staff
Knowledge Area: State Machines, Temporal Workflows
Software or Tool: :gen_statem, Ecto, OTP supervision
Main Book: “Designing for Scalability with Erlang/OTP”

What you will build: An escrow lifecycle through submitted, risk-review, approved, captured, cancelled, expired, and disputed states with state timeouts, postponed events, idempotent commands, durable transition logs, and recovery.

Why it teaches Elixir/BEAM: Erlang behaviour interop, callback modes, state-enter calls, event/state timeouts, postponed events, supervision, and persistence boundaries are learned through an auditable financial workflow.

Core challenges you will face:

Explicit temporal transitions -> OTP + Supervision
Durable recovery -> Ecto + Durable Workflows
Duplicate/stale events -> Process Model

Real World Outcome

$ mix escrow.demo --scenario late-risk-approval
escrow=e_204 transition=submitted->risk_review timer=15m
event=capture postponed reason=awaiting_approval
transition=risk_review->approved replayed=[capture]
transition=approved->captured ledger_version=4
late event=risk_approved ignored category=stale state=captured
recovery replay versions=1..4 state=captured invariant=balanced

The Core Question You Are Answering

“When is state itself insufficient, requiring the event type, timeout kind, and transition actions to be modeled explicitly?”

Concepts You Must Understand First

gen_statem semantics — How do state, event type, data, and actions differ? Book Reference: “Designing for Scalability with Erlang/OTP” - State Machines.
Durable transition identity — How is a command replay recognized? Book Reference: “Enterprise Integration Patterns” - Idempotent Receiver.

Questions to Guide Your Design

Which events are postponed, rejected, or safe to ignore?
Which timeout is relative to a state versus a specific event?
How does process recovery reconstruct timers without replaying effects?

Thinking Exercise

Build a transition table for capture arriving before approval, cancellation during review, and approval after expiration. Include actions and persistence points.

The Interview Questions They Will Ask

“When is gen_statem preferable to GenServer?”
“What are postponed events?”
“How do state and event timeouts differ?”
“How do you recover a timed workflow after restart?”

Hints in Layers

Hint 1: Starting Point Write the complete state/event table before callbacks. Hint 2: Next Level Make transition decisions pure and actions explicit. Hint 3: Technical Details Persist command ID and transition before external effects. Hint 4: Tools/Debugging Use deterministic time in tests and inspect state with system tooling.

Books That Will Help

Topic	Book	Chapter
State machines	“Designing for Scalability with Erlang/OTP”	State Machines
Idempotency	“Enterprise Integration Patterns”	Idempotent Receiver

Common Pitfalls and Debugging

Problem 1: “A recovered escrow expires immediately”

Why: Relative timer duration was restored without accounting for elapsed time.
Fix: Persist deadlines and calculate remaining duration during recovery.
Quick test: Restart midway through a timeout and verify the original deadline.

Definition of Done

Transition table covers every state/event pair
Duplicate and stale commands have deterministic results
Deadlines recover correctly after process/node restart
Postponed events replay only in intended states
Durable log reconstructs state and audit trail

Project 31: Multi-Tenant Dynamic Repo Control Plane

File: P31-multi-tenant-dynamic-repo-control-plane.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Suggested Seniority: Staff
Knowledge Area: Multi-Tenancy, Resource Control
Software or Tool: DynamicSupervisor, Registry, Ecto.Repo
Main Book: “Software Architecture in Practice”

What you will build: A control plane that starts tenant Repo processes on demand, routes without cross-tenant leakage, runs migrations, caps pools, evicts idle tenants, and survives unavailable databases.

Why it teaches Elixir/BEAM: Dynamic supervision, registries, Repo lifecycle, process-local dynamic routing, context propagation, migrations, admission control, and bounded ownership form a staff-level OTP subsystem.

Core challenges you will face:

Dynamic resource supervision -> OTP + Supervision
Tenant-safe routing -> Ecto + Durable Workflows
Connection/migration capacity -> Scheduling + GC

Real World Outcome

$ mix tenants.demo --tenants 250 --max-active 30
requested=250 active_repos=30 queued=14 evicted_idle=206
connections=150/150 migration_concurrency=2
cross_tenant_probe=0 leaks
tenant=acme db=unavailable request=503 retry_after=15 repo_restart=bounded

The Core Question You Are Answering

“How do you turn each tenant database into a supervised, bounded resource without relying on ambient context?”

Concepts You Must Understand First

Dynamic supervision and registration — How are duplicate starts serialized? Book Reference: “Designing for Scalability with Erlang/OTP” - Supervision.
Ecto dynamic repositories — Where is current Repo identity stored and propagated? Book Reference: “Programming Ecto” - Repositories.

Questions to Guide Your Design

What is the admission policy when all pool capacity is used?
Which activity prevents eviction?
How is tenant identity carried into Tasks, jobs, and spawned processes?

Thinking Exercise

Trace two simultaneous first requests for one tenant while eviction selects another tenant with an in-flight query. Identify required serialization and leases.

The Interview Questions They Will Ask

“What is a dynamic Repo?”
“How do Registry and DynamicSupervisor responsibilities differ?”
“Why can process-local tenant context be dangerous?”
“How do you bound tenant connection pools?”

Hints in Layers

Hint 1: Starting Point Start and route one named tenant Repo safely. Hint 2: Next Level Introduce leases/reference counts before idle eviction. Hint 3: Technical Details Put tenant identity in request/job data and set dynamic Repo at the execution boundary. Hint 4: Tools/Debugging Continuously query sentinel rows from randomized tenants to detect leakage.

Books That Will Help

Topic	Book	Chapter
Resource architecture	“Software Architecture in Practice”	Quality Attributes
Ecto repositories	“Programming Ecto”	Repositories and Transactions

Common Pitfalls and Debugging

Problem 1: “A Task queries the default tenant”

Why: Process-local Repo context was not propagated to the spawned Task.
Fix: Pass tenant identity explicitly and select the Repo inside the Task.
Quick test: Spawn concurrent tenant Tasks and assert sentinel isolation.

Definition of Done

Concurrent first access starts one Repo per tenant
Tenant identity is explicit across process boundaries
Repo, connection, migration, and queue limits are enforced
Idle eviction never stops in-use repositories
Cross-tenant probes remain zero under load and failure

Project 32: BEAM Artifact Inspector and Reproducibility Auditor

File: P32-beam-artifact-reproducibility-auditor.md
Main Programming Language: Elixir and Erlang
Alternative Programming Languages: Gleam
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Suggested Seniority: Staff
Knowledge Area: BEAM Artifacts, Reproducible Builds
Software or Tool: :beam_lib, Code, Mix releases
Main Book: “The BEAM Book”

What you will build: A scanner that inventories .beam chunks, exports, attributes, docs, compile metadata, debug/abstract-code exposure, and hashes, then explains differences between two releases without loading untrusted modules.

Why it teaches Elixir/BEAM: Erlang :beam_lib interop, BEAM chunk structure, binaries/tuples, compiler metadata, deterministic inventories, debug information, code-loading risks, and release provenance become observable.

Core challenges you will face:

Safe chunk inspection -> Protocols + Compile-Time Tooling
Canonical build comparison -> Release Handling
Metadata risk classification -> Boundary Engineering

Real World Outcome

$ mix beam.audit rel-a/ rel-b/ --explain
modules_a=428 modules_b=428 byte_identical=401 semantic_identical=421 changed=7
MyApp.Invoice.beam code_hash=same artifact_hash=different
  cause: compile_info source path /build/agent-17 vs /workspace
MyApp.SecretParser.beam abstract_code=present docs=present risk=review
reproducibility_score=98.4% report=beam-audit.json

The Core Question You Are Answering

“Which parts of a BEAM file define executable behavior, and which parts merely make builds byte-different or leak information?”

Concepts You Must Understand First

BEAM chunks — What do code, attributes, compile info, docs, and debug chunks represent? Book Reference: “The BEAM Book” - Code Loading and File Format.
Reproducibility — Which timestamps and paths should be normalized versus reported? Book Reference: “Continuous Delivery” - Build Artifacts.

Questions to Guide Your Design

Can semantic and byte identity be reported separately?
Which chunks are safe to normalize for comparison?
How do you avoid module loading or atom creation from untrusted artifacts?

Thinking Exercise

Compare two modules with identical code but different source paths and docs. Decide which hashes differ and what the report should claim.

The Interview Questions They Will Ask

“What information lives in a BEAM file?”
“Why inspect with :beam_lib instead of loading a module?”
“What is a reproducible build?”
“How can debug info change a release’s security posture?”

Hints in Layers

Hint 1: Starting Point Inventory module, exports, imports, and raw chunk hashes. Hint 2: Next Level Separate execution-relevant and metadata-only comparisons. Hint 3: Technical Details Canonicalize report ordering, not source artifacts. Hint 4: Tools/Debugging Build the same commit in two absolute paths and explain every difference.

Books That Will Help

Topic	Book	Chapter
BEAM artifacts	“The BEAM Book”	Code Loading and BEAM Files
Artifact discipline	“Continuous Delivery”	Managing Binaries

Common Pitfalls and Debugging

Problem 1: “Inspector crashes on a hostile filename/module atom”

Why: It loads code or creates unbounded atoms from external strings.
Fix: Treat paths and chunk values as data and use safe existing-term operations.
Quick test: Scan malformed and adversarial fixtures in an isolated test corpus.

Definition of Done

Inspector never loads scanned modules
Byte and semantic difference categories are separate
Reports are deterministic and explain known path metadata drift
Debug/docs/abstract-code exposure is classified
Malformed artifacts return bounded errors

Project 33: Custom Ecto Adapter for an HTTP Document Store

File: P33-custom-ecto-http-document-adapter.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 5 (See REFERENCE.md)
Difficulty: Level 5 (See REFERENCE.md)
Suggested Seniority: Principal
Knowledge Area: Framework Internals, Query Translation
Software or Tool: Ecto.Adapter, Ecto query AST, Finch
Main Book: “Designing Data-Intensive Applications”

What you will build: A deliberately bounded Ecto adapter supporting schemas, CRUD, filters, ordering, pagination, and type conversion over a mock HTTP document store, with explicit rejection of unsupported joins and transactions.

Why it teaches Elixir/BEAM: Behaviours, macro callbacks, adapter initialization, loaders/dumpers, Ecto query structures, remote protocol translation, supervision, connection ownership, compliance tests, and semantic impedance mismatch culminate in an ecosystem extension.

Core challenges you will face:

Adapter callback contract -> Protocols + Compile-Time Tooling
Query/type translation -> Ecto + Durable Workflows
Honest capability limits -> Boundary Engineering

Real World Outcome

$ mix test test/adapter_compliance
schema_insert_update_delete ............... pass
filters_eq_in_range ....................... pass
ordering_pagination ....................... pass
load_dump_uuid_datetime_decimal ........... pass
unsupported_join ......................... explicit_error
unsupported_transaction .................. explicit_error
86 tests, 0 failures

$ mix ecto_adapter.explain 'from d in Device, where: d.site_id == ^"s1", limit: 20'
GET /documents?filter.site_id=s1&limit=20

The Core Question You Are Answering

“How do you extend a framework without pretending a remote backend provides semantics it does not have?”

Concepts You Must Understand First

Ecto adapter behaviours — Which responsibilities belong to base, schema, queryable, and transaction contracts? Book Reference: “Programming Ecto” - Repositories and Adapters.
Protocol impedance mismatch — How do relational queries map to document HTTP capabilities? Book Reference: “Designing Data-Intensive Applications” - Data Models and Query Languages.

Questions to Guide Your Design

What exact Ecto subset is supported and how is it versioned?
Which Ecto types round-trip through the remote JSON representation?
How are retries, pagination cursors, partial failures, and rate limits surfaced?

Thinking Exercise

Classify ten Ecto queries as exactly translatable, translatable with documented limits, or unsupported. Never add client-side behavior that silently changes semantics.

The Interview Questions They Will Ask

“What does an Ecto adapter implement?”
“How do loaders and dumpers preserve types?”
“Why should unsupported transactions fail explicitly?”
“How would you test query translation independently of HTTP?”
“What makes an adapter ecosystem-compatible?”

Hints in Layers

Hint 1: Starting Point Define a tiny capability matrix and one type round trip. Hint 2: Next Level Translate normalized query structures into an intermediate request plan. Hint 3: Technical Details Keep HTTP execution separate from query translation and result loading. Hint 4: Tools/Debugging Build a compliance suite before increasing the supported query surface.

Books That Will Help

Topic	Book	Chapter
Data/query models	“Designing Data-Intensive Applications”	Data Models and Query Languages
Ecto contracts	“Programming Ecto”	Repositories, Types, and Adapters

Common Pitfalls and Debugging

Problem 1: “A query returns plausible but incomplete results”

Why: Unsupported filtering was applied after paginating remote data.
Fix: Reject queries whose semantics cannot be pushed down correctly.
Quick test: Compare adapter results with a complete reference dataset across page boundaries.

Definition of Done

Supported adapter capability matrix is explicit
Schema CRUD, filters, ordering, pagination, and types pass compliance tests
Unsupported joins/transactions fail before remote execution
Query translation and HTTP execution test independently
Supervision, retries, rate limits, and error taxonomy are documented

Project 34: Crash-Resilient Session Store with ETS and DETS

File: P34-ets-dets-session-store.md
Main Programming Language: Elixir
Alternative Programming Languages: Erlang, Gleam with Erlang interop
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 3 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Stateful Services, Storage Lifecycle, Recovery
Software or Tool: ETS, DETS, Supervisor, Telemetry
Main Book: “Erlang and OTP in Action”

What you will build: A session and feature-entitlement service whose ETS read model is rebuilt from a DETS-backed durable journal after process or VM failure.

Why it teaches Elixir/BEAM: The project forces you to separate process state, shared in-memory tables, disk authority, supervision, replay, idempotency, and acknowledgement semantics.

Core challenges you will face:

Fast reads with one write authority -> ETS ownership and access modes
Honest durability -> DETS lifecycle, sync policy, repair, and replay
Deterministic restart -> Supervisor ordering and idempotent reconstruction

Real World Outcome

$ mix session_store.drill --scenario vm-kill
accepted session=s-104 revision=7 durable=true
forcing ungraceful VM stop...
restarting service...
dets_open ............... ok
journal_replay .......... 184 records
ets_rebuild ............. 43 active sessions
session=s-104 revision=7 recovered=true
recovery_invariant ...... PASS

The operator can distinguish an ETS-owner crash from a whole-VM restart, show which writes were acknowledged durably, inspect repair evidence after an unclean close, and reproduce the same recovered state from the journal.

The Core Question You Are Answering

“Where is the authoritative state, and what exactly survives an owner crash or VM restart?”

Concepts You Must Understand First

ETS ownership and heirs — Why does table survival inside one VM differ from durable recovery? Book Reference: “Erlang and OTP in Action” - ETS.
DETS persistence — What do file open, sync, close, repair, and size/type limits mean operationally? Book Reference: “Programming Erlang” - ETS and DETS.
Replay and idempotency — How does the same journal record remain safe to apply twice? Book Reference: “Designing Data-Intensive Applications” - Encoding and Evolution.

Questions to Guide Your Design

Is DETS a journal, snapshot, or complete canonical table, and when is a write acknowledged?
Which process owns ETS and DETS, and in what supervisor order do storage, replay, API, and cleanup workers start?
How are expiry, compaction, duplicate commands, corrupt records, and version upgrades represented?

Thinking Exercise

Draw timelines for a crash before DETS write, after write but before acknowledgement, and after acknowledgement but before ETS update. State the externally visible result and recovery action for each cut point.

The Interview Questions They Will Ask

“What happens to ETS when its owner dies?”
“Why is an ETS heir not disk persistence?”
“When would DETS be appropriate, and what are its limits?”
“How do you prevent replay from duplicating an effect?”
“What must start before callers can read reconstructed state?”

Hints in Layers

Hint 1: Starting Point Build the service with one ETS owner and make table loss observable. Hint 2: Next Level Append versioned commands to DETS before applying the ETS projection. Hint 3: Technical Details Track command IDs, revisions, replay checkpoints, and a deterministic expiry clock. Hint 4: Tools/Debugging Kill the owner, kill the VM, truncate only disposable fixtures, and compare recovered state with a pure reference fold.

Books That Will Help

Topic	Book	Chapter
ETS/DETS lifecycle	“Erlang and OTP in Action”	ETS and Data Storage
Recovery logs	“Designing Data-Intensive Applications”	Storage and Retrieval
Supervision	“Elixir in Action”	Building a Concurrent System

Common Pitfalls and Debugging

Problem 1: “The API confirms a session that disappears after restart”

Why: The reply was sent after the ETS update but before the durable record reached the chosen sync boundary.
Fix: Define one acknowledgement invariant and order persistence, projection, and reply around it.
Quick test: Inject a VM halt at every boundary and compare acknowledged revisions with recovered revisions.

Definition of Done

ETS and DETS responsibilities are documented and independently tested
Owner crash and whole-VM restart drills produce deterministic evidence
Every acknowledged durable revision is recovered without duplicate application
DETS limits, sync policy, repair behavior, compaction, and backup are documented
Startup gates callers until replay and ETS reconstruction complete

Project 35: Distributed Maintenance Work-Order Ledger with Mnesia

File: P35-mnesia-work-order-ledger.md
Main Programming Language: Elixir/Erlang
Alternative Programming Languages: Erlang
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Suggested Seniority: Staff
Knowledge Area: Distributed State, Transactions, Operational Recovery
Software or Tool: Mnesia, Distributed Erlang, Supervisor
Main Book: “Designing Data-Intensive Applications”

What you will build: A three-node maintenance work-order ledger that atomically claims work, records status transitions, keeps disk-backed replicas, and supports backup, restore, node replacement, and controlled partition drills.

Why it teaches Elixir/BEAM: You will operate an OTP-native distributed database rather than merely mentioning Mnesia as a persistent version of ETS.

Core challenges you will face:

Atomic claims and transitions -> Mnesia transactions and invariants
Durable replicated tables -> schema, disc_copies, ram_copies, and placement
Operational recovery -> startup order, backup/restore, node loss, and netsplit policy

Real World Outcome

$ mix work_orders.drill --scenario lose-node-and-restore
cluster_members .......... [ops1, ops2, ops3]
order=WO-882 claim ....... committed technician=t-17
stopping ops2 ............ done
read_on_ops1 ............. revision=12 status=in_progress
replace_table_copy ....... complete
backup_restore_check ..... 1250/1250 records
duplicate_claims ......... 0
ledger_invariants ........ PASS

The Core Question You Are Answering

“What guarantees do Mnesia transactions and replicas provide, and which partition/recovery policies remain the application’s responsibility?”

Concepts You Must Understand First

Mnesia schema and table copies — How do ram_copies, disc_copies, and disc_only_copies change recovery and performance? Book Reference: “Erlang and OTP in Action” - Mnesia.
Transactions — Which work-order invariants must be checked and written atomically? Book Reference: “Designing Data-Intensive Applications” - Transactions.
Partitions and recovery — Why is replication not consensus or an automatic conflict policy? Book Reference: “Designing Data-Intensive Applications” - Distributed Data.

Questions to Guide Your Design

Which fields form the immutable audit history and which table stores the current projection?
Where are table copies placed, what is the minimum writable topology, and what happens during a minority partition?
How are backups validated, a lost node replaced, and late/stale updates rejected?

Thinking Exercise

Create a failure matrix for one-node loss, two-node loss, asymmetric partition, disk-full, schema mismatch, and interrupted restore. For every case, state whether reads and writes continue, block, or fail closed.

The Interview Questions They Will Ask

“How is Mnesia different from ETS and DETS?”
“What do Mnesia table copy types mean?”
“What can a transaction guarantee during a network partition?”
“How would you recover or replace a failed Mnesia node?”
“How do you prove a backup is restorable?”

Hints in Layers

Hint 1: Starting Point Start one node and one transactional work-order invariant. Hint 2: Next Level Add disk copies and verify clean/unclean restarts before distribution. Hint 3: Technical Details Separate current state, immutable history, idempotency keys, and operator metadata. Hint 4: Tools/Debugging Script partitions and node removal; never rely on a happy-path cluster dashboard as recovery proof.

Books That Will Help

Topic	Book	Chapter
Mnesia	“Erlang and OTP in Action”	Mnesia
Transactions/replication	“Designing Data-Intensive Applications”	Transactions; Replication
Distributed operations	“Designing for Scalability with Erlang/OTP”	Distributed Architectures

Common Pitfalls and Debugging

Problem 1: “Both sides accept conflicting claims during a partition”

Why: The application treated replicated availability as a conflict-resolution guarantee.
Fix: Define a writable-side policy and reject or queue mutations when required members are unavailable.
Quick test: Partition the claimant nodes, attempt the same claim on both sides, heal, and assert one durable winner.

Definition of Done

Work-order transition and claim invariants execute transactionally
Table copy types and placement match a written durability policy
Clean restart, unclean node loss, replacement, and partition drills are repeatable
Backup is restored into an isolated cluster and compared record-for-record
Split-brain, versioning, capacity, and operator runbooks are explicit

Project 36: Self-Contained Mix Release and Air-Gapped Deployment

File: P36-self-contained-mix-release.md
Main Programming Language: Elixir/Erlang
Alternative Programming Languages: Erlang
Coolness Level: Level 4 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 3 (See REFERENCE.md)
Suggested Seniority: Senior
Knowledge Area: Release Engineering, Runtime Configuration, Deployment
Software or Tool: Mix releases, ERTS, tar, system service manager
Main Book: “Release It!”

What you will build: A versioned, checksummed Mix release tarball that includes ERTS and is deployed, operated, health-checked, and rolled back on a clean compatible host with no Erlang or Elixir toolchain.

Why it teaches Elixir/BEAM: It connects applications, boot scripts, ERTS, runtime configuration, node identity, cookies, signals, remote commands, and native compatibility into one observable deployable artifact.

Core challenges you will face:

Correct artifact assembly -> application graph, ERTS inclusion, and tar step
Late-bound operations -> runtime.exs, RELEASE_* variables, secrets, and commands
Portable within a target contract -> OS/CPU/ABI and native/system-library compatibility

Real World Outcome

clean-host$ command -v elixir || echo absent
absent
clean-host$ command -v mix || echo absent
absent
clean-host$ command -v erl || echo absent
absent
clean-host$ tar -xzf fleet_api-1.4.0-linux-arm64.tar.gz
clean-host$ RELEASE_NODE=fleet_api@127.0.0.1 bin/fleet_api start
clean-host$ bin/fleet_api eval 'Fleet.ReleaseCheck.run()'
boot=ok database=ok runtime_config=ok version=1.4.0
clean-host$ bin/fleet_api stop

The Core Question You Are Answering

“What must a BEAM release contain and prove before a clean target can run it without development tools?”

Concepts You Must Understand First

Mix release anatomy — What do lib, releases, bin, boot scripts, and bundled ERTS contribute? Book Reference: “Elixir in Action” - Releases.
Build-target compatibility — Why does including ERTS not erase OS, architecture, libc/ABI, NIF, or shared-library requirements? Book Reference: “Release It!” - Deployment.
Runtime configuration — Which values belong in build config versus runtime.exs and the environment? Book Reference: “Designing Elixir Systems with OTP” - Applications and Configuration.

Questions to Guide Your Design

Which applications and runtime components must be included, excluded, or started in a particular order?
How will the artifact record version, checksum, target triple, toolchain, configuration contract, and provenance?
What constitutes health, graceful shutdown, remote access, failed boot, rollback, and clean-host proof?

Thinking Exercise

Draw the boundary between build-time facts and boot-time facts. Classify database URLs, cookies, node names, TLS paths, feature flags, native libraries, and migration commands, then explain how each is validated without entering the archive as a secret.

The Interview Questions They Will Ask

“Does a Mix release require Elixir or Erlang on the target?”
“What does including ERTS guarantee, and what does it not guarantee?”
“When is runtime.exs evaluated?”
“How do release commands communicate with a running node?”
“How would you make rollback safe when database migrations exist?”

Hints in Layers

Hint 1: Starting Point Assemble one release and inventory every generated directory. Hint 2: Next Level Add the tar step, checksums, runtime validation, and release health command. Hint 3: Technical Details Pin the build target and document native/system-library dependencies explicitly. Hint 4: Tools/Debugging Test inside a clean compatible VM/container image with PATH checks proving that elixir, mix, and erl are absent.

Books That Will Help

Topic	Book	Chapter
Release construction	“Elixir in Action”	Releases
Deployment safety	“Release It!”	Deployment and Stability
Delivery pipeline	“Continuous Delivery”	Managing Binaries

Common Pitfalls and Debugging

Problem 1: “The self-contained release boots on the builder but not the clean host”

Why: The target differs in architecture/ABI or a NIF/shared library is missing.
Fix: Build for an explicit compatible target and include a native dependency audit in artifact verification.
Quick test: Run the exact tarball in the clean-host fixture before promotion.

Definition of Done

Release is assembled and archived with ERTS included
Checksums, version, target contract, configuration contract, and provenance are recorded
Clean compatible host runs it while elixir, mix, and erl are unavailable
Start, health, eval/remote access, graceful stop, failed boot, and rollback are tested
Native dependencies, secrets, migrations, signals, and service-manager integration are documented

Project 37: Distributed Service Directory and Worker Fleet Router

File: P37-distributed-service-directory.md
Main Programming Language: Elixir/Erlang
Alternative Programming Languages: Erlang, Gleam with Erlang interop
Coolness Level: Level 5 (See REFERENCE.md)
Business Potential: Level 4 (See REFERENCE.md)
Difficulty: Level 4 (See REFERENCE.md)
Suggested Seniority: Staff
Knowledge Area: Process Registration, Discovery, Distributed Routing
Software or Tool: Registry, DynamicSupervisor, :pg, :global, Node monitors
Main Book: “Designing for Scalability with Erlang/OTP”

What you will build: A three-node worker fleet whose services register locally, advertise capabilities through distributed groups, route jobs to monitored remote pids, and recover from worker, node, and membership churn.

Why it teaches Elixir/BEAM: Naming becomes an explicit failure-aware system rather than a hidden assumption behind GenServer.call.

Core challenges you will face:

Local identity -> atom names, Registry keys, :via, unique/duplicate modes
Distributed capability discovery -> :pg membership and controlled :global comparison
Safe routing under churn -> monitors, stale-pid races, deadlines, retry, and idempotency

Real World Outcome

$ mix fleet.drill --nodes 3 --scenario route-during-node-loss
capability=image_resize members=6 nodes=3
job=j-901 route=worker-4@fleet2 accepted=true
disconnecting fleet2...
monitor_down ............. worker-4@fleet2
directory_pruned ......... 2 stale members
job=j-901 reroute=worker-1@fleet1 duplicate_effect=false
fleet2 rejoined .......... membership converged
discovery_invariants ..... PASS

The Core Question You Are Answering

“How does a logical service name become a reachable process, and what happens when that answer becomes stale?”

Concepts You Must Understand First

Local registration — What are the scopes and trade-offs of atom names, Registry, and :via tuples? Book Reference: “Elixir in Action” - Registered Processes.
Distributed groups and names — When does capability membership fit :pg, and when is a global singleton dangerous? Book Reference: “Designing for Scalability with Erlang/OTP” - Distribution.
Monitoring and delivery semantics — Why does discovery not guarantee successful or exactly-once work? Book Reference: “Designing Data-Intensive Applications” - Distributed Systems Trouble.

Questions to Guide Your Design

Is each logical identifier local, cluster-wide unique, or a many-member capability?
How does the router select members, validate liveness, monitor them, and bound retries?
What happens during a partition, rolling restart, duplicate registration, late DOWN, or node rejoin?

Thinking Exercise

Classify six services—local cache owner, tenant worker, image-resize pool, leader-only scheduler, PubSub subscribers, and remote job target—against atom registration, Registry unique/duplicate, :pg, :global, or application routing. Defend the failure behavior of each choice.

The Interview Questions They Will Ask

“Is Elixir Registry distributed across nodes?”
“What is a :via tuple?”
“How does :pg differ from :global?”
“Why can a successful lookup still return a dead pid?”
“How do you retry remote work without duplicate effects?”

Hints in Layers

Hint 1: Starting Point Register local workers through Registry and route by logical key. Hint 2: Next Level Join capability groups and observe membership from three connected nodes. Hint 3: Technical Details Treat membership as a hint, monitor selected pids, correlate replies, and preserve idempotency keys across reroutes. Hint 4: Tools/Debugging Inject worker exits, Node.disconnect, simultaneous re-registration, delayed replies, and rolling version changes.

Books That Will Help

Topic	Book	Chapter
Registration and OTP	“Elixir in Action”	Registered Processes and Supervision
Distributed topology	“Designing for Scalability with Erlang/OTP”	Distributed Architectures
Failure semantics	“Designing Data-Intensive Applications”	The Trouble with Distributed Systems

Common Pitfalls and Debugging

Problem 1: “The router sends work to a pid that discovery just returned, but the call exits”

Why: Registration and group membership are not leases; the process died after lookup.
Fix: Monitor selected targets, use deadlines and classified retries, and preserve a job idempotency key.
Quick test: Kill the chosen worker between lookup and send and assert one logical effect after rerouting.

Definition of Done

Local atom names, Registry, unique/duplicate keys, and :via are demonstrated and compared
Three nodes publish and discover capability groups through :pg
A bounded :global singleton experiment documents partition trade-offs
Worker/node loss, stale lookup, rejoin, duplicate registration, and rolling restart drills pass
Routing preserves deadlines and idempotency and exposes membership/route telemetry

Project Comparison Table

Project	Difficulty	Time	Depth of Understanding	Fun Factor
1. Chat System	Level 2	Weekend+	High	★★★★☆
2. Rate Limiter	Level 2	Weekend+	High	★★★☆☆
3. Distributed KV	Level 3	2-3 weeks	Very High	★★★★☆
4. LiveView Dashboard	Level 2	Weekend+	High	★★★★☆
5. GenStage Pipeline	Level 3	2-3 weeks	Very High	★★★★☆
6. Fault Harness	Level 2	Weekend+	High	★★★☆☆
7. Presence Service	Level 3	2-3 weeks	Very High	★★★★☆
8. ETS Cache	Level 2	Weekend+	High	★★★☆☆
9. Upgrade Drill	Level 3	2-3 weeks	Very High	★★★★☆
10. Telemetry Pipeline	Level 3	2-3 weeks	Very High	★★★★☆
11. Multi-Network Cluster Lab	Level 3	2-3 weeks	Very High	★★★★★
12. WAN Netsplit Recovery	Level 4	3-5 weeks	Expert	★★★★★
13. Federated Edge Event Bus	Level 4	3-6 weeks	Expert	★★★★★
14. Bank Reconciliation CLI	Level 1	1 weekend	Foundation	★★☆☆☆
15. iCalendar Conflict Detector	Level 1	1 weekend	Foundation	★★★☆☆
16. Duplicate File Quarantine	Level 1	1 weekend	Foundation	★★☆☆☆
17. Link and TLS Auditor	Level 2	1-2 weeks	High	★★★☆☆
18. Settlement Parser	Level 2	1-2 weeks	High	★★★☆☆
19. Mix SBOM Auditor	Level 2	1-2 weeks	High	★★★☆☆
20. Pricing Policy DSL	Level 3	2-3 weeks	Very High	★★★★☆
21. Webhook Authenticity Gateway	Level 2	1-2 weeks	High	★★★☆☆
22. Inventory Reservation Ledger	Level 3	2-4 weeks	Very High	★★★☆☆
23. Object Storage Library	Level 2	2-3 weeks	High	★★★☆☆
24. Oban Document Workflow	Level 3	2-4 weeks	Very High	★★★★☆
25. TCP Device Gateway	Level 3	3-4 weeks	Very High	★★★★☆
26. Nerves Cold-Chain Gateway	Level 3	3-5 weeks	Very High	★★★★★
27. Rustler Perceptual-Hash NIF	Level 4	3-5 weeks	Expert	★★★★★
28. Compiler-Aware Linter	Level 3	3-4 weeks	Very High	★★★★☆
29. Nx Forecasting Engine	Level 3	3-5 weeks	Very High	★★★★☆
30. :gen_statem Escrow	Level 4	3-5 weeks	Expert	★★★★☆
31. Dynamic Repo Control Plane	Level 4	3-6 weeks	Expert	★★★★☆
32. BEAM Artifact Auditor	Level 4	3-5 weeks	Expert	★★★★☆
33. Custom Ecto HTTP Adapter	Level 5	5-8 weeks	Principal	★★★★★
34. ETS/DETS Session Store	Level 3	3-5 weeks	Very High	★★★★☆
35. Mnesia Work-Order Ledger	Level 4	5-8 weeks	Expert	★★★★★
36. Self-Contained Mix Release	Level 3	3-5 weeks	Very High	★★★★☆
37. Distributed Service Directory	Level 4	4-6 weeks	Expert	★★★★★

Recommendation

If you are new to Elixir: Start with Project 14, then complete Projects 15, 16, and 18 before introducing long-lived processes.

If you know Elixir syntax but not OTP: Move from Project 17 to Projects 1, 5, 25, and 30.

If you build production applications: Focus on Projects 21, 22, 24, and 31 for boundary security, integrity, durable work, and resource isolation.

If you want state and recovery mastery: Move from Project 8 to Projects 34 and 35, proving owner, VM, node, partition, backup, and restore behavior.

If you want distributed systems mastery: Focus on Projects 3, 7, 11, 12, 13, and 37.

If you want release engineering depth: Complete Projects 9, 32, and 36 to distinguish hot upgrades, artifact inspection, and self-contained deployment.

If you want staff/principal ecosystem depth: Complete Projects 27, 28, 32, and 33 and be prepared to defend failure semantics, compatibility, and operational limits.

Final Overall Project: Multi-Tenant BEAM Reliability Platform

The Goal: Combine Projects 21, 22, 24, 30, 31, 34, 36, 37, 10, and 32 into a multi-tenant operational platform whose boundaries, workflows, runtime behavior, discovery, durable recovery, and release artifacts are all auditable.

Authenticate external events through the webhook gateway and preserve rejection evidence.
Apply accepted commands to an inventory ledger whose database constraints prevent overselling.
Run conversion and notification effects through idempotent Oban workflows.
Model long-running approvals with explicit :gen_statem transitions and durable deadlines.
Route each request and job to a bounded tenant-specific Repo.
Expose runtime and workflow signals through the telemetry dashboard.
Rebuild hot session state from a durable journal after process and VM failures.
Discover tenant workers across nodes and reroute safely when membership becomes stale.
Produce two release builds, audit every difference, and deploy the selected tarball to a clean compatible host.

Success Criteria: Under duplicate delivery, concurrent reservations, worker crashes, tenant database outages, and node restarts, the platform preserves ledger and tenant-isolation invariants, resumes workflows without duplicate external effects, exposes recovery evidence, and produces explainable release artifacts.

From Learning to Production: What Is Next

Your Project	Production Equivalent	Gap to Fill
Project 1	Phoenix Presence + Channels	Auth, persistence, scaling
Project 3	Riak / Mnesia-based store	Replication, partition handling
Project 11	Multi-region Erlang cluster with secure distribution	PKI rotation, network policy automation
Project 12	Partition-healing workflow for distributed state	Formal conflict-resolution policy + SLOs
Project 4	LiveView ops dashboards	Auth, multi-tenant UI
Project 9	Release handling pipelines	CI/CD automation
Project 14	Finance operations reconciliation tool	Institution-specific importers, approvals, access control
Project 19	Software supply-chain policy gate	Vulnerability feeds, signature verification, exception workflow
Project 21	Managed webhook security gateway	Key rotation, provider onboarding, tenant isolation, SLOs
Project 22	Inventory/booking ledger	Formal isolation policy, audit retention, capacity planning
Project 24	Document automation platform	Sandboxing, malware scanning, cost controls, operator UI
Project 25	IoT protocol gateway	Mutual authentication, fleet identity, regional sharding
Project 26	Managed Nerves appliance fleet	Secure provisioning, signed OTA service, hardware certification
Project 27	Production native extension	Multi-platform builds, fuzzing, crash containment strategy
Project 29	Forecasting service	Drift monitoring, model registry, human override workflow
Project 31	Database-per-tenant control plane	Global admission, billing, backup/restore, compliance evidence
Project 32	Release provenance auditor	Signed attestations, reproducible builders, policy enforcement
Project 33	Production Ecto adapter	Broad compatibility suite, long-term version policy, support model
Project 34	Durable local session/entitlement service	Encryption, storage rotation, capacity alarms, privacy retention
Project 35	Replicated maintenance/operations ledger	Formal partition policy, geographic recovery, migration automation
Project 36	Self-contained release pipeline	Hermetic multi-target builders, signing, SBOM/attestations, automated rollback
Project 37	Cluster service directory and router	Secure discovery topology, global admission, multi-region policy, SLOs

Summary

This learning path covers Elixir, Erlang/OTP, and the BEAM ecosystem through 37 hands-on projects ranging from junior-friendly transformation tools to principal-level framework extensions and staff-level recovery, release, and discovery drills.

#	Project Name	Main Language	Difficulty	Time Estimate
1	Supervised Chat System	Elixir	Level 2	10-15 hrs
2	Rate Limiter + Circuit Breaker	Elixir	Level 2	12-18 hrs
3	Distributed KV Store	Erlang/Elixir	Level 3	20-30 hrs
4	LiveView Dashboard	Elixir	Level 2	12-20 hrs
5	GenStage Backpressure Pipeline	Elixir	Level 3	18-25 hrs
6	Fault Injection Harness	Elixir	Level 2	10-15 hrs
7	Presence Service	Erlang/Elixir	Level 3	20-30 hrs
8	ETS Cache Service	Erlang/Elixir	Level 2	10-15 hrs
9	Hot Code Upgrade Drill	Erlang	Level 3	15-25 hrs
10	Telemetry Pipeline	Elixir	Level 3	15-25 hrs
11	Multi-Network Cluster Formation Lab	Elixir/Erlang	Level 3	18-28 hrs
12	WAN Netsplit Recovery Drill	Elixir/Erlang	Level 4	24-36 hrs
13	Federated Edge Event Bus	Elixir	Level 4	24-40 hrs
14	Bank Statement Reconciliation CLI	Elixir	Level 1	6-10 hrs
15	iCalendar Conflict Detector	Elixir	Level 1	8-12 hrs
16	Duplicate File Quarantine Planner	Elixir	Level 1	8-12 hrs
17	Concurrent Link and TLS Expiry Auditor	Elixir/Erlang	Level 2	10-16 hrs
18	Fixed-Width Settlement Parser	Elixir	Level 2	12-18 hrs
19	Mix SBOM and License Auditor	Elixir	Level 2	12-18 hrs
20	Compile-Time Pricing Policy DSL	Elixir	Level 3	16-24 hrs
21	Webhook Authenticity Gateway	Elixir	Level 2	12-18 hrs
22	Inventory Ledger and Reservation API	Elixir	Level 3	18-28 hrs
23	Pluggable Object Storage Library	Elixir	Level 2	14-22 hrs
24	Oban Document Workflow	Elixir	Level 3	18-26 hrs
25	TCP Device Telemetry Gateway	Elixir/Erlang	Level 3	20-30 hrs
26	Nerves Cold-Chain Sensor Gateway	Elixir	Level 3	20-32 hrs
27	Rustler Perceptual-Hash NIF	Elixir/Rust	Level 4	24-36 hrs
28	Compiler-Aware Migration Linter	Elixir	Level 3	20-30 hrs
29	Nx Demand Forecasting Engine	Elixir	Level 3	20-32 hrs
30	:gen_statem Escrow Workflow	Elixir/Erlang	Level 4	24-36 hrs
31	Multi-Tenant Dynamic Repo Control Plane	Elixir	Level 4	24-40 hrs
32	BEAM Artifact Reproducibility Auditor	Elixir/Erlang	Level 4	22-34 hrs
33	Custom Ecto HTTP Document Adapter	Elixir	Level 5	35-60 hrs
34	Crash-Resilient ETS/DETS Session Store	Elixir	Level 3	24-36 hrs
35	Distributed Mnesia Work-Order Ledger	Elixir/Erlang	Level 4	35-55 hrs
36	Self-Contained Mix Release Deployment	Elixir/Erlang	Level 3	22-34 hrs
37	Distributed Service Directory and Fleet Router	Elixir	Level 4	28-42 hrs

Expected Outcomes

Design supervision trees that isolate failures
Build distributed BEAM services with clear failure semantics
Implement backpressure pipelines and real-time dashboards
Operate multi-network clusters and recover deterministically from partitions
Build deterministic transformation and binary-parsing tools with explicit failure contracts
Design behaviours, protocols, macros, Mix tasks, compiler tracers, and Hex packages
Protect relational invariants and make durable workflows safe to retry
Engineer HTTP, TCP, native, embedded, and numerical boundaries by blast radius
Extend Ecto honestly through a tested adapter capability contract
Recover supervised services from DETS and Mnesia with explicit durability and partition policies
Ship an ERTS-included release and prove it runs without a target-side Elixir/Erlang installation
Register, discover, monitor, and reroute local and distributed processes safely

Additional Resources and References

Standards and Specifications

Industry Analysis

Books

“Programming Erlang” by Joe Armstrong - the classic reference
“Elixir in Action” by Sasa Juric - practical OTP patterns
“Designing for Scalability with Erlang/OTP” by Francesco Cesarini and Steve Vinoski - supervision and reliability
“Domain Modeling Made Functional” by Scott Wlaschin - explicit domain states and errors
“Enterprise Integration Patterns” by Gregor Hohpe and Bobby Woolf - idempotency and durable messaging patterns
“Designing Data-Intensive Applications” by Martin Kleppmann - transactions, consistency, and data-model trade-offs
“Release It!” by Michael T. Nygard - production failure and stability patterns