Project 37: Distributed Service Directory and Worker Fleet Router

Build an operator-visible directory that discovers capability-specific workers across a BEAM cluster, routes jobs to current process instances, and stays honest about stale PIDs, node partitions, and the different guarantees of Registry, :pg, and :global.

Quick Reference

Attribute	Value
Difficulty	Level 4: Advanced
Suggested Seniority	Staff Elixir or distributed-systems engineer
Time Estimate	28-42 hours
Main Programming Language	Elixir
Alternative Programming Languages	Erlang for direct OTP experiments; Gleam for a typed client
Coolness Level	Level 5: Pure Magic
Business Potential	Level 4: Open Core Infrastructure
Prerequisites	Processes and mailboxes, GenServer, Supervisor, DynamicSupervisor, node naming, distributed Erlang fundamentals, failure injection
Key Topics	process registration, GenServer names, local Registry, `:via`, `:pg`, `:global`, remote PIDs, DynamicSupervisor, process monitors, node monitoring, stale-PID races, service discovery

1. Learning Objectives

By completing this project, you will:

Distinguish a module, a process, a PID, a registered name, a service capability, and a distributed node identity.
Register bounded static processes with atom names and dynamic process instances with a local Registry.
Choose unique or duplicate Registry keys and use {:via, Registry, ...} without creating atoms from external identifiers.
Pair DynamicSupervisor lifecycle management with Registry-based lookup without confusing supervision with discovery.
Connect multiple BEAM nodes and send work to a remote PID or a locally registered name on a named remote node.
Discover interchangeable workers through distributed :pg capability groups and design for temporarily divergent membership views.
Compare :global singleton registration with :pg group membership under normal operation and a controlled partition.
Handle process death, node loss, re-registration, stale lookups, duplicate starts, and rediscovery without caching a PID as permanent identity.
Expose cluster and process-discovery state through an actionable operator CLI and telemetry.
Explain why node discovery, process registration, service discovery, supervision, and durable state recovery solve different problems.

2. All Theory Needed (Per-Concept Breakdown)

Local Process Identity, Registration, and Lifecycle

Fundamentals

A module is code, while a process is a runtime execution context with a mailbox, heap, and lifecycle. A PID identifies one concrete process incarnation; when that process dies, a restarted child receives a different PID. Registration adds an indirect name so callers can locate the current process without retaining its PID forever. Process.register/2 and an atom-valued GenServer :name register one live local process under one bounded atom. Elixir’s Registry supports dynamic term keys, unique or duplicate membership, associated metadata, and {:via, Registry, ...} names, but it remains local to one node. A supervisor owns restart policy, while a registry owns lookup information. Neither mechanism persists business data. Correct systems therefore separate logical identity, current PID, lifecycle ownership, and durable state.

Deep Dive

Start with the identity layers. ThumbnailWorker is a module containing callbacks and public API functions. A worker process running those callbacks has a PID such as #PID<...>. The logical worker identity might be worker-17, while the service capability might be {thumbnail, version_2}. These are not interchangeable. The module can exist when no worker is running; the logical identity can survive many process incarnations; and a capability can be served by many processes. Treating the PID as the business identity creates stale references after every crash or deployment.

The simplest registration mechanism is a local atom name. Process.register/2 binds one live local PID or port to one atom, and a process can have only one such registered name. GenServer exposes this through its :name option. A fixed infrastructure process such as Fleet.TopologyMonitor can sensibly use its module atom as a local name because the set of names is bounded at compile time. Dynamic worker IDs must not become atoms. Atoms are not ordinarily garbage-collected, so converting untrusted device, tenant, job, or worker strings into atoms can exhaust the VM atom table.

GenServer accepts four useful reference forms: a direct PID, a locally registered atom, {registered_name, node} for a local registration on another connected node, and registration abstractions such as {:global, term} or {:via, module, term}. The client API should accept a logical reference and hide these details. This keeps a future naming change from leaking across the application and gives the client one place to classify :noproc, timeout, and :nodedown outcomes.

Elixir’s Registry is a local, decentralized, scalable key-value process store. A unique registry maps a key to at most one PID and is suitable for dynamic names such as {worker, worker_id}. A duplicate registry allows multiple processes under one key and can model local subscriptions or capabilities. Only unique registries work as a :via name because a GenServer call must resolve to one destination. Values attached to entries are routing metadata, not durable records. Keep them compact and safe to expose in diagnostics.

Registry entries belong to the processes that register them. When an owner terminates, its entries are removed automatically. Cleanup does not eliminate the fundamental race: a lookup may return a PID that dies before the next instruction sends a message. No registry can make lookup -> use atomic with the lifetime of another process. Callers must tolerate a dead destination, use synchronous calls with bounded timeouts where appropriate, or create a monitor and handle an immediate :DOWN if the PID has already died. A monitor reports lifecycle; it does not retry work or prove whether a message was processed.

DynamicSupervisor and Registry complement one another. The dynamic supervisor starts and restarts children according to child specifications; Registry makes current children addressable. Registry does not supervise a worker, and DynamicSupervisor does not provide a domain-key lookup index. Starting a missing worker creates a race when two callers both observe no registration and request a child. Resolve this with one explicit ownership protocol: accept an already_started or already_registered result and resolve the winner, serialize creation under a bounded lock, or route all creation through a single partition owner. Never let two apparent owners perform irreversible work while the naming conflict is being resolved.

A registry restart also matters. Local Registry data is runtime state. If the registry process exits, its table contents are not a durable catalog that can simply be trusted after restart. Design the supervision order so workers and their registry have a coherent lifecycle, and decide whether workers re-register, are restarted with the registry, or are reconciled from a durable desired-state source. In this project, the local worker subtree and registry share a failure boundary, while desired worker specifications are deterministic lab configuration rather than Registry metadata.

Generation identity makes stale references observable. Record a random or monotonic generation token when each worker starts, return it in directory output, and include it in telemetry. A route decision reports logical worker ID, node, PID, and generation. After a crash, the logical ID can remain stable while PID and generation change. Tests can then prove that rediscovery happened instead of accidentally reusing a cached destination.

Registration is not persistence. If a process keeps an in-memory count and crashes, registration makes the new process discoverable but does not restore the count. A supervisor restarts code from its start arguments. Durable recovery requires a database, journal, snapshot, or other persistent source plus an explicit recovery protocol. This project persists only job request identity for deduplication in the simulated work adapter; Projects 24 and 30 cover deeper durable workflow recovery.

How this fits on the project

This concept defines the node-local worker subtree, fixed infrastructure names, dynamic :via names, unique and duplicate Registry experiments, start-race handling, monitoring, generation identity, and operator lookup output.

Definitions and key terms

PID: Runtime identifier for one process incarnation.
Logical identity: Stable domain key that can survive process restarts.
Registered name: Indirection from a local or distributed name mechanism to a current PID.
Unique registry: Registry in which one key resolves to zero or one process.
Duplicate registry: Registry in which one key can have multiple process entries.
via tuple: Generic naming contract used by OTP behaviours through a registry module.
Generation: Token distinguishing successive incarnations of one logical worker.
Stale PID: PID retained after its process has terminated or become unreachable.

Mental model diagram

                 stable concepts                       volatile runtime

 [module code]   [logical worker id]   [capability]      [PID + generation]
      |                  |                   |                    |
      |            unique Registry key      |              one incarnation
      |                  |                   |                    |
      +------ start specification ----------+----> DynamicSupervisor
                             |                              |
                             +--- lookup current PID <------+

 Supervisor: starts/restarts processes
 Registry:   resolves current local process entries
 Monitor:    reports termination
 Database:   would restore durable business truth; Registry does not

How it works

Start the local Registry before the dynamic worker supervisor.
Start a worker from a bounded child specification containing logical ID and capabilities.
Register the worker under a term key through {:via, Registry, ...}.
Attach compact diagnostic metadata including generation and capability list.
Resolve the logical ID immediately before a call rather than caching its PID indefinitely.
Monitor the chosen worker during routed work and classify :DOWN separately from call timeout.
On crash, let the supervisor apply restart policy and let the new process claim the same logical key with a new PID and generation.
Invariant: at most one live local worker owns a unique logical key.
Failure modes include dynamic-atom growth, duplicate starts, stale PID use, registry/worker lifecycle mismatch, and assuming registration restores state.

Minimal concrete example

NAMING PLAN, NOT RUNNABLE CODE

fixed infrastructure name:
  Fleet.TopologyMonitor

dynamic worker via name:
  {:via, Registry, {Fleet.LocalWorkers, {:worker, "worker-17"}}}

directory result:
  logical_id="worker-17"
  pid=#PID<...>
  generation="g-0042"
  capabilities=[{"thumbnail", 2}, {"metadata", 1}]

after crash:
  logical_id remains "worker-17"
  pid and generation must change

Common misconceptions

A registered name is a durable record.
A PID remains valid when a supervisor restarts a child.
Registry is distributed across connected nodes.
DynamicSupervisor automatically provides lookup by business key.
A successful lookup guarantees the process will still be alive when called.
Creating atoms from worker IDs is acceptable because old workers eventually stop.

Check-your-understanding questions

Why is a worker ID different from its PID?
When is an atom-valued process name appropriate?
Why can Registry.lookup/2 still return a destination that fails immediately?
What distinct jobs do DynamicSupervisor and Registry perform?
What must be added if process state has to survive VM restart?

Check-your-understanding answers

The logical ID survives restarts; a PID names only one runtime incarnation.
For a bounded, trusted set of static infrastructure names—not unbounded external identifiers.
The process can terminate between lookup and use; distributed scheduling cannot make that pair atomic.
DynamicSupervisor controls lifecycle, while Registry maps keys to current local PIDs and metadata.
A durable store plus explicit load, replay, or reconciliation logic.

Real-world applications

Device and socket session ownership
Per-customer runtime workers
Media conversion fleets
Game match processes
Local job executors and connection managers

Where you will apply it

Project 37 local worker supervisor, Registry, router, start coordinator, monitor, and failure drills

References

Key insight

A name locates the current process incarnation; it does not turn that incarnation into durable identity.

Summary

Model code, logical identity, PID, registration, supervision, and persistence independently. Use bounded atom names for static infrastructure, Registry terms for dynamic local identities, DynamicSupervisor for lifecycle, monitors for termination evidence, and a separate durable mechanism when business state must survive.

Homework/Exercises

Classify eight example names as safe fixed atoms or unsafe dynamic atoms.
Draw the two-caller race that starts the same worker ID and choose a winner-resolution policy.
Write a state table for lookup success followed by normal exit, crash, or node loss before the call.

Solutions

Module-owned infrastructure names are bounded; customer IDs, filenames, UUIDs, device IDs, and request parameters remain strings or tuples.
Both callers may see no entry; one child claims the unique key, while the loser resolves and returns the winner rather than starting irreversible duplicate work.
Every lookup outcome is provisional: a live call may succeed, a dead local PID produces :noproc/:DOWN, and remote loss produces a node/disconnection failure that must be classified and retried only under an idempotency policy.

Distributed Nodes, Remote Processes, and Service Discovery

Fundamentals

Node discovery and process discovery are separate layers. Node discovery identifies BEAM nodes and establishes distribution connections; process discovery locates current processes after those connections exist. Connected nodes can exchange messages using remote PIDs, {registered_name, node} references, or higher-level OTP APIs. :pg supplies distributed named groups for zero or more interchangeable processes, with strongly eventually consistent membership that may temporarily differ across nodes. :global supplies a single distributed name with replicated lookup tables and assumes a reliably fully connected network. Node monitors report nodeup and nodedown; process monitors report one process incarnation ending. A robust router uses these signals as evidence, not as proof that the next message will succeed.

Deep Dive

A distributed Erlang node is a VM with a node name and distribution subsystem. Nodes can connect explicitly or because one references another. The normal lab can use stable names and explicit seeds, while a deployment adapter may source candidates from DNS, orchestration APIs, or static configuration. That adapter only returns candidate node names. It does not discover worker processes. After connection, Node.list/0 describes currently visible peers and :net_kernel.monitor_nodes can deliver nodeup and nodedown events with useful reason metadata.

Distribution makes remote messaging syntactically familiar but not failure-free. A remote PID contains node identity and can be sent messages while the connection exists. A local atom registration on a remote node can be addressed as {name, node}. Remote spawn and distributed Task APIs exist, but production ownership is usually clearer when the remote node’s own supervisor starts its workers. Code versions also matter: sending a function or calling a module that is absent or incompatible on the remote node can fail even when transport is healthy.

Node membership is not worker membership. A node may be connected yet have no thumbnail workers, may be draining, or may serve only an incompatible protocol version. Conversely, a worker can die while its node remains healthy. The service directory therefore models node state and process capability membership separately. nodeup triggers synchronization and probes; it does not imply every service is ready. nodedown invalidates candidates from that node and ends monitors, but an in-flight job still needs outcome classification.

:pg models distributed named process groups. A process joins from its local node, and membership changes propagate to directly connected peers. A worker may join several groups, such as {:service, :thumbnail, 2} and {:zone, :us_east}. Clients obtain members and choose one or more destinations themselves; :pg is not a load balancer and does not send the application message. When a member dies it is removed automatically. Empty groups need no pre-creation ceremony.

The consistency model matters. :pg provides strong eventual consistency: membership views can temporarily diverge when joins or leaves race across nodes. Visibility is not transitive through an intermediate node; nodes that are not directly connected do not necessarily share the same view even if both connect to a third node. The router must not claim a globally instantaneous catalog. It may show per-node observation time, membership source, and convergence status. A stale member can still be returned, so routing retains the same lookup/use race as a local registry.

Group keys should express compatibility, not only a vague service name. Route a version-two thumbnail request to {:service, :thumbnail, 2}, not every process that once called itself a thumbnail worker. Add metadata through a separate local directory record or a capability response rather than making enormous group terms. A member joining a group is an availability advertisement, not an authorization decision. The router still validates that the request is allowed and that the remote node is trusted.

:global solves a different problem. It associates one term with one PID across a distributed system and replicates name tables so lookup can be local. Registration is synchronized across the involved fully connected network, and name conflicts require resolution. Its guarantees depend on the network remaining fully connected under the Kernel settings described by the official documentation. That makes it unsuitable as a casual replacement for every local Registry or for large, partially connected, partition-prone topologies.

This project uses :global only in a controlled experiment for one non-durable lab coordinator. Observe registration, lookup, process death cleanup, competing registration, and a simulated partition. Document what the application does if the singleton becomes unreachable or a partition changes the viable component. Do not call :global a consensus system, durable leader election, or exactly-once owner. A registered singleton can die after lookup, and global naming does not persist its state.

The service router is policy layered over discovery. It queries the versioned :pg group, filters members from draining or recently failed nodes, optionally prefers local capacity, and chooses a candidate by a deterministic policy such as round-robin over a stable ordering. It monitors the selected PID, sends a request carrying a stable request ID, and waits under an overall deadline. If failure happens before an acknowledgement, a retry may select another member only if the operation is idempotent or deduplicated. If a timeout happens after remote execution, the outcome is uncertain; blindly dispatching again may duplicate work.

Do not retain service PIDs in a long-lived cache without invalidation. A short-lived route snapshot can reduce repeated membership queries, but every entry needs node/PID monitoring, an expiry, and a generation-aware diagnostic. The simplest correct first implementation queries :pg for each dispatch. Optimize only after measuring. Membership lookup is generally cheaper than debugging a stale routing cache.

Failure drills reveal the contracts. Killing one worker should remove its local Registry entry and eventually its :pg membership, while its supervisor starts a new PID that rejoins. Disconnecting one node should cause nodedown, invalidate its candidates, and leave workers on other nodes routable. Reconnection should produce nodeup and rediscovery without assuming old PIDs revive. A controlled partition should show temporarily different membership views rather than fabricating one authoritative list.

Secure distribution is mandatory outside an isolated loopback lab. The cookie is not an application authorization model, and the official net_kernel documentation warns that starting distributed nodes without TLS distribution can expose the cluster to complete compromise. The project runbook must require TLS distribution, restricted network reachability, stable certificate identity, and secret handling for non-local environments. Never expose the distribution port directly to the public internet.

How this fits on the project

This concept defines node candidate adapters, connection and node monitoring, remote addressing, :pg capability membership, the :global comparison experiment, route selection, delivery classification, convergence reporting, and network failure drills.

Definitions and key terms

Node discovery: Finding candidate BEAM node names and establishing connections.
Process discovery: Resolving current PIDs by logical identity or service capability.
Remote PID: PID whose process lives on another connected node.
Capability group: :pg group whose name describes a compatible service contract.
Strong eventual consistency: Membership views may diverge temporarily but converge after updates propagate under suitable connectivity.
Fully connected network: Topology in which participating nodes directly maintain the connectivity expected by global naming.
Uncertain outcome: Caller cannot determine whether remote work completed before communication failed.
Rediscovery: Resolving a new current PID rather than reviving a stale one.

Mental model diagram

             NODE DISCOVERY / FORMATION
  [seed config or DNS] -> connect -> nodeup/nodedown -> [node topology]
                                      |
                                      v
               PROCESS / SERVICE DISCOVERY

 node a                         node b                         node c
+------------------+       +------------------+       +------------------+
| DynamicSupervisor|       | DynamicSupervisor|       | DynamicSupervisor|
|  worker a1       |       |  worker b1       |       |  worker c1       |
|  worker a2       |       |                  |       |  worker c2       |
| Local Registry   |       | Local Registry   |       | Local Registry   |
+--------+---------+       +--------+---------+       +--------+---------+
         |                          |                          |
         +--------- join capability-specific :pg group ------+
                                    |
                         [fleet router on any node]
                                    |
                    choose + monitor current remote PID
                                    |
                            dispatch idempotent job

 controlled comparison only:
   {:global, :lab_coordinator} -> zero or one registered PID

How it works

Load candidate node names from the lab discovery adapter.
Connect and subscribe to node state events.
Start each node’s local Registry, DynamicSupervisor, workers, and :pg scope.
Register each worker locally by logical ID and join versioned capability groups.
Query group members at dispatch time and filter incompatible or draining candidates.
Select, monitor, and call a current PID with request ID and deadline.
Classify reply, process death, node loss, timeout, and uncertain outcome separately.
On worker or node recovery, discover new PIDs from fresh Registry/:pg state.
Run the :global singleton experiment separately and record partition assumptions.
Invariant: directory output labels observed state and never claims instantaneous global truth.
Failure modes include node-name mismatch, insecure distribution, partial connectivity, divergent membership, stale PIDs, incompatible capabilities, and duplicate work after ambiguous timeout.

Minimal concrete example

DISCOVERY TRANSCRIPT, NOT RUNNABLE CODE

node candidates:
  fleet-a@127.0.0.1, fleet-b@127.0.0.1, fleet-c@127.0.0.1

:pg group:
  {:service, :thumbnail, 2}

observed members on fleet-a:
  [{fleet-a, pid-a1, generation=g7},
   {fleet-b, pid-b1, generation=g3}]

route(job-42):
  selected pid-b1 on fleet-b
  monitor installed
  request_id=job-42/thumbnail-v2

after fleet-b disconnect:
  invalidate node=fleet-b
  rediscover -> pid-a1
  retry only if request identity is deduplicated

Common misconceptions

Connecting nodes automatically discovers every service process.
Registry entries are visible cluster-wide.
:pg sends messages or load-balances for the application.
Every node sees the same :pg membership at every instant.
:global is a durable consensus or leader-election system.
A nodedown event proves the remote job did not complete.
The distribution cookie alone is sufficient security for an internet-facing cluster.

Check-your-understanding questions

What does node discovery answer that process discovery does not?
Why should capability group names include protocol version?
What consistency behavior must a :pg client tolerate?
When is retrying after a remote timeout safe?
Why is :global only a comparison mechanism in this project?

Check-your-understanding answers

It identifies and connects runtime nodes; it does not identify which worker processes currently serve a capability.
It prevents routing requests to workers with an incompatible message or result contract.
Temporarily divergent membership, stale members, and visibility limited by direct connectivity.
Only when the work is idempotent or deduplicated with a stable request identity and the retry policy respects the overall deadline.
Its singleton naming assumes reliable full connectivity and does not provide durable state, application consensus, or exactly-once ownership.

Real-world applications

Media processing worker fleets
Multi-node device session routing
Distributed game-server match placement
Protocol-version-aware internal service directories
Cluster operation and failure-drill tooling

Where you will apply it

Project 37 topology monitor, group membership reporter, worker router, coordinator comparison, and partition lab

References

Key insight

Cluster connectivity tells you where processes may communicate; service discovery tells you which current processes can perform compatible work.

Summary

Form and monitor the node network first, then discover process capabilities with mechanisms whose guarantees match the topology. Use local Registry for dynamic local identity, :pg for distributed many-member capability discovery, and :global cautiously for a fully connected singleton naming problem. Assume every PID and membership view can become stale.

Homework/Exercises

Draw node and process discovery timelines for a worker crash without node loss.
Draw the same timelines for a node partition after a job acknowledgement is lost.
Choose between Registry, :pg, and :global for five naming scenarios and justify each.

Solutions

Node topology stays unchanged; the old Registry/:pg membership disappears, supervision starts a new PID, and the new process registers and rejoins.
nodedown invalidates the node locally, but remote execution may have completed; the router uses request deduplication before retrying and later reconciles membership after reconnection.
Static local singleton: atom name. Dynamic local identity: unique Registry. Local fan-out: duplicate Registry. Distributed interchangeable workers: :pg. One non-durable name in a reliably full mesh: potentially :global, after documenting partition behavior.

3. Project Specification

3.1 What You Will Build

Build a three-node worker-fleet lab and operator CLI. Each node owns a local Registry and DynamicSupervisor. Workers represent real media operations—thumbnail generation, metadata extraction, and document preview—and advertise protocol-versioned capabilities through :pg. A router discovers a compatible current worker, monitors it, and submits a request with a stable identity. A topology monitor records nodeup/nodedown; failure commands crash workers, disconnect nodes, reconnect them, and compare membership views. A separate experiment registers one disposable lab coordinator through :global and exposes its full-connectivity assumptions.

3.2 Functional Requirements

Start three named BEAM nodes from repeatable lab configuration.
Expose a node-candidate adapter separate from the process directory.
Start a local Registry, DynamicSupervisor, worker subtree, topology monitor, and CLI endpoint on every node.
Register fixed infrastructure with bounded atom names and workers with dynamic term keys through :via.
Demonstrate unique and duplicate local Registry semantics without using dynamic atoms.
Join workers to versioned :pg groups for every supported capability.
List nodes, local registrations, observed distributed groups, PIDs, generations, and observation time.
Route a media job to a compatible local or remote PID under a deterministic selection policy.
Monitor selected processes and classify process crash, node loss, timeout, incompatibility, and uncertain outcome.
Deduplicate simulated media jobs using a stable request ID before allowing retry.
Crash and restart one worker, proving PID/generation replacement and rediscovery.
Disconnect and reconnect one node, proving candidate invalidation and service rediscovery.
Show per-node :pg views during a controlled partition and after convergence.
Run a bounded :global coordinator comparison, including duplicate registration and process-death cleanup.
Require TLS distribution and restricted connectivity for every non-loopback deployment profile.

3.3 Non-Functional Requirements

Correctness: Directory output never presents observed membership as durable or instantly global truth.
Resilience: Remaining compatible workers continue serving when one worker or node fails.
Safety: Retried work is idempotent or deduplicated; dynamic external IDs never become atoms.
Security: Non-local distribution uses TLS and redacts cookies, certificates, and request payloads from logs.
Observability: Every route records logical request, capability, selected node/PID/generation, result class, and timing.
Operability: All drills have deterministic commands, expected output, and cleanup steps.
Scalability: Registry partitions and router lookup policy can be measured before optimization.

3.4 Example Usage / Output

$ fleetctl nodes
NODE                       STATUS   SINCE                 SOURCE
fleet-a@127.0.0.1          up       2026-07-15T16:00:00Z  local
fleet-b@127.0.0.1          up       2026-07-15T16:00:02Z  static_seed
fleet-c@127.0.0.1          up       2026-07-15T16:00:03Z  static_seed

$ fleetctl services thumbnail --version 2
OBSERVED_FROM=fleet-a@127.0.0.1 CONSISTENCY=strong_eventual MEMBERS=3
worker=thumb-a1 node=fleet-a@127.0.0.1 pid=#PID<...> generation=g7 state=ready
worker=thumb-b1 node=fleet-b@127.0.0.1 pid=#PID<...> generation=g3 state=ready
worker=thumb-c1 node=fleet-c@127.0.0.1 pid=#PID<...> generation=g9 state=ready

$ fleetctl route thumbnail job-42 --version 2
request=job-42/thumbnail-v2 selected=thumb-b1 node=fleet-b@127.0.0.1 generation=g3
result=ok artifact=artifacts/job-42-thumb-v2.jpg duration_ms=84 deduplicated=false

3.5 Data Formats / Schemas / Protocols

Node observation

node: atom node identity from trusted configuration
state: connecting | up | down | quarantined
source: local | static_seed | dns_adapter
observed_at: UTC timestamp
connection_id: runtime diagnostic token
down_reason: optional classified reason

Worker descriptor

logical_id: bounded UTF-8 string
node: node identity
pid: runtime PID, never persisted
generation: per-start opaque token
capabilities: [{service, protocol_version}]
state: warming | ready | draining
started_at: UTC timestamp

Route request

request_id: stable string
capability: {service, protocol_version}
input_ref: immutable fixture or artifact reference
deadline_ms: overall remaining budget
attempt: positive integer for diagnostics

Route result classes

ok | unsupported | no_members | process_down | node_down |
timeout_uncertain | rejected | deduplicated_result

3.6 Edge Cases

Two callers request creation of the same logical worker simultaneously.
Lookup returns a PID immediately before that worker exits.
A worker crashes after accepting a request but before replying.
A node disconnects after remote completion but before reply delivery.
A reconnected node starts workers with new PIDs while callers retain old route snapshots.
A worker advertises version 1 while the request requires version 2.
Two nodes observe :pg joins in different orders.
The lab topology becomes only partially connected, so membership views differ.
The :global coordinator dies during lookup or conflicts during reconnection.
Registry restarts while workers are still in an inconsistent lifecycle boundary.
A malicious worker ID attempts to create arbitrary atoms.
Distribution connects with a wrong name domain, cookie, certificate, or code version.

3.7 Real World Outcome

The finished project is a repeatable service-discovery and routing lab that an engineer can operate from one terminal. It displays both node connectivity and worker membership, routes real fixture transformations to remote worker processes, and proves that a logical worker survives as an identity while its PID does not. Operators can inject process and network failures, see exactly which mechanism reports each failure, and verify that healthy workers remain available. The project produces a runbook, event timeline, and capability matrix suitable for reviewing a real distributed Elixir service.

3.7.1 How to Run

Build the lab release and create three node-specific runtime configurations.
Start the nodes on loopback using stable long or short names consistently.
Verify the topology command before starting workers.
Seed deterministic thumbnail, metadata, and preview workers.
Run route and failure-drill commands from the operator CLI.
For non-loopback use, enable TLS distribution and the restricted network profile first.

3.7.2 Golden Path Demo

Start all three nodes, observe three version-two thumbnail workers, route three fixture jobs across different nodes, crash the selected worker, observe its old PID disappear, route through another worker, verify the restarted logical worker has a new PID and generation, disconnect one node, continue routing on two nodes, reconnect, and wait until all per-node group views converge.

3.7.3 Exact Terminal Transcript

$ fleetctl drill kill-worker thumb-b1
event=worker_exit worker=thumb-b1 old_generation=g3 reason=injected_failure
event=worker_started worker=thumb-b1 new_generation=g4 pid_changed=true
event=group_join capability=thumbnail/v2 node=fleet-b@127.0.0.1

$ fleetctl drill disconnect-node fleet-c@127.0.0.1
event=nodedown node=fleet-c@127.0.0.1 reason=disconnect
routes_remaining capability=thumbnail/v2 count=2

$ fleetctl route thumbnail job-43 --version 2
request=job-43/thumbnail-v2 selected=thumb-a1 node=fleet-a@127.0.0.1 generation=g7
result=ok artifact=artifacts/job-43-thumb-v2.jpg duration_ms=71

$ fleetctl drill reconnect-node fleet-c@127.0.0.1 --wait-converged
event=nodeup node=fleet-c@127.0.0.1
event=membership_converged capability=thumbnail/v2 observers=3 members=3

3.8 Scope Boundary Versus Existing Projects

Project 3 builds a distributed key-value store, Project 7 builds end-user presence and notifications, and Projects 11-13 focus on cluster formation, partitions, and cross-site event flow. Projects 30 and 31 use local Registry as one implementation tool. Project 37 is different: process registration and distributed service discovery are the product. It compares naming mechanisms, exposes per-node membership views, routes directly to current remote workers, and makes stale-PID races and rediscovery observable. It does not implement a replicated database, social presence, durable workflow engine, or generic cluster orchestrator.

4. Solution Architecture

4.1 High-Level Design

                              operator
                                 |
                         [fleetctl / API]
                                 |
                 +---------------+----------------+
                 | topology + service directory  |
                 | node views | :pg views | route|
                 +---------------+----------------+
                                 |
         +-----------------------+-----------------------+
         |                       |                       |
   node fleet-a              node fleet-b              node fleet-c
+-------------------+   +-------------------+   +-------------------+
| TopologyMonitor   |   | TopologyMonitor   |   | TopologyMonitor   |
| Local Registry    |   | Local Registry    |   | Local Registry    |
| DynamicSupervisor |   | DynamicSupervisor |   | DynamicSupervisor |
| workers a1, a2    |   | workers b1, b2    |   | workers c1, c2    |
+---------+---------+   +---------+---------+   +---------+---------+
          |                       |                       |
          +------------ versioned :pg groups -----------+
                                  |
                         [select + monitor PID]
                                  |
                         [deduplicated work]

separate experiment: {:global, :fleet_lab_coordinator}

4.2 Key Components

Component	Responsibility	Key Decisions
Node Candidate Adapter	Produce trusted candidate node names	Static lab seeds first; DNS adapter extension
Topology Monitor	Connect and consume nodeup/nodedown	Observation timestamps and reason classification
Local Registry	Resolve logical worker IDs and local metadata	Unique keys for identity; duplicate registry only for local experiment
Worker DynamicSupervisor	Start and restart worker processes	Child restart policy and duplicate-start resolution
Capability Advertiser	Join/leave versioned `:pg` groups	Group naming and readiness transition
Service Directory	Present local and observed distributed membership	Label observation source and consistency
Fleet Router	Filter/select/monitor current workers	Locality, fairness, deadline, stale-PID handling
Work Deduplicator	Return prior result for repeated request identity	Stable identity and bounded retention
Global Comparison	Demonstrate one distributed singleton name	Isolated from routing correctness
Drill Controller	Inject worker crash and node disconnection	Deterministic actions and cleanup
Telemetry Reporter	Emit topology, registration, route, and convergence events	Low-cardinality labels and correlation IDs

4.3 Data Structures (No Full Code)

Node observation map keyed by trusted node atom
Local worker index keyed by {worker, logical_id}
Capability group key {service, service_name, protocol_version}
Worker descriptor containing PID, node, generation, readiness, and capabilities
Route attempt containing request ID, selected descriptor, monitor reference, deadline, and outcome
Failure history keyed by node/PID with monotonic expiry
Per-observer membership snapshot used only for diagnostics
Deduplication record keyed by stable request ID and immutable input digest

4.4 Algorithm Overview

Key Algorithm: Discover, Select, Monitor, and Dispatch

Convert the requested service/version into an exact capability-group key.
Read the current local :pg membership view.
Join each PID with diagnostic metadata from its node-local descriptor response.
Remove incompatible, draining, recently failed, or disconnected candidates.
Order candidates deterministically, then apply locality/fairness policy.
Install a process monitor on the selected current PID.
Send the request with stable request ID and remaining deadline.
On reply, record success and remove the monitor.
On :DOWN or node loss, classify whether retry is safe and rediscover rather than reuse the PID.
On timeout, return uncertain outcome unless the worker’s deduplication query can prove the result.

Complexity Analysis

Membership lookup and filtering: O(M), where M is observed members for one capability.
Deterministic ordering: O(M log M), reducible with measured alternative selection state.
Local Registry unique lookup: expected constant-time lookup for the selected local identity.
Directory snapshot space: O(N + W + G), for nodes, workers, and group observations.

5. Implementation Guide

5.1 Development Environment Setup

Use a currently supported Elixir/OTP combination. Begin with three nodes on loopback and deterministic static seeds. Reserve distinct node names and ports, keep fixture work small, and configure short deadlines so drills complete quickly. Add the TLS distribution profile before moving beyond loopback. The project should run without Kubernetes; orchestration discovery is an extension, not a prerequisite for learning the process model.

5.2 Project Structure

project-root/
├── lib/
│   ├── fleet/application.ex
│   ├── fleet/node_candidates.ex
│   ├── fleet/topology_monitor.ex
│   ├── fleet/local_registry.ex
│   ├── fleet/worker_supervisor.ex
│   ├── fleet/worker.ex
│   ├── fleet/capability_advertiser.ex
│   ├── fleet/service_directory.ex
│   ├── fleet/router.ex
│   ├── fleet/deduplicator.ex
│   ├── fleet/global_comparison.ex
│   ├── fleet/drill_controller.ex
│   └── fleet/cli.ex
├── config/
│   ├── runtime.exs
│   └── tls-distribution.example.conf
├── fixtures/
├── test/
│   ├── unit/
│   ├── integration/
│   └── cluster/
└── docs/
    ├── mechanism-capability-matrix.md
    ├── failure-runbook.md
    └── security-boundary.md

5.3 The Core Question You Are Answering

“How can a caller find a compatible current process anywhere in a BEAM cluster without pretending that names, PIDs, membership views, or network connections are permanent?”

5.4 Concepts You Must Understand First

Processes, mailboxes, PIDs, links, and monitors.
GenServer client/server APIs and supported name forms.
Supervisor child specifications, restart values, and DynamicSupervisor.
Registry unique/duplicate keys, entry ownership, metadata, and :via.
Distributed node names, connections, remote PIDs, and node monitoring.
:pg membership and strong eventual consistency.
:global full-connectivity assumptions and non-durable name semantics.
Idempotency and uncertain outcomes in remote work.

5.5 Questions to Guide Your Design

Which identities are fixed atoms, dynamic terms, PIDs, or durable strings?
Who owns starting a worker, and how do two simultaneous start requests resolve one winner?
What event makes a warming worker eligible to join its capability group?
How does the router distinguish no compatible member from a disconnected node?
What state is authoritative, and what state is only one node’s current observation?
Which outcomes permit safe retry, and which remain uncertain?
How will you prove a restarted worker has a new process incarnation?
What should happen when :pg views differ during a partition?
Why is :global excluded from the ordinary fleet-routing path?
Which runtime secrets and identifiers must be redacted from logs?

5.6 Thinking Exercise

Trace one job through simultaneous worker death and node loss

Draw a timeline containing Registry lookup, :pg membership read, monitor installation, request send, worker acknowledgement, node disconnection, retry decision, supervisor restart, re-registration, group rejoin, and rediscovery.

Questions to answer:

At which points can the selected PID become stale?
Which observer can know whether work completed?
Would retry duplicate an external effect?
Which identity stays stable after restart?
How would the trace differ if only the worker crashed while the node stayed connected?

5.7 The Interview Questions They Will Ask

“What is the difference between node discovery, process registration, and service discovery?”
“When would you use an atom name, Registry, :pg, or :global?”
“Why does Registry lookup not eliminate stale-PID races?”
“How do DynamicSupervisor and Registry divide responsibility?”
“What does strong eventual consistency mean for :pg membership?”
“Why is :global not automatically a consensus-backed leader election?”
“What happens to a call when the remote node disconnects after executing it?”
“How would you secure distributed Erlang outside an isolated network?”

5.8 Hints in Layers

Hint 1: Make identity visible

Print logical ID, node, PID, and generation together. If your CLI cannot show which part changed, your tests cannot prove rediscovery.

Hint 2: Keep the mechanisms separate

Build local supervision and Registry lookup first. Add node monitoring next. Add :pg only after local restarts are correct. Keep the :global experiment outside routing.

Hint 3: Query late

Resolve group membership near dispatch time. A long-lived PID cache creates another invalidation subsystem before the basic directory is correct.

Hint 4: Design the failure result before retry

Give every routed operation a stable request ID and classify process death, node loss, and timeout. Do not add automatic retry until duplicate effects are controlled.

Hint 5: Observe convergence from every node

During a partition drill, record each node’s membership view and observation timestamp. The differences are the lesson, not a test harness inconvenience.

5.9 Books That Will Help

Topic	Book	Chapters / Focus
Processes and OTP	Elixir in Action by Saša Jurić	Processes, generic server processes, fault tolerance, distributed systems
Supervision architecture	Designing for Scalability with Erlang/OTP by Francesco Cesarini and Steve Vinoski	Supervision, process architecture, distribution, operational tradeoffs
Distributed failure	Designing Data-Intensive Applications by Martin Kleppmann	Replication, partitions, consistency, failure reasoning
Erlang concurrency	Programming Erlang by Joe Armstrong	Processes, registered names, distributed programming, OTP

5.10 Implementation Phases

Phase 1: Local Identity and Lifecycle (6-8 hours)

Define worker logical IDs, capabilities, generation tokens, and client APIs.
Start fixed infrastructure, local Registry, and DynamicSupervisor.
Implement unique :via worker names and a duplicate Registry comparison.
Prove worker crash, name cleanup, restart, and new generation.
Add duplicate-start race tests and dynamic-atom safety tests.

Phase 2: Node Formation and Remote Messaging (5-7 hours)

Start three named loopback nodes from deterministic configuration.
Implement candidate-node and topology-monitor boundaries.
Record nodeup/nodedown events and connection reasons.
Send a fixture request to a remote PID and {name, node} reference.
Document code-version and security assumptions.

Phase 3: Distributed Capability Directory (7-10 hours)

Define versioned :pg group names and readiness rules.
Join workers to capability groups and render per-node views.
Implement deterministic route selection, monitors, deadlines, and outcome classification.
Add stable request identity and simulated work deduplication.
Prove local preference and remote fallback without permanent PID caching.

Phase 4: Failure and Convergence Drills (6-9 hours)

Crash a selected worker before and after acknowledgement.
Disconnect and reconnect a node during routed work.
Capture divergent :pg views and wait for convergence.
Verify old PIDs are never reported as revived.
Produce event timelines, telemetry assertions, and operator runbook.

Phase 5: Global Comparison and Hardening (4-8 hours)

Register one disposable lab coordinator through :global.
Exercise duplicate registration, death cleanup, and partition behavior.
Publish a mechanism capability matrix and explain why routing uses :pg.
Add TLS distribution profile, redaction checks, and restricted network guidance.
Measure directory lookup and routing overhead before proposing caches.

5.11 Key Implementation Decisions

Decision	Tempting Alternative	Recommended Choice	Why
Dynamic names	Convert worker IDs to atoms	Term keys in Registry	Avoid atom exhaustion
Lifecycle lookup	Ask DynamicSupervisor for business key	DynamicSupervisor + Registry	Separates lifecycle from naming
Distributed fleet	One `:global` name per worker	Versioned `:pg` groups	Models many interchangeable members
Singleton experiment	Treat as leader election	Isolated `:global` comparison	Avoids overstating guarantees
Route identity	Cache PID indefinitely	Resolve near dispatch	Limits stale references
Retry	Retry every timeout	Stable request ID + classified outcome	Prevents duplicate effects
Membership display	Claim cluster truth	Label observer and timestamp	Reflects eventual membership views
Node discovery	Mix with worker groups	Separate candidate adapter	Makes layers testable
Distribution security	Cookie on open network	TLS + network restrictions	Protects cluster-level authority

6. Testing Strategy

6.1 Test Categories

Category	Goal	Representative Evidence
Pure unit tests	Validate group keys, filtering, ordering, and retry classification	deterministic inputs/outputs
Local registration tests	Prove fixed names, unique/duplicate Registry, `:via`, and cleanup	process lifecycle assertions
Start-race tests	Prove one logical local owner under concurrent creation	barrier-controlled callers
Supervision tests	Prove new PID/generation and correct restart intensity	injected worker exits
Multi-node integration	Prove remote PID and `{name, node}` communication	three loopback nodes
Membership tests	Prove `:pg` join, leave, stale view tolerance, and convergence	per-node snapshots
Network failure tests	Prove nodedown invalidation and reconnect rediscovery	controlled disconnect
Delivery tests	Prove deduplication and uncertain-timeout classification	lost reply fixtures
Global comparison tests	Document singleton registration/death/conflict behavior	isolated experiment
Security tests	Reject unsafe IDs and verify secret redaction	adversarial identifiers/log scan
Performance tests	Measure lookup and route selection under growing membership	latency percentiles and scheduler load

6.2 Critical Test Cases

A fixed infrastructure atom name resolves only on its local node.
A dynamic worker string registers through Registry without increasing atoms based on input.
A unique key rejects a second live owner; the losing start resolves the existing worker.
A duplicate registry returns all local subscribers for one capability key.
Killing a worker removes the old entry and produces a new PID and generation after restart.
A lookup followed by immediate death produces a classified process-down outcome, not a crash in the router.
A remote request succeeds through a PID and through {registered_name, node} for the fixed test service.
All three nodes eventually observe the same stable :pg membership after concurrent joins.
A version-one worker is never selected for a version-two request.
Disconnecting the selected node invalidates it and leaves other compatible workers routable.
Reconnection discovers new process incarnations instead of reusing stored PIDs.
Per-node membership snapshots may differ during a partition and converge afterward.
A request completed before reply loss returns the deduplicated prior result on safe retry.
An operation without deduplication returns uncertain outcome rather than automatic duplicate dispatch.
:global duplicate registration and process death behave as documented in the full-mesh lab.
The non-loopback profile refuses to start without the required secure-distribution configuration.

6.3 Test Data

Use deterministic node names, worker IDs, capability versions, and small immutable media fixtures. Include duplicate worker IDs, hostile Unicode and atom-like IDs, unsupported versions, slow work, crash-before-ack, crash-after-ack, lost reply, node disconnect, reconnect with new generation, concurrent group joins, and partial-connectivity timelines. Store expected artifact digests so a deduplicated retry can prove it returned the original result.

6.4 Definition of Done

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem 1: “The worker restarted, but calls still target the dead PID.”

Why: A caller cached the concrete PID instead of resolving the logical identity again.
Fix: Rediscover near dispatch and invalidate route snapshots through process/node monitors and expiry.
Quick test: Crash a worker and assert the next route reports the same logical ID with different PID and generation.

Problem 2: “Registry works locally but no remote node can see it.”

Why: Elixir Registry is local; connecting nodes does not distribute its entries.
Fix: Use Registry for local identity and :pg for distributed capability groups.
Quick test: Compare local Registry output with per-node :pg membership.

Problem 3: “Two callers started duplicate work for one worker ID.”

Why: Both observed a missing registration and performed side effects before ownership was resolved.
Fix: Make unique registration/start resolution precede irreversible work and return the existing winner to the loser.
Quick test: Release two barrier-controlled start calls simultaneously and count live owners and effects.

Problem 4: “All nodes are connected, but one service is missing.”

Why: Node readiness was confused with process capability readiness, or group propagation is still converging.
Fix: Report node and group layers separately; join only after worker readiness and allow bounded convergence.
Quick test: Delay one worker’s group join while keeping its node connected.

Problem 5: “A timeout caused the same job to run twice.”

Why: The remote side completed, but the reply was lost and the router retried without stable identity.
Fix: Use request deduplication or return uncertain outcome for non-idempotent work.
Quick test: Drop the first successful reply, retry, and compare artifact digest and execution count.

Problem 6: “The partition test produces different member lists.”

Why: :pg views are strongly eventually consistent and visibility depends on direct connectivity.
Fix: Label each observed view and test convergence after connectivity stabilizes rather than demanding instant equality.
Quick test: Capture all observers before, during, and after the partition.

Problem 7: “The global coordinator is being treated as durable leadership.”

Why: Name uniqueness was mistaken for consensus, persistence, or exactly-once ownership.
Fix: Isolate the comparison, document full-mesh assumptions, and keep durable decisions in a suitable durable coordination mechanism.
Quick test: Kill the coordinator and show its name cleanup does not restore its internal state.

7.2 Debugging Strategies

Print node, logical ID, PID, generation, observer, and timestamp on every diagnostic line.
Inspect local Registry and :pg separately before debugging the router.
Subscribe to node events with reasons and correlate them with process :DOWN messages.
Trace one request ID end-to-end instead of relying on aggregate counters.
Compare membership from every node when topology is suspect.
Check node name domains, DNS/hosts mapping, distribution ports, TLS certificates, and clock-independent deadlines.
Use controlled failure hooks rather than random kills so ordering is reproducible.
Verify code and protocol version compatibility before attributing failures to distribution.

7.3 Performance Traps

One unpartitioned local Registry may become a hotspot under extreme local churn; measure and configure partitions deliberately.
Sorting every huge group on every request costs O(M log M); keep the first implementation correct, then benchmark selection alternatives.
Broad :pg groups create propagation traffic and large membership reads; use meaningful capability scopes.
High-cardinality PID, worker ID, or node labels can overwhelm metrics; keep them in traces/logs, not metric dimensions.
Aggressive node reconnect loops amplify outages; use bounded backoff and topology-aware policy.
Monitoring every member permanently duplicates discovery state; monitor selected work or build a deliberate cache subsystem.
Synchronous global-name operations in a large or unstable topology can create latency and operational coupling.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a CLI command that explains why a worker was excluded from selection.
Render fixed names, unique Registry entries, and duplicate Registry entries in separate tables.
Add a readiness delay before a worker joins capability groups.
Export the latest node and membership observations as JSON.

8.2 Intermediate Extensions

Add weighted selection from measured worker capacity without turning membership into mutable load state.
Add zone-aware preference and a fallback budget.
Implement a DNS-based node-candidate adapter behind the existing boundary.
Add graceful draining that leaves capability groups before process shutdown.
Add a short-lived monitored route cache and prove invalidation under every drill.

8.3 Advanced Extensions

Run the lab across containers or virtual machines with TLS distribution and rotated certificates.
Model incompatible rolling versions and verify group-key migration.
Compare :pg with a third-party distributed registry while preserving the same failure contract.
Add PartitionSupervisor for local routing/supervisor scalability and measure contention.
Persist desired worker specifications and implement reconciliation without treating observed membership as desired state.
Integrate OpenTelemetry traces across remote calls while controlling propagation and cardinality.

9. Real-World Connections

9.1 Industry Applications

Distributed service directories appear in media processing, IoT session ownership, communications routing, multiplayer games, industrial gateways, and multi-tenant background execution. BEAM processes make these fleets natural to model, but the engineering challenge is not merely spawning them. Production systems must separate node topology, current process membership, durable desired state, and delivery guarantees. This project builds that vocabulary through observable failures rather than a happy-path cluster demo.

Elixir Registry for local dynamic names and local dispatch
Erlang :pg for distributed named process groups
Erlang :global for global naming under its documented topology assumptions
Phoenix.PubSub for distributed application messaging patterns
libcluster for deployment-specific node-discovery strategies
Horde for distributed supervisor/registry designs and their tradeoffs
OpenTelemetry for cross-process and cross-node trace context

9.3 Interview Relevance

A strong senior-level answer distinguishes lifecycle from lookup, node membership from service membership, and failure notification from delivery certainty. It explains why PIDs are ephemeral, why Registry is local, why :pg is eventually consistent, why :global has topology constraints, why monitors do not retry work, and why a timeout after remote dispatch can be an uncertain outcome. The candidate should be able to draw the worker-crash and node-partition timelines without promising exactly-once behavior.

10. Resources

10.1 Essential Official Reading

Elixir Process — local registration, monitors, process sending, and process lifecycle tools.
Elixir GenServer — client/server APIs, supervision integration, and name forms.
Elixir Registry — local unique/duplicate registries, :via, ownership, dispatch, and partitioning.
Elixir Supervisor — hierarchical trees, restart values, strategies, and intensity.
Elixir DynamicSupervisor — on-demand child lifecycle and local scalability.
Elixir Node — node connection, listing, monitoring, and remote spawning primitives.
Erlang pg — distributed group semantics, scopes, direct connectivity, and consistency.
Erlang global — global names, replicated tables, topology requirements, and partitions.
Erlang net_kernel — distributed node operation, monitoring, and secure-distribution warning.
Erlang distributed systems overview — remote processes and built-in interoperability.
Erlang distribution protocol — transport, EPMD, handshake, and security boundary.

10.2 Books and Deeper Study

Elixir in Action by Saša Jurić — process architecture, OTP, fault tolerance, and distribution.
Designing for Scalability with Erlang/OTP by Francesco Cesarini and Steve Vinoski — supervision, distribution, and operational architecture.
Programming Erlang by Joe Armstrong — processes, registered names, messages, and distributed Erlang.
Designing Data-Intensive Applications by Martin Kleppmann — partial failure, consistency, replication, and retry reasoning.