Project 36: Self-Contained Mix Release and Air-Gapped Deployment

Build, package, verify, deploy, operate, and roll back an Elixir service as a versioned tarball that carries its own ERTS and runs on a clean compatible machine with no system elixir, mix, or erl installation.

Quick Reference

Attribute	Value
Difficulty	Level 3: Advanced
Suggested Seniority	Senior Elixir or platform engineer
Time Estimate	22-34 hours
Main Programming Language	Elixir with Erlang/OTP release tooling
Alternative Programming Languages	Erlang for release internals; shell for deployment orchestration
Coolness Level	Level 3: Genuinely Clever
Business Potential	Level 4: Open Core Deployment Platform
Prerequisites	Mix applications, OTP supervision, environment configuration, Unix processes, archives and checksums
Key Topics	Mix releases, bundled ERTS, `:assemble` and `:tar`, `runtime.exs`, `RELEASE_*`, target triples, NIF/system-library compatibility, provenance, clean-host operation, rollback

1. Learning Objectives

By completing this project, you will:

Explain the difference between source, compiled applications, an OTP release, ERTS, a release directory, and a release tarball.
Configure a release that explicitly includes ERTS and executes the :tar step after :assemble.
Separate compile-time configuration from boot-time configuration and load secrets through config/runtime.exs without depending on Mix at runtime.
Use RELEASE_ROOT, RELEASE_NAME, RELEASE_VSN, RELEASE_NODE, RELEASE_COOKIE, RELEASE_TMP, and distribution settings deliberately.
Produce a checksum and provenance manifest that ties one tarball to its source revision, lockfile, toolchain, target triple, and build policy.
Demonstrate the artifact on a clean compatible target where elixir, mix, and erl are absent from PATH.
Operate the installed release through start, stop, pid, eval, rpc, and remote, and explain which commands use a running node.
Detect target architecture, operating-system, ABI, NIF, and dynamically linked system-library incompatibilities before promotion.
Deploy immutable version directories, switch a stable pointer only after health validation, and roll back without mutating an installed release in place.
Distinguish release assembly from Project 9’s hot-code upgrade mechanics and Project 32’s artifact inspection and reproducibility analysis.

2. All Theory Needed (Per-Concept Breakdown)

Self-Contained Release Assembly, ERTS, and Runtime Configuration

Fundamentals

A Mix release is a deployable target system assembled from the project’s compiled applications, their transitive OTP applications, boot scripts, configuration, management scripts, and—by default—the Erlang Runtime System. ERTS contains the BEAM virtual machine and runtime executables; Elixir and Mix are not the same thing as ERTS. Mix is a build tool and is intentionally unavailable inside the running release. With include_erts: true, the target does not need a separately installed Erlang or Elixir toolchain, but the artifact still depends on a compatible operating system, CPU architecture, ABI, and any dynamically linked libraries. Assembly creates the versioned release tree. A later :tar release step packages that assembled tree for transfer. config/runtime.exs is copied into the artifact and evaluated early on each boot, making it the correct boundary for target-specific endpoints and secrets.

Deep Dive

The release is best understood as an explicit closure over runtime code rather than a compressed source checkout. Mix begins from the selected release and application, resolves the current application plus required dependencies, validates application modes, produces release metadata and boot scripts, consolidates protocols, copies application code and priv assets, generates command wrappers, and optionally copies an ERTS tree. The result normally lives below _build/prod/rel/RELEASE_NAME. It contains bin/RELEASE_NAME, erts-ERTS_VSN/, application directories under lib/, release-version data under releases/RELEASE_VSN/, a cookie fallback, boot files, runtime configuration, VM arguments, and a temporary directory. The learner should inspect this tree before compressing it. A tarball is not the release definition; it is one transport representation of an already assembled release.

Mix release steps make that order visible. The :steps option must contain :assemble, and a :tar step can occur only after assembly. A conceptual configuration for this lab declares include_erts: true, Unix executables for a Unix target, and steps: [:assemble, :tar]. include_erts already defaults to true in current Mix, but stating it explicitly turns the project’s central portability promise into reviewable configuration. Setting it false changes the contract: the target then needs the exact matching ERTS version, and current Mix documentation warns that this also disables hot-code upgrades. This project keeps it true and proves that a bundled erts-* directory and runtime executable are present.

Including ERTS does not mean the archive contains an operating system. The release carries the BEAM runtime and selected OTP/Elixir applications, not the target kernel, C library, certificate store, timezone database, OpenSSL installation, shell, service manager, or arbitrary shared libraries. This distinction prevents the dangerous claim that a release is universally portable. A self-contained Mix release means self-contained with respect to the Erlang/Elixir runtime installation, under a declared target contract.

Configuration has two different moments. config/config.exs and environment-specific imports are evaluated when Mix commands compile or assemble the project. Values read there can become baked into compiled artifacts. config/runtime.exs is copied into the release and evaluated at boot. It must import Config, must not import other configuration files, and must not call Mix because Mix is absent from releases. The project puts target port, public hostname, storage path, and required secret references in runtime configuration. Missing required values must fail boot with a precise diagnostic before the service accepts traffic. Secrets must not appear in the release tar, checksum manifest, provenance file, logs, or command history.

The generated wrapper establishes a release execution environment. RELEASE_ROOT is computed from the extracted release and cannot be relocated by pretending it points somewhere else. RELEASE_NAME, RELEASE_VSN, and RELEASE_COMMAND identify the selected artifact and operation. RELEASE_NODE supplies the node identity, while RELEASE_COOKIE overrides the cookie file and must be protected like a credential. Current Mix writes a random releases/COOKIE fallback during initial assembly when no cookie is configured. An artifact distributed outside a trusted secret channel must either be treated as secret itself or pass through a post-assemble policy step that removes the fallback before :tar and requires RELEASE_COOKIE at operation time. This project chooses the latter and tests the missing-cookie failure path. RELEASE_TMP relocates writable temporary state. RELEASE_MODE selects embedded or interactive code loading. RELEASE_DISTRIBUTION controls long-name, short-name, or disabled automatic distribution. The project records which values are immutable build facts, computed wrapper facts, deployment configuration, and secrets.

Management commands have distinct semantics. start boots the release applications in the foreground and is the preferred shape under a service manager or container supervisor. stop, restart, pid, rpc, and remote communicate with an already running distributed node and therefore depend on matching node identity, distribution reachability, and cookie. eval starts a separate non-booted VM without starting the release applications or distribution, so operational tasks invoked through it must explicitly start any applications they require. Confusing eval with rpc can make a migration or maintenance task run in the wrong runtime context.

Application selection also affects correctness. A release includes the current application and dependency applications recursively, with application modes controlling load and start behavior. Removing an application because it appears unused can break runtime code paths that are not obvious during assembly. Adding every available OTP application increases artifact size and attack surface without proving readiness. The learner should inventory included applications, explain each deliberate override, and smoke-test the actual tarred artifact—not merely run mix test against the development environment.

The core invariants are: ERTS is present in the artifact; the tar step consumes a completed assembly; no source checkout or system Mix command is required at the target; boot-time values come from the target; required secrets are absent from artifact bytes; release commands use one documented node/cookie contract; and the exact extracted artifact is what validation promotes. Failure modes include :tar before :assemble, compile-time secret capture, runtime.exs calling Mix, missing applications or priv files, mismatched remote command identity, writable state inside an immutable version directory, and declaring portability without proving the target platform.

How this fits on the project

This concept defines the release configuration, assembly pipeline, tar contents, runtime configuration boundary, command wrapper contract, clean-host transcript, and operational command tests.

Definitions and key terms

ERTS: Erlang Runtime System, including the BEAM virtual machine and runtime executables.
OTP application: Versioned unit with modules, dependencies, environment, and optional supervision entry point.
Release: A versioned target system composed of ERTS and selected OTP/Elixir applications plus boot/configuration artifacts.
Assembly: Construction of the release directory tree from compiled inputs.
Tar step: Packaging step that archives an assembled release after :assemble.
Runtime configuration: Configuration evaluated on target boot rather than during compilation or release assembly.
Release command wrapper: Generated bin/RELEASE_NAME script used to start and operate the release.
Target system: Installed system built from a release for a declared runtime environment.

Mental model diagram

source + mix.lock + toolchain + release policy
                    |
                    v
            [compile in prod]
                    |
                    v
             [:assemble step]
                    |
    +---------------+----------------+
    | release root                   |
    | bin/app                        |
    | erts-ERTS_VSN/  <-- bundled VM |
    | lib/app-vsn/                   |
    | releases/vsn/runtime.exs       |
    | releases/vsn/{boot,vm,args}    |
    +---------------+----------------+
                    |
                 [:tar]
                    |
                    v
           app-VSN.tar.gz + manifest
                    |
              extract on target
                    |
          runtime.exs reads target env
                    |
            bin/app start / eval

How it works

Lock source, dependency, toolchain, and target inputs.
Compile under the production Mix environment on a target-compatible build host.
Assemble a release with ERTS and intentional application modes.
Inspect the uncompressed tree for ERTS, boot files, application code, runtime configuration, and forbidden files.
Run the tar step only after successful assembly.
Compute artifact digest and attach non-secret provenance.
Extract into a version-specific target directory.
Inject target runtime configuration and secrets outside the artifact.
Start via the generated wrapper and validate commands plus application health.
Invariant: the target never invokes system mix, elixir, or erl.
Failure modes: missing ERTS, baked secrets, absent priv, bad application modes, mismatched remote cookie, or mutable files under the release directory.

Minimal concrete example

CONCEPTUAL RELEASE CONFIGURATION, NOT A COMPLETE mix.exs
release name: airgap_probe
include_erts: true
include_executables_for: [unix]
steps:
  1. assemble
  2. post-assembly policy check and remove generated cookie fallback
  3. tar

RUNTIME CONFIG CONTRACT
PROBE_PORT: required integer, read during boot
PROBE_DATA_DIR: required absolute writable path outside release root
PROBE_SIGNING_SECRET: required secret, never written to artifact

Common misconceptions

A release tarball is a source archive that runs mix after extraction.
Bundled ERTS makes one artifact portable across Linux, macOS, CPU architectures, and ABIs.
runtime.exs can call Mix because it has an .exs extension.
eval executes inside the already running release node.
A generated cookie file is safe to publish with a public artifact.

Check-your-understanding questions

Why does the :tar step have to follow :assemble?
What runtime dependency does include_erts: true remove, and what dependencies remain?
Why should a database URL or signing secret normally be read in runtime.exs?
What is the semantic difference between eval and rpc?

Check-your-understanding answers

Tar packaging archives the completed release tree; without assembly there is no valid target system to package.
It removes the need for a separately installed matching Erlang runtime; OS, architecture, ABI, shared libraries, and native dependency requirements remain.
The target supplies those values at boot, avoiding build-host coupling and secret capture in the artifact.
eval runs a separate non-booted VM; rpc calls the already running distributed node.

Real-world applications

VM or bare-host Elixir deployments without language package managers
Air-gapped or regulated delivery through signed artifact channels
Immutable server images and offline disaster-recovery bundles
Edge installations where build tooling must not exist on production hosts

Where you will apply it

Project 36 release definition, runtime contract, artifact builder, clean-host verifier, and operator runbook

References

Key insight

A self-contained Mix release removes the production Erlang/Elixir installation, not the target-platform contract.

Summary

Assemble first, package second, keep ERTS inside, defer target values to runtime configuration, understand each release command’s execution context, and prove the artifact on a host with no language toolchain.

Homework/Exercises

Classify ten values—release version, HTTP port, source revision, database URL, node name, cookie, ERTS version, data directory, log level, signing secret—as build fact, runtime configuration, computed release value, or secret.
Draw the process and filesystem differences between eval, start, rpc, and remote.

Solutions

Source revision, release version, and ERTS version are provenance/build facts; HTTP port, database URL, data directory, and log level are runtime configuration; release root/name/selected version are computed wrapper facts; cookie and signing secret are runtime secrets; node name is deployment identity.
eval creates a separate clean VM; start creates the service VM and applications; rpc creates a helper VM that calls the service node; remote creates a helper VM with an interactive shell attached to the service node.

Target Compatibility, Artifact Integrity, Operations, and Rollback

Fundamentals

Release portability is bounded by the target triple and runtime dependencies. The build and target must agree on CPU architecture, vendor/operating system, and ABI; in practice, the OS distribution and version should also be kept compatible. ERTS and dependencies may dynamically link to libraries such as OpenSSL, while NIFs and executable assets must be compiled for the same target. Artifact integrity begins with a cryptographic digest but requires provenance to explain what the digest represents. Deployment should extract immutable version directories, validate the exact candidate, and switch a stable pointer only after success. Rollback switches back to a previously validated artifact, but code rollback is safe only when external data and configuration remain backward compatible. Operational commands also rely on node identity and cookie policy, so release engineering includes process operation, not just compression.

Deep Dive

The phrase “works on my machine” becomes a binary compatibility problem once ERTS is bundled. Official Mix release documentation describes compatibility in terms of architecture, vendor/operating system, and ABI, commonly written as a target triple such as x86_64-unknown-linux-gnu or aarch64-unknown-linux-musl. A release assembled on macOS is not a Linux release even when both machines use the same CPU architecture. A glibc-linked Linux artifact is not automatically compatible with musl. A release built on a newer distribution can reference library or symbol versions absent on an older target. The safest baseline is to build inside a controlled environment matching production’s target contract.

Native boundaries add another layer. ERTS itself may dynamically link to OpenSSL for :crypto and :ssl. A project dependency can contain a NIF compiled for a particular CPU/OS/ABI and linked to additional shared libraries. Ports and packaged executables also need the correct target format and executable permissions. Pure BEAM dependencies are more portable within the compatible runtime, but the full release inherits the least portable runtime component. The artifact pipeline should inventory NIFs, shared objects, executable priv assets, and dynamic library requirements, then run load/smoke tests on the target image. A tarball containing ERTS can still fail immediately with “not found” when the missing object is a dynamic loader or shared library.

Compatibility should be machine-readable evidence. Record target triple, OS release identity, CPU architecture, libc family/version where relevant, OTP and Elixir versions, Mix environment, lockfile digest, source revision and dirty status, release name/version, build policy version, and the artifact SHA-256. Record the builder image digest if a container or VM image defines the build environment. Do not record environment values indiscriminately because they can contain secrets. The manifest should distinguish declared target facts from measured build-host facts and should never claim reproducibility merely because two builds have the same version label.

A checksum answers whether bytes changed after the digest was computed. It does not prove who produced the bytes, whether the build was authorized, or whether the artifact is safe. Air-gapped delivery should pair the checksum with an authenticated signature or trusted transfer process as an advanced requirement, and the target should verify before extraction. Archive extraction itself has security concerns: validate destination, ownership, available space, expected top-level layout, file permissions, and absence of dangerous path traversal or links when artifacts are not fully trusted. The core project uses a trusted lab-built archive but documents the stronger production boundary.

The clean-host test is intentionally stricter than launching the artifact on the build machine. Provision a disposable VM or container with the same target contract but no Erlang or Elixir packages. Show that command -v elixir, command -v mix, and command -v erl find nothing. Verify the tar digest, extract into a version directory, assert erts-* is present, supply runtime variables, and exercise eval before starting. Then start the foreground service under a process supervisor or in a dedicated terminal. From another terminal, verify PID, health, RPC, and remote-shell attachment using the same node/cookie contract. Finally stop it through the wrapper and confirm the OS process and listener disappear. The proof must run the transferred tarball, not a nearby _build directory.

Deployment layout is a state machine. An incoming artifact is downloaded to a staging location. Verification establishes checksum, expected release/version, target compatibility, disk space, and secret availability. Extraction creates /opt/airgap_probe/releases/1.4.0 or an equivalent immutable version directory. Preflight eval checks configuration and external prerequisites without pretending the full service is running. The candidate starts, reaches application health, and passes a bounded smoke test. Only then does the stable current pointer move atomically. The previous pointer is retained. Failed candidates are stopped and quarantined; they never partially overwrite the prior installation.

Rollback is artifact selection, not magical time travel. Switching the pointer back restores old code and bundled ERTS. It does not reverse database migrations, messages already published, external API changes, or files written under a shared data directory. Every deployment therefore declares a data compatibility window. Expand-contract database changes, versioned messages, and backward-readable file formats support code rollback. A destructive migration can make the previous release unbootable even though its tarball is intact. The runbook must distinguish “rollback code,” “restore data,” and “compensate external effects.”

Process operation belongs to the target contract. Foreground start integrates naturally with systemd, container supervisors, and log collectors. Shutdown through SIGTERM or the wrapper lets OTP stop applications and supervision trees in reverse startup order. Health checks should distinguish process existence, VM responsiveness, application readiness, and dependency readiness. pid or a successful remote connection does not prove the service is ready to accept business traffic. Conversely, a temporary dependency failure should not necessarily cause a process manager to restart the VM in a loop. The learner defines restart limits at the external service-manager layer separately from OTP child restart strategies.

The core invariants are: artifact bytes match the manifest; extracted version directories are immutable; the target contract is measured and compared; native/system dependencies are inventoried; secrets are injected rather than archived; promotion follows health; rollback retains a known-good artifact; application data lives outside release roots; and clean-host evidence proves no system language runtime was used. Failure modes include building on the wrong target, missing OpenSSL or libc symbols, wrong NIF architecture, checksum verification after extraction, reusing one mutable directory, switching before readiness, losing the previous version, incompatible data migration, and remote operations failing because the helper command uses a different cookie or node identity.

How this fits on the project

This concept defines the provenance manifest, target preflight, native dependency inventory, air-gapped transcript, immutable deployment layout, service-manager boundary, health gate, and rollback drill.

Definitions and key terms

Target triple: Compact description of architecture, vendor/OS, and ABI.
ABI: Binary interface contract between compiled code, runtime, loader, and libraries.
NIF: Native function loaded into the BEAM VM; it inherits target and shared-library requirements.
Dynamic dependency: Shared library resolved by the target loader at runtime.
Provenance manifest: Non-secret evidence tying artifact bytes to inputs, toolchain, target, and build policy.
Immutable deployment: Installation model where released version directories are never edited in place.
Promotion: Making a validated candidate the active release.
Rollback: Re-selecting a previously validated release under an explicit data-compatibility policy.

Mental model diagram

controlled compatible builder
  target=x86_64-linux-gnu
  source + lock + OTP + Elixir
            |
            v
    tarball + checksum + provenance
            |
      authenticated transfer
            |
            v
clean compatible target (no elixir/mix/erl)
            |
     verify before extract
            |
  /opt/app/releases/1.4.0  <--- immutable candidate
            |
      preflight + start + health
            |
            v
 current -> releases/1.4.0
 previous -> releases/1.3.2
            |
      failure after promotion?
            |
            +---- stop 1.4.0 ----> current -> 1.3.2
                                      |
                           shared data must remain compatible

How it works

Define one target contract before selecting a builder.
Build on a matching host/image and inventory native/system dependencies.
Generate artifact digest plus sanitized provenance.
Transfer artifact, digest, and provenance through the air-gap process.
Verify identity and compatibility before extraction.
Extract into a new immutable version directory.
Run configuration and dependency preflight through eval.
Start candidate, then test VM, application, and endpoint readiness.
Atomically promote only after the health gate passes.
Retain previous artifact and test a deliberate failed-candidate rollback.
Invariant: shared data remains readable by every release in the rollback window.
Failure modes: target mismatch, missing library, NIF load error, corrupted transfer, premature switch, or irreversible data change.

Minimal concrete example

PROVENANCE RECORD, NOT A BUILD SCRIPT
release: airgap_probe 1.4.0
artifact: airgap_probe-1.4.0.tar.gz
sha256: <64 hexadecimal characters>
source_revision: <commit id>
source_dirty: false
lock_digest: <sha256>
elixir: 1.x.y
otp: 2x.y
target: x86_64-unknown-linux-gnu
builder_image_digest: sha256:<digest>
native_assets: [one declared NIF or "none"]
required_system_libraries: [declared names and minimum policy]

Common misconceptions

A matching CPU architecture is sufficient for release compatibility.
SHA-256 proves artifact authorship and safety.
A passing pid command proves the application is ready.
Keeping the previous tarball guarantees rollback after any database migration.
Editing runtime.exs inside an installed version directory is harmless configuration management.

Check-your-understanding questions

Why can a bundled-ERTS release fail on a target with no missing BEAM modules?
What does a checksum prove, and what does it not prove?
Why should deployment use version directories plus a stable pointer?
What non-code state can prevent rollback?

Check-your-understanding answers

ERTS, NIFs, Ports, or executable assets can require a different ABI, dynamic loader, CPU, OS, or shared library.
It proves transferred bytes match the digested bytes; it does not prove producer identity, authorization, safety, or reproducibility.
It prevents partial in-place replacement, makes the candidate inspectable, and keeps the previous artifact available for an atomic switch back.
Database schemas/data, shared files, messages, external side effects, and changed runtime configuration can all make old code incompatible.

Real-world applications

Regulated offline installations with controlled artifact import
Fleet deployment to VMs without development toolchains
Release promotion across staging and production from one immutable artifact
Disaster-recovery bundles retained with provenance and runbooks

Where you will apply it

Project 36 compatibility manifest, dependency preflight, clean-host test, promotion transaction, service operation, and rollback drill

References

Key insight

The deployable unit is not “Elixir code”; it is one immutable artifact plus a declared platform, configuration, data, and operations contract.

Summary

Build against the target, inventory native dependencies, attach integrity and provenance, validate the transferred artifact on a toolchain-free host, promote immutably, and treat rollback as a compatibility exercise across code and data.

Homework/Exercises

Compare x86_64-apple-darwin, x86_64-unknown-linux-gnu, x86_64-unknown-linux-musl, and aarch64-unknown-linux-gnu; mark which pairs can safely share one release artifact.
Draw a failed deployment in which code rollback succeeds but the service remains broken because data is incompatible.

Solutions

Treat every distinct triple as requiring its own build artifact; even matching triples should retain an OS/system-library compatibility policy.
Candidate code applies a destructive migration, fails health, and pointer rollback starts old code that cannot read the new schema. Prevent this with expand-contract migrations or a coordinated restore plan.

3. Project Specification

3.1 What You Will Build

Build AirgapProbe, a small supervised HTTP/status service plus a release/deployment harness. The service exposes build identity and a health result, owns no important state inside the release directory, and includes bounded release-task entry points for configuration preflight and readiness inspection. The harness assembles a production Mix release with ERTS, creates a tarball after assembly, emits checksum/provenance evidence, transfers it into a clean compatible target, installs it under an immutable version directory, exercises release commands, promotes it, and proves rollback with a deliberately unhealthy candidate.

The application is deliberately simple because release engineering—not HTTP framework breadth—is the learning target. Use a minimal endpoint or supervised listener already familiar to the learner. Do not add clustering, a database migration framework, hot-code upgrades, or a custom release packager to the core scope.

3.2 Functional Requirements

Assemble a named production release with include_erts: true.
Execute :assemble before :tar and produce a versioned .tar.gz artifact.
Include only required applications, compiled code, priv assets, configuration templates, management scripts, and bundled ERTS.
Remove or reject the generated cookie fallback after assembly and require an injected RELEASE_COOKIE before release operations.
Read port, data directory, build-display policy, and a required secret sentinel through config/runtime.exs.
Define and document the required RELEASE_* environment contract for service and remote commands.
Emit a SHA-256 file and sanitized provenance manifest for the tarball.
Record target architecture, OS/vendor, ABI, OTP, Elixir, lock digest, source revision, and native/system dependency inventory.
Verify artifact digest and target compatibility before extraction.
Extract into a version-specific immutable directory outside shared data/config paths.
Prove elixir, mix, and erl are absent on the target while the bundled release starts.
Exercise eval, foreground start, pid, rpc, remote, and stop with observable results.
Reject boot when required runtime configuration is absent or malformed without exposing secret values.
Promote a healthy candidate through an atomic stable pointer.
Deploy a deliberately unhealthy next version, refuse promotion or roll it back, and restore service through the last known-good artifact.
Preserve logs, data, and deployment evidence outside immutable release directories.

3.3 Non-Functional Requirements

Portability honesty: Every artifact declares exactly one target contract; no cross-platform claim is implied.
Security: Secrets never enter the archive, provenance, checksum, or logs; cookies are handled as credentials.
Reliability: Failed verification or health checks cannot alter the current known-good installation.
Reproducibility evidence: Inputs and toolchain are recorded without claiming byte reproducibility until P32 proves it.
Operability: Start, stop, PID, remote inspection, logs, health, and rollback have bounded documented procedures.
Auditability: Every promotion records artifact digest, version, operator/test identity, target, and result.

3.4 Example Usage / Output

$ release-lab build --version 1.4.0 --target x86_64-unknown-linux-gnu
compiled mix_env=prod
assembled release=airgap_probe version=1.4.0 include_erts=true
policy cookie_fallback=removed runtime_cookie=required
packaged artifact=airgap_probe-1.4.0.tar.gz step_order=assemble,policy,tar
manifest artifact_sha256=9d6b...7e21 source_dirty=false native_assets=0

$ release-lab verify airgap_probe-1.4.0.tar.gz
checksum=ok layout=ok bundled_erts=present forbidden_secrets=0
target expected=x86_64-unknown-linux-gnu actual=x86_64-unknown-linux-gnu compatible=true

3.5 Data Formats / Schemas / Protocols

Artifact manifest: release name/version, filename, byte size, SHA-256, source revision/dirty state, lock digest, OTP/Elixir, target triple, builder identity, policy version.
Dependency inventory: bundled ERTS version, OTP applications, NIF/shared objects, executable priv assets, declared system libraries.
Runtime contract: variable name, classification, required/default status, validation rule, secret flag, safe diagnostic name.
Deployment record: target identity, candidate path, previous path, verification/health timestamps, promotion result, rollback reason.
Health result: release identity, VM up, application ready, listener ready, dependency status, no secret fields.

3.6 Edge Cases

Artifact built for macOS but imported into Linux.
Linux artifact has correct CPU but wrong glibc/musl ABI.
NIF has wrong architecture or cannot resolve a shared library.
Required runtime variable is absent, empty, malformed, or points inside the release root.
Release cookie differs between service and remote/rpc commands.
Tar transfer is truncated or checksum file is paired with the wrong version.
Archive extracts successfully but lacks executable permissions.
Health endpoint responds before the application is truly ready.
Candidate starts, writes an incompatible shared-data format, then fails.
Disk fills between verification and extraction.
Operator invokes eval assuming all applications are started.
Stable pointer changes but process manager still runs the previous executable.

3.7 Real World Outcome

3.7.1 How to Run

Use a controlled builder matching the chosen Linux target. Transfer only the tarball, checksum, provenance manifest, and public verification material into a disposable clean VM/container. The target must have basic OS tools and declared shared libraries but no installed Erlang or Elixir packages. Keep two terminals: Terminal A runs the foreground service; Terminal B performs health and release-management commands.

3.7.2 Golden Path Demo

The learner proves four claims in order: the transferred bytes match the manifest; the artifact contains ERTS; the target has no system BEAM/Elixir commands; and the generated release wrapper can evaluate a preflight, start the application, accept a remote shell/RPC, report its PID and health, and stop cleanly. A second candidate intentionally fails its readiness gate, leaving or restoring the stable pointer to version 1.4.0.

3.7.3 Exact Clean-Host Terminal Transcript

$ command -v elixir >/dev/null || echo "elixir=absent"
elixir=absent
$ command -v mix >/dev/null || echo "mix=absent"
mix=absent
$ command -v erl >/dev/null || echo "erl=absent"
erl=absent

$ sha256sum -c airgap_probe-1.4.0.tar.gz.sha256
airgap_probe-1.4.0.tar.gz: OK
$ mkdir -p /opt/airgap_probe/releases/1.4.0
$ tar -xzf airgap_probe-1.4.0.tar.gz -C /opt/airgap_probe/releases/1.4.0
$ cd /opt/airgap_probe/releases/1.4.0
$ test -d erts-* && echo "bundled_erts=present"
bundled_erts=present

$ export PROBE_PORT=4100
$ export PROBE_DATA_DIR=/var/lib/airgap_probe
$ export PROBE_SIGNING_SECRET='<injected-by-target-secret-store>'
$ export RELEASE_NODE=airgap_probe@127.0.0.1
$ export RELEASE_DISTRIBUTION=name
$ export RELEASE_COOKIE='<injected-release-cookie>'
$ bin/airgap_probe eval 'AirgapProbe.ReleaseTasks.preflight()'
preflight=ok release=1.4.0 data_dir=writable secret=present

Terminal A:
$ bin/airgap_probe start
12:00:00.000 [info] release=airgap_probe version=1.4.0 bundled_erts=true
12:00:00.020 [info] listening=127.0.0.1:4100 readiness=ready

Terminal B:
$ bin/airgap_probe pid
4127
$ curl --silent http://127.0.0.1:4100/health
{"status":"ready","release":"1.4.0","erts":"bundled"}
$ bin/airgap_probe rpc 'AirgapProbe.ReleaseTasks.identity()'
release=airgap_probe version=1.4.0 node=airgap_probe@127.0.0.1
$ bin/airgap_probe remote
Interactive Elixir (press Ctrl+C to exit)
iex(airgap_probe@127.0.0.1)1> AirgapProbe.ReleaseTasks.identity()
release=airgap_probe version=1.4.0 node=airgap_probe@127.0.0.1
iex(airgap_probe@127.0.0.1)2> <Ctrl+C>
$ bin/airgap_probe stop
ok
$ curl --silent --max-time 1 http://127.0.0.1:4100/health || echo "service=stopped"
service=stopped

3.8 Scope Boundary Versus Projects 9, 26, and 32

Project 9 teaches in-place OTP release handling, appup/relup, state migration, and hot upgrade/rollback. Project 36 deploys whole immutable artifacts and restarts processes; it does not implement hot code loading.
Project 26 produces Nerves firmware containing a Linux system, ERTS, and application for embedded hardware. Project 36 produces a conventional Mix release tarball for a declared host OS; it does not build firmware, a kernel, or a root filesystem.
Project 32 inspects and compares existing release artifacts for BEAM chunks and provenance. Project 36 creates, packages, installs, and operates the release; P32 is an optional downstream audit gate.

4. Solution Architecture

4.1 High-Level Design

+---------------------- compatible build environment ----------------------+
| source revision + mix.lock + pinned Elixir/OTP + target policy            |
|                |                                                          |
|          compile(prod) -> assemble(include ERTS) -> inspect -> tar         |
|                                                        |                   |
|                         checksum + provenance <--------+                   |
+--------------------------------------------------------+-------------------+
                                                         |
                                                controlled transfer
                                                         |
+------------------------- clean target -----------------v-------------------+
| verify bytes + compare target + inventory shared libraries                |
|                              |                                             |
|                   extract releases/1.4.0                                  |
|                              |                                             |
| runtime env -> preflight -> start candidate -> readiness gate              |
|                              | pass                 | fail                  |
|                              v                      v                       |
|                   current -> 1.4.0          stop/quarantine candidate       |
|                              |                                             |
|                 service manager + logs + health                           |
|                              |                                             |
|                 rollback -> previous known-good                           |
+----------------------------------------------------------------------------+

4.2 Key Components

Component	Responsibility	Key decision
AirgapProbe Application	Supervised service and health identity	Keep business scope intentionally small
Release Definition	Application modes, ERTS, executables, steps	Explicit `include_erts: true`; tar after assembly
Runtime Contract	Parse and validate boot configuration	Fail closed without revealing secrets
Artifact Builder	Compile, assemble, sanitize/inspect, tar	Run only in matching pinned environment
Provenance Writer	Bind artifact to inputs and target	Sanitize environment and record policy version
Target Preflight	Compare platform and native dependencies	Reject before extraction/promotion where possible
Installer	Extract immutable version directory	Never overwrite current directory in place
Health Gate	Verify VM, application, endpoint, dependencies	Promotion requires bounded readiness evidence
Service Manager Boundary	Own foreground process, logs, restart policy	Keep OS restart policy separate from OTP tree
Promotion/Rollback Controller	Switch current/previous pointers	Preserve known-good and data compatibility evidence

4.3 Data Structures (No Full Code)

BuildIdentity: source revision, dirty flag, lock digest, Elixir/OTP, target, builder image, policy version.
ArtifactIdentity: release name/version, filename, size, digest, release-tree inventory digest.
NativeRequirement: path, kind (NIF, Port, executable, ERTS object), target, required shared libraries.
RuntimeVariable: name, class, required/default, validator, secret flag, safe failure label.
TargetIdentity: architecture, OS/vendor/version, ABI/libc, available library policy, filesystem capabilities.
DeploymentState: staged, verified, extracted, preflighted, started, healthy, promoted, failed, rolled_back.
DeploymentRecord: candidate, current, previous, checks, timestamps, result, rollback/data compatibility note.

4.4 Algorithm Overview

Build and deployment are two explicit state machines. Build moves from pinned inputs through compile, assemble, layout inspection, tar, digest, and provenance. Deployment moves from received through verified, compatible, extracted, configured, started, healthy, and promoted. Every failure produces a terminal record and leaves current unchanged. Rollback stops the failed current candidate, validates the known-good runtime contract, switches the pointer, starts it, and rechecks health.

Artifact hashing is O(B) time in artifact bytes and bounded streaming memory. Layout/native inventory is O(F) in files plus external loader inspection. Extraction is O(B). Promotion is constant-time when implemented as an atomic pointer switch on one filesystem. Retained storage is O(K * B) for K kept release versions, excluding shared application data.

5. Implementation Guide

5.1 Development Environment Setup

Choose one supported Elixir/OTP pair and one target contract. Use a disposable VM, image, or container matching the production architecture/OS/ABI for builds. Provision a second clean target image from the same contract without Erlang/Elixir packages. Provide checksum/archive/platform inspection tools, a service manager or two-terminal foreground harness, and a writable external data directory.

5.2 Project Structure

airgap_probe/
├── config/
│   ├── config.exs
│   └── runtime.exs
├── lib/airgap_probe/
│   ├── application.ex
│   ├── endpoint.ex
│   ├── health.ex
│   ├── build_identity.ex
│   └── release_tasks.ex
├── rel/
│   ├── env.sh.eex
│   ├── vm.args.eex
│   ├── remote.vm.args.eex
│   └── overlays/
├── release_lab/
│   ├── target-contract.md
│   ├── build-policy.md
│   ├── manifest-schema.md
│   ├── deployment-state-machine.md
│   └── rollback-runbook.md
├── test/
│   ├── runtime_contract/
│   ├── release_layout/
│   ├── clean_target/
│   └── rollback/
└── mix.exs

5.3 The Core Question You Are Answering

“What exactly must an Elixir release contain—and what must the target still provide—so one verified tarball can run and be operated without installing Erlang, Elixir, Mix, or application source?”

5.4 Concepts You Must Understand First

OTP applications, dependency modes, supervision startup, and shutdown order.
Mix compile environment versus config/runtime.exs boot configuration.
ERTS, boot scripts, release directory layout, and release command wrappers.
CPU architecture, OS/vendor, ABI/libc, dynamic loaders, shared libraries, NIFs, and Ports.
Cryptographic digests versus signatures, provenance, and reproducibility claims.
Unix foreground processes, signals, service managers, writable state, and immutable directories.
Distribution node names/cookies and the execution contexts of eval, rpc, and remote.
Backward-compatible configuration and data evolution needed for whole-artifact rollback.

5.5 Questions to Guide Your Design

What exact target triple and OS/system-library policy does one artifact support?
Which included applications are required, and which mode starts each one?
How will the build prove ERTS exists before packaging?
Which values are compile-time facts, runtime configuration, computed release variables, or secrets?
How will remote commands receive the same node/cookie contract without leaking it into shell history?
Which files must be writable, and why are none inside the version directory?
What evidence distinguishes VM alive, application ready, and dependency ready?
Which native libraries or executables constrain portability?
What exact checks happen before extraction, before start, and before promotion?
What data/configuration compatibility must remain true for the previous artifact to boot?

5.6 Thinking Exercise

Draw two deployment timelines. In the first, a correct 1.4.0 artifact is verified, started, promoted, stopped, and restarted without a system toolchain. In the second, 1.5.0 passes checksum but fails because one NIF needs an unavailable shared library. Mark every filesystem path, process, pointer value, health result, and deployment record. Explain why current must remain on 1.4.0 and why checksum success did not imply runtime compatibility.

5.7 The Interview Questions They Will Ask

“What does include_erts: true include, and why does it not make a release platform-independent?”
“What is the difference between assembly and the tar release step?”
“Why is runtime.exs different from config.exs, and why can it not use Mix?”
“How do eval, rpc, and remote differ?”
“Why can a pure BEAM application still depend on target system libraries?”
“How would you prove an Elixir release works on a server with no Erlang installation?”
“What does a checksum prove compared with provenance or a signature?”
“Why can code rollback fail after a successful database migration?”

5.8 Hints in Layers

Hint 1: Inspect before compressing — Treat the uncompressed release tree as the first artifact. Verify ERTS, boot/config files, applications, priv, and forbidden paths.

Hint 2: Make target identity an input — Refuse to build a generically named “Linux artifact.” Name and record the target contract.

Hint 3: Test the transferred bytes — Copy only tar/manifest into the clean host and move the original _build tree out of reach.

Hint 4: Keep eval honest — Preflight tasks must explicitly start only what they need and must not assume the service node exists.

Hint 5: Fail before switching — Extraction and candidate boot happen under a version path; the stable pointer changes only after health.

Hint 6: Debug native failures from the loader outward — Inspect executable format, target triple, NIF/Port location, dynamic dependencies, permissions, and only then application code.

5.9 Books That Will Help

Topic	Book	Precise reading
Deployment pipelines	Continuous Delivery — Jez Humble and David Farley	Chapter 5 “Anatomy of the Deployment Pipeline” and Chapter 10 “Deploying and Releasing Applications”
Production process behavior	Release It!, 2nd Edition — Michael T. Nygard	“Processes on Machines” and stability-pattern sections; connect to health and restart boundaries
Linux processes and shared libraries	How Linux Works, 3rd Edition — Brian Ward	Chapters on process management, shared libraries, and system startup
Shell artifact operations	The Linux Command Line, 2nd Edition — William E. Shotts	Chapters on archives, checksums, permissions, environment, and process control

5.10 Implementation Phases

Phase 1: Release Anatomy and Runtime Contract (5-7 hours)

Build the supervised probe service and distinguish liveness/readiness.
Define required runtime values and safe failure diagnostics.
Initialize release templates and document RELEASE_* policy.
Assemble with ERTS and inspect the release tree.

Checkpoint: The uncompressed release starts through its wrapper, reports build identity, rejects missing runtime values, and contains no secret fixture.

Phase 2: Tar, Integrity, and Target Evidence (5-8 hours)

Add :tar after :assemble.
Stream artifact checksum and write sanitized provenance.
Inventory release files, OTP applications, NIF/shared objects, Ports, and executable assets.
Define target preflight and artifact naming.

Checkpoint: A verifier rejects altered bytes, wrong release identity, missing ERTS, wrong target identity, and forbidden files.

Phase 3: Clean-Host Operations (6-9 hours)

Provision matching clean target without Erlang/Elixir/Mix.
Verify, extract, inject runtime values, and execute preflight.
Exercise start, PID, health, RPC, remote shell, stop, signals, logs, and writable paths.
Capture the deterministic terminal transcript.

Checkpoint: The exact transferred tarball completes the transcript while all three system toolchain commands remain absent.

Phase 4: Immutable Promotion and Rollback (6-10 hours)

Add version directories plus current/previous pointers.
Implement bounded readiness and promotion records.
Inject wrong-ABI/native-library and unhealthy-candidate failures.
Roll back and prove service/data compatibility.
Optionally feed artifact into Project 32’s auditor.

Checkpoint: A failed 1.5.0 candidate cannot corrupt or replace healthy 1.4.0, and the rollback record explains every decision.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
ERTS	bundled / target-installed	Bundled explicitly	Proves no target Erlang installation is required
Packaging	ad hoc tar / Mix `:tar` step	Mix `:tar` after `:assemble`	Preserves official release pipeline semantics
Runtime secrets	compile config / fallback file / runtime provider	Runtime provider; remove cookie fallback	Prevents artifact capture and build-host coupling
Builder	developer laptop / matching pinned image	Matching pinned image	Enforces target architecture/OS/ABI contract
Installation	overwrite current / version directories	Immutable version directories	Makes failure and rollback bounded
Execution	background daemon by default / foreground service	Foreground under supervisor	Clear logs, signals, and restart ownership
Promotion	process started / bounded readiness	Bounded readiness	PID does not equal application readiness
Rollback	old tar exists / tested compatibility window	Tested compatibility window	Code alone cannot reverse external state

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Runtime Contract	Prove boot-time parsing and secrecy	required/malformed vars, no secret echo
Release Layout	Verify assembled target system	ERTS, boot files, apps, `priv`, executable wrapper
Step Ordering	Verify package pipeline	tar cannot be accepted without completed assembly evidence
Artifact Integrity	Detect transfer/change	valid, truncated, bit-flipped, wrong manifest
Compatibility	Enforce target boundary	wrong architecture, OS, ABI, missing library
Native Boundary	Exercise NIF/Port inventory	correct asset, wrong ELF architecture, missing dependency
Clean Target	Prove no system runtime	absent `elixir`/`mix`/`erl`, bundled service boots
Command Semantics	Test operations	eval before start; pid/rpc/remote/stop after start
Process Lifecycle	Verify signals and external supervision	SIGTERM, graceful shutdown, restart limits
Deployment	Verify state transitions	verify, extract, preflight, health, promote
Rollback	Preserve last known-good	unhealthy candidate, pointer restoration, data readability
Provenance	Validate evidence	revision, lock/toolchain/target, sanitized environment

6.2 Critical Test Cases

Release tree contains an erts-* directory and runs when system erl is absent.
Tar artifact is produced only from a successful assembled release and contains one coherent top-level layout.
Generated cookie fallback is absent from the tar, a missing injected cookie fails closed, and the injected cookie enables service and remote commands.
Missing runtime secret fails boot with variable name but not value or surrounding environment.
A value changed only at target boot changes behavior without rebuilding the tarball.
eval preflight succeeds before the service starts and does not claim the application is running.
pid, rpc, and remote fail with a classified identity/cookie diagnostic when policy is intentionally mismatched.
Correct node/cookie settings make PID, RPC, remote shell, and stop work against the running release.
Wrong target triple is rejected before promotion.
Missing system library or wrong-architecture NIF is detected by preflight/smoke testing.
Bit-flipped tarball fails digest verification before extraction.
Release source, Mix cache, dependency cache, and build tree are unavailable on clean-host test.
SIGTERM produces orderly application shutdown and bounded exit.
Candidate health failure leaves current unchanged.
Post-promotion failure switches to previous, starts it, and proves shared data remains readable.
Provenance contains no cookie, secret, database URL, or complete unfiltered environment dump.

6.3 Test Data

Maintain two valid versions, one deliberately unhealthy candidate, one truncated archive, one wrong-target manifest, one missing-library fixture, one malformed runtime configuration set, and one small shared-data format explicitly readable by both valid versions. Use deterministic release identity and health outputs while allowing timestamps/PIDs to be normalized only in assertions where they are inherently variable.

6.4 Definition of Done

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem	Why it happens	Fix	Quick test
Target says executable not found although file exists	Wrong loader, ABI, architecture, or shared library	Compare target triple and loader dependencies	Inspect executable format and dependency resolution
Release works only on build machine	Tested `_build` tree with installed toolchain	Transfer only tar/manifest to clean target	Hide source/build paths and remove toolchain from target
Secret appears in archive	Read or generated during build	Move to `runtime.exs`/provider and rebuild from clean tree	Scan archive and manifest for secret canary
`runtime.exs` crashes on Mix call	Mix does not exist in release runtime	Use Config/System/application APIs only	Boot on clean target
Remote shell cannot connect	Node name, distribution, cookie, hostname, or ports differ	Publish one remote-command environment contract	Compare service and helper identities safely
`eval` cannot call application service	Applications are not started in eval VM	Explicitly start required apps or keep task pure	Run preflight with service stopped
Tarball misses files	Ad hoc archive or undeclared `priv`/overlay	Use official release tar step and layout allow-list	Compare assembled and tar inventories
Health passes too early	PID or socket-open used as readiness	Gate on application/dependency invariants	Delay dependency and observe readiness
Rollback old code fails	Candidate changed shared data incompatibly	Expand-contract/versioned data or restore plan	Run old release against post-candidate fixture
Release directory changes over time	Logs/data/config written beneath current root	Externalize writable state	Snapshot file hashes before/after run

7.2 Debugging Strategies

Start with artifact identity: filename, digest, release name/version, target, and selected RELEASE_VSN.
Inspect the extracted erts-*, lib, releases, wrapper permissions, and start_erl.data before application logs.
Separate wrapper/environment failures, VM loader failures, boot-script failures, runtime configuration failures, application startup failures, and readiness failures.
For native load errors, inspect file architecture and dynamic dependencies on both builder and target; do not repeatedly change Elixir code first.
Use safe release identity/RPC tasks rather than printing the full environment or cookie.
Run eval preflight with service stopped, then run rpc inspection with service running to expose context confusion.
Compare immutable release-tree hashes before and after execution to locate accidental writes.
Preserve failed-candidate logs and deployment records without leaving the failed process running.

7.3 Performance Traps

Recompressing already compressed large assets without measuring build time or size
Including unused applications, debug/source artifacts, caches, or fixtures
Hashing by reading the whole tar into memory instead of streaming
Running expensive external checks during every health probe
Starting remote helper VMs in high-frequency monitoring loops
Retaining unlimited versions or logs beneath release storage
Rebuilding on every target rather than promoting one target-specific artifact

8. Extensions & Challenges

8.1 Beginner Extensions

Add an operator command that prints safe release/build identity.
Add a release-tree allow-list and forbidden-file canary scan.
Add a human-readable compatibility report alongside JSON-like manifest data.

8.2 Intermediate Extensions

Add a systemd unit and verify foreground logs, SIGTERM, restart limits, and writable directories.
Build separate glibc and musl artifacts with unambiguous names and manifests.
Add an offline signature verification step with a protected signing workflow.
Use P32 to compare two builds and distinguish integrity from reproducibility.

8.3 Advanced Extensions

Build an ARM artifact on a native matching builder and deploy it to an isolated ARM target.
Add a Rustler NIF, record its native dependencies, and prove wrong-target rejection.
Add SBOM/provenance attestations connected to Project 19 without leaking credentials.
Design a multi-application umbrella with two different release subsets and application modes.
Add a staged deployment controller with bounded canary traffic and automatic rollback evidence.
Compare immutable restart deployment with P9 hot upgrade, documenting operational and state-migration trade-offs.

9. Real-World Connections

9.1 Industry Applications

Mix releases are used to deploy Elixir systems onto VMs, bare hosts, containers, appliances, and regulated networks without installing a language toolchain on production machines. The release becomes the promoted unit across environments, while runtime configuration and secrets remain target-owned. Air-gapped environments make the implicit artifact contract impossible to ignore: every library, native dependency, configuration value, operation, and rollback step must be available without reaching package repositories or source-control systems.

Mix release tooling for assembly, tar packaging, runtime configuration, and management scripts
Erlang/OTP systools, boot scripts, ERTS target-system structure, and release metadata
systemd or another external process supervisor for foreground service operation
Project 19’s SBOM/license auditor for dependency evidence
Project 32’s BEAM artifact/reproducibility auditor for post-build inspection
Nix/Guix or controlled container builders for repeatable target environments, without confusing an image with the Mix release itself

9.3 Interview Relevance

This project demonstrates that the learner can bridge Elixir code, OTP target systems, binary compatibility, configuration security, Linux process operation, supply-chain evidence, and deployment rollback. Strong answers state the boundaries: ERTS inclusion removes one installation dependency; it does not erase OS/ABI/native-library requirements, and a preserved tarball does not guarantee data rollback.

10. Resources

10.1 Essential Official Reading

Mix release task — self-contained assembly, running commands, deployment requirements, options, steps, runtime configuration, environment variables, and directory layout.
Mix release.init task — generating vm.args, remote VM args, and environment templates.
Mix.Release — release structure available to custom steps.
Elixir configuration and Mix configuration — build/runtime configuration background.
Erlang/OTP release structure — release resource, ERTS, applications, and target layout.
Erlang/OTP system principles — starting/stopping target systems and boot concepts.
Erlang/OTP creating a target system — target-system installation model and runtime layout.
Erlang erl runtime flags — VM argument reference used by release templates.

10.2 Books and Deeper Study

Continuous Delivery by Jez Humble and David Farley — Chapters 5 and 10 for deployment pipelines, immutable candidates, promotion, and release strategies.
Release It!, 2nd Edition by Michael T. Nygard — process-on-machine and stability-pattern sections for health, shutdown, restart, and production failure boundaries.
How Linux Works, 3rd Edition by Brian Ward — process management, startup, shared libraries, and system runtime foundations.
The Linux Command Line, 2nd Edition by William E. Shotts — archives, checksums, permissions, environment variables, and process operation.

Confirm the title, number, filename, target, and release name agree.
Measure Fundamentals and Deep Dive word counts for both theory chapters.
Show ERTS presence and absent system elixir, mix, and erl in one clean-host transcript.
Show :assemble before :tar, and verify the actual transferred artifact.
Exercise target-time configuration without Mix and without embedding secrets.
Demonstrate eval, start, pid, rpc, remote, and stop under one documented identity contract.
Record target architecture, OS/vendor, ABI, native assets, and required shared libraries.
Prove corrupted transfer and wrong target fail before promotion.
Prove unhealthy candidate cannot destroy the known-good release.
Test rollback against shared data/configuration, not just pointer movement.
Keep all examples as configuration fragments, pseudocode, architecture, transcripts, and expected outputs rather than a runnable solution.

This guide expands Project 36 from the BEAM, Elixir, and Erlang learning journey. For the complete project sequence, see the expanded project index.