Project 31: Multi-Tenant Dynamic Repo Control Plane

Build a database-per-tenant control plane that starts Ecto repositories on demand, routes work without tenant bleed, constrains total pools, coordinates migrations, and evicts only resources that are provably idle.

Quick Reference

Attribute Value
Difficulty Level 4: Expert
Suggested Seniority Staff Elixir engineer
Time Estimate 24-40 hours
Main Programming Language Elixir
Alternative Programming Languages Erlang for lifecycle components
Coolness Level Level 4: Hardcore Tech Flex
Business Potential Level 4: Open Core Infrastructure
Prerequisites Ecto.Repo, PostgreSQL, supervision, Registry, process context, migrations, resource pooling
Key Topics dynamic repos, process-local routing, tenant isolation, pool lifecycle, leases, bounded capacity, migration orchestration

1. Learning Objectives

By completing this project, you will:

  1. Explain Ecto’s default repository identity and dynamic repository routing.
  2. Route each database operation through an authenticated tenant context.
  3. Preserve and restore process-local repo selection safely across nested operations.
  4. Understand why spawned tasks do not automatically inherit the caller’s routing context.
  5. Start, monitor, lease, idle, evict, and restart tenant repository supervision trees.
  6. Bound total connections and startup concurrency across many tenant databases.
  7. Coordinate per-tenant migrations with resumable, observable failure states.
  8. Prove tenant isolation with adversarial concurrency and fault tests.

2. All Theory Needed (Per-Concept Breakdown)

Dynamic Repository Routing and Tenant Context Safety

Fundamentals

An Ecto repository module usually points to one running repository instance, but Ecto.Repo.put_dynamic_repo/1 can select another repository name or PID for subsequent calls made through that module in the current process. The selection is process-local. It is not a global switch, is not automatically inherited by newly spawned processes, and remains in effect until changed. A safe tenant-routing boundary authenticates a tenant ID, resolves it to a live repo instance, saves the caller’s previous dynamic repo, installs the tenant repo, executes the operation, and restores the previous value in an unconditional cleanup path. The tenant ID itself is not a repo name and must never be turned into an unbounded atom. Routing context is security-sensitive state.

Deep Dive

Dynamic repo routing looks deceptively small because the public operation is one call. The architectural work surrounds that call. Every database access path must establish an authenticated tenant context, and no background task or callback may accidentally fall back to the default repository. Treat a missing context as an error in tenant-scoped code rather than a convenience fallback. A default repo is useful for control-plane metadata but dangerous as an implicit data-plane destination.

Begin with identity separation. External requests carry an untrusted tenant reference. Authentication and authorization resolve that reference to an internal stable tenant ID. The control plane looks up connection metadata and a running repo lease. Only then does the execution boundary install the repo PID. Never accept a PID, atom, database URL, or schema prefix directly from a client. Avoid creating atoms from tenant strings because atoms are not garbage-collected in the ordinary way and an attacker can exhaust the VM atom table. Repo PIDs and opaque references are safer runtime identities.

put_dynamic_repo associates routing with the current process. Nested tenant contexts must be stack-safe: save the prior repo, set the new one, and restore the exact prior value in after, including exceptions, throws, and exits that cleanup can observe. If policy forbids switching tenants within one request, detect and reject nesting rather than silently allowing it. Restoring blindly to the module’s default can break an outer legitimate context.

Process boundaries require explicit propagation. A Task started inside a tenant context begins as another process and does not inherit arbitrary process dictionary entries. An asynchronous job also runs later with no request process at all. Pass a validated tenant ID or a lease token as explicit data, then re-enter the tenant boundary in the new process. Do not pass a transient repo PID into a durable queue payload; it will be meaningless after restart. Background workers resolve the tenant again and can decide what to do if the tenant is suspended or migrating.

Transactions and streams add lifetime constraints. Every query inside an Ecto transaction must use the same selected repo and checked-out connection semantics. The tenant lease must outlive the transaction. A database stream can retain a connection while enumerated and cannot escape the routing boundary safely. Document that lazy database streams must be fully consumed inside the tenant context, or return materialized domain values instead.

Observability must carry tenant identity without exposing credentials. Attach internal tenant ID, repo generation, operation class, and correlation ID to telemetry metadata. Do not log database passwords or full connection URLs. If a query reaches the default data-plane repo, emit a high-severity isolation diagnostic. In tests, give each tenant distinct sentinel records so a routing leak is immediately visible.

Isolation includes error handling. A disconnected tenant database should produce a tenant-scoped availability error, not trigger a fallback to another tenant or global database. A repo that dies mid-operation invalidates its lease generation. Retried work must re-resolve through the control plane. This avoids using stale PIDs and creates a clean point for circuit state, suspension, or migration checks.

How this fits on the project

This concept defines the request boundary, background-task contract, lease token, transaction safety, telemetry context, and isolation tests.

Definitions and key terms

  • Dynamic repo: Repository identity selected for calls made by a Repo module in one process.
  • Tenant context: Authenticated, explicit routing scope for one tenant’s operation.
  • Lease: Proof that a repo generation is active and may not be evicted during work.
  • Repo generation: Identity that changes when a tenant repo restarts.
  • Tenant bleed: A data operation reaching the wrong tenant’s database.
  • Control-plane repo: Separate database containing tenant metadata, not tenant business data.

Mental model diagram

untrusted request tenant_ref
          |
          v
[authn/authz] --> internal tenant_id
          |
          v
[control plane acquire lease] --> {repo_pid, generation, lease_token}
          |
          v
[tenant execution boundary]
  save previous dynamic repo
  install repo_pid
  run complete transaction/stream
  restore previous repo in cleanup
          |
          v
[release lease]

spawned task: pass tenant_id explicitly --> acquire its own context

How it works

  1. Authenticate and authorize the external tenant reference.
  2. Resolve an internal tenant record and current operational state.
  3. Acquire a lease for a live repo generation.
  4. Save the process’s previous dynamic repo.
  5. Install the leased repo PID and run the bounded operation.
  6. Restore the previous value and release the lease even on failure.
  7. Spawned work receives tenant identity explicitly and repeats the boundary.
  8. Invariant: tenant-scoped query code cannot run with an absent or different tenant context.
  9. Failure modes include stale PID, forgotten cleanup, nested-context corruption, Task context loss, lazy stream escape, and unsafe fallback.

Minimal concrete example

PSEUDOCODE
with authorized tenant_id,
     lease <- control_plane.acquire(tenant_id):
  previous = repo.get_dynamic_repo()
  try:
    repo.put_dynamic_repo(lease.repo_pid)
    assert current_tenant_context == lease.tenant_id
    execute bounded database operation
  after:
    repo.put_dynamic_repo(previous)
    control_plane.release(lease.token)

Common misconceptions

  • Dynamic repo selection changes routing globally for every process.
  • A Task automatically inherits the caller’s selected repo.
  • Restoring to the default repo is equivalent to restoring the previous repo.
  • Tenant IDs are safe to convert into atoms because they came from authentication.

Check-your-understanding questions

  1. Why must a lazy database stream stay inside the tenant boundary?
  2. What should a durable job store instead of a repo PID?
  3. Why does a lease include a repo generation?

Check-your-understanding answers

  1. Enumeration may issue queries and retain a checked-out connection after context or lease release.
  2. A stable internal tenant ID plus operation data; it re-resolves the runtime repo later.
  3. A restarted repo can reuse the tenant identity while invalidating old PID-based assumptions.

Real-world applications

  • Database-per-customer SaaS
  • Regional or regulated data residency
  • Enterprise customer isolation
  • On-premises customer database gateways

Where you will apply it

  • Project 31 request routing, background workers, transactions, streams, and leak tests

References

Key insight

Dynamic repo routing is process-local capability state and must be established, propagated, and cleaned up like a security boundary.

Summary

Authenticate tenant identity, acquire a generation-aware lease, scope dynamic routing with guaranteed restoration, and explicitly re-establish context across every process boundary.

Homework/Exercises

  1. Trace a request that starts two Tasks and explain the correct repo in each process.
  2. Design a test that detects restoration to default instead of restoration to an outer context.

Solutions

  1. The request and each Task independently acquire tenant context using the explicit internal tenant ID; no Task depends on inherited process state.
  2. Enter tenant A, nest an allowed tenant B test boundary, exit B, and assert a sentinel query still reaches A before exiting the outer boundary.

Repository Lifecycle, Capacity, and Migration Orchestration

Fundamentals

Each running tenant Ecto.Repo owns a supervision tree and a database connection pool. Starting every tenant permanently can exhaust connections, file descriptors, memory, and database limits. A control plane therefore tracks tenant repo states such as stopped, starting, ready, draining, migrating, degraded, and failed. DynamicSupervisor starts and stops repo children, Registry or an internal index maps tenant IDs to entries, and leases prevent eviction during work. Capacity policy bounds total pools, connections, and concurrent starts. Idle eviction first marks a repo draining, rejects or queues new leases, waits for active leases to reach zero, and then stops the correct generation. Migrations are separately locked, versioned operations with per-tenant progress and global concurrency limits.

Deep Dive

The resource unit is not “one tenant process”; it is an entire repository supervision tree plus its pool and database-side connections. If 5,000 tenants each receive a pool of ten, a naive eager strategy demands 50,000 connections. The control plane needs a capacity model expressed in connections and memory, not only repo count. It may assign pool sizes by tenant tier while enforcing a hard global budget. Starting a repo consumes capacity before it becomes ready, so pending starts count against a startup reservation.

Model lifecycle as explicit state. A stopped tenant has metadata but no runtime pool. Starting means one authorized starter owns initialization while other callers wait within a bounded queue or receive a retryable response. Ready accepts leases. Degraded may accept limited work or fail fast based on health. Draining accepts no new leases and waits for existing work. Migrating blocks ordinary writes and follows a documented read policy. Failed stores a classified reason and backoff, preventing every request from launching a connection storm.

Single-flight startup is essential. Concurrent first requests for one tenant should converge on one repo start. Use a control-plane entry or dedicated lifecycle process to serialize transitions, not a check-then-start sequence vulnerable to races. The startup result publishes a new generation and pool metadata. Waiters have deadlines and cancellation; an abandoned caller must not cancel a startup needed by others.

Eviction requires more than last_used_at. A query may be running, a transaction may hold a connection, or a stream may still be active. Lease count captures control-plane-known work. Transition ready to draining atomically, preventing new leases. When count reaches zero, stop the exact PID/generation. If work never releases, emit diagnostics and apply a deliberate maximum-drain policy rather than silently killing transactions. Update last-used using monotonic time inside one runtime; persistent idle decisions after restart need wall-clock records with cautious interpretation.

Capacity saturation is normal, not an exception. If all repos are leased and the budget is full, decide between bounded wait, victim selection among idle ready repos, or rejection with retry guidance. Never evict an active tenant to admit another without an explicit higher-level preemption policy. LRU is only a candidate ordering; eligibility requires ready state, zero leases, no migration, and no pinned status.

Credentials should arrive from a runtime provider and remain outside logs and durable lifecycle events. Rotation can create a new generation: mark current repo draining, start a replacement under the new credentials within a temporary rotation budget, switch new leases, then retire old generation after its leases finish. If simultaneous generations are too expensive, schedule maintenance and fail predictably.

Migrations are a fleet operation. Determine desired migration version, acquire a per-tenant migration lock, mark lifecycle state, run migrations using the target repo/configuration, record version and outcome, and restore ready or failed state. Bound global migration concurrency to protect both the BEAM node and database servers. A failed tenant does not roll back unrelated tenants. Store a migration campaign with pending, running, succeeded, failed, and skipped counts so it can resume after control-plane restart.

Schema compatibility governs rolling application deployment. New application code may run while some tenant databases remain on the old version. Prefer expand/contract migrations and publish minimum/maximum compatible schema versions. The router can reject a feature requiring a newer version or direct the tenant through an upgrade path. “Run all migrations at deploy time” is not viable when thousands of isolated databases have different availability windows.

Observability should include repo state, generation, pool size, active lease count, waiters, startup duration, last use, migration version, failure class, and retry time. Aggregate metrics need tenant-cardinality controls; operator detail can be queried by tenant rather than placing every tenant ID on every global metric label.

How this fits on the project

This concept defines lifecycle state, single-flight startup, capacity admission, draining eviction, credential rotation, and migration campaigns.

Definitions and key terms

  • Single-flight start: One shared startup attempt for concurrent requests to the same tenant.
  • Draining: State that blocks new leases while allowing existing work to finish.
  • Capacity budget: Hard limit expressed in repo pools, database connections, and startup reservations.
  • Pinned tenant: Repo excluded from idle eviction by policy.
  • Migration campaign: Resumable fleet operation over many tenant databases.
  • Expand/contract: Schema evolution that preserves compatibility across staged application deployment.

Mental model diagram

                 startup failure
                      |
[stopped] -> [starting] -> [ready] -> [draining] -> [stopped]
                 |           |  ^          |
                 |           v  |          +-- waits for leases=0
                 |       [migrating]
                 |           |
                 +------> [failed] -- backoff/manual repair --> [starting]

GLOBAL CAPACITY
sum(pool_size for ready/starting/draining generations) <= connection budget

ADMISSION
tenant request -> ready lease | single-flight start | bounded wait | retryable reject

How it works

  1. Load tenant metadata and current lifecycle entry.
  2. If ready and healthy, increment lease count for current generation.
  3. If stopped, reserve capacity and start exactly once.
  4. If full, evict an eligible idle repo or apply bounded admission policy.
  5. Release leases after operations and update idle metadata.
  6. Drain eligible repos before stopping the exact generation.
  7. Run migration campaigns with per-tenant lock and global concurrency cap.
  8. Invariant: allocated pool capacity never exceeds the declared budget except a documented rotation reserve.
  9. Failure modes include startup stampede, stale stop, leaked lease, active eviction, connection storm, migration split ownership, and cardinality-heavy metrics.

Minimal concrete example

LIFECYCLE TRANSCRIPT
tenant=T-204 state=stopped request=acquire
capacity reserved connections=5 total=45/50
generation=8 state=starting waiters=3
repo ready pid=<runtime-pid> state=ready leases=3
idle timer elapsed but leases=1 -> eviction deferred
leases=0 state=draining -> repo stopped -> capacity=40/50

Common misconceptions

  • An idle timestamp proves no transaction or stream is active.
  • A bound on repo count is equivalent to a database connection budget.
  • Migrations across tenants should be one all-or-nothing transaction.
  • A failed connection should trigger an immediate start on every request.

Check-your-understanding questions

  1. Why must starting repos reserve capacity?
  2. What conditions make a repo eligible for LRU eviction?
  3. Why does a migration campaign need application/schema compatibility ranges?

Check-your-understanding answers

  1. Pool connections and memory are consumed during initialization, before ready state.
  2. Ready, zero leases, not pinned, not migrating, healthy enough to stop, and selected under the idle policy.
  3. Application versions and thousands of tenant schemas cannot change atomically; staged compatibility prevents outages.

Real-world applications

  • Enterprise SaaS with one database per customer
  • Customer-managed database connectivity
  • Regional database fleets
  • Test-environment database orchestration

Where you will apply it

  • Project 31 lifecycle manager, admission controller, evictor, credential rotation, and migrator

References

Key insight

A tenant repo is a leased pool with a lifecycle and capacity cost, not a hash-map entry that can be started or deleted casually.

Summary

Serialize startup, budget real connections, lease active generations, drain before eviction, classify failures with backoff, and operate migrations as resumable bounded campaigns.

Homework/Exercises

  1. Calculate a pool budget for 40 active standard tenants and 5 premium tenants.
  2. Trace credential rotation while two long transactions hold leases.

Solutions

  1. Multiply each tier’s active repo count by its pool size, reserve startup/rotation headroom, and keep the total below BEAM and database limits.
  2. Start or schedule the new generation under rotation policy, route new leases only after readiness, keep old generation for existing leases, then stop it when both release.

3. Project Specification

3.1 What You Will Build

Build a control-plane application that stores tenant connection metadata separately from tenant data, starts Ecto Repo instances on demand, grants generation-bound leases, scopes dynamic repo routing, enforces a global connection budget, drains idle repos, and runs resumable migration campaigns.

3.2 Functional Requirements

  1. Register tenant metadata without storing plaintext credentials in logs or status events.
  2. Start one repo generation for concurrent first requests.
  3. Acquire and release leases around every tenant operation.
  4. Route dynamic repo calls with guaranteed previous-context restoration.
  5. Pass tenant identity explicitly across Task and background-job boundaries.
  6. Enforce pool and startup budgets with bounded admission behavior.
  7. Drain and evict eligible idle repos without interrupting leased work.
  8. Detect repo death, invalidate its generation, and classify recovery state.
  9. Run per-tenant migrations with a global concurrency cap and resumable results.
  10. Expose operator status without high-cardinality global metrics.

3.3 Non-Functional Requirements

  • No tested request can read or write another tenant’s sentinel data.
  • Capacity remains within the declared connection budget under load and failure.
  • Startup and migration stampedes are prevented.
  • Control-plane restart reconstructs lifecycle truth from metadata and live children.
  • Tenant credentials are requested at runtime and redacted everywhere else.
  • Saturation returns bounded, explainable outcomes rather than unbounded queues.

3.4 Example Usage / Output

$ tenantctl acquire T-204
tenant=T-204 generation=8 state=ready repo_pid=<0.481.0> lease=L-991

$ tenantctl capacity
repos ready=8 starting=1 draining=1 failed=2
connections allocated=45 budget=50 startup_reserved=5 waiters=3
eviction_candidate=T-099 idle=43m leases=0 pinned=false

3.5 Data Formats / Schemas / Protocols

TENANT METADATA
{tenant_id, credential_reference, database_host_class, pool_tier,
 desired_schema_version, status, residency, pinned}

RUNTIME ENTRY
{tenant_id, generation, repo_pid, lifecycle_state, pool_size,
 leases, waiters, last_used_monotonic, retry_at, failure_class}

LEASE
{opaque_token, tenant_id, generation, repo_pid, issued_at, owner_pid}

MIGRATION RESULT
{campaign_id, tenant_id, from_version, to_version, state,
 started_at, finished_at, failure_summary}

3.6 Edge Cases

  • Two first requests arrive for a stopped tenant
  • Tenant is disabled between authentication and lease acquisition
  • Repo dies after lease grant but before query
  • Task starts without explicit tenant propagation
  • Nested tenant contexts attempt a forbidden switch
  • Lazy stream escapes its lease
  • Capacity full with every repo leased
  • Eviction timer races with a new lease
  • Migration runner crashes after applying migration but before recording success
  • Credential rotation overlaps active transactions

3.7 Real World Outcome

3.7.1 How to Run

$ mix deps.get
$ mix test
$ mix tenant.demo --tenants 20 --connection-budget 25 --concurrency 100

3.7.2 Golden Path Demo

One hundred concurrent operations target twenty tenants while only five pools of five connections may be allocated. The control plane shares same-tenant starts, leases ready repos, evicts only zero-lease idle candidates, and completes operations without reading any wrong-tenant sentinel.

3.7.3 Exact Terminal Transcript

$ mix tenant.demo --tenants 20 --connection-budget 25 --concurrency 100
operations=100 succeeded=100 tenant_bleed=0
repo_starts=20 shared_start_waiters=37 duplicate_starts=0
max_connections_allocated=25 budget=25 violations=0
evictions=15 active_repo_evictions=0
stale_generation_retries=2
result=PASS

3.8 Scope Boundary Versus Projects 1-13

The first thirteen projects teach generic supervision, distributed nodes, ETS, LiveView, backpressure, fault injection, and telemetry. This project uses supervision as one tool but centers on database tenancy, process-local routing as a security boundary, resource leases, pool capacity, and fleet migrations. It is neither a distributed KV store nor an ETS cache.

4. Solution Architecture

4.1 High-Level Design

                   [control-plane Repo]
                 tenant metadata/campaigns
                           |
request -> auth -> [Tenant Control Plane] <---- runtime credential provider
                         /   |    \
                admission leases migrator
                    |        |       |
             [DynamicSupervisor: tenant Repo generations]
                    |
              [Registry / runtime index]
                    |
          TenantContext.run(lease, operation)
                    |
      Ecto Repo module --dynamic--> leased tenant Repo PID

4.2 Key Components

Component Responsibility Key decision
Tenant Catalog Durable identity and policy Never mix control-plane and tenant data
Lifecycle Manager Serialize start/drain/fail transitions Generation-aware single-flight operations
Capacity Manager Reserve and release pool budget Count connections, not only repos
Lease Manager Protect active repo generations Zero leases required before eviction
Tenant Context Scope put_dynamic_repo Guaranteed exact restoration
Credential Provider Supply runtime connection secret No secret in lifecycle events
Idle Evictor Select eligible candidates LRU orders only eligible zero-lease repos
Migrator Run resumable bounded campaigns Per-tenant lock, global cap

4.3 Data Structures (No Full Code)

  • TenantPolicy: tier, residency, pool size, pinned flag, compatibility range.
  • RuntimeEntry: state machine for one current repo generation.
  • CapacityLedger: allocated, reserved, rotation reserve, and budget.
  • LeaseRecord: owner monitor, generation, issue time, release status.
  • MigrationCampaign: desired version plus per-tenant outcomes.
  • TenantExecutionError: unavailable, saturated, migrating, disabled, stale generation, or context violation.

4.4 Algorithm Overview

On acquisition, authorize tenant state and inspect its runtime entry. A ready repo grants a monitored lease. A stopped repo attempts a single-flight start after capacity reservation. Saturation selects an eligible idle victim or returns bounded wait/retry. Every operation installs the leased dynamic repo only inside a cleanup-protected context. Release decrements the exact generation. Eviction atomically drains, waits for zero leases, then stops and releases capacity.

Typical ready acquisition is O(1). Candidate eviction is O(log R) with an indexed idle ordering or O(R) for a simple scan over R running repos. Capacity memory is O(R + L + W) for repos, leases, and bounded waiters.

5. Implementation Guide

5.1 Development Environment Setup

Use a current Elixir, Ecto, Ecto SQL, and PostgreSQL. For deterministic tests, start multiple logical test databases or schemas while preserving a database-per-tenant runtime abstraction. Configure very small pool sizes to make capacity behavior observable.

5.2 Project Structure

tenant_control_plane/
├── lib/tenant_control_plane/
│   ├── application.ex
│   ├── catalog.ex
│   ├── lifecycle_manager.ex
│   ├── capacity_manager.ex
│   ├── lease_manager.ex
│   ├── tenant_context.ex
│   ├── credential_provider.ex
│   ├── idle_evictor.ex
│   ├── migrator.ex
│   ├── repo_factory.ex
│   └── operator_status.ex
├── priv/repo/migrations/
├── test/fixtures/
└── test/

5.3 The Core Question You Are Answering

“How can one BEAM application safely operate thousands of isolated tenant databases when only a bounded subset may hold live pools at once?”

5.4 Concepts You Must Understand First

  1. Ecto Repo lifecycle, pool behavior, transactions, and migrations.
  2. Process-local dynamic repo selection and cleanup.
  3. DynamicSupervisor, Registry, monitors, and generation identity.
  4. Resource leasing, draining, capacity reservation, and backoff.
  5. Database-per-tenant migration compatibility.

5.5 Questions to Guide Your Design

  1. Which process proves tenant authorization before routing?
  2. How are Tasks and durable jobs forced to re-establish context?
  3. What exact resource does the global budget count?
  4. How can eviction race with acquisition without killing active work?
  5. Which application versions can operate each schema version during a campaign?

5.6 Thinking Exercise

Trace three simultaneous requests for a stopped tenant while capacity is full. One idle repo is eligible, one repo has a long transaction, and one is migrating. Identify the only legal victim, when the startup reservation is made, and what each waiter sees if startup fails.

5.7 The Interview Questions They Will Ask

  1. “How does put_dynamic_repo scope repository selection?”
  2. “Why is process dictionary routing dangerous across Tasks?”
  3. “How do you prevent two first requests from starting two pools?”
  4. “What proves a repo is safe to evict?”
  5. “How would you rotate credentials without interrupting transactions?”
  6. “How do expand/contract migrations apply to database-per-tenant SaaS?”

5.8 Hints in Layers

Hint 1: Make missing context fail — Tenant-scoped modules should not quietly use the default repo.

Hint 2: Lease generations — A tenant ID alone cannot prove that a PID is still the current repo.

Hint 3: Drain before stop — Atomically block new leases, then wait for existing leases to reach zero.

Hint 4: Campaigns are data — Persist per-tenant migration state so a control-plane restart can resume safely.

5.9 Books That Will Help

Topic Book Precise reading
Ecto repositories Programming Ecto — Darin Wilson and Eric Meadows-Jönsson Chapters on repositories, queries, transactions, and Multi
Multi-tenancy Programming Ecto Chapter/section “Multi-Tenancy with Query Prefixes,” contrasted with database-per-tenant dynamic repos
Process topology Elixir in Action, 3rd ed. — Saša Jurić Chapters 7 “Building a Concurrent System” and 9 “Isolating Error Effects”
Resource thinking Designing for Scalability with Erlang/OTP Chapters on supervision architectures and overload protection

5.10 Implementation Phases

Phase 1: Safe Routing Boundary (5-8 hours)

  • Create catalog, internal tenant identity, and context API.
  • Add exact previous-repo restoration and nesting policy.
  • Prove Tasks require explicit tenant propagation.

Phase 2: Lifecycle and Leases (7-11 hours)

  • Add single-flight repo startup and generation identity.
  • Grant monitored leases and recover leaked owners.
  • Handle repo death and stale-generation retry.

Phase 3: Capacity and Eviction (6-10 hours)

  • Budget pools and startup reservations.
  • Add bounded admission, eligible-idle ordering, draining, and stop.
  • Run saturation and eviction race tests.

Phase 4: Fleet Migrations and Operations (6-11 hours)

  • Add resumable migration campaigns and compatibility checks.
  • Add credential rotation policy and failure backoff.
  • Build operator status and cardinality-safe metrics.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Runtime repo identity tenant atom / registered global name / PID+generation PID+generation Avoids atom growth and stale identity
Context propagation implicit / explicit tenant identity Explicit Process-local routing is not inherited reliably
Eviction signal idle time only / draining + leases Draining + zero leases Protects transactions and streams
Capacity unit repo count / total connections Total connections with reservations Models actual scarce resource
Migration execution deploy-wide blocking / resumable campaign Resumable bounded campaign Isolates failures across tenant fleet

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Isolation Prevent tenant bleed unique sentinels under concurrency
Context Verify routing cleanup exception, nested context, Task boundary
Lifecycle Validate state transitions concurrent start, repo death, retry backoff
Capacity Enforce hard budget saturation, reservation, rotation headroom
Lease/Eviction Protect active work long transaction, leaked owner, drain race
Migration Verify campaign recovery partial success, crash, version compatibility
Security Protect credentials and identity redaction, untrusted tenant reference, atom growth

6.2 Critical Test Cases

  1. Hundreds of mixed-tenant queries always return the correct unique sentinel.
  2. An exception restores the exact previous dynamic repo.
  3. A Task without explicit tenant context fails closed.
  4. Concurrent first requests produce one repo generation.
  5. Repo death invalidates leases and does not route to another tenant.
  6. Capacity never exceeds the budget during concurrent startup.
  7. A leased repo is never selected or stopped by eviction.
  8. New acquisition during draining cannot attach to the draining generation.
  9. Migration crash resumes from observed database version without blind reapplication.
  10. Arbitrary tenant strings do not create atoms or select connection parameters.

6.3 Test Data

Provision tenants with unmistakable sentinel rows and different schema versions. Use fake credential providers, controllable repo starters, monitored long transactions, leaked lease-owner processes, and migration fixtures with one intentional failure. Capture maximum simultaneous pool allocation during load tests.

6.4 Definition of Done

  • Tenant identity is authenticated before catalog and runtime resolution.
  • Dynamic repo selection is always restored in cleanup.
  • Background processes re-enter context explicitly.
  • No adversarial concurrency test observes tenant bleed.
  • Same-tenant startup is single-flight and bounded.
  • Capacity counts pools/connections and never exceeds declared limits.
  • Eviction requires draining and zero leases for the exact generation.
  • Repo death, startup failure, and saturation have distinct retryable outcomes.
  • Migration campaigns resume and isolate per-tenant failure.
  • Credentials and high-cardinality identifiers are handled safely in observability.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem Why it happens Fix Quick test
Cross-tenant query Missing context falls back to default Fail closed in tenant data modules Invoke without context
Outer routing lost Cleanup restores default, not previous Save and restore exact prior repo Nested-context fixture
Task uses wrong database Assumed process context inheritance Pass tenant ID and reacquire Spawn Task in tenant scope
Connection budget exceeded Starting repos not reserved Reserve pool cost before start Concurrent cold-start load
Active transaction killed LRU uses timestamp only Lease and drain before stop Hold transaction past idle timer
Startup storm Check-then-start race Serialize lifecycle single flight Launch simultaneous first requests

7.2 Debugging Strategies

  • Include tenant ID, repo generation, lease token hash, and operation correlation in debug traces.
  • Query lifecycle state and capacity ledger before inspecting low-level pool processes.
  • Verify the current dynamic repo at the entry to tenant data modules in test mode.
  • Monitor lease owners and log leaked-lease cleanup separately from normal release.
  • Compare catalog schema version, live database version, and campaign record when migration status disagrees.

7.3 Performance Traps

  • Starting large pools for low-traffic tenants
  • Serializing all tenants through one lifecycle process instead of per-tenant coordination plus a capacity authority
  • High-cardinality tenant labels on every global metric
  • Repeated credential fetch and TLS setup without bounded caching policy
  • Migrating too many databases concurrently

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a status command showing repo generation and lease count.
  • Add configurable pinned tenants.
  • Add a dry-run eviction candidate report.

8.2 Intermediate Extensions

  • Add tiered pool sizes and per-tier budgets.
  • Add tenant suspension with graceful draining.
  • Add a migration canary cohort before fleet rollout.

8.3 Advanced Extensions

  • Add regional control-plane shards while preserving one owner per tenant.
  • Rotate credentials through overlapping generations and a bounded reserve.
  • Add adaptive admission using observed pool utilization without violating hard caps.
  • Support customer-managed database network tunnels as separately leased resources.

9. Real-World Connections

9.1 Industry Applications

Enterprise SaaS frequently chooses database-per-tenant isolation for residency, customer backup/restore, noisy-neighbor control, or contractual boundaries. The architecture trades simple shared-pool operation for a fleet of scarce, stateful resources. BEAM supervision helps manage the runtime topology, but isolation depends on disciplined context and lifecycle contracts.

  • Ecto dynamic repositories and Repo lifecycle APIs
  • DBConnection pool behavior underneath Ecto SQL adapters
  • DynamicSupervisor and Registry for runtime children and discovery
  • Ecto.Migrator for controlled schema changes

9.3 Interview Relevance

The project tests staff-level judgment about process-local context, security boundaries, connection economics, eviction races, migration compatibility, and failure isolation. A strong answer discusses leases and generations rather than only “start a Repo per customer.”

10. Resources

10.1 Essential Official Reading

10.2 Books and Deeper Study

  • Programming Ecto by Darin Wilson and Eric Meadows-Jönsson — repository, transactions, Multi, and multi-tenancy sections.
  • Elixir in Action, 3rd edition, by Saša Jurić — Chapters 7 and 9.
  • Designing for Scalability with Erlang/OTP by Francesco Cesarini and Steve Vinoski — supervision architecture and overload sections.

10.3 Verification Checklist Before Sharing the Project

  • Publish the capacity calculation and saturation policy.
  • Demonstrate exact previous-repo restoration and Task fail-closed behavior.
  • Show an active transaction surviving an eviction attempt.
  • Prove migration campaigns resume after control-plane restart.
  • Keep examples as lifecycle transcripts, schemas, pseudocode, and CLI output rather than full implementation code.