Project 33: Custom Ecto Adapter for an HTTP Document Store

Build a deliberately bounded Ecto adapter that maps schemas, changesets, CRUD operations, and a documented query subset onto a revision-aware HTTP document store—while rejecting every semantic promise the backend cannot honestly keep.

Quick Reference

Attribute Value
Difficulty Level 5: Master
Suggested Seniority Principal Elixir/platform engineer
Time Estimate 35-60 hours
Main Programming Language Elixir
Alternative Programming Languages Erlang for transport internals
Coolness Level Level 5: Pure Magic
Business Potential Level 4: Open Core Infrastructure
Prerequisites Advanced Ecto queries/schemas, behaviours, macros, supervision, HTTP semantics, type systems, distributed failure
Key Topics Ecto.Adapter callbacks, Queryable and Schema contracts, query translation, loaders/dumpers, capability boundaries, optimistic concurrency, compliance tests

1. Learning Objectives

By completing this project, you will:

  1. Explain the responsibilities divided across Ecto.Adapter, Ecto.Adapter.Schema, and Ecto.Adapter.Queryable.
  2. Initialize an adapter supervision tree and metadata without pretending an HTTP client is a transactional database connection.
  3. Translate a bounded Ecto query representation into a documented HTTP request plan.
  4. Implement type loaders and dumpers that preserve Ecto schema expectations at the JSON boundary.
  5. Map remote validation, conflict, stale revision, missing document, timeout, and protocol errors into stable Ecto-facing results.
  6. Reject joins, fragments, locks, transactions, or consistency guarantees the backend cannot support.
  7. Build a reusable conformance suite that distinguishes adapter bugs from backend limitations.
  8. Publish a capability matrix and compatibility contract suitable for application teams.

2. All Theory Needed (Per-Concept Breakdown)

Ecto Adapter Contracts, Types, and Query Planning

Fundamentals

Ecto separates application-facing repository operations from backend implementations through adapter behaviours. Ecto.Adapter defines lifecycle, metadata, checkout, and type loader/dumper responsibilities. Ecto.Adapter.Schema covers schema inserts, updates, deletes, and related returned values. Ecto.Adapter.Queryable covers preparing and executing query operations. Optional capabilities such as transactions have additional behaviours and should be implemented only when the backend can satisfy their semantics. An adapter is not a thin URL wrapper: it receives planned Ecto structures, parameters, schema metadata, and type expectations, then must return precisely shaped results or documented errors. The project should support a small capability set completely and fail early for unsupported constructs rather than issuing approximate remote requests.

Deep Dive

The adapter module sits beneath Ecto.Repo and participates in compile-time and runtime contracts. Its base initialization returns children and adapter metadata used by the Repo supervision tree. For an HTTP backend, children may include a supervised Finch pool, configuration holder, endpoint health component, or bounded request telemetry process. Secrets belong in runtime configuration and must not appear in adapter metadata rendered in logs. The adapter’s lifecycle should validate endpoint URI, authentication provider, timeouts, and declared backend API version before accepting queries.

Type translation is foundational. Ecto schemas describe fields using Ecto types; JSON and the document API have their own representations. Dumpers transform values heading toward the backend, while loaders transform backend values into schema values. Booleans, integers, floats, decimals, UUIDs, dates, UTC datetimes, naive datetimes, binaries, arrays, maps, and custom parameterized types require explicit policy. JSON numbers do not by themselves preserve every integer/decimal distinction, timestamps may omit offsets or precision, and binary data needs an agreed encoding. A wrong loader can make a query look successful while corrupting domain meaning.

Define an adapter type matrix. For every supported primitive, specify remote JSON shape, nullability, precision, ordering behavior, and round-trip tests. Unsupported custom types must return a clear dump/load error. Binary IDs should use an explicit UUID representation. Field source names may differ from Elixir field names, so request/response mapping uses schema source metadata rather than assuming atom-to-string conversion.

Queryable support requires a planning boundary. Ecto query expressions represent sources, fields, parameters, boolean expressions, ordering, limits, offsets, selections, and potentially joins, subqueries, fragments, grouping, locking, windows, updates, and combinations. The remote document store supports only a subset. During prepare or translation, walk the planned query and construct an internal request plan. A capability validator rejects unsupported nodes with an actionable Ecto.QueryError that names the construct and adapter limitation. Never silently drop an order_by, convert an inner join to client filtering, or ignore a lock.

Parameter handling must separate query structure from values. Translate fields and operators into a canonical remote filter representation while dumping parameter values through the correct Ecto types. Do not interpolate values into URL strings manually. Encode them through the HTTP/JSON layer and apply explicit URL-length policy. Complex filters may use POST to a search endpoint if the backend defines it, but the HTTP method’s retry and cache implications must be documented.

Selection shapes affect results. A query selecting whole schemas expects field values that can be loaded into schema structs through Ecto’s pipeline. A projection selecting a tuple or map expects rows in the order and shape Ecto planned. The adapter should begin with whole-schema and simple field projections, then add composite selections only with dedicated tests. Aggregate queries require backend support and precise empty-set/null semantics; omit them from the core rather than computing them client-side over one page.

Prepared query caching can store the canonical request-plan structure independently from parameter values. Cache keys must include backend API/capability version and adapter translation version. A plan prepared under one capability set cannot be reused after configuration changes that alter supported operators. Avoid retaining tenant secrets or large query ASTs in cache metadata.

The adapter must also distinguish Ecto source metadata from backend collection names. Validate collection identifiers under a safe pattern or explicit mapping; never let a dynamic source become arbitrary URL path traversal. Atom creation from response keys is forbidden. Decode into known schema/source fields and preserve unknown-field policy deliberately.

How this fits on the project

This concept defines adapter initialization, type conversion, schema/source mapping, query capability validation, request plans, parameter encoding, and result shaping.

Definitions and key terms

  • Adapter metadata: Runtime map returned during adapter initialization and used by Repo operations.
  • Loader/dumper: Conversion pipeline between Ecto types and backend representations.
  • Prepared query: Backend-oriented plan derived from an Ecto query before parameter values execute.
  • Capability matrix: Versioned declaration of supported and rejected Ecto constructs.
  • Projection: Selected result shape such as schema, field, tuple, or map.
  • Source: Backend collection/table identity associated with an Ecto schema.

Mental model diagram

application
  Repo.all / insert / update / delete
           |
           v
      Ecto planning
           |
  +--------+------------------+
  | Adapter behaviours        |
  | base lifecycle + types    |
  | Schema CRUD               |
  | Queryable prepare/execute |
  +--------+------------------+
           |
   [capability validator]
      | supported       | unsupported
      v                 v
[HTTP request plan]  Ecto.QueryError with exact construct
      |
dump params -> JSON -> load response -> Ecto result shape

How it works

  1. Initialize supervised transport and versioned capability metadata.
  2. Receive schema or planned query input from Ecto.
  3. Validate sources, selection shape, operators, and optional constructs.
  4. Translate supported structure into a canonical request plan.
  5. Dump typed parameter values into remote JSON representations.
  6. Execute through the supervised transport.
  7. Load response fields and return the exact Ecto result shape.
  8. Invariant: unsupported semantics fail before a misleading request is issued.
  9. Failure modes include lossy type conversion, ignored clauses, unsafe sources, response-key atom growth, projection mismatch, and stale prepared plans.

Minimal concrete example

REQUEST PLAN, NOT RUNNABLE CODE
source: invoices
filter: and(equal(field=status, parameter=0), greater(field=total_cents, parameter=1))
order: [{issued_at, descending}, {id, ascending}]
limit: 50
selection: full_schema
typed_parameters: [{0, string}, {1, integer}]

unsupported node: join
result: query_error(adapter=EctoHttpDoc, capability=joins, supported=false)

Common misconceptions

  • Implementing Ecto.Adapter alone automatically supplies schema and query support.
  • JSON round trips preserve every Ecto type without policy.
  • Unsupported clauses can be ignored if most queries still return useful documents.
  • Client-side filtering of one response page is semantically equivalent to backend filtering.

Check-your-understanding questions

  1. Why must query capabilities be validated before transport execution?
  2. What belongs in a prepared plan versus execution parameters?
  3. Why can arbitrary response-key atomization become a VM safety issue?

Check-your-understanding answers

  1. Issuing an approximate request can return silently incorrect data and violate Ecto expectations.
  2. Stable translated structure belongs in the plan; typed runtime values remain parameters.
  3. Atoms are not ordinarily garbage-collected, so untrusted keys can exhaust the atom table.

Real-world applications

  • Ecto integrations for search/document services
  • Legacy SaaS data APIs exposed through an Ecto-shaped application boundary
  • Specialized test or offline adapters
  • Internal data-platform clients with compile-time query validation

Where you will apply it

  • Project 33 base, Schema, Queryable, type, cache, and capability modules

References

Key insight

An adapter is correct when it preserves supported Ecto semantics and rejects the rest, not when it makes every query appear to run.

Summary

Treat behaviours, types, planned queries, and result shapes as contracts. Translate a bounded subset completely, make capabilities versioned, and reject unsupported semantics before transport.

Homework/Exercises

  1. Define the type contract for decimal, UTC datetime, UUID, and binary fields.
  2. Classify where, two-field order_by, join, aggregate, fragment, and lock under the core capability set.

Solutions

  1. Specify exact JSON encoding, precision, nullability, loader/dumper order, invalid value, and round-trip property for each.
  2. Support documented comparisons and deterministic ordering; reject joins, aggregates, fragments, and locks unless the backend later gains exact semantics.

HTTP Document Semantics, Concurrency, and Failure Honesty

Fundamentals

An HTTP document store has different semantics from a relational database. CRUD maps naturally only when document identity, field representation, revision handling, and status codes are explicit. Network calls can time out after the server committed work, responses can be retried, pages can change during traversal, and the backend may not offer multi-document transactions. ETags or revision tokens can implement optimistic concurrency for one document. Conditional requests prevent stale updates, while stable idempotency keys protect retryable creates when the backend supports them. The adapter must classify transport failure, protocol failure, validation error, not-found, conflict, stale revision, rate limit, and uncertain outcome separately. It must not implement Ecto.Adapter.Transaction merely to run a callback without atomicity.

Deep Dive

Schema operations begin with a backend contract. An insert chooses document identity from a client-supplied primary key or a server-generated value returned in the response. The adapter dumps fields, sends one create request, and returns generated fields in Ecto’s expected format. A duplicate ID maps to a constraint-like conflict only when a named uniqueness contract exists; otherwise it is a backend conflict error. Validation failures should identify fields and stable codes where the remote API provides them, but the adapter cannot invent database constraint names not guaranteed by the service.

Updates and deletes need stale-write protection. Load a remote revision or ETag into a designated schema field or adapter metadata policy, then send it through If-Match or the backend’s revision parameter. A precondition failure maps to an Ecto stale result. Without a revision token, the adapter offers last-write-wins and must state that optimistic_lock cannot be honored. Never read, merge, and write as though the sequence were atomic unless the backend explicitly provides conditional update.

HTTP retry policy is operation-specific. Safe reads may be retried under bounded time and attempt budgets. A create is not safely retryable after an ambiguous timeout unless it uses a backend-supported stable idempotency key or client-assigned deterministic document ID. An update with a revision precondition may be retried only after classifying whether the same request can be recognized. The adapter should surface uncertain outcomes rather than returning a generic timeout that encourages blind application retries.

Rate limiting and overload are normal remote states. Respect Retry-After when valid, cap waits, and do not sleep indefinitely inside a checked-out Repo call. Return a classified retryable error with metadata or integrate a bounded caller policy. Circuit breaking is outside the adapter’s core semantic contract; if added, it must not turn backend unavailability into empty query results.

Pagination is a consistency contract. Offset pagination can duplicate or skip documents when concurrent writes reorder results. Cursor pagination is preferable when the backend supplies stable cursors tied to a deterministic order. Ecto’s stream expectation may imply lazy pages, resource cleanup, and failure propagation. The core project supports bounded all with limit and cursor traversal under one call, while stream is explicitly unsupported unless implemented with documented consistency and cleanup semantics. Never present one page as the complete result.

Transactions are the brightest boundary. If the remote service lacks atomic multi-operation transactions, do not implement the transaction behaviour. Repo.transaction should be unavailable rather than a function that executes sequential HTTP calls and rolls nothing back. Ecto.Multi can still serve as an application planning structure only if executed by a higher-level saga that documents compensation; it is not an adapter transaction.

Transport supervision should use a bounded HTTP pool and runtime authentication provider. Timeouts need connect, receive, and overall operation budgets. Responses require maximum body size, content type, JSON shape, and backend API version validation. Do not decode unknown keys into atoms. Authentication refresh must not recursively invoke the adapter or create a retry storm.

Telemetry records operation class, collection, status class, duration, retry count, page count, and response size, while avoiding document IDs or sensitive query parameters as high-cardinality labels. Logs include request correlation but redact authorization and bodies. A mock reference server should inject delayed commit, response loss, conflicting revisions, malformed JSON, wrong content type, partial pagination, and rate-limit responses.

Compatibility is two-dimensional: Ecto version and backend API version. Maintain a tested matrix and fail initialization if the configured backend version lacks required capabilities. Adapter package releases must state which query/Schema operations are stable and which are experimental. Principal-level engineering here means saying “unsupported” early and precisely.

How this fits on the project

This concept defines CRUD requests, revision mapping, retries, pagination, transaction absence, transport supervision, errors, telemetry, and compatibility policy.

Definitions and key terms

  • ETag/revision: Backend token identifying one document version.
  • Conditional request: HTTP operation performed only if a revision condition holds.
  • Uncertain outcome: Client cannot determine whether the server committed an operation.
  • Idempotency key: Stable request identity recognized across retries.
  • Cursor pagination: Continuation based on a server-issued stable position token.
  • Capability honesty: Refusal to emulate guarantees the backend cannot provide.

Mental model diagram

Ecto schema operation
        |
 [dump types + validate capability]
        |
 [request builder] -- auth/idempotency/revision --> [HTTP document store]
        |                                                |
        |<-- status + headers + bounded JSON body --------+
        |
 [response classifier]
  | success -> load fields/generated values
  | 404 -> not found contract
  | 409/412 -> conflict or stale revision
  | 422 -> validation mapping
  | 429/5xx/timeout -> retryable, terminal, or uncertain outcome

How it works

  1. Validate backend capability and dump typed fields.
  2. Build path and body from trusted source metadata.
  3. Add stable idempotency or revision conditions when supported.
  4. Execute under bounded transport and response limits.
  5. Classify status, headers, content type, and body schema.
  6. Retry only operations proven safe under policy.
  7. Load successful fields or return precise Ecto/backend error.
  8. Invariant: the adapter never reports atomicity, completeness, or success it cannot establish.
  9. Failure modes include ambiguous commit, stale revision, pagination drift, malformed response, auth refresh storm, excessive body, rate limit, and unsupported transaction expectation.

Minimal concrete example

PROTOCOL TRANSCRIPT
UPDATE /documents/invoices/INV-42
If-Match: revision-7
Idempotency-Key: operation-C91
body: documented dumped field map

response 412 Precondition Failed
remote_revision: revision-8
adapter_result: stale_entry(expected=revision-7, observed=revision-8)
retry_policy: do_not_overwrite; caller must reload and decide

Common misconceptions

  • HTTP status 500 proves the server made no change.
  • Running functions sequentially inside a callback is an Ecto transaction.
  • GET requests can always be retried without an overall deadline.
  • Offset pagination is a stable snapshot of a changing collection.

Check-your-understanding questions

  1. When is a timed-out insert safe to retry?
  2. Why should a backend without atomicity omit the transaction behaviour?
  3. How does a revision token map to Ecto stale-write semantics?

Check-your-understanding answers

  1. When deterministic identity or a backend-recognized stable idempotency key guarantees one create outcome.
  2. Sequential execution would mislead callers into assuming rollback and isolation that do not exist.
  3. Send the known token conditionally; a failed precondition becomes a stale-entry result rather than an overwrite.

Real-world applications

  • Document database adapters
  • Search-index Ecto integrations
  • Remote SaaS persistence gateways
  • Migration bridges from relational applications to service-owned data

Where you will apply it

  • Project 33 transport, Schema callbacks, revision policy, pagination, retries, and compliance server

References

Key insight

The adapter’s most important feature is the guarantee it refuses to fake.

Summary

Map CRUD with explicit revisions and types, classify ambiguous outcomes, retry only proven-idempotent work, implement pagination honestly, and omit transaction support when atomicity does not exist.

Homework/Exercises

  1. Classify retry safety for GET, deterministic-ID create, random server-ID create, revision-guarded update, and delete.
  2. Design the adapter response to a malformed 200 JSON body.

Solutions

  1. GET is bounded-retryable; deterministic/idempotent create may be; random-ID create is uncertain; guarded update depends on stable operation identity; delete depends on backend idempotency contract.
  2. Return a protocol error containing status, content type, correlation, and bounded diagnostic evidence; never treat it as an empty success.

3. Project Specification

3.1 What You Will Build

Build EctoHttpDoc, a Hex-ready adapter plus a deterministic mock document server. The core supports schema insert/update/delete, get by primary key, equality/range boolean filters, stable field ordering, limit, cursor pagination, and full-schema/simple-field selections. It explicitly rejects joins, fragments, aggregates, locks, bulk updates, streams, and transactions.

3.2 Functional Requirements

  1. Start supervised HTTP transport through adapter initialization.
  2. Validate endpoint, backend API version, auth provider, timeouts, and capabilities.
  3. Load and dump the documented Ecto primitive type matrix.
  4. Support revision-aware schema CRUD and generated fields.
  5. Prepare and execute the documented Queryable subset.
  6. Reject unsupported AST constructs before any request.
  7. Map response status/body into stable Ecto or adapter errors.
  8. Apply operation-specific bounded retry and uncertainty policy.
  9. Emit redacted, cardinality-safe telemetry.
  10. Publish a compatibility/capability matrix and conformance suite.

3.3 Non-Functional Requirements

  • No query clause is silently dropped or evaluated client-side.
  • No untrusted string becomes an atom or arbitrary URL source.
  • Request/response sizes, retries, pages, and total duration are bounded.
  • Adapter initialization and every public error are diagnosable.
  • Supported values round-trip under the declared type contract.
  • Tests distinguish backend limitation, transport failure, and adapter defect.

3.4 Example Usage / Output

$ mix ecto_http_doc.conformance --server http://127.0.0.1:4100
backend_api=v1 ecto=3.x adapter=0.1
schema_insert PASS
schema_update_revision PASS
schema_delete_stale PASS
query_filter_boolean PASS
query_order_limit PASS
query_join_rejected PASS no_request_issued=true
transaction_capability UNSUPPORTED expected=true
result=PASS supported=27 rejected_as_designed=11 failed=0

3.5 Data Formats / Schemas / Protocols

CAPABILITY DOCUMENT
{adapter_version, ecto_range, backend_api_range,
 schema_ops, query_operators, selections, pagination,
 supported_types, transaction_support=false, stream_support=false}

REMOTE DOCUMENT
{id, revision, fields, inserted_at, updated_at}

ADAPTER ERROR
{class, operation, retryability, uncertainty, http_status?,
 backend_code?, correlation_id?, safe_message, cause_summary}

3.6 Edge Cases

  • Backend returns generated ID with wrong type
  • Decimal loses scale during JSON conversion
  • Update receives stale ETag
  • Timeout occurs after create committed
  • order_by field lacks remote sortable capability
  • Query contains nested unsupported fragment inside a boolean expression
  • Cursor repeats or cycles
  • 200 response has wrong content type or excessive body
  • Authentication refresh fails under concurrent requests
  • Backend API version changes capability at startup

3.7 Real World Outcome

3.7.1 How to Run

$ mix deps.get
$ mix test
$ mix ecto_http_doc.mock_server
$ mix ecto_http_doc.conformance --server http://127.0.0.1:4100

3.7.2 Golden Path Demo

Insert a document, load it through an Ecto schema, update with the correct revision, reject a stale second update, query a stable ordered page, and prove that a join raises a capability error while the mock server records no request.

3.7.3 Exact Terminal Transcript

$ mix ecto_http_doc.demo
insert invoice=INV-42 revision=1 returned_fields=[id,revision,inserted_at]
get invoice=INV-42 total_cents=12500 status=open
update expected_revision=1 new_revision=2 result=ok
stale_update expected_revision=1 observed_revision=2 result=stale
query status=open order=[issued_at:desc,id:asc] rows=10 next_cursor=C2
unsupported join result=Ecto.QueryError requests_sent=0

3.8 Scope Boundary Versus Projects 1-13

This project does not build another KV store, ETS cache, event pipeline, or distributed cluster. It extends the Ecto ecosystem itself and focuses on framework contracts, type semantics, query planning, remote failure, and honest capability boundaries. P1-P13 runtime patterns remain supporting knowledge rather than the learning target.

4. Solution Architecture

4.1 High-Level Design

Ecto.Repo API
     |
 [Ecto planner]
     |
+---- Adapter ------------------------------------------+
| base lifecycle -> supervised Finch/auth/config        |
| Schema callbacks -> CRUD request plans                 |
| Queryable -> capability validator -> filter/projection |
| types -> dump/load pipelines                           |
| response classifier -> Ecto result or precise error    |
+-------------------------+------------------------------+
                          |
                   HTTP document API
                          |
             revision + cursor + typed JSON contract

4.2 Key Components

Component Responsibility Key decision
Base Adapter Lifecycle, metadata, loaders/dumpers Expose no unsupported behaviour
Schema Adapter Insert/update/delete Use revision preconditions
Queryable Adapter Prepare and execute subset Reject unsupported nodes early
Capability Validator Versioned semantic gate Never silently approximate
Query Translator Canonical remote request plan Keep values parameterized and typed
Transport Bounded HTTP, auth, body validation Operation-specific retries
Response Classifier Stable error/result mapping Preserve uncertainty and correlation
Conformance Suite Backend-independent contract Positive and negative capability tests

4.3 Data Structures (No Full Code)

  • AdapterMeta: transport reference, capability version, endpoint identity, timeout policy.
  • TypeContract: Ecto type, remote representation, loader, dumper, precision/null rules.
  • RequestPlan: operation, source, filter tree, ordering, pagination, selection, typed parameters.
  • BackendResponse: status, headers, bounded decoded body, correlation.
  • AdapterError: public classification including retryability and uncertainty.
  • Capability: supported operations/operators/selections keyed by backend version.

4.4 Algorithm Overview

Validate planned queries recursively and translate supported nodes into a canonical request plan. Dump parameters using Ecto type metadata, execute under a total deadline, classify response, load result fields, and shape rows as Ecto expects. Schema writes add revision/idempotency policy. Unsupported nodes fail before transport.

Translation is O(Q) in query-expression nodes. Loading is O(R * F) for R result rows and F selected fields. Memory is bounded by configured response/page size; multi-page all must enforce maximum pages and rows.

5. Implementation Guide

5.1 Development Environment Setup

Use a supported Ecto version, Finch, and a local deterministic mock server. Lock the backend API schema and expose failure injection controls only in the test server.

5.2 Project Structure

ecto_http_doc/
├── lib/ecto_http_doc/
│   ├── adapter.ex
│   ├── schema.ex
│   ├── queryable.ex
│   ├── capability.ex
│   ├── type_contract.ex
│   ├── query_validator.ex
│   ├── query_translator.ex
│   ├── transport.ex
│   ├── response_classifier.ex
│   └── error.ex
├── test/support/mock_document_server/
├── test/conformance/
└── test/

5.3 The Core Question You Are Answering

“How can an Ecto adapter make a non-relational remote service feel coherent without lying about queries, transactions, types, or failure?”

5.4 Concepts You Must Understand First

  1. Ecto Repo, Schema, Changeset, Query, and adapter behaviour boundaries.
  2. Ecto types, parameter dumping, result loading, and projections.
  3. HTTP methods, conditional requests, retries, status classification, and JSON limits.
  4. Optimistic concurrency, idempotency, and uncertain outcomes.
  5. Behaviour compatibility, package versioning, and conformance testing.

5.5 Questions to Guide Your Design

  1. Which Ecto semantics can the backend satisfy exactly?
  2. Where is unsupported syntax rejected, and what evidence reaches the user?
  3. Which types round-trip without loss?
  4. Which write failures are safe to retry and which are uncertain?
  5. How will applications discover capabilities before production?

5.6 Thinking Exercise

Trace a where plus ordered, limited selection through Ecto planning, capability validation, typed parameters, remote request, JSON response, loaders, and final projection. Then insert a join and identify the earliest correct rejection point.

5.7 The Interview Questions They Will Ask

  1. “What responsibilities belong to each Ecto adapter behaviour?”
  2. “How do loaders and dumpers protect type semantics?”
  3. “Why is client-side filtering not an acceptable query translation fallback?”
  4. “How do ETags implement optimistic concurrency?”
  5. “When is an HTTP write retry safe?”
  6. “Why should an adapter omit transaction support rather than emulate it?”

5.8 Hints in Layers

Hint 1: Publish the matrix first — Decide supported types and query nodes before callbacks.

Hint 2: Plan before transport — Translate into an internal request plan that can be tested without a server.

Hint 3: Negative tests are features — Prove joins and transactions fail clearly with zero requests.

Hint 4: Preserve uncertainty — A timeout after a write is not automatically a safe retry or clean failure.

5.9 Books That Will Help

Topic Book Precise reading
Ecto internals Programming Ecto — Darin Wilson and Eric Meadows-Jönsson Repository, schema, query composition, changeset constraint, and transaction/Multi chapters
Data-model mismatch Designing Data-Intensive Applications — Martin Kleppmann Chapter 2 “Data Models and Query Languages”
Distributed failure Designing Data-Intensive Applications Chapter 8 “The Trouble with Distributed Systems”
HTTP contract HTTP: The Definitive Guide — Gourley and Totty Chapters on methods/status, caching, authentication, and conditional requests; verify details against RFC 9110

5.10 Implementation Phases

Phase 1: Contracts and Types (8-12 hours)

  • Publish capability and type matrices.
  • Initialize adapter children/metadata and implement type round trips.
  • Create deterministic mock server protocol.

Phase 2: Schema Operations (8-12 hours)

  • Add insert/get/update/delete request plans.
  • Add generated fields, revision handling, and error classification.
  • Inject ambiguous timeout and stale-write cases.

Phase 3: Queryable Subset (10-16 hours)

  • Validate and translate filters, ordering, limit, cursor, and selections.
  • Add prepared-plan cache versioning.
  • Prove unsupported constructs issue no request.

Phase 4: Hardening and Packaging (9-20 hours)

  • Add bounded retries, telemetry, compatibility checks, and resource limits.
  • Complete conformance and performance fixtures.
  • Build Hex package, documentation, and upgrade policy.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Query scope broad approximation / narrow exact subset Narrow exact subset Avoids silent semantic corruption
Transactions sequential emulation / unsupported Unsupported Backend cannot provide rollback/isolation
Concurrency last-write-wins / revision precondition Revision precondition Maps stale writes honestly
Retries generic middleware / operation-specific Operation-specific Safety depends on method and idempotency
Type keys atomize response / known field map Known field map Prevents atom exhaustion and schema drift

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Behaviour Validate callback contracts init, checkout, loaders/dumpers
Type Property Prove supported round trips UUID, dates, decimal, arrays, null
Query Translation Validate plans without network nested booleans, order, selection
Negative Capability Reject unsupported semantics join, fragment, lock, aggregate, transaction
Protocol Classify remote behavior status, headers, malformed JSON, oversized body
Failure/Retry Preserve write outcomes delayed commit, response loss, rate limit
Compatibility Test Ecto/backend matrix supported and rejected versions

6.2 Critical Test Cases

  1. Every supported primitive round-trips within its precision policy.
  2. Unknown response keys never create atoms.
  3. Unsupported nested query nodes fail before transport.
  4. Insert returns generated ID/revision in the shape Ecto expects.
  5. Stale revision maps to an Ecto stale result without overwrite.
  6. Ambiguous create timeout is classified uncertain unless protected by idempotency.
  7. Cursor cycle and page limits terminate with explicit error.
  8. Malformed 200 response is not treated as empty success.
  9. Repo.transaction capability is absent rather than falsely sequential.
  10. Backend version mismatch fails adapter startup with capability evidence.

6.3 Test Data

Use schemas covering every supported type, deterministic document fixtures, translation fixtures with nested booleans, and mock-server scenarios for all status/error classes. The server records request count and normalized request plans so negative tests prove zero network calls.

6.4 Definition of Done

  • Base, Schema, and Queryable callback responsibilities are documented.
  • Capability and type matrices are versioned and tested.
  • Supported values round-trip under explicit precision/null policies.
  • Unsupported query constructs fail early with no remote request.
  • Revision-aware writes prevent stale overwrite.
  • Retry policy distinguishes safe, terminal, and uncertain outcomes.
  • Pagination, response size, attempt count, and total time are bounded.
  • Transaction support is explicitly unavailable.
  • Conformance tests pass against the deterministic mock backend.
  • Package documentation states tested Ecto and backend API versions.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem Why it happens Fix Quick test
Query returns wrong rows Unsupported clause was ignored Recursive capability validation before request Add nested fragment fixture
Decimal/time drifts Generic JSON conversion Explicit type matrix and round-trip properties Boundary precision values
Duplicate document Blind retry after ambiguous timeout Stable idempotency/deterministic ID or uncertain result Commit then drop response
Lost update No revision precondition Map ETag/revision and stale result Two concurrent updates
Memory/atom growth Atomizes arbitrary JSON keys Map only known schema fields Random key flood
False transaction confidence Sequential callback called “transaction” Omit transaction behaviour Assert capability unavailable

7.2 Debugging Strategies

  • Inspect the canonical request plan before examining HTTP wire output.
  • Log capability rule ID for every rejection.
  • Correlate adapter operation, remote request ID, revision, and retry attempt without bodies/secrets.
  • Replay recorded bounded responses through the classifier independently.
  • Compare expected Ecto projection shape with loaded row shape field by field.

7.3 Performance Traps

  • N+1 remote requests disguised behind association access
  • Client-side collection of unbounded pages
  • Rebuilding translation plans for identical queries
  • Repeated authentication refresh under concurrency
  • Excessive JSON decode followed by discarding most fields
  • Retrying rate-limited calls without total budgets

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a capability-report Mix task.
  • Add request-plan pretty printing with redacted parameters.
  • Add deterministic-ID insert mode.

8.2 Intermediate Extensions

  • Add backend-supported aggregates one operation at a time.
  • Add cursor-backed stream with cleanup and consistency documentation.
  • Add optional changes-feed integration outside Ecto query semantics.

8.3 Advanced Extensions

  • Add a second backend version adapter with compatibility negotiation.
  • Build a saga library above the adapter for compensatable Ecto.Multi plans without calling them transactions.
  • Add differential tests against a reference in-memory semantic model.
  • Submit the adapter to a real backend’s ecosystem after an external contract/security review.

9. Real-World Connections

9.1 Industry Applications

Ecto adapters let applications retain schemas, changesets, query composition, and repository boundaries while integrating nontraditional stores. The difficult work is not HTTP plumbing; it is reconciling type, query, consistency, and failure semantics so application developers are never surprised by a promise that only relational adapters normally provide.

  • Ecto and Ecto SQL adapters as contract references
  • Postgrex for studying a mature adapter/driver boundary without copying SQL-specific assumptions
  • Finch and Mint for supervised HTTP transport
  • Backend-specific Ecto adapters as comparative capability designs

9.3 Interview Relevance

This project demonstrates principal-level framework extension, semantic negotiation, public compatibility design, distributed failure judgment, and negative capability discipline. The strongest answer is often a precise refusal rather than an ingenious approximation.

10. Resources

10.1 Essential Official Reading

10.2 Books and Deeper Study

  • Programming Ecto by Darin Wilson and Eric Meadows-Jönsson — repository, schema, query, changeset, constraint, transaction, and Multi chapters.
  • Designing Data-Intensive Applications by Martin Kleppmann — Chapters 2 and 8.
  • HTTP: The Definitive Guide by David Gourley and Brian Totty — methods/status, authentication, caching, and conditional request chapters, checked against current RFC 9110.

10.3 Verification Checklist Before Sharing the Project

  • Publish the exact positive and negative capability matrix.
  • Demonstrate a stale update, uncertain create, and zero-request unsupported query.
  • Prove every supported type round-trips under documented limits.
  • Verify transaction APIs are not falsely offered.
  • Keep examples as request plans, protocol transcripts, schemas, pseudocode, and CLI output rather than a runnable adapter solution.