Project 23: Publishable Pluggable Object Storage Library

Design a Hex-ready Elixir library with a stable object-storage contract, local-disk and in-memory adapters, an HTTP-compatible adapter, and one reusable conformance suite.

Quick Reference

Attribute Value
Difficulty Level 2
Suggested Seniority Mid-level Elixir developer
Time Estimate 14-22 hours
Main Programming Language Elixir
Alternative Programming Languages Erlang, Gleam
Coolness Level Level 3
Business Potential Level 2
Prerequisites Modules, structs, result tuples, ExUnit, dependency basics
Key Topics Behaviours, protocols, typespecs, adapter contracts, doctests, semantic versioning, Hex packaging

1. Learning Objectives

By completing this project, you will be able to:

  1. Define a small stable storage API using behaviours, callbacks, typespecs, and explicit capability reporting.
  2. Explain when Elixir behaviours and protocols solve different extension problems.
  3. Implement local-disk, in-memory, and HTTP-compatible adapter designs without leaking backend details into callers.
  4. Run one adapter conformance suite against every implementation.
  5. Separate required, optional, and unsupported operations with stable error semantics.
  6. Prepare documentation, package metadata, file inclusion, semantic-version policy, and build verification for Hex publication.

2. All Theory Needed (Per-Concept Breakdown)

Stable Behaviours, Protocols, and Adapter Contracts

Fundamentals

An Elixir behaviour specifies callbacks that implementing modules promise to provide. It is a module-level contract: the caller chooses or configures an adapter module and invokes operations whose inputs and outputs are documented by typespecs. A protocol dispatches on the data type of its first argument and is appropriate when many data structures need the same operation. These mechanisms are related but not interchangeable. A storage backend is normally selected by configuration and maintains backend-specific options, making a behaviour the central boundary. Protocols can complement it for values such as upload bodies or object identifiers, but should not obscure the adapter contract. A publishable library also needs behavioral compatibility: every adapter must agree on key validation, missing-object errors, metadata, streaming expectations, and capability reporting, not merely compile with callback names.

Deep Dive into the concept

Library design starts from the caller’s stable needs, not from one vendor’s SDK. A useful object-storage core might support put, get, delete, stat, and bounded list. The contract needs to answer questions hidden by those verbs. Are keys arbitrary binaries or normalized path-like identifiers? Does put replace existing content? Is a missing delete successful? Does get return one binary, an enumerable stream, or a handle? Are metadata names normalized? Is listing lexicographic and paginated? A behaviour callback without these semantics is only a type-shaped suggestion.

Define canonical domain values owned by the library: an object key, object metadata, page token, put result, and closed error taxonomy. Adapter-specific response structs must not cross the boundary. Convert an HTTP 404, filesystem missing path, and absent in-memory entry into the same not_found result. Preserve useful causes in a safe details field or exception chain for debugging without making callers pattern-match vendor errors. Separate domain failures such as invalid key and unsupported capability from temporary transport errors and backend authorization failures.

Capabilities matter because storage systems are not identical. One backend may support atomic conditional writes, signed URLs, range reads, or server-side copy, while another does not. Pretending the least common denominator supports everything produces runtime surprises. Keep the required core small and expose an explicit capability query for optional operations. Optional callbacks may provide a compile-time signal, but callers still need a runtime way to decide whether a configured adapter supports a feature. The default result for unsupported functionality should be explicit and documented, never silent emulation with different guarantees.

Configuration should create an adapter reference that bundles the implementation module and validated backend options. Avoid hiding all configuration in application environment because it makes multiple storage instances and tests awkward. The public API can accept a storage handle and delegate to the configured module. Validate configuration at startup or handle construction, not on the thousandth request. Never require callers to know that the disk adapter wants a root path while the HTTP adapter wants an endpoint, credential provider, and request pool.

Protocols are useful at a different seam. An upload body might be a binary, an enumerable of iodata chunks, or a file reference. A protocol can convert supported data types into a bounded source description, while the adapter behaviour consumes that canonical description. This permits third-party types to implement body conversion without changing adapter callbacks. Do not create a protocol solely to dispatch on an adapter struct when a behaviour and explicit module already express the dependency; protocol consolidation and extension rules can surprise library authors and consumers.

Resource ownership needs a clear policy. The in-memory adapter may use an Agent or ETS table; the HTTP adapter may need a supervised client pool; the disk adapter owns no long-running process. A library should not unexpectedly start global named processes when a caller invokes put. Define a child specification or adapter child_spec capability so host applications place required processes in their supervision tree. Tests can start uniquely named instances. Ownership is part of the public contract because it determines shutdown, isolation, and multi-instance behavior.

The adapter conformance suite is the executable specification. It accepts a setup function that yields a clean adapter reference, then runs the same scenarios: round-trip bytes and metadata, overwrite semantics, missing lookup, delete policy, key rejection, listing order/page boundaries, large/streamed body behavior, and declared optional capabilities. Backend-specific tests still exist for transport errors or filesystem containment, but no adapter is accepted until the shared suite passes. A conformance suite prevents the common failure where the in-memory fake behaves more conveniently than production and allows application tests to pass against false semantics.

Disk storage is a security exercise. Mapping an object key to a filesystem path must prevent absolute paths, .. traversal, separator ambiguity, and symlink escape. Prefer an encoded or segmented internal representation rather than concatenating untrusted keys. Write to a temporary file in the intended root, flush according to the documented durability contract, and rename atomically where the filesystem permits. State clearly whether readers can observe old or new content during replacement.

HTTP-compatible storage introduces transient failures, authentication, timeouts, and response-body cleanup. Its adapter maps requests into the canonical contract, streams or bounds response bodies, and never reports success merely because a connection opened. Retries belong only around operations proven safe: a put with an idempotency token may be retryable, while an ambiguous non-idempotent operation may not. Keep vendor signing or credential refresh in the adapter.

Publication turns code into a promise. Typespecs aid tools and readers but do not enforce runtime semantics. @impl checks callback intent. Doctests keep examples honest. Package metadata declares name, description, licenses, links, and included files. mix hex.build --unpack exposes exactly what consumers receive; secrets, fixtures, and local artifacts must be absent, while README, license, source, and essential docs are present. Semantic versioning means changing a return tuple, key policy, error atom, or callback requirement can be breaking even when compilation succeeds.

The project teaches that an abstraction is valuable when it makes differences visible in controlled places. It should remove accidental vendor coupling while preserving meaningful differences through capabilities and documented guarantees. A small, precise contract with strong tests is more reusable than a large “universal” API that every adapter interprets differently.

How this fits on projects

The behaviour defines backend operations, typespecs document canonical values, an optional protocol normalizes upload bodies, and the shared conformance suite proves equivalent observable semantics before package publication.

Definitions & key terms

  • Behaviour: Module callback contract implemented by adapter modules.
  • Protocol: Polymorphic dispatch based on the first argument’s data type.
  • Adapter: Module translating a stable library contract to one backend.
  • Capability: Explicit optional feature with documented guarantees.
  • Conformance suite: Shared tests executed unchanged against every adapter.
  • Semantic versioning: Compatibility policy using major, minor, and patch versions.
  • Package manifest: Files and metadata included in the published artifact.

Mental model diagram

Application
    |
    v
[Public Storage API] -- canonical keys/results/errors --> [Adapter Behaviour]
       |                                                    /    |      \
       |                                        [Memory] [Local Disk] [HTTP]
       |
[UploadSource Protocol] <--- binary / file ref / chunk enumerable

Shared Conformance Suite ---------------------------------> every adapter
Hex package build -> unpack -> inspect docs/source/license only

How it works (step-by-step, with invariants and failure modes)

  1. Construct a validated storage reference from module and adapter options.
  2. Validate and normalize the public key and operation input.
  3. Convert supported body data into a canonical upload source.
  4. Delegate to the behaviour callback.
  5. Map backend responses into canonical successes or closed errors.
  6. Run identical conformance scenarios against all adapters.
  7. Invariant: callers never need backend response types or configuration shapes.
  8. Failure modes: fake/production semantic drift, path traversal, leaked response resources, unsupported feature assumptions, package file leakage, and breaking result changes.

Minimal concrete example

PSEUDOCODE — public contract

Storage.put(storage_ref, key, upload_source, metadata)
  -> accepted result: {ok, ObjectInfo}
  -> rejected result: {error, invalid_key | unsupported | unauthorized |
                              not_found | conflict | temporarily_unavailable}

Adapter callbacks:
  validate_options(options)
  put(adapter_state, canonical_key, canonical_source, metadata)
  get(adapter_state, canonical_key, range_or_all)
  stat(adapter_state, canonical_key)
  delete(adapter_state, canonical_key)
  list(adapter_state, prefix, page_request)
  capabilities(adapter_state)

Common misconceptions

  • “Behaviours enforce all semantics.” They check callback shape; tests and documentation define behavior.
  • “Protocols are interfaces.” They dispatch on data, while configured services generally fit behaviours.
  • “An in-memory fake may be simpler.” It must still reproduce externally observable contract semantics.
  • “A minor version may add a required callback.” That breaks existing adapter implementers.

Check-your-understanding questions

  1. Why is a configured storage backend better modeled by a behaviour than a protocol?
  2. Which differences should capabilities expose instead of hide?
  3. Why must the conformance suite test the in-memory adapter as strictly as HTTP storage?

Check-your-understanding answers

  1. Selection is module/configuration based, not based on the type of an arbitrary stored value.
  2. Features with meaningfully different guarantees, such as conditional writes, ranges, and signed URLs.
  3. Application tests rely on the fake; friendlier semantics would give false confidence.

Real-world applications

  • Attachment libraries that support cloud and local development.
  • Backup/export components with swappable destinations.
  • Open-source integrations that cannot choose a consumer’s vendor.

Where you’ll apply it

  • Canonical API and capabilities in Sections 3–4.
  • Behaviours, body protocol, and packaging in Section 5.
  • Shared conformance and package inspection in Section 6.

References

  • Elixir behaviours and typespecs: https://hexdocs.pm/elixir/typespecs.html#behaviours
  • Protocol: https://hexdocs.pm/elixir/Protocol.html
  • mix hex.build: https://hexdocs.pm/hex/Mix.Tasks.Hex.Build.html
  • mix hex.publish: https://hexdocs.pm/hex/Mix.Tasks.Hex.Publish.html

Key insight

A pluggable library succeeds when every adapter shares observable guarantees, not merely callback names.

Summary

Use a behaviour for configured backend modules, protocols only at data-polymorphism seams, canonical domain types at the public boundary, capabilities for honest differences, and one conformance suite as the executable contract.

Homework/Exercises to practice the concept

  1. Classify range reads, signed URLs, and conditional puts as core or optional capabilities.
  2. Define missing-object behavior for get, stat, and delete.
  3. List three changes that require a new major version.

Solutions to the homework/exercises

  1. Keep them optional unless all required adapters can provide identical guarantees.
  2. get and stat return not_found; choose and document idempotent delete or not_found, then enforce it everywhere.
  3. Adding a required callback, changing key normalization, or changing a public result/error shape is breaking.

3. Project Specification

3.1 What You Will Build

Create a library tentatively named ObjectDock with a public storage reference, required adapter behaviour, upload-source protocol, canonical structs/errors, and three adapters: memory, safe local disk, and a simulated S3-compatible HTTP service. Prepare—but do not actually publish—a Hex package.

3.2 Functional Requirements

  1. Provide put, get, stat, delete, and paginated list core operations.
  2. Normalize keys, metadata, success values, and errors.
  3. Report optional capabilities explicitly.
  4. Support multiple independently configured storage instances.
  5. Run one conformance suite against all adapters.
  6. Include doctested public examples and complete typespecs.
  7. Build and unpack a clean Hex artifact with correct metadata.

3.3 Non-Functional Requirements

  • Public documentation contains no backend-specific requirement for core operations.
  • Disk adapter cannot escape its configured root.
  • HTTP response bodies and pools have explicit ownership.
  • The package has zero compiler warnings and passes formatting, tests, docs, and package validation.

3.4 Example Usage / Output

$ objectdock put local:photos portraits/a.bin --from fixture.bin
stored key=portraits/a.bin bytes=4096 etag=sha256:9d... adapter=local

$ objectdock stat memory:test portraits/a.bin
error=not_found

$ objectdock capabilities http:staging
core=[put,get,stat,delete,list]
optional=[range_read,conditional_put]

3.5 Data Formats / Schemas / Protocols

Canonical ObjectInfo contains key, byte size, content type, normalized metadata, opaque etag, and modification time. Page results contain objects and an opaque continuation token. HTTP adapter details never appear in these structs.

3.6 Edge Cases

  • Empty, absolute, traversal, Unicode, and separator-ambiguous keys.
  • A chunk source failing midway through upload.
  • Replacing an object while another reader is active.
  • Listing exactly at a page boundary.
  • HTTP timeout after request bytes were sent.
  • An adapter declaring a capability but returning unsupported.
  • Package build accidentally including local credentials or large fixtures.

3.7 Real World Outcome

3.7.1 How to Run (Copy/Paste)

$ mix deps.get
$ mix test
$ mix docs
$ mix hex.build --unpack
$ objectdock-contract --all-adapters

3.7.2 Golden Path Demo (Deterministic)

The contract runner stores the same fixture through all three adapters, verifies byte/content metadata round-trip, replacement, listing, and deletion, then inspects the unpacked package allow-list.

3.7.3 Exact terminal transcript

$ objectdock-contract --all-adapters
adapter=memory cases=24 passed=24 failed=0
adapter=local cases=24 passed=24 failed=0
adapter=http_compatible cases=24 passed=24 failed=0
optional range_read memory=unsupported local=supported http_compatible=supported
package files=31 forbidden_files=0 docs=true license=true
CONTRACT PASS

4. Solution Architecture

4.1 High-Level Design

                    +---------------------+
Caller ------------> ObjectDock Public API|
                    +----------+----------+
                               |
              normalize key/result/error + upload source protocol
                               |
                    +----------v----------+
                    | Adapter Behaviour   |
                    +-----+----------+-----+
                          |          |
                 +--------v--+  +----v------+  +-------------+
                 | Memory    |  | Safe Disk |  | HTTP Client |
                 +-----------+  +-----------+  +-------------+
                          ^          ^               ^
                          +---- shared contract -----+

4.2 Key Components

Component Responsibility Key Decision
Public API Stable validation/delegation boundary Backend types never escape
Adapter Behaviour Required callbacks Small core, explicit optional features
UploadSource Protocol Normalize supported input data Stream ownership documented
Storage Reference Module plus validated state Support multiple instances
Contract Suite Executable cross-adapter semantics Same cases, backend-specific setup
Package Configuration Hex metadata and file allow-list Inspect unpacked artifact

4.3 Data Structures (No Full Code)

  • Storage reference with adapter module and opaque validated adapter state.
  • Object key smart value or validated canonical binary.
  • Object info and page structs owned by the library.
  • Closed core error atoms plus optional safe details.
  • Upload source descriptor containing size knowledge and chunk acquisition policy.

4.4 Algorithm Overview

Core lookups are backend-dependent. Local stat/get is expected O(1) path access; memory lookup O(1); HTTP one network request. Listing is O(k) per page to callers. Key normalization is O(length(key)) and occurs before delegation.

5. Implementation Guide

5.1 Development Environment Setup

Use an actively supported Elixir/OTP pair, ExUnit, ExDoc, Bypass or a local deterministic HTTP fixture server, and Hex tooling. Use temporary directories outside the repository for disk tests.

5.2 Project Structure

object_dock/
├── lib/object_dock.ex
├── lib/object_dock/{adapter,upload_source,key,object_info,page,error}.ex
├── lib/object_dock/adapters/{memory,local,http_compatible}.ex
├── test/contracts/object_storage_contract.exs
├── test/adapters/
├── guides/adapter_authoring.md
├── README.md
├── LICENSE.md
└── mix.exs

5.3 The Core Question You’re Answering

“How do I publish an Elixir abstraction whose adapters can differ internally without surprising callers externally?”

5.4 Concepts You Must Understand First

  1. Behaviours and typespecs — Elixir in Action, 3rd Edition, Chapter 5, “Higher-level abstractions”.
  2. Protocol dispatch — Programming Elixir 1.6, Chapter 20, “Protocols”.
  3. Package compatibility — Semantic Versioning 2.0.0 specification.
  4. Contract testing — Growing Object-Oriented Software, Guided by Tests, Chapter 8, adapted to adapters.

5.5 Questions to Guide Your Design

  1. What is the smallest useful required API?
  2. Which backend differences are genuine capabilities?
  3. Who owns processes, streams, temporary files, and HTTP bodies?
  4. Can a third party author an adapter using only public documentation and tests?
  5. What observable changes would break consumers?

5.6 Thinking Exercise

Compare a disk rename, an in-memory map replacement, and an HTTP conditional put. Write the exact guarantee the common put can promise. Move stronger behavior into a capability rather than quietly overstating the core contract.

5.7 The Interview Questions They’ll Ask

  1. What is the difference between an Elixir behaviour and protocol?
  2. How do optional callbacks differ from runtime capabilities?
  3. How do you keep a test adapter faithful to production semantics?
  4. What does @impl check, and what does it not check?
  5. How would you evolve a callback without breaking external adapters?
  6. What should be inspected before publishing a Hex package?

5.8 Hints in Layers

Hint 1: Write the contract prose first — Decide key, overwrite, delete, and missing semantics before callbacks.

Hint 2: Let memory expose ambiguity — Build the simplest adapter and make every vague behavior a test decision.

Hint 3: Treat disk as hostile — Test traversal and symlink boundaries before happy-path performance.

Hint 4: Consume your package — Unpack the artifact and run a tiny external fixture project against its public API.

5.9 Books That Will Help

Topic Book Chapter
Elixir abstractions Elixir in Action, 3rd Edition Ch. 5
Protocols Programming Elixir 1.6 Ch. 20
API evolution Building Evolutionary Architectures, 2nd Edition Ch. 4–5
Testing contracts Growing Object-Oriented Software, Guided by Tests Ch. 8

5.10 Implementation Phases

Phase 1: Contract and Memory Adapter (4-6 hours)

  • Define canonical types, behaviour, memory implementation, and first contract cases.

Phase 2: Real Boundaries (6-10 hours)

  • Build safe-disk and HTTP-compatible adapters; add capabilities and ownership tests.

Phase 3: Publishability (4-6 hours)

  • Complete docs, doctests, adapter guide, changelog policy, Hex metadata, and artifact inspection.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Backend dispatch Protocol, behaviour Behaviour Configured module contract
Upload bodies Binary only, protocol-normalized sources Protocol-normalized bounded set Extensible data seam
Optional features Pretend, callbacks/capabilities Explicit capabilities Honest guarantees
Configuration Global environment, storage reference Storage reference Multiple instances and tests
Package contents Broad defaults, allow-list Intentional allow-list Prevent leakage

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Conformance Same semantics for all adapters round-trip, overwrite, delete
Key security Protect disk and canonical identity traversal, absolute, Unicode
Ownership Close resources and isolate instances stream failure, HTTP body cleanup
Capability Keep optional behavior truthful declared/implemented agreement
Documentation Keep examples executable doctests and external consumer
Package Verify published artifact unpacked allow-list and metadata

6.2 Critical Test Cases

  1. Each adapter round-trips arbitrary binary content and normalized metadata.
  2. Missing-object and idempotent-delete policy is identical everywhere.
  3. Disk keys cannot escape the root through traversal, separators, or symlinks.
  4. Midstream source failure leaves no successful partial object under the target key.
  5. Page continuation yields every object exactly once in stable order.
  6. Every declared capability executes; undeclared features return unsupported.
  7. Two storage instances with different configuration do not share state accidentally.
  8. Unpacked Hex artifact contains source, docs, README, and license but no credentials, build artifacts, or large fixtures.

6.3 Test Data

Use empty and multi-megabyte binaries, iodata-like chunks, Unicode metadata, path-like malicious keys, hundreds of sorted keys, forced HTTP failures, and temporary roots with symlink traps.

6.4 Definition of Done

  • Required behaviour and all canonical return/error types are documented and typed.
  • Memory, safe-disk, and HTTP-compatible adapters pass one shared contract suite.
  • Capabilities accurately represent optional operations.
  • Disk adapter passes traversal and symlink-escape tests.
  • Multiple adapter instances have explicit supervised ownership and isolation.
  • Public examples pass as doctests and from an external fixture project.
  • mix hex.build --unpack yields only intended files and valid metadata.
  • Semantic-version policy identifies callback and result-shape breaking changes.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem: “Tests pass in memory and fail in production.”

  • Why: The fake invents friendlier overwrite, ordering, or missing-object behavior.
  • Fix: Run the exact same contract cases against every adapter.
  • Quick test: Produce a per-adapter contract matrix and compare case names.

Problem: “Disk keys write outside the root.”

  • Why: Untrusted keys are concatenated into paths.
  • Fix: Validate/encode keys and verify resolved containment; test symlinks.
  • Quick test: Attempt absolute, .., mixed-separator, and symlink escape cases.

Problem: “External adapters break after a minor release.”

  • Why: A callback became required or a return shape changed.
  • Fix: Treat implementers as consumers and follow semantic versioning.
  • Quick test: Compile an adapter fixture written against the prior contract.

7.2 Debugging Strategies

  • Log canonical operation, adapter module, key digest, and stable error—never credentials.
  • Re-run one failing contract case with adapter-specific transport tracing.
  • Inspect the storage reference to confirm the intended adapter instance.
  • Unpack the Hex tarball rather than assuming repository contents equal package contents.

7.3 Performance Traps

  • Converting streamed bodies into one large binary without declaring a limit.
  • Listing an entire bucket to implement one page.
  • Starting a new HTTP pool per operation.
  • Serializing all memory operations through one process when ETS semantics would suffice.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add content-type inference as an opt-in helper outside adapter callbacks.
  • Add a package changelog generated from compatibility categories.

8.2 Intermediate Extensions

  • Add range-read and conditional-put capabilities with shared optional tests.
  • Publish a separate adapter package that depends only on the core library.
  • Add property tests for key normalization and page traversal.

8.3 Advanced Extensions

  • Design multipart-upload capability and explicit abandoned-upload cleanup.
  • Add observability hooks without forcing a telemetry dependency on adapter authors.
  • Build a compatibility checker that compares two public callback/type snapshots.

9. Real-World Connections

9.1 Industry Applications

Elixir applications need interchangeable attachment, backup, archival, and media stores across local development, private infrastructure, and public cloud. A precise adapter library can become a reusable open-source foundation or supported commercial integration layer.

  • Ecto adapters demonstrate behaviour-driven ecosystem extension at a larger scale.
  • Req and Finch demonstrate public request APIs and supervised HTTP ownership.
  • ExAws-style libraries illustrate vendor integration, while this project focuses on a vendor-neutral core.

9.3 Interview Relevance and Scope Boundary

This project teaches library API design, behaviours, protocols, package compatibility, and contract testing. It does not overlap with Project 3’s distributed KV store or Project 8’s ETS cache because it neither implements storage internals nor distributed state; it standardizes access to external object stores.

10. Resources

10.1 Essential Reading

  • Elixir behaviours: https://hexdocs.pm/elixir/typespecs.html#behaviours
  • Protocol: https://hexdocs.pm/elixir/Protocol.html
  • Module callbacks and @impl: https://hexdocs.pm/elixir/Module.html
  • ExUnit doctests: https://hexdocs.pm/ex_unit/ExUnit.DocTest.html
  • Hex build task: https://hexdocs.pm/hex/Mix.Tasks.Hex.Build.html
  • Hex publish task: https://hexdocs.pm/hex/Mix.Tasks.Hex.Publish.html

10.2 Standards and Further Study

  • Semantic Versioning 2.0.0: https://semver.org/
  • Hex package policies: https://hex.pm/policies/codeofconduct
  • Elixir library guidelines: https://hexdocs.pm/elixir/library-guidelines.html
  • Building Evolutionary Architectures, 2nd Edition, Chapters 4–5.