Sprint: JIDO Mastery - Real World Projects

Goal: Build a deep, Elixir-first command of the JIDO ecosystem by combining req_llm (provider-agnostic model orchestration) and jido (pure-functional agent runtime). You will learn how to design deterministic agent logic, safe side-effect handling, workflow composition, and production-grade observability with BEAM strengths. By the end, you will be able to ship practical LLM workflows that are testable, policy-aware, resource-efficient, and resilient under retries, model outages, and multi-agent concurrency.

Introduction

  • What is JIDO? A growing ecosystem around jido (core agent behavior) and req_llm (provider-agnostic LLM API access) for building practical AI products in Elixir.
  • What problem does it solve today? It replaces duplicated HTTP glue, ad-hoc message formats, and side-effect-heavy agent code with standardized models, command contracts, and OTP supervision.
  • What you will build across the projects: a portfolio of agentic systems from API routers to autonomous workflows, each with measurable behavior and clear failure handling.
  • In scope: practical architecture, state and directive design, tool integration, streaming, testing, and production operations. Out of scope: deep tuning of every model, and building brand-new LLM providers.

Big-picture model:

┌──────────────────────────────────────────────────────────────┐
│                    Product Intent / Policy                    │
│  (business rules, budgets, safety boundaries, success criteria)│
└───────────────────────────────┬──────────────────────────────┘
                                │
               ┌────────────────▼────────────────┐
               │        jido Agents + Signals    │
               │ (pure commands, deterministic)   │
               └────────────────┬────────────────┘
                                │
               ┌────────────────▼────────────────┐
               │         Workflow Engine          │
               │ (plans, routing, checkpoints)    │
               └────────────────┬────────────────┘
                                │
               ┌────────────────▼────────────────┐
               │      req_llm Provider Layer      │
               │ (models, tools, output schemas)  │
               └────────────────┬────────────────┘
                                │
               ┌────────────────▼────────────────┐
               │   Observability + Persistence    │
               │ (cost, tokens, traces, states)   │
               └─────────────────────────────────┘

JIDO Ecosystem Big Picture

Use this as your default debugging lens: if a behavior is surprising, locate the boundary that crossed from policy -> command, command -> directive, or directive -> execution.

How to Use This Guide

  • Read the primer before any project. Keep a running notes file for each concept chapter because projects reference chapter names directly.
  • Pick one of the learning paths by your role and start from the first project in that path.
  • Validate progress after each project using its Definition of Done section and keep test evidence (transcripts, outputs, and decision logs).
  • Alternate between two project modes:
    • Build mode: implement the project.
    • Hardening mode: run through failure scenarios, retry logic, and cost checks before moving forward.

Prerequisites & Background Knowledge

Essential Prerequisites (Must Have)

  • Strong reading comprehension of Elixir syntax and data transformation.
  • Comfortable with maps, structs, protocols, and pattern matching.
  • Basic understanding of HTTP APIs and JSON payload design.
  • Familiarity with command-line debugging and process-based systems.
  • Recommended Reading: “The Little Elixir & OTP Guidebook” by Benjamin Tan Wei Hao - Ch. 1-6

Helpful But Not Required

  • Supervision trees in Phoenix/BEAM applications (learn during Projects 6-8)
  • Structured data validation (learn during Projects 1-4)

Self-Assessment Questions

  1. Can you explain how a function can be pure and still produce useful side-effectful behavior in an agent system?
  2. Can you identify where model provider differences leak into your product and where they should be abstracted?
  3. Can you propose a minimal checkpoint model for retries and timeouts across multiple steps?

Development Environment Setup

Required Tools:

  • Elixir 1.17+
  • Erlang/OTP 26+
  • Git
  • mix

Recommended Tools:

  • PostgreSQL or SQLite for state logs
  • curl for API smoke checks
  • telemetry stack for tracing

Testing Your Setup:

$ mix hex.info elixir
$ mix local.hex
$ mix local.rebar

Expected output snippets:

Erlang/OTP 26+
Elixir 1.17+
Rebar and Hex: ok

Time Investment

  • Simple projects: 4-8 hours each
  • Moderate projects: 10-20 hours each
  • Complex projects: 20-40 hours each
  • Total sprint: ~10-14 weeks for all 8 projects and a review project

Important Reality Check The hardest part is not calling LLM APIs; it is maintaining invariants: what is state, what is side effect, and what is guaranteed to be reproducible under retries.

Big Picture / Mental Model

The ecosystem split is simple:

  • req_llm solves how to talk to different model providers without your application owning all protocol differences.
  • jido solves how to think, decide, and route these model outputs through deterministic agents under OTP.

The two fit when agents emit structured plans and directives instead of raw imperative behavior.

+-------------------+      +--------------------+      +------------------+
|  req_llm Providers| ---> |  Agent Commands    | ---> | Runtime Directives|
|  (openai,
|  anthropic,
|  gemini, etc.)    |      | cmd/2 + state      |      | spawn, emit,
+-------------------+      +--------------------+      | schedule, stop    |
          │                        │                      +---------+--------+
          ▼                        ▼                                |
+-------------------+      +--------------------+                   |
|  Provider Sync    | ---> | Workflow + Signals  | <-----------------+
|  Model metadata   |      | routing + state     |
+-------------------+      +--------------------+

ReqLLM and Jido Boundary

This makes the system resilient to model churn:

  • Model selection changes stay in provider layer.
  • Business behavior stays in agent/action layer.
  • Runtime behavior stays in directives and OTP policies.

Theory Primer

Chapter 1: Canonical Provider Abstraction with req_llm

Fundamentals req_llm exposes one high-level path for different providers while preserving provider-specific capabilities where needed. It supports a model spec format and standardized request/response shapes. In practice, this means your product can start with two API clients and scale to many. The key idea is not “one universal API always works,” but “one universal contract plus deterministic provider translation.” The two-layer design lets you choose between generate_text/3, generate_object/4, and stream_text/3 helpers and direct low-level Req plugin use when you need custom HTTP behavior.

Deep Dive Provider abstraction is an economic and operational contract. In production code, every direct model call creates three risk planes: compatibility, semantics, and cost control. Compatibility risk includes schema drift across vendors (parameter names, token accounting, response format). Semantic risk includes differences in tool-calling and JSON support. Cost risk includes hidden charges from tool usage or image generation. The package addresses all three with three mechanisms:

  1. Canonical models and model specs. By pinning each call to explicit model identifiers and context structures, you avoid hard-coded request payload assumptions.
  2. Provider metadata and sync. Model registry data (cost, limits, context windows, modalities) is synchronized and can power budget decisions and runtime guards.
  3. Canonical/ provider-aware translation layers. Default options are translated into provider-specific field names and behavior, while your application writes against stable high-level terms.

The practical effect is visible in testability. You can test with fixtures, stubs, and cached responses because your call shape is stable. You can also route workloads by provider capabilities: for example, send tool-heavy extraction jobs to providers with stronger structured output support, while sending long-context summarization to models with favorable context windows. req_llm’s feature set in v1.5.1 also adds streaming adapters and explicit usage telemetry, so you can track both UX responsiveness and spend.

Failure modes to design for:

  • Model downgrade or outage: fallback path from preferred provider to secondary.
  • Invalid schema returns: structured output mode fails hard if schema assumptions are wrong.
  • Timeout amplification: long prompts with strict synchronous calls can stall if connection pools are misconfigured.
  • Key precedence ambiguity: wrong key source injection causing tests to use production keys.

A stable strategy is to treat provider differences as policy-managed capabilities, not as call-site conditionals.

Definitions & key terms

  • Canonical model: standardized schema used for request and response.
  • Model spec: <provider>:<model> shorthand plus optional config.
  • Structured generation: map/object output constrained by schema.
  • Streaming session: token-level transport that preserves immediate UX feedback.

Mental model diagram

User request
     |
     v
+-------------------+      +-----------------+
| req_llm high layer | ---> | provider-specific |
| generate_text/3     |      | translation       |
+-------------------+      +-----------------+
     |   
     v    
+----------------------------+
| canonical message / context |
| schema validation + retries |
+----------------------------+
     |
     v
+----------------------------+
| provider API + telemetry     |
| usage, cost, tool metadata   |
+----------------------------+

ReqLLM Canonical Translation Flow

How it works

  1. Parse incoming model spec and messages.
  2. Resolve provider module and model metadata.
  3. Apply provider options and capability constraints.
  4. Generate request payload in canonical form, then translate.
  5. Execute request and decode into canonical response.
  6. Return deterministic response with usage/cost metadata.

Invariants: invalid spec fails early; structured mode cannot silently return malformed shape; streaming responses expose tokens and metrics. Failure modes: provider HTTP faults, schema decode failures, key resolution errors.

Minimal concrete example

$ mix deps.get
$ mix req_llm.gen "Summarize last 3000 chars" --model anthropic:claude-3-7
{agent: ok}
$ mix req_llm.gen "Generate JSON invoice line" --json --model openai:gpt-4o-mini
{"invoice_id":"...","total":12.34}

Common misconceptions

  • “One provider abstraction means all providers behave identically.” They don’t; abstraction hides differences but must expose capability differences.
  • “Structured outputs are always guaranteed.” Schema mode can still produce edge-case validation failures.
  • “Streaming is only for UI.” Streaming is also a diagnostic channel for partial inference timing.

Check-your-understanding questions

  1. What does provider translation buy you that a raw HTTP client cannot?
  2. When would low-level Req usage be mandatory?
  3. How do you detect and react to usage cost drift over time?

Check-your-understanding answers

  1. Stable call contract and capability-aware options.
  2. Custom headers/auth flows, unusual endpoints, or strict request tracing.
  3. Compare usage metadata over history and route model/provider decisions with cost budgets.

Real-world applications

  • LLM routers in SaaS support portals.
  • Multi-provider fallback in enterprise copilots.
  • API translation layers for tool-calling toolchains.

Where you’ll apply it

  • Projects 1, 2, 3, 4.

References

Key insight Provider abstraction succeeds when it reduces cognitive load without hiding required capability distinctions.

Summary Without canonical contracts, each model call becomes bespoke technical debt. With req_llm, the contract is standardized, and capability differences become explicit strategy data.

Homework/Exercises to practice the concept

  1. List three providers and map which option names differ.
  2. Sketch a fallback matrix for latency, cost, and capability.
  3. Design a policy that blocks image generations after budget threshold.

Solutions to the homework/exercises

  1. Example names: max_tokens, max_completion_tokens, image endpoint variant names.
  2. Use a table with score-based provider ranking per task type.
  3. Add middleware before request execution that checks cumulative cost before every generate_* call.

Chapter 2: Typed Message, Tool, and Output Modeling

Fundamentals LLM workflows break when teams mix free-text prompts, non-deterministic tool shapes, and ad-hoc JSON assumptions. req_llm offers canonical types (Context, Message, ContentPart, Tool, Response, Usage) and integrates with structured generation. This pushes your architecture toward contracts, which is especially important as multiple agents begin producing output that feeds other agents.

Deep Dive Type modeling in agentic systems is not about syntax aesthetics; it is about blast-radius control. An untyped output from one model becomes a fault source in all downstream workers. A typed canonical layer solves this by moving uncertainty to validation boundaries.

In this ecosystem, typed modeling appears at two layers:

  1. Provider payload normalization in req_llm.
  2. Agent command/state semantics in jido.

For req_llm, schema-driven generation is your defense against “almost JSON.” This means declaring shape expectations before generation and letting the library validate decoded output. For long pipelines, the output model should be versioned because fields evolve while tasks remain stable.

For JIDO-style agents, message and signal typing is similarly central. Signals are envelopes for routing, audit, and observability. When each signal has a predictable envelope shape, supervisors, bus adapters, and external tools can interoperate without implicit assumptions.

Failure modes and patterns:

  • Loose contracts create silent downstream failures.
  • Overly strict contracts reject recoverable provider variants and trigger unnecessary retries.
  • Unversioned schemas make migrations painful and break older persisted outputs.

Design pattern: dual-layer schema strategy.

  • Internal schemas: strict and typed for business logic.
  • External schemas: permissive ingestion to canonical projection layer.

This allows you to preserve compatibility and reliability while still improving contract quality over time.

Definitions & key terms

  • Canonical schema: stable internal representation.
  • Schema drift: incompatible change in field presence or meaning.
  • Tool schema: declaration of function/tool input expected by model or agent.
  • Envelope: wrapper metadata around content, source, and trace context.

Mental model diagram

Incoming payload
   |
   v
+------------------+      +--------------------+
| Typed request     |---->| Validation boundary |
| schema + context  |     | canonical output    |
+------------------+      +--------------------+
         |                        |
         v                        v
+------------------+      +--------------------+
| Tool call payload |      | Signal / directive |
| model output      |      | execution metadata |
+------------------+      +--------------------+

Typed Contract Validation Boundary

How it works

  1. Define contracts for each stage before implementation.
  2. Encode expected output shape in req_llm schema options.
  3. Enforce validation at generation boundaries.
  4. Emit typed signals from agent actions.
  5. Translate typed outputs into agent state transitions.

Invariants: no stage may consume unknown/unvalidated payloads. Retry only idempotent steps. Failure modes include schema mismatch, unknown provider fields, and signal routing to non-existent handlers.

Minimal concrete example

Schema draft:
- invoice_id (required, string)
- total_cents (required, integer)
- currency (required, one of usd/eur/gbp)

Run:
$ mix req_llm.gen "Parse invoice" --json --model openai:gpt-4o-mini
{"invoice_id":"INV-1001","total_cents":1299,"currency":"usd"}

Common misconceptions

  • “Schemas remove creativity.” They don’t; they constrain just the machine contract.
  • “Validation is optional in prototypes.” It is optional only for throwaway scripts.
  • “One schema per project is enough.” High maturity requires multiple versions.

Check-your-understanding questions

  1. Why should schema versions exist in long-lived workflows?
  2. Where is the best place to validate external tool results?
  3. What is the failure mode when strict schema mode is too rigid?

Check-your-understanding answers

  1. Persistence and replay become predictable.
  2. At boundary modules that own the external integration.
  3. Retry loops and user-visible false failures when model output format changes.

Real-world applications

  • Invoice, incident, and procurement parsers.
  • Policy-driven orchestration where signals route by typed tags.
  • Contract tests for multi-agent tool pipelines.

Where you’ll apply it

  • Projects 1, 2, 6, 7.

References

Key insight Strong interfaces are your leverage point: they let agents reason with less ambiguity and recover faster from drift.

Summary Use strict contracts where behavior matters, and resilient parsing at external boundaries where reality is noisy.

Homework/Exercises to practice the concept

  1. Draft two versions of an invoice output schema.
  2. Propose backward compatibility for a changed total field name.
  3. Design one signal envelope for human approval.

Solutions to the homework/exercises

  1. v1 (total_cents, currency), v2 (line_total, currency_code) with explicit migration.
  2. Keep aliases at translation stage with deprecation logging.
  3. Include approval_id, actor, resource, and deadline_ms fields in signal envelope.

Chapter 3: Pure Command + Directive Runtime in jido

Fundamentals jido centers on the idea that an agent command should be pure state transition logic, while effects remain explicit directives. cmd/2 returns updated agent state and a directive list. This separation enables deterministic tests and predictable execution semantics.

Deep Dive In raw GenServer code, logic and side effects often co-locate. That makes testing difficult because every branch may touch external systems. jido formalizes a command architecture where each action’s role is clear: transform state; emit intent. This is conceptually similar to Elm and Redux reducers paired with effect descriptors.

For practical systems, this has three payoffs:

  1. Determinism: same input -> same state change.
  2. Auditability: directives reveal external impact intent.
  3. Runtime swapability: directive execution policy can change without changing business logic.

A typical confusion is thinking directives are mandatory external code inside action functions. In this ecosystem, directives are returned data. Execution occurs through runtime strategies (AgentServer, strategy modules, supervisors). This makes it easier to test with fixtures and to introduce policy changes.

Failure modes:

  • State + directive ambiguity: writing external APIs directly inside action functions breaks determinism.
  • Over-directed commands: returning too many directives without explicit ordering can hide flow bugs.
  • Unhandled directive types: runtime may drop unknown directives silently if not registered.

Countermeasures:

  • Keep each action small and composable.
  • Classify directives by effect domain (emit/schedule/spawn/stop).
  • Add compile-time and runtime guards for directive schema.

Definitions & key terms

  • cmd/2: core operation returning {agent, directives}.
  • Action: command unit with schema and run/2 behavior.
  • Directive: external effect description.
  • StateOp: in-command internal state update.

Mental model diagram

+-----------+      +-----------------+      +--------------------+
| Action fn | ---> | cmd/2 contract  | ---> | Directives Pipeline |
| run/2     |      | (state only)    |      | runtime interpreter  |
+-----------+      +-----------------+      +--------------------+
                                              |
                                              v
                                       +----------------------+
                                       | External side effects |
                                       +----------------------+

Jido cmd/2 Directive Pipeline

How it works

  1. Receive command and validate action schema.
  2. Compute next state from immutable rules.
  3. Construct directive list with no direct side effects.
  4. Return state + directives.
  5. Runtime strategy executes directives atomically or with policies.

Invariant: state changes are pure functions of input. Failure modes: invalid action params, missing directive handler, directive execution timeout.

Minimal concrete example

$ make run-counter
before: count=0
command increment(+1)
after:  count=1 directive=[Emit(:progress_event)]

Common misconceptions

  • “Directives are slow because they are indirect.” Indirection increases clarity and testability.
  • “Pure logic means no useful side effects.” It means side effects are explicit and controlled.
  • “Actions must be classes/modules only.” They are composable units with schema and deterministic behavior.

Check-your-understanding questions

  1. Why return directives instead of executing actions directly?
  2. What is the value of StateOps?
  3. How do you test action purity?

Check-your-understanding answers

  1. So runtime can control ordering, retries, and supervision.
  2. They represent safe, internal state transition operations.
  3. Snapshot-before/snapshot-after comparisons and property-style invariants.

Real-world applications

  • Financial workflows with audit trails.
  • Approval systems where actions should be reproducible.
  • Internal tool orchestration with explicit side effects.

Where you’ll apply it

  • Projects 5, 6, 7, 8.

References

Key insight The hardest production bugs disappear when you force state changes to be pure and effects to be declared.

Summary cmd/2 is the seam between reasoning and action. Guard it well and most system behavior becomes inspectable.

Homework/Exercises to practice the concept

  1. Convert one side-effecting action into command+directive form.
  2. Define three directive types for one domain.
  3. Define test case for idempotence.

Solutions to the homework/exercises

  1. Move API call out of action, emit HttpCall directive.
  2. Emit, Schedule, Stop.
  3. Run command twice on same state and verify expected stable state transitions.

Chapter 4: OTP Runtime, State Machines, and Signals

Fundamentals jido runs on BEAM/OTP, so you inherit supervision, fault recovery, and scheduling primitives. The differentiator is that agent behavior is structured before supervision: directives and signals are first-class messages and execution strategies are pluggable (Direct execution, FSM, custom strategies).

Deep Dive In OTP systems, resilience is achieved by supervision and restart patterns; in agent systems, resilience is also about intent preservation. If an agent step crashes, should you replay from the start or checkpoint and resume? If messages are unordered, can causality still hold? jido addresses this by coupling immutable agent state with explicit signals and strategies.

A practical framework for workflow reliability has four layers:

  1. Agent state: pure domain state and transition invariants.
  2. Signal envelope: typed routing metadata.
  3. Strategy: execution policy including retries, queueing, and timing.
  4. Supervisor tree: process lifecycle.

In a multi-agent setup, signals are crucial. They carry operation intent across boundaries without forcing each agent to know every peer’s internals. This allows horizontal growth and easier policy enforcement.

Failure modes:

  • Signal loss from misconfigured routes.
  • Signal storms from unbounded retries.
  • Unsupervised children that survive only because upstream state still references them.

Mitigation:

  • Explicit backoff and max-retry budgets.
  • Supervisor limits and child lifecycle tracing.
  • Bounded queues and dead-letter/error policies.

Definitions & key terms

  • FSM strategy: finite-state-machine based execution style.
  • Signal: portable message envelope for eventing/routing.
  • Restart policy: how failed processes are recovered.
  • Supervision tree: hierarchical ownership model of BEAM.

Mental model diagram

+----------------+
|  AgentServer   |
|  cmd/state map |
+--------+-------+
         |
  +------+-------------------+
  |      strategy layer     |
  +------+-------------------+
         |
   +-----+------+      +-------------------+
   | Directive  | ---> | Signal Router     |
   | Execution  |      | (priority+route)  |
   +-----------+      +-------------------+
         |
   +-----+------+
   | Supervisor |
   | restart    |
   +------------+

AgentServer Strategy and Signal Router

How it works

  1. Receive a command and produce directives.
  2. Strategy chooses execution order (direct/fsm/custom).
  3. Signals route to other agents or worker adapters.
  4. On failure, supervisor applies strategy.
  5. Metrics and traces record each lifecycle transition.

Invariants: no uncontrolled process spawning and no indefinite loops; signals must be routeable. Failure modes: unbounded queue growth, invalid restart thresholds, poison messages.

Minimal concrete example

$ mix agent.harness.run --scenario timeout-retry
step1: command accepted
step2: worker timeout -> retry with delay
step3: max_retries=2 reached -> emit error directive

Common misconceptions

  • “OTP is enough; no extra strategy needed.” Strategy is the policy layer between generic OTP and domain flow control.
  • “Signals are just messages.” A signal is message + metadata + intended route semantics.
  • “FSMs are overkill.” They are useful where retries and transitions must be visible.

Check-your-understanding questions

  1. What happens to command determinism when strategy changes?
  2. Why are signals better than direct pid calls for scale?
  3. Where do you enforce idempotency?

Check-your-understanding answers

  1. Keep command semantics stable; strategy can alter execution only.
  2. Signals keep topology decoupled and routeable.
  3. In state transitions and deduplication keys.

Real-world applications

  • Asynchronous workflows in support desks.
  • Multi-step automation that requires pause/resume behavior.
  • Human-in-the-loop escalation chains.

Where you’ll apply it

  • Projects 6, 7, 8.

References

  • Jido Core Loop / Agents / Signals docs links in hexdocs
  • OTP principles from “Designing for Scalability with Erlang/OTP” (secondary)

Key insight OTP gives failure recovery; jido’s strategy and signal layers give intentional agent behavior.

Summary Your systems become scalable when process control and business semantics are separated into explicit layers.

Homework/Exercises to practice the concept

  1. Draw a restart policy for a transient API timeout.
  2. Define route rules for two failure signals.
  3. Propose max queue depth and backoff values.

Solutions

  1. 1) retry, 2) wait-and-retry, 3) dead-letter after budget.
  2. :agent_busy, :downstream_timeout with different retry budgets.
  3. Start with 200 queue depth; exponential backoff with cap.

Chapter 5: Workflow Orchestration + Testing + Observability

Fundamentals Observability and testability are non-negotiable when multiple models and agents coordinate. A workflow is only useful if you can answer: what happened, why, and how much it cost. jido workflows provide structure; req_llm provides request-level telemetry and usage metadata.

Deep Dive Workflows become brittle when branching and tool calls are embedded in opaque functions. In this ecosystem, use explicit instruction graphs and bounded transitions. You then test three layers:

  1. Command tests: deterministic state transitions.
  2. Workflow tests: sequencing, timeouts, and failure branching.
  3. Runtime tests: directive execution and strategy semantics.

Cost observability is similar: collect token usage and tool usage per stage. If stream usage jumps after a code change, you have a regression. If a model route changes due to config, usage should be expected and visible.

Failure modes:

  • Latent nondeterminism from non-typed inputs.
  • Observability gaps where side effects happen without trace IDs.
  • Test suites that only pass in happy-path mode.

Countermeasures:

  • Tag each workflow run with correlation IDs.
  • Assert telemetry events in tests (token counts, latency buckets).
  • Inject fixture responses and run provider matrix tests.

Definitions & key terms

  • Instruction sequence: ordered action list executed by workflow engine.
  • Workflow telemetry: execution-level events beyond model tokens.
  • Fixture matrix: deterministic recorded responses for provider simulation.
  • Correlation id: request-level link across stages.

Mental model diagram

Run Request
   |
   v
+----------------------+      +--------------------+
| Validation + Contract | ---> | Workflow Executor  |
+----------------------+      +--------------------+
                                  |
                      +-----------+-----------+
                      |                       |
                [Branch A]               [Branch B]
                      |                       |
             +--------v---------+     +------v--------+
             | req_llm call      |     | jido cmd      |
             +------------------+     +---------------+
                      
                      v
                Trace + Usage Ledger

Workflow Branch Execution Ledger

How it works

  1. Define workflow graph and policy constraints.
  2. Validate each step input/ output contract.
  3. Execute with timeout and retry options.
  4. Emit events at step boundaries.
  5. Aggregate cost and state transitions for audits.

Invariants: every executed step must emit an event; costs must be logged; failed steps should be classifiable. Failure modes: missing events, lost trace context, provider timeout storms.

Minimal concrete example

$ mix test test/workflow_matrix_test.exs
test: workflow_rejects_invalid_payload ... ok
test: workflow_handles_llm_timeout ... ok
metrics: avg_cost_delta_ms +2ms

Common misconceptions

  • “If it works locally, it works under load.” Workflows fail differently under concurrency.
  • “Telemetry slows things down and should be deferred.” Light traces are essential for production behavior.
  • “Integration tests replace unit tests.” They are complementary.

Check-your-understanding questions

  1. Why must workflow tests include timeout and retry cases?
  2. How does token telemetry influence architecture?
  3. Why version workflow outcomes?

Check-your-understanding answers

  1. They reproduce production failure modes.
  2. It reveals cost regressions and provider route shifts.
  3. For replay, rollback, and incident postmortems.

Real-world applications

  • Customer support agent fleets.
  • Data extraction + verification systems.
  • Internal copilots with approval gates.

Where you’ll apply it

  • Projects 1, 2, 3, 4, 6, 8.

References

  • ReqLLM usage and streaming sections — ReqLLM 1.5.1
  • Jido testing guides — jido testing
  • “Site Reliability Engineering” by Beyer et al.

Key insight If a workflow is not observable and testable as a sequence, it is not production safe.

Summary Treat each run as a financial and reliability unit, not just an execution.

Homework/Exercises to practice the concept

  1. Build three traces for one workflow: success, timeout, retry.
  2. Add correlation ids to every step.
  3. Add a simple budget rule and observe behavior.

Solutions

  1. Sequence with success and two failure branches.
  2. Prefix logs with run id and action id.
  3. Rule: abort if cumulative token cost exceeds threshold.

Chapter 6: BEAM Production Tradeoffs: Concurrency, Isolation, and Throughput

Fundamentals The BEAM runtime is strong at lightweight processes, fault isolation, and supervisor-driven recovery. jido + req_llm succeeds when architecture is built to exploit these properties: many short-lived agents, bounded concurrency, and isolation boundaries.

Deep Dive This is where pragmatic architecture matters. A single long-running task with shared mutable state is fragile; many small isolated agents with explicit messages are safer. jido gives explicit state and directive models; req_llm can be configured with connection pools and stream handling for throughput control.

A mature design pattern uses five knobs:

  1. Concurrency caps to avoid provider saturation.
  2. Pooling strategy for HTTP clients.
  3. Backpressure in signals and queues.
  4. Hibernation / persistence boundaries for inactive agents.
  5. Route-level cost budgets before request dispatch.

You are balancing latency and reliability tradeoffs. Direct synchronous calls improve simplicity but block throughput; asynchronous streaming and queueing improve throughput but require stronger ordering semantics.

Failure patterns:

  • Overcommitted pools causing request queuing and timeouts.
  • Retry loops creating thundering herds.
  • Starvation in shared scheduler when one project over-runs resources.

Use req_llm streaming and Finch pool configuration as operational controls, and use jido strategies to serialize sensitive transitions.

Definitions & key terms

  • Backpressure: explicit control to slow intake when downstream is saturated.
  • Process isolation: independent state and failure boundary per BEAM process.
  • Prefetch/concurrency cap: maximum in-flight work.
  • Hibernation: memory/CPU optimization for idle agent processes.

Mental model diagram

Incoming jobs      Concurrency queue       Agent pools
   |                    |                     |
   +------------------->+-------------------->+
                        v                     |
                  +-----------+        +-------------------+
                  |  Strategy | ----->  | req_llm workers    |
                  |  policy   |        | Finch connections  |
                  +-----------+        +-------------------+
                        |                     |
                        v                     v
                 +-------------------------------+
                 | Backpressure + Retry Controls |
                 +-------------------------------+

BEAM Throughput and Backpressure Controls

How it works

  1. Accept job and classify with policy (rate, priority).
  2. Route to bounded queue and agent strategy.
  3. Apply concurrency cap before dispatch.
  4. Execute in worker with timeout + backoff.
  5. Record metrics and release capacity.

Invariants: bounded memory usage, bounded queue, and bounded retries. Failure modes: unbounded queue, dropped signals, incorrect fairness.

Minimal concrete example

$ mix jido.ops.report --window 1m
throughput_ok: 142 req/min
timeouts: 1
queue_depth: 12/200
retry_count: 8

Common misconceptions

  • “BEAM guarantees no bottlenecks.” It guarantees isolation, not infinite throughput.
  • “More concurrency always means faster.” Without budget controls, it means more failures.
  • “Streaming is cheap.” Streams require proper pool sizing and lifecycle control.

Check-your-understanding questions

  1. What does bounded concurrency prevent?
  2. Why are provider pools part of business reliability?
  3. Why do we need queue depth limits in agent systems?

Check-your-understanding answers

  1. Prevents collapse under bursty traffic.
  2. Provider limits directly impact success rates and latency budgets.
  3. To protect downstream and preserve fairness under load.

Real-world applications

  • High-volume support and QA automation.
  • Document processing with model-heavy stages.
  • Asynchronous enterprise copilots used across teams.

Where you’ll apply it

  • Projects 1, 3, 4, 7, 8.

References

  • Finch and streaming notes in ReqLLM overview
  • OTP/BEAM architecture references from Erlang/OTP runtime docs
  • “The Book of Elixir” by Dave Thomas (supervision semantics)

Key insight Throughput is a contract: every layer must expose and enforce limits.

Summary Use bounded concurrency and explicit scheduling as first-class design parameters, not tuning after launch.

Homework/Exercises to practice the concept

  1. Estimate safe max in-flight requests for provider pools.
  2. Simulate burst load and note queue depth behavior.
  3. Introduce a fallback route when queue is near capacity.

Solutions

  1. Start low, then raise via controlled load testing.
  2. Keep queue and error metrics visible during test.
  3. Route low-priority requests to delayed queue with backoff.

Glossary

  • Action: A validated operation that returns an agent state transition and optional directives.
  • Canonical model: Standardized internal representation for model requests and outputs.
  • Directive: Declarative instruction describing a side effect.
  • Directive strategy: Runtime policy that executes directives.
  • FSM: Finite state machine execution style.
  • Provider adapter: Module translating canonical request to provider-specific protocol.
  • Signal: Structured event envelope used to route between agents and services.
  • StreamResponse: Streaming response abstraction that yields tokens and usage metadata.
  • Workflow: Ordered set of actions and instructions.

Why JIDO Matters

By 2026, AI systems have moved from single-shot prompts to orchestrated agentic loops. The ecosystem response has shifted to reusable, observable runtimes that can operate across providers and agents.

  • Standards pressure: interoperability moved from a “nice-to-have” to a necessity as tool ecosystems grow.
  • Cost pressure: providers vary in latency, output quality, and pricing; observability is required.
  • Reliability pressure: agents execute with side effects; non-determinism must be bounded.

A practical old-vs-new view:

Old Stack                                  New Stack
----------                                  --------
Prompt scripts only                         Standardized LLM client + command agents
Provider-specific request code               Provider-agnostic + model metadata
Side effects inline                         Pure cmd + explicit directives
Implicit logs                               Trace-aware signals + cost telemetry
Best-effort retries                         Strategy/rate-aware retries with policies

Old vs New Agent Stack

Real-world stats and sources (as of February 14, 2026):

  • ReqLLM package page lists v1.7.0 and about 26.8K downloads on Hex, indicating active adoption for provider-agnostic Elixir LLM integration. Source: hex.pm/packages/req_llm
  • Jido package page lists v1.5.0 and about 7.5K downloads on Hex, reflecting sustained usage of the core agent runtime. Source: hex.pm/packages/jido
  • Jido.Signal package page lists v1.0.2 and about 0.6K downloads, showing an emerging but growing event/signals layer for multi-agent dispatch. Source: hex.pm/packages/jido_signal
  • On GitHub, the agentjido/jido repository shows about 1.7K stars, and agentjido/req_llm shows about 294 stars, suggesting healthy community visibility around both runtime and LLM abstraction layers. Sources: github.com/agentjido/jido, github.com/agentjido/req_llm
  • Context7-indexed ReqLLM docs describe a provider-agnostic registry with 45+ providers and 665+ models, plus usage/cost normalization and streaming abstractions. Source: context7 req_llm index
  • Public ecosystem shift: major model vendors moved toward protocol-based tool interoperability (MCP-era integrations), reinforcing the need for adapter-driven architecture. Sources: Google Gemini updates (2025), Anthropic MCP context (Nov 25, 2024)

The practical conclusion: if your project depends on one provider or one side-effect pattern, you will rework often. A disciplined ecosystem-level design compounds once.

Concept Summary Table

Concept Cluster What You Need to Internalize
Provider-agnostic LLM abstraction Use req_llm to normalize call patterns while preserving provider-specific capabilities and controls.
Typed contracts and schemas Treat prompts, tool arguments, and outputs as explicit schemas with versioned boundaries.
Pure cmd and directive pattern Keep state transitions deterministic, route side effects through explicit directive descriptions.
Signals, workflows, and orchestration Model agent-to-agent communication and multi-step flow with routing, retries, and checkpoints.
Observability and testing Measure usage, latency, and failure classes at both workflow and model boundaries.
BEAM-driven production controls Use supervision, bounded concurrency, and process isolation as architectural primitives, not infrastructure afterthoughts.

Project-to-Concept Map

Project Concepts Applied
Project 1: Multi-Model Gateway with req_llm Provider-agnostic LLM abstraction, Typed contracts, BEAM production controls
Project 2: Schema-First Contract Extractor Typed contracts and schemas, Observability and testing
Project 3: Streaming Copilot Operations Center Provider-agnostic LLM abstraction, Observability and testing
Project 4: Provider Failover and Cost Router Provider-agnostic LLM abstraction, BEAM production controls
Project 5: Deterministic Counter Agent Pure cmd and directive pattern
Project 6: Workflow-First Orchestrator Signals, workflows, and orchestration, Pure cmd and directive pattern
Project 7: Signal Mesh for Multi-Agent Delegation Signals, workflows, and orchestration, BEAM-driven production controls
Project 8: Audit-Ready State Machine Agent Pure cmd/directive, Observability/testing, Signals/workflows

Deep Dive Reading by Concept

Concept Book and Chapter Why This Matters
Provider-agnostic LLM abstraction “Designing Data-Intensive Applications” by Martin Kleppmann - Ch. 3 Helps evaluate tradeoffs in protocol layering and data contracts.
Typed contracts and schemas “Domain-Driven Design” by Eric Evans - Ch. 4-6 Prevents semantic drift when integrating LLM outputs with business logic.
Pure cmd and directive pattern “Designing Elixir Systems with OTP” by James Edward Gray II - Ch. 5-8 Grounds immutability and explicit side-effect boundaries in production systems.
Signals and workflow orchestration “Building Microservices” by Sam Newman - Ch. 8 Clarifies message-driven integration, routing, and failure domains.
Observability and testing “Site Reliability Engineering” by Betsy Beyer et al. - Ch. 4 Establishes practical measurement and reliability patterns for operations.
BEAM production controls Elixir/Erlang docs (Supervision and tasks) - v1.17+ Explains process isolation, supervision, and failure recovery models.

Quick Start: Your First 48 Hours

Day 1:

  1. Read this guide up to Chapter 6.
  2. Install prerequisites and scaffold one mix app with jido and req_llm dependencies.
  3. Start Project 1 and Project 5 to internalize two different abstraction styles.

Day 2:

  1. Validate Project 1 by dispatching same prompt across two providers.
  2. Validate Project 5 by proving command determinism with repeated runs.
  3. Log one failure and one retry event.

Day 3–4:

  1. Move to Project 2 and Project 6.
  2. Define schema contracts and workflow checkpoints.

End of weekend: you should be able to explain why directive-based designs remain testable where pure functions end and process runtime begins.

Path 1: Product Builder (Recommended)

  • Project 1 → Project 2 → Project 3 → Project 6 → Project 8

Path 2: Platform/LLM Operator

  • Project 1 → Project 4 → Project 3 → Project 7 → Project 8

Path 3: Agent Architect

  • Project 5 → Project 6 → Project 7 → Project 8

Success Metrics

  • You can design and explain at least three provider-selection policies that include fallback and budget rules.
  • At least 90% of project workflows include explicit failure handling paths with recorded outcomes.
  • You can run a mixed test matrix for at least one workflow with happy-path, timeout, invalid payload, and directive-missing cases.
  • You can route between at least two agents using typed signals and verify end-to-end trace continuity.

Project Overview Table

# Project Package Focus Difficulty Time Key Focus
1 Multi-Model Gateway with req_llm req_llm 2 1.5-2 weeks Provider abstraction + unified contracts
2 Schema-First Contract Extractor req_llm 3 2 weeks Structured outputs + schema contracts
3 Streaming Copilot Operations Center req_llm 3 1.5-2 weeks Streaming + telemetry + UX feedback
4 Provider Failover and Cost Router req_llm + jido 4 2-3 weeks Cost/routing + key management
5 Deterministic Counter Agent jido 2 1.5 weeks cmd/2 and directive separation
6 Workflow-First Orchestrator jido 4 2-3 weeks Workflow engine + action sequencing
7 Signal Mesh for Multi-Agent Delegation jido_signal 4 2-3 weeks Signals, routing, and parent-child dynamics
8 Audit-Ready State Machine Agent jido 5 3-4 weeks FSM strategies + observability + production hardening

Project List

The following projects guide you from a provider-first architecture to agentic production systems.

Project 1: Multi-Model Gateway with req_llm

  • File: P01-multi-model-gateway.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Gleam, F#
  • Coolness Level: 4/5 (high)
  • Business Potential: 4/5
  • Difficulty: 2
  • Knowledge Area: Protocol adaptation, LLM orchestration
  • Software or Tool: req_llm, mix
  • Main Book: The Little Elixir and OTP Guidebook

What you will build: A provider-agnostic inference gateway that normalizes prompts, logs standardized usage, and fails over between providers.

Why it teaches JIDO: It shows how stable provider contracts make agent workflows portable and controllable.

Core challenges you will face:

  • Model portability -> map providers without branching logic at call sites.
  • Rate and reliability policy -> define fallback and quotas.
  • Usage accountability -> expose token and cost trace lines per request.

Real World Outcome

You run a single CLI command with one prompt and see two providers return the same canonical response envelope, each with independent latency and usage fields. A successful run includes a deterministic routing decision and a fallback branch test.

For CLI projects - exact output shape:

$ mix run -e "Gateway.demo()"
[router] run_id=gw_001 policy=latency_then_cost
[router] primary=openai:gpt-4o-mini secondary=anthropic:claude-haiku-4-5
[provider:openai] status=ok latency_ms=612 input_tokens=122 output_tokens=74 total_cost_usd=0.0018
[provider:anthropic] status=ok latency_ms=703 input_tokens=122 output_tokens=79 total_cost_usd=0.0021
[gateway] canonical_response.id=resp_9f2a source=primary
[gateway] fallback_triggered=false

The Core Question You Are Answering

“How can one product endpoint stay stable while model providers, model IDs, and pricing keep changing?”

This matters because most LLM platform rewrites happen at integration boundaries, not business logic boundaries.

Concepts You Must Understand First

  1. Canonical Request/Response Contracts
    • What fields are internal invariants vs provider-specific metadata?
    • Book Reference: “Designing Data-Intensive Applications” by Martin Kleppmann - Ch. 3
  2. Provider Capability Matrices
    • Which tasks need tool calling, JSON mode, or long context?
    • Book Reference: “Building Microservices” by Sam Newman - Ch. 8
  3. Cost and Latency Budgets
    • What thresholds trigger fallback or route denial?
    • Book Reference: “Site Reliability Engineering” by Beyer et al. - Ch. 4

Questions to Guide Your Design

  1. Contract Boundary
    • Which fields are always present in canonical output?
    • How do you preserve provider diagnostics without leaking them upstream?
  2. Routing Policy
    • What is your deterministic tie-break order?
    • How do you test fallback behavior without real outages?

Thinking Exercise

Route Before You Build

Draw a decision table with rows as request types (chat, extract, tool-heavy) and columns as providers. For each cell, assign latency, cost, and reliability scores, then define a deterministic winner.

The Interview Questions They Will Ask

  1. “How do you design a provider-agnostic layer without hiding critical provider differences?”
  2. “What should happen when the primary provider times out after partial streaming output?”
  3. “How do you audit model routing decisions for compliance and cost governance?”
  4. “What signals prove your abstraction reduced operational risk?”
  5. “How do you test fallback logic deterministically?”

Hints in Layers

Hint 1: Stable Envelope First
Define your canonical output map before writing any provider selection code.

Hint 2: Separate Policy From Transport
Treat routing as data and keep transport/retry code in a separate module.

Hint 3: Deterministic Fallback Pseudocode

if primary.status in [:timeout, :rate_limited] then use secondary
else return primary

Hint 4: Debugging Emit run_id, selected provider, fallback reason, and token/cost deltas on every request.

Books That Will Help

Topic Book Chapter
Interface contracts “Designing Data-Intensive Applications” by Kleppmann Ch. 3
Reliability budgets “Site Reliability Engineering” by Beyer et al. Ch. 4

Common Pitfalls and Debugging

Problem 1: “Different provider responses break downstream parser”

  • Why: Canonicalization happened after business logic.
  • Fix: Canonicalize immediately after provider call.
  • Quick test: run one prompt against two providers and diff canonical JSON.

Problem 2: “Fallback loops forever under repeated timeouts”

  • Why: Retry and fallback policies are coupled incorrectly.
  • Fix: Add explicit max attempts and terminal failure class.
  • Quick test: inject three timeout faults and confirm hard stop.

Definition of Done

  • One endpoint supports at least two providers with identical canonical output shape.
  • Fallback executes deterministically for at least two failure classes.
  • Token and cost fields are logged per request and validated in tests.
  • A contract test fails if canonical schema changes unintentionally.

Project 2: Schema-First Contract Extractor

  • File: P02-schema-first-contract-extractor.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: TypeScript, Python
  • Coolness Level: 5/5
  • Business Potential: 4/5
  • Difficulty: 3
  • Knowledge Area: Validation, structured outputs, contracts
  • Software or Tool: req_llm, JSON tooling
  • Main Book: Domain-Driven Design

What you will build: A contract parser that turns messy text into validated typed outputs with schema versioning and rejection pathways.

Why it teaches JIDO: It maps model outputs into deterministic agent inputs and demonstrates contract governance.

Core challenges you will face:

  • Structured output strictness -> avoid false positives and silent drift.
  • Schema evolution -> backward-compatible migration.
  • Downstream safety -> prevent malformed payload propagation.

Real World Outcome

You submit noisy text (emails, ticket descriptions, invoices) and receive either a valid normalized object or an explicit validation error with reason and schema version. Every result is replayable by run_id.

$ mix run -e "Extractor.demo()"
[extractor] run_id=ex_017 schema=v2
[input] source=ticket_442 raw_chars=1381
[result] status=valid confidence=0.93
[result] payload={"priority":"high","category":"billing","customer_tier":"pro"}
[audit] schema_version=v2 migrated_from=v1 fields_added=1

The Core Question You Are Answering

“How do we make model output reliable enough to become a typed contract between agents?”

The answer determines whether your workflow fails early and safely or fails late and expensively.

Concepts You Must Understand First

  1. Schema Versioning
    • How do you evolve fields without breaking old runs?
    • Book Reference: “Domain-Driven Design” by Eric Evans - Ch. 4-6
  2. Validation Boundaries
    • Where should invalid payloads be rejected?
    • Book Reference: “Release It!” by Michael Nygard - Ch. 8
  3. Deterministic Rejection Paths
    • How do you classify recoverable vs terminal validation failures?
    • Book Reference: “Site Reliability Engineering” by Beyer et al. - Ch. 5

Questions to Guide Your Design

  1. Schema Design
    • Which fields are required vs derived?
    • How do you represent confidence and provenance?
  2. Migration Policy
    • How do you upgrade v1 payloads to v2?
    • What test guarantees no silent field loss?

Thinking Exercise

Failure Taxonomy Draft

Create three categories: recoverable_format_error, unknown_enum, and missing_required_field. For each, define retry policy, operator alerting, and user-facing message.

The Interview Questions They Will Ask

  1. “How do you avoid accepting ‘almost valid’ JSON from an LLM?”
  2. “What is your strategy for schema migration over six months?”
  3. “How do you detect semantic drift even when shape validation passes?”
  4. “What metrics indicate extraction quality is degrading?”
  5. “Where do you place this validation in a multi-agent pipeline?”

Hints in Layers

Hint 1: Version Every Contract
Treat schema version as first-class metadata, not documentation.

Hint 2: Two-Phase Parsing
Use permissive parse then strict canonical validation.

Hint 3: Rejection Pseudocode

if required_field_missing -> terminal_invalid
if enum_unknown -> recoverable_with_mapping

Hint 4: Debugging Persist invalid payload samples with reason tags for weekly drift review.

Books That Will Help

Topic Book Chapter
Domain contracts “Domain-Driven Design” by Eric Evans Ch. 4-6
Failure containment “Release It!” by Michael Nygard Ch. 8

Common Pitfalls and Debugging

Problem 1: “Extractor passes tests but fails in production text”

  • Why: Test corpus does not cover real noise patterns.
  • Fix: Add adversarial fixtures from real anonymized failures.
  • Quick test: run nightly validation against top 50 failure samples.

Problem 2: “Schema migration drops field semantics”

  • Why: Field rename happened without migration alias.
  • Fix: Add explicit alias map and deprecation log.
  • Quick test: replay v1 fixtures and compare semantic equivalence.

Definition of Done

  • Validator returns deterministic pass/fail with reason codes.
  • At least two schema versions are supported with documented migration.
  • Invalid payloads are logged with reproducible fixture IDs.
  • Downstream workflow consumes only canonical validated objects.

Project 3: Streaming Copilot Operations Center

  • File: P03-streaming-copilot-operations-center.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: JavaScript, Go
  • Coolness Level: 4/5
  • Business Potential: 5/5
  • Difficulty: 3
  • Knowledge Area: Streaming, observability, UX feedback
  • Software or Tool: req_llm, dashboard tooling
  • Main Book: Site Reliability Engineering

What you will build: A terminal-style operations copilot with real-time streaming, usage reporting, and cost-aware cutoffs.

Why it teaches JIDO: It teaches production telemetry and responsive workflows for model calls.

Core challenges you will face:

  • Stream lifecycle -> avoid blocking token loops.
  • Telemetry completeness -> collect usage even on streamed flows.
  • Quality thresholds -> enforce response quality controls.

Real World Outcome

You type a question and see token chunks appear live, plus end-of-run usage and cost totals. If latency or budget thresholds are exceeded, the stream is interrupted with an explicit policy message.

$ mix run -e "OpsCopilot.demo()"
[stream] run_id=sc_201 provider=anthropic:claude-haiku-4-5
token> To
token> investigate
token> this
token> incident,
[stream] complete=true duration_ms=1680
[usage] input_tokens=210 output_tokens=132 total_cost_usd=0.0034
[policy] within_budget=true within_latency_slo=true

The Core Question You Are Answering

“How can streaming improve operator response time without sacrificing traceability and cost control?”

Streaming is only a production win when partial output, cancellation, and usage attribution are all handled correctly.

Concepts You Must Understand First

  1. Stream Lifecycle States
    • What are start, chunk, complete, and cancel transitions?
    • Book Reference: “Site Reliability Engineering” by Beyer et al. - Ch. 4
  2. Correlation IDs
    • How do you trace a streamed response across logs and dashboards?
    • Book Reference: “Designing Data-Intensive Applications” by Kleppmann - Ch. 11
  3. Budget Guardrails
    • Where do you enforce token and cost limits in streamed mode?
    • Book Reference: “Release It!” by Michael Nygard - Ch. 11

Questions to Guide Your Design

  1. Streaming UX
    • What should users see when stream is partial or interrupted?
    • How do you render provider switch events?
  2. Operational Observability
    • Which metrics are emitted per chunk vs per run?
    • How do you preserve final usage when stream is cancelled?

Thinking Exercise

Chunk Timeline Analysis

Draw a timeline for one request with 12 chunks. Mark where backpressure, timeout, and user cancellation may happen. Decide which events must be persisted.

The Interview Questions They Will Ask

  1. “What metrics are unique to streaming versus non-streaming inference?”
  2. “How do you ensure no token chunks are lost during provider reconnect?”
  3. “What is your cancellation strategy to prevent orphan requests?”
  4. “How do you communicate partial answer confidence?”
  5. “How would you test stream resilience under packet loss?”

Hints in Layers

Hint 1: Treat Stream as State Machine
Model transitions explicitly: starting -> active -> finished|cancelled|failed.

Hint 2: Separate Chunk Rendering From Accounting
Do not tie usage accounting to UI update loop.

Hint 3: Stream Buffer Pseudocode

for each chunk: append_to_buffer; emit_ui_event; track_chunk_count
on_complete: finalize_usage; persist_summary

Hint 4: Debugging Inject artificial delay and cancellation in tests to validate end-of-run accounting.

Books That Will Help

Topic Book Chapter
Runtime telemetry “Site Reliability Engineering” by Beyer et al. Ch. 4
Event streams “Designing Data-Intensive Applications” by Kleppmann Ch. 11

Common Pitfalls and Debugging

Problem 1: “UI shows tokens but no final usage summary”

  • Why: Completion event path is not guaranteed for early exits.
  • Fix: Add finalize hook on all terminal states.
  • Quick test: cancel at chunk 3 and verify usage row exists.

Problem 2: “Stream stalls under high concurrency”

  • Why: Shared worker pool is saturated.
  • Fix: Add per-stream timeout and bounded concurrency.
  • Quick test: run 50 concurrent streams and observe queue depth/latency.

Definition of Done

  • Live stream renders incremental output for at least one provider.
  • Terminal state always emits usage, cost, and duration summary.
  • Cancellation and timeout are observable and test-covered.
  • Dashboard view can filter runs by run_id and status.

Project 4: Provider Failover and Cost Router

  • File: P04-provider-failover-and-cost-router.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Kotlin, Rust
  • Coolness Level: 4/5
  • Business Potential: 5/5
  • Difficulty: 4
  • Knowledge Area: Reliability, cost governance, routing
  • Software or Tool: req_llm, jido
  • Main Book: Designing Data-Intensive Applications

What you will build: A policy engine that switches providers by latency, budget, failure history, and capability.

Why it teaches JIDO: It combines req_llm routing with jido policy-driven execution.

Core challenges you will face:

  • Routing policy ambiguity -> choose deterministic tie-breakers.
  • Budget enforcement -> protect operations from runaway tokens.
  • Error class taxonomy -> differentiate retryable and terminal faults.

Real World Outcome

Given a prompt and policy profile, the router chooses a provider, executes request, and returns a scored decision record that includes fallback rationale, SLO status, and budget impact.

$ mix run -e "CostRouter.demo()"
[router] request=rt_88 task=extract_contract priority=high
[policy] max_cost_usd=0.01 p95_latency_ms=1800
[score] openai:gpt-4o-mini=0.82 anthropic:claude-haiku-4-5=0.79 google:gemini-2.5-flash=0.75
[decision] selected=openai:gpt-4o-mini reason=best_score
[execution] status=timeout fallback=anthropic:claude-haiku-4-5
[final] status=ok total_cost_usd=0.0063 fallback_used=true

The Core Question You Are Answering

“How do we turn provider choice from ad-hoc preference into explicit, testable policy?”

This is where platform reliability and unit economics meet.

Concepts You Must Understand First

  1. Policy Scoring Models
    • How do latency, cost, and reliability become one routing score?
    • Book Reference: “Designing Data-Intensive Applications” by Kleppmann - Ch. 3
  2. Failure Taxonomy
    • Which errors are retryable, fallback-only, or terminal?
    • Book Reference: “Release It!” by Michael Nygard - Ch. 5
  3. Budget Enforcement
    • How do you fail closed when expected cost exceeds budget?
    • Book Reference: “Site Reliability Engineering” by Beyer et al. - Ch. 6

Questions to Guide Your Design

  1. Score Computation
    • What weights are static and what weights are adaptive?
    • How do you prevent score oscillation?
  2. Fallback Semantics
    • Do you retry primary first or switch immediately?
    • How many fallback levels are allowed per request?

Thinking Exercise

Scoreboard Stress Test

Design a scenario where the cheapest provider has poor reliability. Show how your scoring still picks a more expensive provider under incident conditions.

The Interview Questions They Will Ask

  1. “How do you calibrate routing weights without overfitting?”
  2. “What data do you need to justify a fallback in production?”
  3. “How do you avoid silent cost regressions after model updates?”
  4. “How would you implement canary routing for a new provider?”
  5. “What is your plan when all providers violate SLO?”

Hints in Layers

Hint 1: Start With Static Weights
Ship simple deterministic scoring before adaptive learning.

Hint 2: Classify Errors First
Routing depends more on error class quality than on scoring formula complexity.

Hint 3: Scoring Pseudocode

score = w_latency*latency_norm + w_cost*cost_norm + w_reliability*reliability_norm
select max(score) that satisfies policy constraints

Hint 4: Debugging Persist score cards for every decision so postmortems can replay route choices.

Books That Will Help

Topic Book Chapter
Policy systems “Designing Data-Intensive Applications” by Kleppmann Ch. 3
Resilience patterns “Release It!” by Michael Nygard Ch. 5

Common Pitfalls and Debugging

Problem 1: “Router flips providers too often”

  • Why: Small latency jitter causes unstable scores.
  • Fix: Add hysteresis and minimum hold time.
  • Quick test: replay 1,000 decisions and measure provider switch frequency.

Problem 2: “Budget cap not enforced in fallback path”

  • Why: Cost guard only checks primary route.
  • Fix: Re-evaluate budget after each fallback stage.
  • Quick test: trigger fallback and assert cumulative cost never exceeds cap.

Definition of Done

  • Router emits deterministic score cards and final decisions.
  • At least three failure classes are handled with explicit policy.
  • Budget guardrail blocks out-of-policy execution.
  • Replay test reproduces routing outcome from stored telemetry.

Project 5: Deterministic Counter Agent

  • File: P05-deterministic-counter-agent.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Elixir scripts, Lua
  • Coolness Level: 3/5
  • Business Potential: 3/5
  • Difficulty: 2
  • Knowledge Area: cmd/2, StateOp, directives
  • Software or Tool: jido, jido_action
  • Main Book: Programming Elixir

What you will build: A pure action-based agent whose behavior is tested via command outputs and generated directives.

Why it teaches JIDO: It teaches the core cmd/2 mental model and why purity enables repeatability.

Core challenges you will face:

  • Pure state design -> no side effects in action logic.
  • Directive design -> clear effect contracts.
  • Test determinism -> stable assertions under repeated commands.

Real World Outcome

You run a deterministic harness: same initial state and same command sequence always produce same final state and same directive list. This gives you a concrete baseline for larger agent systems.

$ mix run -e "CounterAgent.demo()"
[state] initial=%{count: 0}
[cmd] increment amount=2
[result] state=%{count: 2} directives=[emit:counter.updated]
[cmd] decrement amount=1
[result] state=%{count: 1} directives=[emit:counter.updated]
[assert] deterministic_replay=true

The Core Question You Are Answering

“Can we prove business behavior with pure state transitions before we add side effects?”

If this is not true in a toy agent, it will definitely fail in a multi-agent production system.

Concepts You Must Understand First

  1. Pure Command Functions
    • What is the exact input/output contract of cmd/2?
    • Book Reference: “Programming Elixir” by Dave Thomas - Ch. 8
  2. Directive as Data
    • Why should side effects be returned, not executed inline?
    • Book Reference: “Designing Elixir Systems with OTP” - Ch. 5
  3. Idempotence and Replay
    • Which commands are safe to replay?
    • Book Reference: “Release It!” by Michael Nygard - Ch. 4

Questions to Guide Your Design

  1. State Invariants
    • What transitions are allowed from each state?
    • Which commands should be rejected?
  2. Directive Contract
    • What metadata must each directive include?
    • How do you test directive order?

Thinking Exercise

Replay Audit

Write a command log of 10 operations. Replay it twice from same snapshot. If final states differ, identify where hidden nondeterminism leaked in.

The Interview Questions They Will Ask

  1. “What does cmd/2 buy you over direct GenServer state mutation?”
  2. “How do directives improve reliability and testability?”
  3. “How do you detect nondeterminism in command handlers?”
  4. “When is it acceptable to execute side effects inline?”
  5. “How would you evolve this agent to support approvals?”

Hints in Layers

Hint 1: Start With One Invariant
Example invariant: count can never go below zero.

Hint 2: Keep Command Scope Small
One command should change one concern.

Hint 3: Transition Pseudocode

validate(params) -> compute(next_state) -> return {next_state, directives}

Hint 4: Debugging Snapshot pre/post state for every command in tests.

Books That Will Help

Topic Book Chapter
Elixir state modeling “Programming Elixir” by Dave Thomas Ch. 8
OTP architecture “Designing Elixir Systems with OTP” Ch. 5

Common Pitfalls and Debugging

Problem 1: “Tests pass once but fail on replay”

  • Why: Command reads wall-clock time directly.
  • Fix: Inject time source as data/context.
  • Quick test: replay same log with fixed clock and compare outputs.

Problem 2: “Directive order changes under refactor”

  • Why: Directives are appended from unordered map traversal.
  • Fix: Build directives from explicit ordered list.
  • Quick test: assert exact directive sequence in unit tests.

Definition of Done

  • Command handlers are pure and deterministic.
  • At least one invariant is enforced and test-covered.
  • Directives are explicit, ordered, and validated.
  • Replay of command logs is deterministic.

Project 6: Workflow-First Orchestrator

  • File: P06-workflow-first-orchestrator.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Go, Java
  • Coolness Level: 4/5
  • Business Potential: 5/5
  • Difficulty: 4
  • Knowledge Area: workflows, instruction sets, agent cooperation
  • Software or Tool: jido, req_llm, req_llm tools
  • Main Book: Monolith to Microservices

What you will build: A multi-step workflow engine using instructions and timeout-aware branching.

Why it teaches JIDO: It demonstrates how real workflows differ from linear scripts.

Core challenges you will face:

  • Branching logic -> reproducible retries.
  • Timeouts and fallback -> maintain checkpoint integrity.
  • Composability -> reuse actions with clean schemas.

Real World Outcome

A single run executes classify -> extract -> verify -> route with checkpoints after each step. If one step times out, recovery resumes from the last completed checkpoint.

$ mix run -e "WorkflowOrchestrator.demo()"
[workflow] run_id=wf_900 started steps=4
[step:classify] status=ok duration_ms=240
[step:extract] status=timeout retry=1
[step:extract] status=ok duration_ms=710
[step:verify] status=ok confidence=0.92
[step:route] status=ok destination=finance_queue
[workflow] completed checkpoints=4 resumed_from=extract

The Core Question You Are Answering

“How do we build multi-step agent workflows that can fail and resume without corrupting state?”

This is the operational difference between demos and production systems.

Concepts You Must Understand First

  1. Checkpointing
    • What state do you persist per step?
    • Book Reference: “Monolith to Microservices” by Sam Newman - Ch. 7
  2. Branching and Compensation
    • Which branches need rollback/compensation semantics?
    • Book Reference: “Release It!” by Michael Nygard - Ch. 9
  3. Workflow Invariants
    • What conditions must remain true regardless of retries?
    • Book Reference: “Designing Data-Intensive Applications” by Kleppmann - Ch. 11

Questions to Guide Your Design

  1. Execution Model
    • Which steps can run in parallel vs strictly sequential?
    • How do you enforce max retry budgets?
  2. Recovery Model
    • How do you detect stale checkpoints?
    • How do you prevent duplicate side effects on resume?

Thinking Exercise

Failure Branch Walkthrough

Choose one step and simulate three outcomes: success, timeout, terminal validation error. For each, define next state and whether workflow continues, retries, or stops.

The Interview Questions They Will Ask

  1. “How do you model workflow checkpoints to allow safe resume?”
  2. “What is your compensation strategy for partial side effects?”
  3. “How do you prove retries are idempotent?”
  4. “How do you test branching explosion without brittle tests?”
  5. “How do you monitor workflow health at scale?”

Hints in Layers

Hint 1: Make Step Results Explicit
Use a normalized step result structure: status, payload, error_class, metrics.

Hint 2: Persist Checkpoints Early
Persist after each successful step before moving forward.

Hint 3: Branch Pseudocode

case step_result.status of
  :ok -> next_step
  :retryable_error -> retry_with_backoff
  :terminal_error -> halt_and_emit

Hint 4: Debugging Build a replay mode that reads checkpoint logs and prints branch decisions.

Books That Will Help

Topic Book Chapter
Workflow architecture “Monolith to Microservices” by Sam Newman Ch. 7
Failure handling “Release It!” by Michael Nygard Ch. 9

Common Pitfalls and Debugging

Problem 1: “Workflow resumes but duplicates downstream actions”

  • Why: Resume logic does not mark side effects as committed.
  • Fix: Add idempotency keys and committed-step ledger.
  • Quick test: simulate crash after step 3 and verify no duplicate side effects.

Problem 2: “Branch conditions become unmaintainable”

  • Why: Business rules encoded directly in nested conditionals.
  • Fix: Externalize branch policy into decision tables.
  • Quick test: table-driven tests cover all policy branches.

Definition of Done

  • Workflow executes all steps with checkpoint persistence.
  • Retryable and terminal failures are handled distinctly.
  • Resume from checkpoint is deterministic and test-covered.
  • Branch decisions are traceable through logs/metrics.

Project 7: Signal Mesh for Multi-Agent Delegation

  • File: P07-signal-mesh-for-multi-agent-delegation.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Rust, Python
  • Coolness Level: 5/5
  • Business Potential: 5/5
  • Difficulty: 4
  • Knowledge Area: signals, routing, pub/sub, delegation
  • Software or Tool: jido, jido_signal
  • Main Book: Release It!

What you will build: A delegated agent mesh with typed signals, route policies, and parent-child lifecycle tracking.

Why it teaches JIDO: It shows how to scale from one agent to many without shared hidden coupling.

Core challenges you will face:

  • Routing correctness -> deterministic destination logic.
  • Event storms -> guardrails and dead-letter handling.
  • Policy drift -> consistent behavior across many agents.

Real World Outcome

One coordinator agent receives a request, dispatches typed signals to three specialist agents, and aggregates results. You can trace each signal by ID from dispatch to handler completion.

$ mix run -e "SignalMesh.demo()"
[signal] id=sig_301 type=task.created source=/coordinator
[route] sig_301 -> /agent/classifier status=delivered
[route] sig_301 -> /agent/extractor status=delivered
[route] sig_301 -> /agent/verifier status=delivered
[aggregate] request_id=req_44 completed=3 failed=0
[mesh] dead_letter_count=0 avg_dispatch_ms=18

The Core Question You Are Answering

“How do we coordinate many agents through messages without introducing hidden point-to-point coupling?”

The quality of your signal design determines whether the system scales cleanly or collapses into routing chaos.

Concepts You Must Understand First

  1. Signal Envelopes and Metadata
    • Which envelope fields are mandatory for tracing and routing?
    • Book Reference: “Building Microservices” by Sam Newman - Ch. 8
  2. Routing Patterns
    • When should you use broadcast, direct route, or topic pattern?
    • Book Reference: “Release It!” by Michael Nygard - Ch. 10
  3. Backpressure and Dead Letters
    • What happens when a subscriber is slow or unavailable?
    • Book Reference: “Site Reliability Engineering” by Beyer et al. - Ch. 21

Questions to Guide Your Design

  1. Signal Contracts
    • How do you version signal types safely?
    • What schema validation runs before dispatch?
  2. Mesh Reliability
    • What queue limits trigger throttling?
    • How are poison signals quarantined?

Thinking Exercise

Signal Topology Map

Draw your mesh with at least five signal types. For each edge, specify delivery mode (sync, async) and failure policy.

The Interview Questions They Will Ask

  1. “Why choose typed signals over direct process calls?”
  2. “How do you prevent signal storms during incidents?”
  3. “What observability fields are mandatory in every signal?”
  4. “How do you evolve signal schemas across teams?”
  5. “How do you test routing correctness under fan-out?”

Hints in Layers

Hint 1: Normalize Envelope First
Keep type, source, id, correlation_id, and dispatch metadata mandatory.

Hint 2: Route by Pattern + Policy
Use explicit routing tables with fallback handlers.

Hint 3: Dispatch Pseudocode

validate_signal -> resolve_subscribers -> dispatch_with_mode -> collect_delivery_results

Hint 4: Debugging Record per-signal delivery attempts and dead-letter reasons.

Books That Will Help

Topic Book Chapter
Message routing “Building Microservices” by Sam Newman Ch. 8
Failure isolation “Release It!” by Michael Nygard Ch. 10

Common Pitfalls and Debugging

Problem 1: “Signals arrive but handlers ignore payload fields”

  • Why: Producer and consumer signal schemas diverged.
  • Fix: Add shared schema version and compatibility tests.
  • Quick test: contract-test producer fixtures against consumer validator.

Problem 2: “Dispatch latency spikes under fan-out”

  • Why: Unbounded async dispatch saturates executors.
  • Fix: Introduce bounded pools and priority queues.
  • Quick test: load-test 1,000 signals/minute and track p95 dispatch latency.

Definition of Done

  • Mesh routes typed signals to multiple agent roles deterministically.
  • Delivery outcomes are traceable with correlation IDs.
  • Dead-letter path exists and is test-covered.
  • Backpressure guardrails prevent queue runaway.

Project 8: Audit-Ready State Machine Agent

  • File: P08-audit-ready-state-machine-agent.md
  • Main Programming Language: Elixir
  • Alternative Programming Languages: Clojure, Scala
  • Coolness Level: 5/5
  • Business Potential: 4/5
  • Difficulty: 5
  • Knowledge Area: FSM strategy, supervision, auditing, enterprise controls
  • Software or Tool: jido, telemetry, req_llm
  • Main Book: Software Engineering at Google (casebook reading)

What you will build: A production-like multi-stage state machine agent with approval gates, audit trails, and failure recovery.

Why it teaches JIDO: It integrates all concepts into a realistic operational system.

Core challenges you will face:

  • Compliance visibility -> every side effect must be accounted for.
  • Concurrent updates -> safe transitions under load.
  • Incident replayability -> deterministic trace and state snapshots.

Real World Outcome

You execute regulated workflow runs (draft -> review -> approved -> executed) where each transition is logged with actor, reason, and timestamp. Failed transitions generate immutable audit events and operator alerts.

$ mix run -e "AuditFsm.demo()"
[fsm] case_id=case_901 state=draft -> review actor=agent_router
[fsm] case_id=case_901 state=review -> approved actor=human_approver_42
[fsm] case_id=case_901 state=approved -> executed actor=executor_worker
[audit] events_written=3 immutable_log=true
[replay] case_id=case_901 deterministic=true

The Core Question You Are Answering

“How do we make autonomous agent behavior auditable enough for high-stakes production environments?”

This project closes the gap between technical correctness and governance requirements.

Concepts You Must Understand First

  1. Finite State Machine Design
    • Which transitions are legal and who can trigger them?
    • Book Reference: “Software Engineering at Google” - Ch. 16
  2. Audit Logging Semantics
    • What makes an audit event immutable and replayable?
    • Book Reference: “Designing Data-Intensive Applications” by Kleppmann - Ch. 11
  3. Supervisor Strategy and Recovery
    • How do crashes affect state consistency guarantees?
    • Book Reference: “Designing Elixir Systems with OTP” - Ch. 8

Questions to Guide Your Design

  1. Transition Governance
    • How are manual approvals represented and validated?
    • How do you block unauthorized transitions?
  2. Replay and Forensics
    • What minimum fields are required to replay a case?
    • How do you prove event ordering is correct?

Thinking Exercise

Compliance Walkthrough

Take one failed transition scenario and write the exact evidence an auditor needs: prior state, attempted transition, actor identity, policy rule, and remediation path.

The Interview Questions They Will Ask

  1. “How do you encode governance rules in an FSM without hardcoding policy everywhere?”
  2. “What makes an audit trail legally/operationally useful?”
  3. “How do you recover from mid-transition crashes?”
  4. “How do you guarantee replay fidelity under schema evolution?”
  5. “How does supervision strategy impact compliance outcomes?”

Hints in Layers

Hint 1: Define State Graph Before Coding
Explicitly list allowed transitions and forbidden transitions.

Hint 2: Treat Audit Event as Product Artifact
Design event schema with forensic and operational use in mind.

Hint 3: Transition Pseudocode

if transition_allowed?(from,to,actor,policy) then write_audit_event and commit_state
else reject_and_write_violation_event

Hint 4: Debugging Add replay command that rebuilds state from audit events only.

Books That Will Help

Topic Book Chapter
Large-scale governance “Software Engineering at Google” Ch. 16
Event replay “Designing Data-Intensive Applications” by Kleppmann Ch. 11

Common Pitfalls and Debugging

Problem 1: “State changes happen but audit trail is incomplete”

  • Why: Logging is best-effort and not in transition transaction boundary.
  • Fix: Make state commit contingent on durable audit write.
  • Quick test: force log storage failure and assert transition aborts.

Problem 2: “Replay reconstructs wrong final state”

  • Why: Events are missing ordering or version metadata.
  • Fix: Add monotonic sequence IDs and schema version field.
  • Quick test: replay historical events and compare expected terminal state.

Definition of Done

  • FSM enforces legal transitions and rejects invalid transitions with reason.
  • Every accepted and rejected transition emits immutable audit events.
  • Replay command reconstructs final state deterministically.
  • Approval and policy checks are explicit and test-covered.

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor
1. Multi-Model Gateway with req_llm Medium 1.5-2 weeks High ★★★★☆
2. Schema-First Contract Extractor Medium-High 2 weeks High ★★★★★
3. Streaming Copilot Operations Center Medium 1.5-2 weeks High ★★★★☆
4. Provider Failover and Cost Router High 2-3 weeks Very High ★★★★☆
5. Deterministic Counter Agent Low-Medium 1.5 weeks Medium ★★★★☆
6. Workflow-First Orchestrator High 2-3 weeks Very High ★★★★★
7. Signal Mesh for Multi-Agent Delegation High 2-3 weeks Very High ★★★★★
8. Audit-Ready State Machine Agent Very High 3-4 weeks Very High ★★★☆☆

Recommendation

If you are new to agentic systems: start with Project 5, then Project 1, then Project 2. This avoids heavy orchestration before you own invariants.

If you are a platform engineer: start with Projects 4, 7, and 8 to build routing and reliability patterns.

If you want production outcomes quickly: start with Projects 3, 6, and 8.

If you want to build robust LLM products: follow Projects 1, 2, 3, 4, then 6.

Final Overall Project: JIDO Incident Response Product

The Goal: Build a single integrated system combining all 8 patterns:

  1. Provider abstraction and contract parsing.
  2. Streaming ops UI.
  3. Cost routing and fallback.
  4. Directive-led deterministic core agent.
  5. Workflow orchestration with signal routing.
  6. Audit-grade FSM for human approvals.

Success Criteria: the final output must process 100+ sample tickets, produce reproducible logs, enforce cost caps, and resume correctly after simulated worker restarts.

From Learning to Production: What Is Next

Your Project Production Equivalent Gap to Fill
Projects 1-4 Provider gateway service Multi-region failover policies
Projects 5-8 Agent coordination platform Centralized policy service, security hardening
Full sprint Internal automation platform SSO, multi-tenant quotas, release governance

Suggested production upgrades:

  • Add distributed tracing (OpenTelemetry).
  • Add queue-based backlog + DLQ.
  • Introduce per-team permission model and audit export.
  • Add chaos testing for provider and process faults.

Summary

This learning path covers the JIDO ecosystem end-to-end through practical projects. You practiced provider abstractions, typed contracts, directive-driven agents, workflow orchestration, signal routing, observability, and production reliability under BEAM.

# Project Name Main Language Difficulty Time Estimate
1 Multi-Model Gateway with req_llm Elixir 2 1.5-2 weeks
2 Schema-First Contract Extractor Elixir 3 2 weeks
3 Streaming Copilot Operations Center Elixir 3 1.5-2 weeks
4 Provider Failover and Cost Router Elixir 4 2-3 weeks
5 Deterministic Counter Agent Elixir 2 1.5 weeks
6 Workflow-First Orchestrator Elixir 4 2-3 weeks
7 Signal Mesh for Multi-Agent Delegation Elixir 4 2-3 weeks
8 Audit-Ready State Machine Agent Elixir 5 3-4 weeks

Expected Outcomes

  • You can design deterministic, testable agent commands.
  • You can route LLM usage safely across providers.
  • You can produce operational telemetry and replay traces.

Additional Resources and References

Standards and Specifications

Package and Repository Signals

Industry Analysis

  • Google Gemini MCP support announcement (2025): blog.google
  • Anthropic MCP announcement coverage (2024): techcrunch

Books

  • The Little Elixir and OTP Guidebook — Ben Brummelen
  • The Book of Elixir — Dave Thomas
  • Designing Data-Intensive Applications — Martin Kleppmann