Project 8: Demand-Aware Fanout Bridge

Build a controlled bridge from push-based Pub/Sub into demand-driven processing, then prove—under repeatable load—which guarantees exist at each boundary and which guarantees do not.

Quick Reference

Item	Detail
Primary outcome	A benchmarkable bridge that compares raw Pub/Sub, competing GenStage consumers, broadcast GenStage consumers, and a Broadway batch pipeline
Main language	Elixir
Core tools	Phoenix.PubSub, GenStage, Broadway, Telemetry, ExUnit
Difficulty	Level 3 — Advanced
Estimated time	18–26 hours
Knowledge area	Flow control, backpressure, fanout, batching, acknowledgements, ordering, overload policy
Observable proof	Queue depth, event age, in-flight demand, per-consumer throughput, batch latency, acknowledgement counts, and edge-policy decisions
Central constraint	A push source cannot become backpressured merely because the downstream system is demand-aware
Completion artifact	A reproducible comparison report for 50,000 events plus a documented overload decision table

1. Learning Objectives

By completing this project, you will be able to:

1. Explain the difference between broadcast delivery and work distribution without treating them as interchangeable forms of Pub/Sub.
2. Trace demand from a GenStage consumer back to its producer and identify every point where that demand signal stops.
3. Compare GenStage.DemandDispatcher with GenStage.BroadcastDispatcher, especially their behavior when one consumer is slow.
4. Explain what Broadway adds—processor pools, batching, acknowledgement hooks, telemetry, and failure classification—and what it deliberately leaves to the message producer.
5. Design a bounded bridge between a push-based Phoenix.PubSub source and a demand-driven pipeline.
6. Choose an explicit overload policy: coalesce replaceable state, reject work before accepting it, or spill durable work to a durable system.
7. Reason about ordering by key, concurrency, retries, failures, and repartitioning rather than claiming global ordering.
8. Prove behavior with event-age percentiles, queue measurements, demand counters, acknowledgement totals, and controlled slow-consumer experiments.
9. Separate flow control from durability: bounded queues and demand prevent uncontrolled admission, but they do not make accepted work survive a crash.
10. State the operational contract of a push-to-pull bridge precisely enough that another engineer can decide whether it is safe for a given event class.

2. Theoretical Foundation

Push and pull are different admission contracts

Phoenix.PubSub is push-oriented. A publisher broadcasts a message to the processes subscribed to a topic. The broadcaster does not wait for each subscriber to advertise capacity, and a successful broadcast does not mean that every subscriber has processed the message. Delivery places work into process mailboxes; the subscribers still own scheduling, processing, failure handling, and overload behavior.

GenStage is demand-oriented between its connected stages. A consumer asks for a bounded number of events. A producer may dispatch up to that requested amount to that subscription. As the consumer completes work, it asks for more. This creates a feedback loop that controls the amount of work in flight inside the GenStage topology.

The distinction matters because demand is not contagious across arbitrary boundaries. If Pub/Sub pushes 20,000 events into a bridge process while the downstream GenStage consumer has demand for only 200, the remaining events still exist somewhere. Unless the bridge rejects, coalesces, spills, or otherwise bounds admission, its mailbox or internal buffer becomes the uncontrolled queue.

PUSH SIDE                                      DEMAND SIDE

device publishers                              processors
      |                                            ^
      | broadcast: no capacity negotiation        | demand: 160
      v                                            |
+------------------+      admission edge      +------------------+
| Phoenix.PubSub   | ------------------------> | bridge producer  |
| topic subscription|                          | bounded capacity |
+------------------+                           +------------------+
      |                                            |
      | mailbox growth can occur here              | dispatch <= demand
      v                                            v
+------------------+                           +------------------+
| bridge subscriber|                           | GenStage/Broadway|
| push receiver     |                           | processing pool  |
+------------------+                           +------------------+

Demand controls the right-hand topology.
It does not travel backward through Phoenix.PubSub to the original publishers.

Demand, buffers, and in-flight work

A GenStage subscription is governed by demand. Consumers typically request up to a configured maximum, then ask for more after demand falls below a configured minimum. Those thresholds trade throughput for bounded concurrency:

• A high maximum demand can keep processors busy and amortize coordination overhead.
• A low maximum demand reduces in-flight work and often shortens recovery after failures.
• A high minimum replenishes work early but permits more queued work at the consumer.
• A low minimum waits longer before requesting more and may expose idle gaps.

Demand is a count, not a guarantee of latency, memory usage, or durability. One event may cost one microsecond while another costs five seconds or holds a large binary. A capacity plan must therefore pair event counts with event size, processing-time distributions, queue age, and failure behavior.

A useful accounting identity is:

accepted events
    = completed
    + failed and accounted for
    + currently processing
    + buffered at an explicitly measured boundary

If events appear outside those categories, the system has an invisible queue or an unaccounted loss path.

Competing consumers with DemandDispatcher

GenStage.DemandDispatcher distributes events across consumers according to their outstanding demand. This models a shared work queue: one event is assigned to one consumer subscription. Faster consumers tend to request more work over time, so the dispatcher naturally sends more events to available capacity.

                         event stream
                              |
                              v
                   +--------------------+
                   | DemandDispatcher   |
                   | distribute once    |
                   +--------------------+
                     /        |        \
                 demand 40 demand 40 demand 40
                   v          v          v
              +---------+ +---------+ +---------+
              | worker A| | worker B| | worker C|
              +---------+ +---------+ +---------+

event 417 is sent to A, B, or C—not to all three.

This is appropriate when consumers are interchangeable workers performing the same logical task. It is not appropriate when each consumer represents a distinct projection, notification channel, or external system that must independently observe every event.

Even in a competing-consumer topology, “distributed once” is narrower than “processed exactly once.” A process may fail after producing an external side effect but before recording completion. Another system may redeliver the event. End-to-end correctness still depends on acknowledgement semantics, idempotency, and the durability contract of the upstream producer.

Broadcast consumers with BroadcastDispatcher

GenStage.BroadcastDispatcher sends each event to every subscribed consumer. Demand still exists, but the dispatcher can emit only when the subscribed consumers have capacity for the broadcast. Consequently, a slow consumer can constrain the entire broadcast group.

                              event 417
                                  |
                                  v
                     +------------------------+
                     | BroadcastDispatcher    |
                     | all subscribers demand?|
                     +------------------------+
                       /          |          \
                      v           v           v
                +----------+ +----------+ +----------+
                | fast A   | | fast B   | | slow C   |
                | 2 ms     | | 3 ms     | | 250 ms   |
                +----------+ +----------+ +----------+
                                                |
                                                v
                          group throughput is limited here

This coupling is not automatically a defect. It can be the desired policy when every consumer must stay aligned and the producer should slow down rather than let subscribers diverge. It becomes dangerous when an optional analytics or debugging consumer accidentally throttles a critical workflow. Separate stages, topics, or durable streams are often safer when subscribers have different service-level objectives.

Broadway as a processing topology

Broadway organizes message processing into a supervised topology:

1. A producer receives or pulls messages from a source.
2. Processor stages execute per-message work concurrently.
3. Optional batcher stages group messages by a batch key or size/time trigger.
4. Batch processors perform batch-oriented work.
5. An acknowledger reports successful and failed messages to the source-specific acknowledgement mechanism.

durable or custom producer
          |
          v
+-------------------+
| producer stage    |
| emits Broadway    |
| Message structs   |
+-------------------+
          |
          v
+-------------------+       partition/key
| processor pool    | --------------------------+
| validate/enrich   |                            |
+-------------------+                            v
          |                              +-------------------+
          v                              | batcher           |
+-------------------+                    | size or timeout   |
| success / failure |                    +-------------------+
| classification    |                            |
+-------------------+                            v
          |                              +-------------------+
          +----------------------------> | batch processor   |
                                         +-------------------+
                                                  |
                                                  v
                                         +-------------------+
                                         | acknowledger      |
                                         | success / failed  |
                                         +-------------------+

Broadway gives structure and observability, but it does not invent persistence or retry guarantees. Retries are normally the responsibility of the message producer or broker integration. The acknowledger tells that integration which messages succeeded or failed; the integration decides what acknowledgement, rejection, redelivery, or dead-letter behavior follows.

For a custom in-memory producer, a successful acknowledgement can be purely informational. If the node crashes after accepting an event but before acknowledgement, the event may disappear. For a durable broker producer, the same failure can lead to redelivery. The visible Broadway topology may look similar while the end-to-end semantics differ radically.

The push_messages trap

Broadway.push_messages/2 is useful for injecting messages into a running Broadway pipeline, but it is a push API. It does not make the caller wait for downstream demand. A process that subscribes to Phoenix.PubSub and immediately invokes this function for every incoming event has not propagated backpressure to Pub/Sub publishers. It has only moved the uncontrolled admission point.

PubSub burst
    |
    v
subscriber mailbox
    |
    | push_messages/2
    v
Broadway topology
    |
    v
demand-aware processors

Downstream demand exists, but upstream publication continues.
The mailbox or push boundary must still have a bounded policy.

The bridge in this project therefore treats the push-to-demand edge as a first-class component with measurable capacity and an explicit overload decision.

Three legitimate overload policies

There is no universal correct behavior when the edge is full. The event’s meaning determines the safe policy.

Event class	Safe candidate policy	Why
Replaceable current state, such as device temperature	Coalesce by entity key	Only the newest not-yet-processed value may matter
Command requiring immediate admission decision	Reject before acceptance	The caller can retry or surface overload without believing work was accepted
Durable business fact, such as payment captured	Spill to durable broker/log	The fact must survive process and node failure
Best-effort UI hint	Drop with a metric	Loss may be acceptable if authoritative state can be reloaded

Coalescing is not generic dropping. It is a domain-specific compaction rule: multiple pending updates for one key are replaced by the latest state because intermediate values are explicitly declared non-essential.

Rejecting after the publisher was already told “accepted” violates the contract. If reliable admission matters, rejection must occur synchronously at a boundary capable of responding to the caller; ordinary fire-and-forget Pub/Sub broadcast cannot provide that acknowledgement by itself.

Spilling to disk or a broker creates new responsibilities: durable writes, identifiers, replay, poison-message handling, retention, and capacity alarms. It should not be described as a tiny implementation detail.

Ordering is scoped, not global

Concurrent demand-driven pipelines usually improve throughput by relaxing ordering. Ordering claims must specify:

• Which key? All events, one customer, one device, or one source partition?
• Which interval? Normal processing only, or also restart and retry paths?
• Which boundary? Dispatch order, handler-start order, side-effect order, or acknowledgement order?
• Which failure model? No failures, worker crashes, producer restart, or broker redelivery?

Broadway processors run concurrently by default. Partitioning by a stable key can route related messages consistently to the same processor stage and preserve more local sequencing under ordinary operation. It does not create a universal exactly-once, global-order guarantee. Failures, redelivery, asynchronous side effects, and configuration changes can still reorder visible results.

input order:      A1  B1  A2  B2  A3
partition by key: |A| |B| |A| |B| |A|
                  v    v   v    v   v
lane A:           A1 ---- A2 ---- A3
lane B:                B1 ---- B2

Possible completion order across lanes:
                  B1, A1, B2, A2, A3

Per-key sequencing may be preserved in the normal path.
Global completion order is intentionally not preserved.

Acknowledgement and idempotency

An acknowledgement is a protocol message about processing outcome. It is not proof that a side effect occurred exactly once. Consider this failure window:

1. receive event E-417
2. charge external service successfully
3. process crashes before acknowledgement
4. source redelivers E-417
5. charge executes again unless protected

The standard defense is a stable event or operation identifier plus an idempotent effect boundary. The handler records or checks the identifier in the same consistency boundary as the side effect when possible. When that is impossible, the external service should receive an idempotency key and the system must reconcile ambiguous outcomes.

Key insights

• Backpressure is an end-to-end property only when every admission boundary participates.
• Demand bounds in-flight work inside a topology; it does not make messages durable.
• Competing consumers distribute work, while broadcast consumers replicate it.
• Broadcast demand can couple group throughput to the slowest consumer.
• Broadway structures concurrent processing and acknowledgements, but source integrations define persistence and redelivery.
• Every push-to-pull bridge needs an explicit, domain-specific overload policy.

3. Project Specification

The core question you are answering

How can a system cross from push-based fanout into demand-driven processing without hiding an unbounded queue or claiming guarantees that stop at the bridge?

What you will build

Create a comparison harness with four execution modes over the same deterministic 50,000-event workload:

1. Raw Pub/Sub push — subscribers process messages directly from their mailboxes.
2. GenStage competing demand — a producer uses demand to distribute each event to one worker.
3. GenStage broadcast demand — every event goes to all consumers, including one deliberately slow consumer.
4. Broadway batch processing — messages pass through concurrent processors, a keyed batcher, and a recording acknowledger.

Add a bounded push-to-demand bridge that supports a configured policy for each event class:

• coalesce pending device-state updates by device_id;
• reject commands when reliable admission cannot be granted;
• spill durable business events to a durable adapter interface;
• count and report every decision.

Functional requirements

• Generate a deterministic event sequence with stable IDs, entity keys, creation timestamps, event classes, and configurable processing cost.
• Run each mode independently with the same seed and workload profile.
• Record accepted, processed, failed, acknowledged-success, acknowledged-failed, rejected, coalesced, spilled, and unaccounted counts.
• Record maximum queue depth, maximum in-flight demand, throughput, and event-age p50/p95/p99.
• Configure one slow consumer in broadcast mode and identify its effect on group throughput.
• Configure failures in Broadway mode and report them through a custom recording acknowledger.
• Preserve the declared normal-path order for events sharing a partition key, then document what changes during failure and retry.
• Produce a machine-readable result file and a human-readable comparison table.
• Exit non-zero when accounting invariants fail or an unbounded boundary exceeds its safety limit.

Non-functional requirements

• The benchmark must be reproducible from a clean checkout with a fixed random seed.
• Metrics collection must not log one line per event during the measured interval.
• The bridge must have a hard, configurable capacity.
• No policy may silently lose a durable event.
• Every reported percentile must identify the population and timestamp boundaries used.
• The comparison must distinguish throughput from correctness and flow control from durability.

Event envelope

Use a conceptual envelope with these fields:

event_id         stable unique identifier, for example EVT-0000417
event_class      device_state | command | durable_fact
entity_key       device or aggregate identifier
created_mono     monotonic creation timestamp for age measurement
sequence         source-local sequence number
payload_bytes    deterministic payload size
work_cost_ms     simulated handler cost
failure_mode     none | transient | permanent

This is a data-shape specification, not runnable Elixir code.

Required command transcript

Your finished harness must produce output equivalent to:

$ mix flow.compare --events 50000
MODE pubsub_push      queue_max=38112 age_p99_ms=9440 backpressure=false
MODE genstage_demand  in_flight_max=160 age_p99_ms=218 distributed_once=true
MODE broadcast_demand consumers=4 throughput_limited_by=slow_consumer
MODE broadway_batch   processed=50000 ack_success=49992 ack_failed=8 batch_p99=100
EDGE full policy=coalesce_by_device coalesced=6840 durable_events_dropped=0

The exact timings may vary by machine. The field names, accounting relationships, and qualitative conclusions must remain stable.

Acceptance invariants

For every mode, report enough counters to verify:

generated
    = rejected_before_acceptance
    + accepted

accepted
    = processed_success
    + processed_failed
    + coalesced_under_documented_rule
    + spilled_and_confirmed
    + still_in_flight_at_snapshot

At final quiescence, still_in_flight_at_snapshot must be zero. durable_events_dropped must always be zero.

4. Solution Architecture

Big-picture architecture

                         +----------------------+
                         | deterministic source |
                         | 50,000 envelopes     |
                         +----------+-----------+
                                    |
                   +----------------+----------------+
                   |                |                |
                   v                v                v
          +----------------+ +-------------+ +----------------+
          | raw Pub/Sub    | | GenStage    | | Broadway       |
          | push baseline  | | comparison  | | batch pipeline |
          +-------+--------+ +------+------+ +-------+--------+
                  |                 |                |
                  v                 v                v
          +---------------------------------------------------+
          | normalized measurement collector                  |
          | counts | queue | demand | age | throughput | acks |
          +-------------------------+-------------------------+
                                    |
                                    v
                         +----------------------+
                         | comparison reporter  |
                         | text + data artifact |
                         +----------------------+

Separate edge experiment:

Pub/Sub topic -> bounded bridge -> policy router -> demand producer
                                     |   |   |
                                  merge reject durable spill

Components and responsibilities

Component	Responsibility	Must not do
Workload generator	Produce identical seeded envelopes for each mode	Depend on wall-clock randomness
Raw Pub/Sub runner	Broadcast events and sample subscriber mailboxes	Pretend mailbox delivery means completion
Demand producer	Emit only up to outstanding demand	Dispatch beyond demand
Competing workers	Process mutually exclusive assignments	Count one event at multiple workers
Broadcast workers	Process a copy at every subscriber	Hide the slow consumer
Broadway producer	Convert source records to message envelopes	Promise retries without a supporting source
Processors	Validate, classify, and partition work	Perform unbounded synchronous logging
Batcher	Group compatible work with size/time limits	Mix incompatible entity or destination keys
Recording acknowledger	Record success and failure IDs	Claim durability it does not own
Bounded edge	Enforce capacity and invoke event-class policy	Accumulate unbounded state
Measurement collector	Aggregate counters and histograms	Become the benchmark bottleneck
Reporter	Validate invariants and render results	Omit failed or rejected categories

Bounded-edge state machine

                    incoming event
                          |
                          v
                 +------------------+
                 | capacity free?   |
                 +------------------+
                    | yes       | no
                    v           v
               +---------+  +------------------+
               | accept  |  | classify event   |
               +---------+  +------------------+
                                /      |       \
                               v       v        v
                          replaceable command durable
                               |       |        |
                               v       v        v
                         coalesce by  reject   spill and
                         entity key   now      confirm write
                               |       |        |
                               +-------+--------+
                                       |
                                       v
                              increment decision metric

Key algorithms in pseudocode

Demand-safe producer dispatch

ON downstream_demand(amount):
    increase outstanding_demand by amount
    WHILE outstanding_demand > 0 AND bounded_queue is not empty:
        event = remove oldest eligible event
        dispatch event
        decrement outstanding_demand

Keyed coalescing

ON full_edge(device_state event):
    key = event.entity_key
    IF pending_state contains key:
        replace pending_state[key] with newer event
        increment coalesced
    ELSE IF coalescing_slots remain:
        pending_state[key] = event
    ELSE:
        apply the separately documented overflow policy

Acknowledgement accounting

ON acknowledge(successful, failed):
    record every successful event_id once
    record every failed event_id with failure class
    reject duplicate terminal accounting
    assert no event_id appears in both terminal sets

Architectural decisions to record

Write a short decision record for each item:

1. Why a given flow uses work distribution or broadcast.
2. Where the hard capacity limit lives.
3. Which event classes may be coalesced and why intermediate values are dispensable.
4. How commands receive synchronous admission rejection.
5. Which durable adapter owns persistence and redelivery.
6. What ordering scope is promised.
7. Which component owns retries and dead-letter handling.
8. What metrics prove that the edge is healthy.

5. Implementation Guide

Suggested project structure

Use this as an organizational outline; adapt names to your application while keeping responsibilities separated.

lib/
  flow_compare/
    workload_generator
    event_envelope
    measurement_collector
    invariant_checker
    report_renderer
    pubsub_runner
    demand_runner
    broadcast_runner
    broadway_runner
    bounded_edge
    policy_router
    durable_adapter_behaviour
    recording_acknowledger
test/
  flow_compare/
    workload_generator_test
    accounting_test
    demand_test
    broadcast_test
    broadway_test
    bounded_edge_test
    recovery_test

These are conceptual module names, not a code listing.

Phase 1: Establish a trustworthy workload

1. Define the event envelope and validate all required fields.
2. Generate a fixed sequence from a seed.
3. Mix event classes and entity keys deliberately.
4. Add configurable processing-cost bands: fast, medium, slow.
5. Add exactly eight deterministic failure events for the standard 50,000-event run.
6. Store expected counts by event class before starting a runner.

Checkpoint

$ mix flow.fixture --events 50000 --seed 417
events=50000 devices=1000 commands=5000 durable_facts=5000
failure_none=49992 failure_injected=8 checksum=8f3c2d71

Running the command twice must produce the same checksum and category counts.

Phase 2: Build the raw Pub/Sub baseline

1. Start a dedicated Pub/Sub instance for the experiment.
2. Start configurable subscribers with deterministic processing delays.
3. Broadcast the workload without waiting for subscriber capacity.
4. Sample each subscriber’s mailbox length at a fixed interval.
5. Measure event age when processing begins and ends.
6. Wait for quiescence and reconcile generated, delivered, and processed IDs.

Checkpoint

The experiment must visibly show a growing mailbox when publication rate exceeds processing rate. The report must label backpressure=false.

Phase 3: Build competing demand

1. Create a producer that owns a bounded input collection.
2. Subscribe multiple identical consumers with explicit min/max demand.
3. Use DemandDispatcher semantics so one event goes to one consumer.
4. Track outstanding demand and maximum in-flight events.
5. Vary worker speed and show that available workers receive more work over time.
6. Verify that every accepted event ID is assigned exactly once in the failure-free run.

Checkpoint

mode=genstage_demand generated=50000 processed=50000 duplicates=0 missing=0
workers=4 in_flight_max=160 demand_violations=0

Phase 4: Expose broadcast coupling

1. Replace the dispatcher with broadcast semantics.
2. Use three fast consumers and one slow consumer.
3. Give every consumer the same initial demand configuration.
4. Record per-consumer completion rates and group throughput.
5. Stop the slow consumer in a controlled experiment and document the changed subscription behavior.
6. Decide whether the optional consumer belongs in the same broadcast group.

Checkpoint

The report must identify the slow consumer by measurement, not by configuration knowledge alone.

Phase 5: Build the Broadway batch pipeline

1. Create message envelopes with stable IDs and event timestamps.
2. Configure processor concurrency intentionally.
3. Partition by entity key for the event class that needs per-key normal-path order.
4. Add a batcher with both size and timeout triggers.
5. Mark the eight deterministic failure events as failed during processing.
6. Record success and failure lists through the acknowledger.
7. Attach aggregate telemetry for processing and batching duration.

Checkpoint

mode=broadway_batch processed=50000
ack_success=49992 ack_failed=8 duplicate_terminal_ids=0
batches_by_size=492 batches_by_timeout=11 batch_p99_ms=100

Document that these acknowledgements are observations unless the chosen producer maps them to a durable source.

Phase 6: Implement the bounded push-to-demand edge

1. Subscribe the edge receiver to a dedicated Pub/Sub topic.
2. Set a hard capacity for immediately admissible events.
3. Add a separate bounded keyed area for coalescible device state.
4. Route commands through a request/reply admission API rather than relying on fire-and-forget broadcast.
5. Define a durable adapter contract whose successful response means the spill is durably accepted.
6. Emit one aggregate metric for each admission decision.
7. Drain only when downstream demand is available.
8. Prove that all internal collections remain within their configured bounds.

Checkpoint

edge_capacity=1000 edge_max_observed=1000
coalesce_slots=1000 coalesce_max_observed=997 coalesced=6840
commands_rejected=127 durable_spilled=412 durable_spill_failed=0
durable_events_dropped=0

Phase 7: Failure and recovery drills

Run these drills separately:

• terminate a demand consumer while it is processing;
• restart the in-memory producer;
• fail the durable adapter temporarily;
• delay the recording acknowledger;
• make one partition disproportionately hot;
• disconnect the slow broadcast consumer;
• restart the Broadway supervision tree.

For each drill, record:

failure injected -> observable symptom -> ownership decision -> recovery action
-> event accounting after recovery -> guarantee that did or did not hold

Phase 8: Produce the comparison report

The final report must include:

• topology and admission-boundary diagram for each mode;
• configuration values;
• workload seed and checksum;
• throughput and age percentiles;
• maximum queue and in-flight counts;
• slow-consumer evidence;
• acknowledgement reconciliation;
• overload policy counts;
• ordering scope;
• durability and retry owner;
• limitations of the benchmark.

Questions to guide your design

1. Where can events accumulate before demand is observed?
2. Can the original publisher know that the bridge is full?
3. Which event class is safe to compact, and what business rule proves that safety?
4. Does every broadcast subscriber have the same service-level objective?
5. Is the desired property distribution to one worker or replication to all consumers?
6. Who owns an event after the bridge says it was accepted?
7. What happens if processing succeeds but acknowledgement fails?
8. Does the ordering requirement apply across keys or only within one key?
9. Can one hot key starve unrelated keys?
10. Which metric detects overload sooner: throughput, queue length, or event age?

Thinking exercise: trace one event through every mode

Take EVT-0000417 for device-042. Draw its path through all four modes and answer:

• How many process mailboxes can contain a copy?
• Which component decides when it may advance?
• What happens if its handler takes 500 milliseconds?
• Who records success or failure?
• What survives a node crash?
• Can the same event be processed again?
• What ordering relationship exists with EVT-0000418 for the same device?

The interview questions they will ask

1. “Why does putting Broadway after Phoenix.PubSub not automatically backpressure publishers?”
2. “When would you use DemandDispatcher instead of BroadcastDispatcher?”
3. “How can one slow consumer affect a demand-driven broadcast topology?”
4. “What does a Broadway acknowledger guarantee?”
5. “How would you preserve ordering per customer while retaining concurrency across customers?”
6. “What policy would you use when a bounded bridge is full?”

Hints in layers

Hint 1: Start with accounting

Before optimizing throughput, make every event land in exactly one observable accounting category.

Hint 2: Draw the feedback path

Use arrows to show where demand travels. Stop the arrow at the first push-only API.

Hint 3: Measure age, not only queue length

A queue of 1,000 two-millisecond events differs from a queue of 1,000 five-second events. Event age exposes user-visible staleness.

Hint 4: Separate event classes

Do not use one overflow rule for replaceable state, commands, and durable facts.

Books that will help

Topic	Book	Suggested focus
Messaging patterns	Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf	Message channels, competing consumers, publish-subscribe channels, idempotent receivers
Reliability boundaries	Release It!, Second Edition by Michael T. Nygard	Stability patterns, capacity, timeouts, bulkheads, operational visibility
Streams and delivery semantics	Designing Data-Intensive Applications, Second Edition by Martin Kleppmann and Chris Riccomini	Replication, stream processing, delivery semantics, partitioning
Phoenix messaging	Real-Time Phoenix by Stephen Bussey	PubSub, distributed real-time systems, operational behavior
Live systems on the BEAM	Programming Phoenix LiveView by Bruce A. Tate and Sophie DeBenedetto	Event-driven UI boundaries and process-oriented design

6. Testing Strategy

Test layers

Layer	What to prove	Example failure caught
Unit	Event classification, coalescing choice, accounting equations	A durable fact enters the drop path
Component	Demand never dispatches above the requested count	Producer oversends during replenishment
Integration	Dispatcher and worker topology has intended delivery shape	Broadcast used where competing delivery was intended
Load	Queue, age, and in-flight bounds under a 50,000-event burst	Hidden mailbox grows without limit
Failure	Crashes and acknowledgements produce documented outcomes	Event disappears between effect and ack
Recovery	Restart behavior matches durability claim	In-memory accepted work is incorrectly described as durable
Property	Generated IDs reconcile into terminal categories	Duplicate or missing terminal accounting

Required deterministic test data

Use at least these fixtures:

Fixture	Count	Purpose
Normal device-state events	40,000	Sustained keyed coalescing and partition load
Commands	5,000	Synchronous acceptance/rejection semantics
Durable facts	5,000	Spill and zero-loss invariant
Injected transient failures	5	Failed acknowledgement and retry-owner analysis
Injected permanent failures	3	Dead-letter or terminal failure analysis
Entity keys	1,000	Parallelism and per-key ordering
Deliberately hot key	1	Partition skew experiment
Slow broadcast subscriber	1 of 4	Slowest-consumer coupling

The eight failures are included within the 50,000 total events, not added afterward.

Critical tests

1. Demand bound — dispatch never exceeds cumulative demand for a subscription.
2. Competing delivery — each event reaches one worker in a failure-free run.
3. Broadcast delivery — each event reaches every active broadcast subscriber.
4. Slow-consumer coupling — group throughput falls when the slow subscriber remains active.
5. Push-boundary overload — Pub/Sub publication outpaces the bridge without corrupting accounting.
6. Coalescing safety — only replaceable state with the same key is superseded.
7. Command rejection — rejected commands are never counted as accepted.
8. Durable spill — a durable fact is accepted only after the adapter confirms durable admission.
9. Acknowledgement partition — success and failed ID sets are disjoint and complete.
10. Per-key normal order — same-key processing follows declared order in the no-failure case.
11. Failure-order caveat — the report demonstrates how retry or crash can change completion order.
12. Restart semantics — in-memory queued work may be lost and the report labels that result correctly.

Load profiles

Run all four profiles:

STEADY     input rate < processing capacity for 120 seconds
BURST      50,000 events as fast as the source can publish
SUSTAINED  input rate 20% above processing capacity for 180 seconds
RECOVERY   source stops; measure time until queue age and depth return to baseline

Measurements and assertions

Collect:

• publication rate and completion rate;
• mailbox length where available;
• outstanding demand and in-flight count;
• accepted-edge and coalescing-slot occupancy;
• event age at handler start and completion;
• per-worker and per-subscriber counts;
• batch size and wait-time distributions;
• acknowledgement success and failure totals;
• scheduler and memory context sufficient to interpret the run.

Avoid asserting exact millisecond values across machines. Assert bounds, accounting identities, and qualitative relationships such as “broadcast throughput decreases when the slow subscriber participates.”

Example final verification transcript

$ mix test test/flow_compare
78 tests, 0 failures

$ mix flow.compare --events 50000 --seed 417
fixture checksum=8f3c2d71 generated=50000
accounting missing=0 duplicate_terminal=0 durable_events_dropped=0
comparison written=tmp/flow_compare/seed-417.json
RESULT pass

7. Common Pitfalls & Debugging

Problem 1: “Broadway is downstream, so Pub/Sub is backpressured”

• Symptom: The Broadway processor queue looks bounded, but the subscriber mailbox grows rapidly.
• Why: Demand stops at the push boundary; publication continues independently.
• Fix: Add a bounded edge with a class-specific coalesce, reject, or durable-spill policy.
• Quick test: Pause downstream demand and confirm that every accepted event is still bounded and accounted for.

Problem 2: Using push_messages/2 as a flow-control mechanism

• Symptom: The bridge injects messages faster than the topology can process them.
• Why: The API pushes messages and does not wait for downstream demand.
• Fix: Treat it as an injection mechanism only, or place a real demand-aware producer behind a bounded admission edge.
• Quick test: Set processor work cost to 500 milliseconds and publish a burst while sampling the bridge mailbox.

Problem 3: Broadcasting work that should be distributed

• Symptom: The same side effect runs once per consumer.
• Why: Broadcast semantics replicate every event.
• Fix: Use competing-consumer semantics for interchangeable workers and retain idempotency at the effect boundary.
• Quick test: Compare unique event IDs with total handler invocations.

Problem 4: An optional subscriber throttles critical consumers

• Symptom: All broadcast consumers slow when an analytics subscriber falls behind.
• Why: Broadcast dispatch waits for participating subscribers’ demand.
• Fix: Move consumers with different service-level objectives to separate flows or a durable stream.
• Quick test: Remove the slow consumer and compare group throughput.

Problem 5: Treating acknowledgement as exactly-once execution

• Symptom: An external side effect occurs twice after a crash.
• Why: The process failed between the side effect and acknowledgement.
• Fix: Use stable identifiers and an idempotent effect boundary; document the producer’s redelivery contract.
• Quick test: Crash immediately after the external-effect stub records success but before acknowledgement.

Problem 6: Assuming Broadway retries failed messages

• Symptom: Failed messages never return, or they return differently across producers.
• Why: Retry and redelivery behavior belong to the source integration or custom producer design.
• Fix: State the retry owner explicitly and test that integration’s acknowledgement mapping.
• Quick test: Fail one deterministic message and inspect the source-side disposition.

Problem 7: Claiming global ordering after partitioning

• Symptom: Events from different keys complete in a different order from input.
• Why: Partitioning permits concurrency across keys; failures may also disturb local completion order.
• Fix: Promise only the required per-key, per-boundary ordering scope.
• Quick test: Give one key slow events and observe completion order across another key.

Problem 8: Coalescing business facts

• Symptom: Intermediate payment, inventory, or audit events vanish.
• Why: A replaceable-state optimization was applied to immutable facts.
• Fix: Restrict coalescing by event class and prove the rule with domain invariants.
• Quick test: Attempt to classify a durable fact as coalescible and require the test to reject it.

Problem 9: The metrics collector becomes the bottleneck

• Symptom: Enabling metrics changes throughput dramatically.
• Why: Per-event synchronous logging or high-cardinality labels add excessive work.
• Fix: Aggregate counters and histograms, sample queue state, and keep event IDs out of metric labels.
• Quick test: Compare a measurement-only run with a no-output run and report overhead.

Problem 10: Queue length appears healthy while users see stale results

• Symptom: Queue depth is stable but event latency grows.
• Why: Event cost or an upstream hidden queue changed.
• Fix: Measure event age at multiple boundaries using monotonic time.
• Quick test: Add a fixed upstream delay without changing queue capacity and observe age percentiles.

Debugging sequence

When behavior is unclear, inspect in this order:

1. Reconcile event IDs and terminal categories.
2. Draw the actual demand feedback path.
3. Locate every mailbox and internal buffer.
4. Measure input rate, service rate, and event age.
5. Confirm dispatcher semantics.
6. Confirm acknowledgement and retry ownership.
7. Inspect per-key skew and slow consumers.
8. Only then tune concurrency, batch size, or demand thresholds.

8. Extensions & Challenges

Extension 1: Durable broker adapter

Replace the recording spill adapter with RabbitMQ, Kafka, Redis Streams, or another durable log. Document persistence, publisher confirmation, consumer acknowledgement, redelivery, retention, and dead-letter behavior.

Extension 2: Adaptive admission controller

Adjust command admission using queue age and processing-rate trends rather than queue length alone. Add hysteresis so the system does not oscillate between open and closed states.

Extension 3: Hot-key mitigation

Detect a key that dominates one partition. Compare keeping strict per-key order with splitting safe sub-work across secondary keys.

Extension 4: Multi-tenant fairness

Add tenant identifiers and prevent one noisy tenant from consuming all edge capacity. Compare reserved quotas, weighted fairness, and per-tenant queues.

Extension 5: End-to-end idempotency ledger

Model a durable idempotency ledger and inject crashes at every step between receive, effect, ledger write, and acknowledgement. Write a failure-window table.

Extension 6: Autoscaling signal study

Compare scaling decisions based on CPU, queue depth, outstanding demand, completion rate, and event age. Identify signals that react too late.

Extension 7: Cross-node experiment

Run producers and consumers across multiple BEAM nodes. Introduce network delay and partitions, then separate local GenStage behavior from the source’s distributed durability contract.

Extension 8: Property-based accounting

Generate random sequences of admission, demand, failure, restart, acknowledgement, and recovery actions. Assert that no durable event enters an unaccounted state.

Extension 9: Alternative flow technologies

Reproduce the comparison with Flow, a broker-native Broadway producer, or an Oban-backed durable workflow. Compare semantic boundaries rather than only benchmark numbers.

Extension 10: Operator runbook

Write a production runbook for rising event age, a full edge, a failing durable adapter, a slow broadcast subscriber, and a growing failed-acknowledgement count.

9. Real-World Connections

Where this architecture appears

• Telemetry ingestion: Device readings may be coalescible, while alarms and audit facts require durable delivery.
• Notification systems: A broadcast event can feed email, push, and webhook pipelines, but each channel often needs independent durable buffering.
• Data enrichment: Competing consumers distribute independent records across available CPU or I/O capacity.
• Search indexing: Batching improves throughput, while document IDs provide a natural partition and idempotency key.
• Payments: Durable facts, idempotency, and acknowledgement ownership are more important than raw low latency.
• Collaborative applications: UI hints can be ephemeral, while authoritative mutations belong in durable state.
• ETL pipelines: Batch size, timeout, partitioning, and retry ownership determine operational behavior.

Mapping to industry systems

Project concept	Industry analogue
DemandDispatcher	Competing queue consumers
BroadcastDispatcher	A synchronized fanout stage with coupled subscriber demand
Broadway producer	Broker/source connector
Broadway processor pool	Concurrent record handlers
Broadway batcher	Bulk database/API writer
Acknowledger	Commit, ack, reject, or disposition protocol adapter
Bounded edge	Admission controller or load shedder
Coalesce by key	Last-value compaction for replaceable state
Durable spill	Append to broker/log before asynchronous processing
Partition by entity	Per-key processing lane

What this project teaches beyond Elixir

The central lesson applies to any distributed system: flow-control guarantees end where the feedback signal ends. Adding a queueing library, reactive API, or worker pool downstream cannot repair an unbounded upstream admission contract. Engineers must name the boundary, own the queue, define overflow behavior, and measure the age of accepted work.

Production review questions

Before approving a real bridge, ask:

• Can the caller distinguish accepted from rejected work?
• Does accepted work survive a process crash, node crash, and network partition?
• Which events may be compacted without losing business meaning?
• Is every queue bounded and observable?
• Can optional consumers affect critical-path throughput?
• Who retries, for how long, and where do poison messages go?
• What identifier makes side effects idempotent?
• What exact ordering is required and proven?

10. Resources

Official documentation

• GenStage documentation — stage roles, demand, subscriptions, and producer-consumer behavior.
• GenStage dispatcher behaviour — dispatcher responsibilities and demand contracts.
• GenStage BroadcastDispatcher — broadcast delivery and demand coordination.
• Broadway documentation — topology, processors, batching, partitioning, telemetry, and message injection.
• Broadway acknowledger behaviour — success and failure acknowledgement callbacks.
• Broadway message documentation — message data, status, metadata, batching, and acknowledger fields.
• Phoenix.PubSub documentation — local and distributed publish-subscribe semantics.
• Telemetry documentation — event handlers, measurements, metadata, and synchronous execution.

Books

• Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf.
• Release It!, Second Edition by Michael T. Nygard.
• Designing Data-Intensive Applications, Second Edition by Martin Kleppmann and Chris Riccomini.
• Real-Time Phoenix by Stephen Bussey.
• Programming Phoenix LiveView by Bruce A. Tate and Sophie DeBenedetto.

Suggested study order

1. Read GenStage’s producer/consumer and demand sections.
2. Compare the two dispatcher documents.
3. Read Broadway topology, partitioning, batching, and acknowledgement sections.
4. Revisit Phoenix.PubSub and mark where no demand signal exists.
5. Build the accounting model before starting the load harness.

11. Self-Assessment Checklist

• I can explain why a push source remains push-based when Broadway is added downstream.
• I can draw the path of demand and mark where it stops.
• I can distinguish broadcast fanout from competing work distribution.
• I can explain how a slow broadcast consumer affects the group.
• I can define min/max demand as flow-control settings rather than durability guarantees.
• I can explain what Broadway processors, batchers, and acknowledgers each own.
• I can identify the retry owner for every producer used in my project.
• I can state why push_messages/2 does not propagate backpressure to Pub/Sub publishers.
• I have a hard capacity at the push-to-demand boundary.
• I use different overload policies for replaceable state, commands, and durable facts.
• I can prove that no durable fact is silently dropped.
• I can reconcile every generated event into an observable category.
• I measure event age as well as queue depth.
• I can state my exact ordering scope and its failure caveats.
• I can explain why acknowledgement does not imply exactly-once side effects.
• I have tested a slow consumer, a processor crash, a producer restart, and a durable-adapter failure.
• I can reproduce the 50,000-event comparison with a fixed seed.
• My metrics do not create high-cardinality labels or per-event log pressure.
• I can defend every throughput optimization in terms of preserved correctness.
• I can describe the project’s limitations without overstating its guarantees.

12. Submission / Completion Criteria

Submit the following artifacts:

1. A runnable project harness organized around the responsibilities in the architecture section.
2. A deterministic 50,000-event fixture with seed, checksum, class counts, and eight injected failures.
3. Raw Pub/Sub, GenStage competing-demand, GenStage broadcast-demand, and Broadway batch modes.
4. A bounded push-to-demand edge with coalesce, reject, and durable-spill decisions.
5. Automated tests covering demand bounds, delivery shape, accounting, ordering scope, acknowledgement, overload, and restart behavior.
6. A comparison report containing the required command transcript fields and all accounting invariants.
7. A failure-window table for side effect, acknowledgement, crash, and retry sequences.
8. Architecture decision records for delivery semantics, capacity, overload policy, durability, retry ownership, and ordering.
9. An operator runbook with thresholds and first-response actions for rising event age, full capacity, failed durable spill, and slow broadcast consumers.
10. A short reflection answering the project’s core question in your own words.

The project is complete when:

• all tests pass from a clean checkout;
• the same seed produces the same fixture checksum and accounting totals;
• all four modes complete or fail explicitly without hidden event categories;
• durable_events_dropped=0 in every test and load run;
• the edge never exceeds configured capacity;
• the report demonstrates, with measurements, that demand controls the GenStage/Broadway topology but not the upstream Pub/Sub publishers;
• the report identifies the slow broadcast consumer as the throughput constraint;
• successful and failed acknowledgement sets are complete and disjoint;
• ordering claims are scoped by key, boundary, and failure model;
• no section claims exactly-once execution, automatic retry, or durability without supporting evidence.

Continue with the Pub/Sub in Elixir Deep Dive or return to the expanded project index.