Project 11: Tenant-Safe Broker Edge Bridge

Build a RabbitMQ and Broadway ingestion edge that validates tenant envelopes, applies idempotent durable projections, classifies failures, dead-letters poison messages, and emits compact Phoenix.PubSub refreshes.

Quick Reference

Attribute	Value
Difficulty	Level 4 — Expert
Time Estimate	24–36 hours
Language	Elixir (alternatives: Erlang, Java)
Prerequisites	RabbitMQ routing, manual acknowledgements, Broadway, Ecto transactions, Phoenix.PubSub, multi-tenant authorization
Key Topics	Publisher confirms, consumer acknowledgements, prefetch/demand, redelivery, idempotent receiver, DLQ, tenant isolation, broker-to-PubSub boundary

1. Learning Objectives

By completing this project, you will be able to:

Explain publisher confirms and consumer acknowledgements as orthogonal transfers of responsibility.
Design a tenant-scoped event envelope whose identity and authorization can be checked independently of routing metadata.
Use Broadway demand and RabbitMQ prefetch to bound in-flight work rather than move overload into BEAM mailboxes.
Place consumer acknowledgement after the chosen durable effect and idempotency record.
Demonstrate at-least-once redelivery with one effective projection.
Classify transient, permanent, unauthorized, and poison failures into retry, reject, or dead-letter outcomes.
Derive Phoenix.PubSub topics from authenticated server context rather than trusting the broker routing key or payload.
Propagate trace and event identity across publisher, broker, Broadway, database, PubSub, and UI.
Distinguish broker backlog, in-flight work, processor saturation, subscriber mailbox lag, and UI event age.
Produce chaos evidence for processor crashes, malformed events, forged tenants, and broker outage.

2. Theoretical Foundation

2.1 Core Concepts

Two acknowledgement directions

RabbitMQ publisher confirms tell a publisher whether the broker accepted responsibility for a publication. For persistent messages routed to durable queues, the confirmation point depends on the queue type and persistence path; for quorum queues, it involves acceptance by the required replicas. A socket write alone is not this evidence.

Consumer acknowledgements flow in the other direction. They tell the broker that the consumer has completed enough processing for the delivery to be removed. Official RabbitMQ documentation emphasizes that publisher confirms and consumer acknowledgements are orthogonal: a publisher confirm knows nothing about downstream consumers, and a consumer acknowledgement knows nothing about the original publisher.

This creates a responsibility chain rather than one end-to-end transaction:

publisher durable intent
       │ publish + confirm
       v
broker accepts responsibility
       │ delivery + manual ack
       v
consumer accepts responsibility after durable effect

Each boundary has an uncertainty window and can produce duplicates. A publisher may retransmit when a confirm is lost. A broker automatically requeues an unacknowledged delivery after consumer connection or process failure. Therefore immutable event identity and idempotent effects are required at both sides.

Broadway demand and acknowledger semantics

Broadway provides a supervised processing topology with producers, processors, optional batchers, and an acknowledger. A Broadway.Message carries acknowledgement metadata. Successful and failed messages eventually reach the acknowledger, which communicates the outcome to the source connector. Broadway itself does not create universal retry semantics; source/producer integration and acknowledgement configuration decide whether failed work requeues, rejects, or dead-letters.

Demand and RabbitMQ prefetch bound work at different points. Prefetch limits unacknowledged broker deliveries in flight on a consumer/channel. Broadway demand limits how many messages move through stages. Processor and batch concurrency determine how much work can execute at once. These controls should align. Unlimited prefetch followed by a small processor pool moves the backlog from a durable broker into application memory.

Idempotent projection

The reference durable effect is a tenant-scoped projection row. The projection transaction first claims or checks the immutable event ID, validates that the recorded digest matches, applies the projection revision, and records completion. If the processor crashes after commit and before acknowledgement, RabbitMQ redelivers. The next attempt finds the inbox/idempotency record and reports already_applied without repeating the business effect.

Deduplication cannot be a global in-memory set. It must survive restart and include the authorization boundary. A robust key is {tenant_id, event_id, consumer_name} with a payload digest and outcome. The digest detects malicious or accidental reuse of an event ID for different content.

Failure taxonomy and dead lettering

Not all failures should requeue. A temporary database outage or rate limit may be retried with backoff. Unsupported schema, malformed required fields, invalid signature, or tenant mismatch are permanent for that delivery and should fail closed. Requeueing permanent failures creates a hot poison loop that consumes capacity without progress.

A dead-letter queue is a quarantine and diagnostic boundary, not an infinite retry strategy. The system records reason code, event ID, schema version, first/last observed time, and redacted context. Operators need an explicit inspect, repair, replay, or discard workflow. Replays preserve the original identity and create audited operator action.

Tenant isolation across two buses

The broker and Phoenix.PubSub are separate routing domains. An event may present tenant identity in credential scope, exchange/vhost/queue binding, routing key, envelope field, and database row. These values must be checked for agreement; none should silently override the others.

After the durable projection commits, the edge publishes a compact refresh to a topic derived from the server-validated tenant and resource. It never concatenates an arbitrary payload tenant into a topic. The LiveView or Channel performs its own authorization during connection/mount and reloads the current projection. This prevents a compromised publisher from converting a forged routing key into cross-tenant UI leakage.

Observability boundaries

Broker ready count shows queued work; unacknowledged count shows in-flight responsibility; redelivery count indicates retries; DLQ count shows permanent failures; consumer capacity and processing latency show whether consumption can keep up. Broadway demand, stage processing time, database transaction time, and BEAM mailbox length identify application bottlenecks. PubSub subscriber lag and UI event age describe the transient edge.

One metric cannot stand in for another. A low BEAM mailbox with 12,400 ready broker messages is healthy backpressure but poor freshness. A zero broker backlog with a LiveView mailbox of 50,000 is not healthy end-to-end delivery.

2.2 Why This Matters

A BEAM cluster is an excellent low-latency fanout domain, but it is not always the ownership boundary between independently deployed services, languages, teams, or regions. Durable brokers provide retained cross-service delivery, routing, acknowledgements, and independent consumer pace. Phoenix.PubSub provides efficient transient fanout inside the application cluster. A broker edge intentionally terminates one contract and starts another.

The bridge must not forward blindly. External events are untrusted input. Schema, size, tenant, signature, event identity, and authorization are validated before any effect or PubSub broadcast. The durable projection gives reconnecting UIs a source of truth. The compact refresh minimizes latency for currently connected clients.

This project turns “RabbitMQ plus PubSub” from a component diagram into a failure-tested responsibility chain. You will know exactly when the broker may delete a message, when a duplicate is harmless, why one message entered the DLQ, and why forged tenant data produced zero UI delivery.

2.3 Historical Context / Background

Message brokers evolved to decouple producers and consumers in time and deployment. AMQP formalized exchanges, queues, routing, delivery modes, and acknowledgements. Publisher confirms were introduced as a more practical data-safety mechanism than heavyweight per-message channel transactions.

Broadway applies BEAM supervision and demand-driven processing to external data sources. Phoenix.PubSub serves a different purpose: ephemeral fanout to local and distributed subscribers. Combining them is common, but only correct when acknowledgement and authority boundaries remain explicit.

2.4 Common Misconceptions

“Publisher confirm means a consumer processed the event.” It confirms broker responsibility, not application completion.
“Consumer ack means the publisher received a business response.” The acknowledgement is local to broker-consumer responsibility.
“Broadway retries failed messages automatically in every connector.” Retry behavior depends on the source and acknowledger configuration.
“Durable queue plus persistent message guarantees no duplicates.” Reliability mechanisms intentionally retransmit uncertain deliveries.
“DLQ means retry later automatically.” It is quarantine requiring an explicit workflow.
“Routing-key tenant is already authorized.” Routing metadata is untrusted input unless bound to authenticated publisher scope and validated.
“Low BEAM mailbox means the system is caught up.” Backlog may be retained safely in RabbitMQ.
“Forward the original event to PubSub.” UIs need a compact authorized refresh, not an untrusted integration payload.

3. Project Specification

3.1 What You Will Build

Build a disposable RabbitMQ environment, a confirmed test publisher, a Broadway ingestion pipeline, a PostgreSQL projection/inbox store, a dead-letter quarantine, Phoenix.PubSub refreshes, and an operator dashboard.

Tenant 42’s publisher credential can publish only to the intended exchange/vhost scope. Events carry a versioned envelope and routing key. The consumer derives its authorized tenant context from queue binding/configuration, compares it with the routing key and envelope, validates schema and size, and opens a projection transaction.

The projection transaction records {tenant_id, event_id, consumer} uniquely, verifies the payload digest, and applies a monotonic resource revision. Only after commit does the edge emit a compact PubSub refresh and allow successful acknowledgement. A crash switch after projection commit but before acknowledgement proves safe redelivery.

The dashboard shows ready, unacknowledged, confirmed, redelivered, rejected, DLQ, processor latency, projection latency, PubSub refresh, and end-to-end event age. Chaos controls kill a processor, publish an unsupported schema, forge tenant combinations, and pause the broker.

3.2 Functional Requirements

Confirmed publishing: Track positive, negative, returned/unroutable, and timed-out publications separately.
Durable topology: Declare exchange, queue, bindings, queue type, persistence, and dead-letter policy reproducibly.
Versioned envelope: Validate event ID, tenant, type, schema, occurred-at, trace context, resource identity, revision, and payload size.
Tenant agreement: Credential/queue scope, routing key, envelope, and target database tenant must agree.
Bounded consumption: Configure prefetch, Broadway demand, processor concurrency, and database pool coherently.
Idempotent projection: Duplicate deliveries produce one durable effect.
Monotonic revision: Stale events are classified; gaps trigger repair policy rather than silent overwrite.
Acknowledgement policy: Success only after durable projection; failures map deterministically to requeue, reject, or DLQ.
Compact PubSub refresh: Emit only validated tenant/resource/revision/event identity after commit.
Operator quarantine: Inspect and replay/discard DLQ entries through audited commands.
Outage behavior: Broker pause builds visible durable backlog without unbounded BEAM mailboxes.
Cross-tenant test matrix: Every mismatched tenant combination fails closed with zero projection and zero PubSub leakage.

3.3 Non-Functional Requirements

Data safety: Confirm and acknowledgement outcomes are measured independently.
Idempotency: Redelivery count may exceed zero; duplicate effect count remains zero.
Isolation: Cross-tenant projection, PubSub, metrics-label, and log leakage remain zero.
Capacity: Maximum in-flight work and stage concurrency are documented and enforced.
Recovery: Broker and processor restarts resume from durable queue state.
Latency: Under reference load, end-to-end UI refresh p99 remains below the stated local target.
Diagnostics: Every event is traceable by immutable ID without logging sensitive payloads.
Operability: DLQ and exhausted retry paths have owner, alert, retention, and replay procedure.

3.4 Example Usage / Output

$ mix broker.edge.drill
topology exchange=tenant.events queue=edge.tenant42 type=quorum durable=true PASS
publish event=evt-901 tenant=42 confirm=ack routed=1
consume event=evt-901 redelivered=false schema=1 tenant_check=PASS
projection event=evt-901 revision=71 result=applied
pubsub topic=tenant:42:device:abc refresh_revision=71 sent=1
consumer_ack event=evt-901 result=success

inject fault=crash_after_projection_before_ack event=evt-902
redelivery event=evt-902 redelivered=true projection=already_applied
pubsub refresh=coalesced consumer_ack=success effective_projections=1

publish event=evt-903 schema=99
classification=permanent reason=unsupported_schema requeue=false dlq_count=1

publish event=evt-904 credential_tenant=42 routing_tenant=42 payload_tenant=77
authorization=DENY projection_changes=0 pubsub_deliveries=0

broker paused duration=45s ready=12400 unacked=0
beam_mailbox_max=72 processor_status=waiting alert=UPSTREAM_LAG
broker resumed drained_in=38.4s duplicate_effects=0
drill result=PASS

3.5 Real World Outcome

Open the operator dashboard beside RabbitMQ’s management view. Publish evt-901 with tenant 42 credentials. The publisher panel shows one positive confirm; RabbitMQ briefly shows ready/unacknowledged movement; the projection table advances device abc to revision 71; and the authorized tenant-42 LiveView refreshes. Tenant 77’s board remains unchanged.

Trigger the processor crash after evt-902 commits. The RabbitMQ delivery remains unacknowledged until the connection closes and is then redelivered. The dashboard shows redelivered=true, two processing attempts, one inbox record, one projection effect, and a coalesced or repeated harmless UI refresh.

Publish schema 99. It appears exactly once in the DLQ with unsupported_schema; it does not hot-loop. Publish a payload claiming tenant 77 through tenant 42’s route. The authorization card shows denial, and automated probes confirm no tenant-77 database row, PubSub event, metric label, or sensitive log entry was created.

Pause RabbitMQ or disconnect the application. The ready count grows at the broker while Broadway and BEAM mailbox bounds remain stable. Resume the broker and watch bounded consumers drain backlog. The final report separates upstream backlog, consumer in-flight work, processor latency, PubSub refresh age, and duplicate-effect count.

4. Solution Architecture

4.1 High-Level Design

       PUBLISHER RESPONSIBILITY                    CONSUMER RESPONSIBILITY

┌──────────────┐  publish   ┌──────────────────┐  deliver  ┌────────────────┐
│ Tenant source│───────────▶│ RabbitMQ         │──────────▶│ Broadway       │
│ outbox/client│◀─confirm───│ exchange + queue │◀────ack───│ processors     │
└──────────────┘            │ DLX + DLQ        │           └───────┬────────┘
                            └──────────────────┘                   │ validate
                                                                  v
                                                       ┌────────────────────┐
                                                       │ PostgreSQL         │
                                                       │ inbox + projection │
                                                       └─────────┬──────────┘
                                                                 │ after commit
                                                                 v
                                                       ┌────────────────────┐
                                                       │ Phoenix.PubSub     │
                                                       │ compact refresh    │
                                                       └─────────┬──────────┘
                                                                 v
                                                       ┌────────────────────┐
                                                       │ Authorized UI      │
                                                       │ reload projection  │
                                                       └────────────────────┘

Poison/auth failure -> reject without requeue -> DLX/DLQ -> audited review
Transient failure -> bounded retry/requeue policy -> same event identity

4.2 Key Components

Component	Responsibility	Key Decisions
Confirmed publisher	Retain publication intent until broker confirm	Correlate confirm sequence to immutable event ID
Broker topology	Route and retain tenant event classes	Quorum/durable queue, policy-defined DLX, least privilege
Broadway producer	Pull under prefetch/demand	Align in-flight limit with processors and DB pool
Envelope validator	Enforce size, schema, identity, and tenant agreement	Fail closed before effect
Projection transaction	Dedupe and apply monotonic revision	Unique inbox key plus payload digest
Acknowledger policy	Translate outcomes to ack/reject/requeue/DLQ	Explicit permanent/transient classification
PubSub refresher	Notify authorized cluster subscribers	Server-derived compact topic and payload
Quarantine console	Inspect and replay DLQ work	Immutable evidence and audited operator action
Metrics pipeline	Measure each responsibility boundary	Bounded labels; no tenant secrets/high cardinality IDs

4.3 Data Structures

IntegrationEnvelope
  event_id              immutable globally unique ID
  tenant_id             asserted tenant, validated against context
  event_type            e.g. device.status_changed
  schema_version        supported contract version
  resource_type/id      projection target
  resource_revision     monotonic source revision
  occurred_at           source event time
  published_at          source send time
  traceparent           bounded trace context
  payload               size-limited schema body

InboxRecord
  tenant_id + event_id + consumer_name   unique key
  payload_digest
  first_received_at
  applied_at
  outcome
  resource_revision

QuarantineRecord
  event_id
  reason_code
  redacted_envelope_summary
  first/last_seen
  source_queue
  operator_resolution

4.4 Algorithm Overview

Algorithm: Consume, project, acknowledge

RECEIVE delivery with broker metadata and authorized queue context
CHECK byte size before decoding expensive payload
DECODE versioned envelope
VALIDATE schema and tenant agreement
TRANSACTION
  lookup inbox key {tenant,event,consumer}
  if existing digest matches: return already_applied
  if existing digest differs: return identity_conflict
  verify revision policy
  apply projection
  insert inbox outcome
COMMIT
BEST-EFFORT publish compact authorized refresh
ACK success to broker

Algorithm: Failure classification

temporary DB/network/rate limit -> failed delivery with bounded retry/requeue policy
unsupported schema/malformed payload -> reject requeue=false -> DLQ
tenant/signature/identity conflict -> fail closed -> security quarantine/alert
application bug -> limited retry, then quarantine with stack fingerprint
successful/already_applied -> ack

Complexity analysis

Envelope validation is O(payload bytes), with size checked first.
Inbox lookup and projection update are O(log n) with tenant/event/resource indexes.
Per-message processing space is O(payload size); bounded demand makes total in-flight space O(prefetch × max payload).
DLQ and inbox retention are O(events retained) and require lifecycle policies.

5. Implementation Guide

5.1 Development Environment Setup

Use disposable local services and generated training credentials. Prefer policy-based RabbitMQ topology where operational changes are expected.

$ docker compose up -d rabbitmq postgres
rabbitmq health=healthy management=http://127.0.0.1:15672
postgres health=healthy port=5432

$ ./scripts/provision-training-topology
vhost=/edge-lab exchange=tenant.events
queue=edge.tenant42 type=quorum dlx=tenant.events.dlx
user=tenant42-publisher permission_scope=publish-only
provision result=PASS

$ mix broker.edge.doctor
confirm_mode=enabled manual_ack=true prefetch=100
broadway_processor_concurrency=10 db_pool=15
doctor result=PASS

Never put real broker credentials in the guide, evidence, or committed configuration. Record resolved RabbitMQ, Broadway, AMQP client, Elixir, OTP, Ecto, Phoenix, and Phoenix.PubSub versions.

5.2 Project Structure

broker_edge_bridge/
├── lib/
│   ├── broker/topology.ex
│   ├── broker/confirmed_publisher.ex
│   ├── edge/pipeline.ex
│   ├── edge/envelope.ex
│   ├── edge/tenant_policy.ex
│   ├── edge/projection.ex
│   ├── edge/ack_policy.ex
│   └── web/edge_dashboard_live.ex
├── priv/repo/migrations/
├── config/
│   └── runtime.exs
├── scripts/
│   ├── provision-training-topology
│   └── broker-edge-drill
├── test/
│   ├── contract/
│   ├── integration/
│   ├── isolation/
│   └── chaos/
├── docs/
│   ├── responsibility-chain.md
│   ├── failure-classification.md
│   └── dlq-runbook.md
└── evidence/

5.3 The Core Question You’re Answering

“How do durable broker guarantees terminate safely at an Elixir application and become transient UI fanout without leaking tenants or duplicating effects?”

Answer by naming the responsibility transfer at confirm, delivery, projection commit, consumer acknowledgement, and PubSub refresh. If your answer says “the event is delivered exactly once,” replace it with the observed delivery and effect semantics.

5.4 Concepts You Must Understand First

Publisher confirms
- When has the broker accepted responsibility?
- How do mandatory returns and confirms differ for unroutable messages?
- Reference: RabbitMQ Consumer Acknowledgements and Publisher Confirms.
Manual consumer acknowledgements
- Why must acknowledgement happen on the delivery’s channel/context?
- What happens to unacknowledged messages after connection loss?
- Reference: RabbitMQ reliability and acknowledgements guides.
Broadway demand and acknowledgers
- Where do successful and failed messages terminate?
- Which connector/source policy supplies retries?
- Reference: Broadway and Broadway.Acknowledger documentation.
Idempotent Receiver
- Which durable unique key turns redelivery into one effect?
- How will conflicting event-ID reuse be detected?
- Book Reference: “Enterprise Integration Patterns” — Idempotent Receiver.
Tenant authorization
- Which context is authenticated, and which tenant fields are merely claims?
- How are PubSub topics derived server-side?
- Reference: OWASP authorization guidance and Phoenix LiveView security model.

5.5 Questions to Guide Your Design

Topology and responsibility
- Which queue type and durability settings match the lab’s guarantee?
- What does a positive publisher confirm actually prove?
Capacity
- How do prefetch, Broadway demand, processor concurrency, and DB pool size relate?
- Where should backlog remain during downstream slowdown?
Failure policy
- Which outcomes requeue, dead-letter, quarantine, or page an operator?
- What prevents requeue storms?
Tenant isolation
- Which independent tenant values must agree?
- Could metrics, logs, or DLQ tooling expose cross-tenant data even if projection is safe?
PubSub handoff
- Is refresh sent before or after durable projection commit?
- How does a missed refresh repair on reconnect?

5.6 Thinking Exercise

Trace one event through this chain:

publisher intent -> TCP write -> broker confirm -> ready queue -> delivery
-> Broadway demand -> validation -> projection transaction -> PubSub refresh
-> consumer acknowledgement -> broker deletion

Place a crash before and after every arrow. For each point, predict:

whether the publisher retransmits;
whether RabbitMQ redelivers;
whether the projection exists;
whether the UI may have refreshed;
whether a duplicate effect can occur;
which durable evidence resolves uncertainty.

Then repeat with mismatched credential tenant, routing tenant, payload tenant, and database tenant.

5.7 The Interview Questions They’ll Ask

“What is the difference between publisher confirms and consumer acknowledgements?”
“Why does RabbitMQ at-least-once delivery require idempotency?”
“Does Broadway implement retries, and who decides failed-message behavior?”
“How do prefetch and Broadway demand prevent overload?”
“What belongs in a dead-letter queue, and how is it replayed safely?”
“How do you prevent forged broker tenant data from reaching a Phoenix topic?”

5.8 Hints in Layers

Hint 1: Build the failure table before the pipeline

List every outcome with ack, requeue, dead-letter, or security quarantine. Make ambiguous cases explicit.

Hint 2: Acknowledge after the durable boundary

The reference durable boundary is the committed inbox/projection transaction. PubSub refresh is best-effort and does not delay broker acknowledgement indefinitely.

Hint 3: Validate tenant from both directions

Compare authenticated queue/credential context with routing key and envelope. Derive the final PubSub topic from the validated server context.

Hint 4: Make backlog visible where it belongs

Pause processors and confirm ready messages rise at RabbitMQ while BEAM mailbox length stays bounded.

5.9 Books That Will Help

Topic	Book	Chapter / Pattern
Messaging guarantees	“Enterprise Integration Patterns”	Messaging Channels, Guaranteed Delivery, Idempotent Receiver
Streams and redelivery	“Designing Data-Intensive Applications, 2nd Edition”	Stream Processing
Service boundaries	“Building Microservices, 2nd Edition”	Asynchronous Communication
Stability and capacity	“Release It!, 2nd Edition”	Stability Patterns

5.10 Implementation Phases

Phase 1: Broker Responsibility and Contract Validation (8–12 hours)

Goals

Provision durable topology and confirmed publishing.
Validate event and tenant contracts before effects.

Tasks

Document exchange, queue, binding, DLX, persistence, and credential scopes.
Implement a confirmed test publisher with correlation evidence.
Define versioned envelope and size limits.
Build tenant-agreement and schema test matrices.
Verify mandatory/unroutable and negative/timeout publication paths.

Checkpoint: Valid events are confirmed and routed; unroutable and forged events have deterministic, distinct outcomes with no projection.

Phase 2: Bounded Processing and Idempotent Projection (10–16 hours)

Goals

Process under explicit demand and acknowledgement.
Prove redelivery yields one effect.

Tasks

Configure prefetch, demand, concurrency, and database pool budgets.
Add inbox/projection transaction and digest conflict check.
Configure explicit acknowledgement/failure policy.
Inject crash after projection before ack.
Add compact PubSub refresh after commit.

Checkpoint: evt-902 is delivered at least twice, projected once, acknowledged successfully, and never leaks to another tenant.

Phase 3: Quarantine, Outage, and Operational Evidence (10–16 hours)

Goals

Make poison handling and backlog operable.
Certify the edge under failures.

Tasks

Add DLQ inspection and audited replay/discard workflow.
Instrument confirms, ready/unacked, redelivery, DLQ, stage latency, projection, and PubSub age.
Pause broker and processors under fixed-rate publishing.
Run cross-tenant probes across data, UI, logs, and metrics.
Produce the responsibility-chain and chaos report.

Checkpoint: Broker outage creates bounded upstream lag, recovery drains predictably, poison work is quarantined once, and observed tenant leaks remain zero.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Queue type	Classic; quorum; stream	Quorum queue for reference durable-work lab	Makes replicated broker responsibility explicit
Acknowledgement	Automatic; manual	Manual after durable projection	Prevents deletion before responsibility transfers
Deduplication	Memory cache; durable inbox	Durable tenant/event/consumer inbox	Survives restart and detects conflict
Failure handling	Requeue all; classify	Explicit transient/permanent/security classes	Avoids poison loops and silent drops
DLX configuration	Hard-coded x-args; broker policy	Policy where operationally appropriate	Easier controlled evolution
PubSub payload	Original envelope; compact refresh	Validated ID + revision refresh	Minimizes trust and disclosure
Backlog location	BEAM mailbox; broker queue	Durable broker with bounded in-flight demand	Preserves work and application health

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Contract	Validate envelope and versioning	Size, required fields, unsupported schema
Broker integration	Verify confirms and acknowledgements	Ack/nack/return, connection loss, redelivery
Projection integration	Verify idempotency and ordering	Duplicate event, digest conflict, stale revision
Broadway capacity	Verify bounds	Prefetch, demand, slow database, processor crash
Tenant isolation	Verify fail-closed routing	Credential/routing/payload/database matrix
DLQ operations	Verify quarantine lifecycle	Poison event, inspect, repair, replay, discard
End-to-end	Verify broker-to-UI journey	Confirm through authorized LiveView reload
Chaos	Verify recovery	Broker pause, app kill, DB outage, subscriber slowdown

6.2 Critical Test Cases

Confirmed valid publish: Publisher receives positive confirm and exactly one queue receives the event.
Unroutable mandatory publish: Return and confirm semantics are recorded without pretending a consumer exists.
Normal consume: Projection commits before successful acknowledgement.
Crash after projection: Delivery redelivers and inbox returns already_applied.
Duplicate ID/different body: Digest conflict fails closed and alerts.
Unsupported schema: One dead-letter outcome, no requeue loop.
Transient database outage: Delivery remains recoverable and later succeeds with the same ID.
Tenant mismatch matrix: Every mismatch creates zero projection and zero PubSub delivery.
Stale revision: Classified according to documented skip/reconcile policy.
Broker pause: Ready backlog rises while application in-flight/mailbox bounds hold.
Slow subscriber: Broker/projection succeeds; PubSub lag is separately visible and bounded.
DLQ replay: Repaired event preserves original ID and operator audit identity.

6.3 Test Data

authorized_context:
  vhost=/edge-lab
  queue=edge.tenant42
  credential_tenant=42

valid_event:
  event_id=evt-901
  tenant_id=42
  type=device.status_changed
  schema_version=1
  resource=device:abc
  revision=71
  payload={status: degraded}

redelivery_event: evt-902 revision=72 fault=after_projection_before_ack
poison_event: evt-903 schema_version=99 expected=DLQ
forged_event: evt-904 routing_tenant=42 payload_tenant=77 expected=DENY
identity_conflict: evt-901 altered_resource=device:xyz expected=SECURITY_ALERT

load:
  rate=500 events/second
  duration=60 seconds
  max_payload=64 KiB
  prefetch=100
  processor_concurrency=10

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Treating confirm as end-to-end success	Publisher says success while projection is absent	Name broker acceptance separately from consumer effect
Ack before commit	Crash loses uncommitted projection permanently	Ack only after durable effect
Requeue every failure	Poison message consumes the queue repeatedly	Classify permanent failures and dead-letter
Memory-only dedupe	Duplicates reapply after restart	Persist inbox key and digest
Unlimited prefetch	App memory/mailboxes grow under slow DB	Bound prefetch and Broadway demand
Trust routing tenant	Forged payload reaches another tenant	Cross-check authenticated context and derive topic server-side
High-cardinality metrics	Metrics backend cost explodes	Keep event IDs in traces/logs, not metric labels
Blind DLQ replay	Bad event loops again or changes identity	Repair with audit, preserve ID, validate before replay

7.2 Debugging Strategies

Follow one event ID across publish intent, confirm, queue delivery, inbox, projection, refresh, and ack.
Compare ready vs unacked: ready indicates queued backlog; unacked indicates in-flight consumer responsibility.
Inspect redelivered flag and attempt history before blaming the publisher for duplicates.
Check connector acknowledgement configuration because Broadway failure alone does not define broker retry.
Freeze the database to see whether demand/prefetch boundaries keep application memory stable.
Run the tenant matrix whenever routing, envelope, or authorization code changes.
Examine DLQ reason codes rather than replaying raw entries until one succeeds.

7.3 Performance Traps

Prefetch far above useful concurrency increases redelivery and memory cost during consumer failure.
Batching unrelated tenants can create fairness and isolation problems.
Large payload decoding before size checks exposes CPU and memory denial of service.
Synchronous PubSub/UI work before broker acknowledgement lengthens unacked time unnecessarily.
Per-event metric labels overwhelm telemetry backends; use trace correlation instead.
Hot retry without backoff amplifies database or broker recovery pressure.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a UI journey view for one event ID.
Add an operator-visible reason dictionary for every failure class.
Compare automatic and manual acknowledgement in a disposable loss drill.

8.2 Intermediate Extensions

Add per-tenant batching keys while preserving fairness and isolation.
Add schema upcasting for one old supported version.
Implement rate-limited audited DLQ replay.
Add broker and subscriber lag panels to the same timeline.

8.3 Advanced Extensions

Use an upstream transactional outbox to feed the confirmed publisher.
Compare quorum queues with RabbitMQ streams for high-volume replay use cases.
Run a broker-node failure drill and measure confirm and redelivery behavior.
Add OpenTelemetry propagation across publisher, broker, Broadway, database, and LiveView.

9. Real-World Connections

9.1 Industry Applications

Payments: Broker deliveries drive idempotent ledger projections and transient operator refreshes.
Logistics: Device/fleet updates cross services durably before appearing on realtime boards.
Healthcare integration: Versioned events require quarantine and audit rather than blind retries.
Multi-tenant SaaS: Routing, database, UI, and observability must share an isolation contract.
Incident platforms: Durable escalation work and ephemeral dashboard freshness use different mechanisms.

RabbitMQ: Durable routing, confirms, acknowledgements, and dead lettering.
Broadway: Concurrent, multi-stage data ingestion and processing.
BroadwayRabbitMQ: RabbitMQ producer integration for Broadway.
Phoenix.PubSub: Transient cluster and UI fanout.
OpenTelemetry Erlang/Elixir: Cross-boundary tracing.

9.3 Interview Relevance

This project provides evidence for a common staff-level system-design discussion: where broker guarantees end, how at-least-once becomes one effect, how backpressure remains durable, and how multi-tenant routing fails closed. Draw the responsibility chain and state the acknowledgement point instead of listing technologies.

10. Resources

10.1 Essential Reading

“Enterprise Integration Patterns” by Gregor Hohpe and Bobby Woolf — messaging channels, routing, guaranteed delivery, and idempotent receiver.
“Designing Data-Intensive Applications, 2nd Edition” by Martin Kleppmann et al. — stream processing and distributed failure.
“Building Microservices, 2nd Edition” by Sam Newman — asynchronous service integration.
“Release It!, 2nd Edition” by Michael Nygard — stability, backpressure, and recovery.

10.2 Video Resources

RabbitMQ official talks on quorum queues, confirms, acknowledgements, and flow control.
ElixirConf talks on Broadway pipelines, demand, and production observability.

10.3 Tools & Documentation

RabbitMQ reliability guide — responsibility and failure recovery.
Consumer acknowledgements and publisher confirms — orthogonal confirmation semantics.
RabbitMQ consumers — manual ack, prefetch, and consumer capacity.
RabbitMQ dead lettering — DLX/DLQ behavior and policies.
Broadway.Acknowledger — successful and failed message outcomes.
Broadway.Message — message failure and acknowledgement configuration.
Phoenix LiveView security — authentication and authorization on connected mount.

Project 7: Mailbox Pressure Laboratory introduces bounded BEAM-side overload controls.
Project 5: LiveView Operations Board establishes reconnect behavior, while Project 3: Contract-Aware Domain Event Hub establishes secure topic derivation.
Project 9: Durable Notification Outbox solves the producer-side database handoff.
Project 10: Three-Node PubSub Laboratory tests cluster transient behavior.
Project 12: Multi-Tenant Real-Time Event Platform combines the full responsibility chain.

11. Self-Assessment Checklist

11.1 Understanding

I can state what publisher confirm proves and does not prove.
I can state when the broker may delete a consumed delivery.
I can explain how prefetch and Broadway demand interact.
I can identify the crash window that causes redelivery after projection.
I can distinguish retry, reject, DLQ, and security quarantine.
I can explain why tenant identity is validated at multiple independent boundaries.
I can distinguish broker lag from subscriber lag.

11.2 Implementation

Confirmed publishing reports ack, nack, return, and timeout separately.
Durable topology and least-privilege credentials are reproducible.
Envelope size, schema, identity, and tenant are validated before effects.
Projection and inbox record commit together.
Post-commit crash causes redelivery and one effective projection.
Unsupported schema reaches DLQ without a hot loop.
Forged tenant tests produce zero leakage.
Broker outage keeps application memory/mailboxes bounded.

11.3 Growth

I documented one previous misconception about broker guarantees.
I can defend the chosen prefetch and concurrency with measurements.
I can operate the DLQ without blind replay.
I can present the event responsibility chain in a system-design interview.

12. Submission / Completion Criteria

Minimum Viable Completion

One confirmed tenant event is consumed, projected durably, acknowledged, and refreshed through PubSub.
A duplicate delivery produces one projection effect.
An unsupported schema reaches a DLQ.
One forged tenant event produces zero projection and zero UI delivery.
Prefetch and Broadway demand are explicitly bounded.

Full Completion

All minimum criteria plus confirm-return/timeout handling, digest conflict detection, monotonic revisions, and audited DLQ replay/discard.
Processor crash after projection causes safe redelivery.
Broker and processor outage drills separate ready, unacked, stage, mailbox, and UI lag.
Cross-tenant tests cover credentials, routing, payload, database, PubSub, logs, metrics, and quarantine tooling.
A redacted end-to-end report traces immutable event IDs through every responsibility boundary.

Excellence (Going Above & Beyond)

Upstream transactional outbox and publisher confirms are connected and failure-tested.
Broker-node failure and quorum recovery meet documented RTO/RPO expectations.
OpenTelemetry traces span publisher, RabbitMQ, Broadway, PostgreSQL, PubSub, and LiveView.
A capacity model predicts safe rate, in-flight limits, drain time, and alert thresholds and matches load evidence.

This guide was expanded from LEARN_ELIXIR_PUBSUB_DEEP_DIVE.md. For the complete expanded project index, see README.md.

Project 11: Tenant-Safe Broker Edge Bridge

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Historical Context / Background

2.4 Common Misconceptions

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Structures

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: Broker Responsibility and Contract Validation (8–12 hours)

Phase 2: Bounded Processing and Idempotent Projection (10–16 hours)

Phase 3: Quarantine, Outage, and Operational Evidence (10–16 hours)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

7.3 Performance Traps

8. Extensions & Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools & Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria