Project 21: Multi-Provider Webhook Authenticity Gateway

Build a Plug-based gateway that proves who sent each webhook before parsing or forwarding it, rejects replays, and leaves an auditable explanation for every decision.

Quick Reference

Attribute Value
Difficulty Level 2
Suggested Seniority Mid-level Elixir developer
Time Estimate 12-18 hours
Main Programming Language Elixir
Alternative Programming Languages Erlang, Gleam
Coolness Level Level 3
Business Potential Level 3
Prerequisites Pattern matching, result tuples, HTTP fundamentals, basic cryptography vocabulary
Key Topics Plug, raw request bodies, HMAC, constant-time comparison, replay windows, behaviours

1. Learning Objectives

By completing this project, you will be able to:

  1. Preserve the exact HTTP body bytes used by a provider when calculating a signature.
  2. Model Stripe-like, GitHub-like, and custom providers behind one Elixir behaviour without erasing provider-specific rules.
  3. Verify HMAC signatures with constant-time comparison and distinguish malformed, stale, unknown-key, and mismatched requests.
  4. enforce timestamp windows and event-identifier deduplication as two separate replay defenses.
  5. Use with, tagged tuples, structs, and multi-clause functions to make a security decision pipeline explicit.
  6. Produce rejection records that are useful to operators without storing secrets or full sensitive payloads.

2. All Theory Needed (Per-Concept Breakdown)

Authenticity Pipelines over Exact Bytes

Fundamentals

A webhook signature is evidence about a specific byte sequence, not about the Elixir map produced after JSON decoding. A provider usually combines the raw body with a timestamp, version marker, or delivery identifier and computes a message authentication code using a shared secret. The receiver reconstructs exactly the same signed material and compares the expected digest with the supplied signature. Three properties must remain separate: authenticity asks whether a party holding the secret created the message; integrity asks whether the signed bytes changed; freshness asks whether an authentic message is too old or has already been processed. Parsing before verification, comparing ordinary strings, or treating a valid HMAC as proof of freshness breaks those properties. The gateway therefore needs a byte-preserving ingress boundary, explicit provider rules, constant-time comparison, and replay state.

Deep Dive into the concept

HTTP frameworks make application development convenient by consuming a request body and decoding it into data structures. Signature verification reverses the usual order of concern. The verifier needs the original sequence exactly as it arrived, including whitespace, escaping, and field order. Two JSON documents can decode to equal maps while having different bytes and therefore different signatures. A Plug body reader may return the body in chunks; a correct gateway accumulates chunks up to a strict limit, retains the resulting binary in private connection state, and makes it available to the verifier. It does not reconstruct JSON afterward and call that the raw body.

Each provider defines a signing grammar. One may sign timestamp + "." + body, another may sign only the body and prefix its header with an algorithm name, and a third may allow multiple active signatures during key rotation. The common abstraction should not be “all providers use the same header.” It should be a behaviour whose callback receives a constrained verification context and returns a normalized decision. Provider modules own header parsing, signed-material construction, accepted algorithms, timestamp extraction, and provider event identifiers. The gateway owns common limits, secret lookup, audit redaction, replay policy, and forwarding.

An HMAC is not encryption and does not hide the payload. It combines a secret key with a cryptographic hash so that someone without the key cannot feasibly produce a matching tag for changed bytes. The expected tag should be computed with OTP :crypto.mac using a configured algorithm and then compared to the decoded supplied tag. Ordinary equality may stop at the first differing byte, exposing timing differences correlated with prefix length. Plug.Crypto.secure_compare/2 is intended for equal-length binaries; the verifier must decode and validate the supplied representation first, reject wrong lengths, and only then perform constant-time comparison. Never “secure compare” unbounded attacker strings.

Key rotation means verification may need a small ordered set of active secrets. Associate each secret with an opaque key identifier and activation interval; do not log its value. If the provider does not transmit a key identifier, try the bounded active set and record which identifier matched. Keep this work bounded because every extra HMAC calculation is attacker-triggerable. An unknown version, unsupported algorithm, malformed hexadecimal/base64 tag, or missing timestamp should be rejected before event parsing.

Freshness uses two controls. First, compare the signed timestamp against a trusted server clock and an explicit tolerance. Reject timestamps too far in the past and usually too far in the future, allowing only documented clock skew. Because the timestamp itself is signed, an attacker cannot change it without invalidating the tag. Second, claim a provider delivery ID in a deduplication store with an expiry longer than the replay window. Timestamp validation prevents an old captured message from being accepted; deduplication prevents the same fresh message from being accepted repeatedly. The claim must be atomic. A read followed by a write permits concurrent duplicates to pass.

The decision order matters. Apply transport and body-size limits first, identify the provider from the route rather than an attacker-controlled body, read the raw bytes once, parse only the provider’s signature metadata, look up an active secret, validate freshness, compute and compare the tag, atomically claim the delivery ID, and only then decode and normalize the event. This keeps expensive or unsafe parsing behind authentication. Whether to claim before or after payload normalization is a policy tradeoff: claiming immediately after authenticity prevents repeated poison payload work, but operators need a recovery mechanism for an authentic event that fails schema validation.

Audit records should describe the decision without becoming a secret warehouse. Record a generated request ID, provider, delivery ID hash or allowed identifier, received time, declared timestamp, matched key ID, payload digest, body size, decision code, and latency. Do not record signing secrets, raw authorization headers, or complete payloads by default. Keep the taxonomy stable: malformed_signature, unsupported_version, unknown_key, stale_timestamp, future_timestamp, signature_mismatch, duplicate_delivery, payload_invalid, and accepted. Stable categories make metrics and incident investigation possible.

Failure handling must be fail-closed for authenticity. A secret-store timeout is not a reason to accept a request. A deduplication-store outage may be handled as 503 retry later, preserving the provider’s retry semantics, rather than as an authentication failure. The gateway should return intentionally boring external responses while retaining rich internal reasons, because detailed cryptographic errors can help attackers probe the system. This project’s core lesson is that security comes from preserving boundaries and sequencing evidence, not from sprinkling a hash function into an HTTP controller.

How this fits on projects

The complete gateway is one authenticity pipeline. Plug establishes the byte-preserving boundary; provider behaviours isolate signing grammars; result tuples carry precise failures; and replay state converts a valid signature into a one-time processing decision.

Definitions & key terms

  • Raw body: Exact request bytes before JSON or form decoding.
  • HMAC: Keyed message authentication code that provides integrity and authenticity.
  • Signed material: Precisely specified bytes supplied to the MAC operation.
  • Constant-time comparison: Comparison designed not to reveal matching-prefix length through timing.
  • Replay window: Maximum accepted age and future skew for a signed message.
  • Delivery ID: Provider identifier used to deduplicate attempts of one logical event.
  • Fail closed: Reject or defer when required verification evidence is unavailable.

Mental model diagram

Untrusted HTTP
     |
     v
[route selects provider] -> [bounded raw-body reader] -> [signature metadata parser]
                                                            |
                                                            v
                          [secret IDs] -> [rebuild signed bytes] -> [HMAC]
                                                                     |
header signature ------------------------------------------------> [secure compare]
                                                                     |
signed timestamp -> [freshness] -> delivery ID -> [atomic claim] ----+
                                                                     |
                                        accepted bytes only ---------v
                                                        [decode + normalize]
                                                                     |
                                             [audit] <--- decision ---> [forward]

How it works (step-by-step, with invariants and failure modes)

  1. Select a provider adapter from a fixed route.
  2. Read at most the configured number of raw bytes.
  3. Parse only signature headers and enforce format/length limits.
  4. Reconstruct the provider-defined signed material.
  5. Compute HMAC for the bounded active key set and compare safely.
  6. Validate signed time against server time.
  7. Atomically claim the delivery ID.
  8. Decode, normalize, audit, and forward the authenticated event.
  9. Invariant: no payload reaches a downstream handler without an accepted verification result.
  10. Failure modes: consumed body, key-store outage, clock skew, duplicate racing requests, oversized input, malformed encoding, and audit-store pressure.

Minimal concrete example

PSEUDOCODE — provider callback contract

verify(context):
  metadata <- parse_signature_headers(context.headers)
  secret_set <- lookup_active_secrets(context.provider, metadata.key_id)
  signed_bytes <- construct_signed_material(metadata.timestamp, context.raw_body)
  matched_key <- compare_expected_macs(secret_set, signed_bytes, metadata.signature)
  assert_fresh(metadata.timestamp, context.received_at, tolerance)
  return accepted(provider_event_id, matched_key.id, digest(context.raw_body))

Common misconceptions

  • “Equivalent decoded JSON produces the same signature.” Signatures cover bytes, not semantic maps.
  • “A valid HMAC prevents replay.” It proves origin and integrity; freshness needs signed time and deduplication.
  • “Hash equality is harmless.” Attacker-controlled tag comparison should use a suitable constant-time primitive.
  • “Logging rejected payloads is always useful.” It can turn observability into a sensitive-data leak.

Check-your-understanding questions

  1. Why must verification occur before JSON decoding?
  2. Why are timestamp validation and delivery-ID deduplication both required?
  3. What should happen when the deduplication store is unavailable?

Check-your-understanding answers

  1. Decoding loses the exact bytes whose signature the provider calculated.
  2. The timestamp rejects old captures; atomic deduplication rejects repeated fresh deliveries.
  3. Fail closed or return a retryable service error; never silently accept without replay protection.

Real-world applications

  • Payment, source-control, identity, and logistics webhook ingestion.
  • A shared authenticity edge in a multi-service platform.
  • Auditable ingress for regulated workflows.

Where you’ll apply it

  • Raw body capture and routing in Sections 3 and 4.
  • Provider behaviour and result taxonomy in Section 5.
  • adversarial replay and timing-safe tests in Section 6.

References

  • Plug documentation: https://hexdocs.pm/plug/Plug.html
  • Plug.Conn: https://hexdocs.pm/plug/Plug.Conn.html
  • Plug.Crypto.secure_compare/2: https://hexdocs.pm/plug_crypto/Plug.Crypto.html#secure_compare/2
  • OTP crypto: https://www.erlang.org/doc/apps/crypto/crypto.html

Key insight

Authenticity is a chain of byte-preserving, time-aware, fail-closed decisions; the HMAC operation is only one link.

Summary

Verify exact bytes through provider-owned signing rules, compare bounded binary tags safely, enforce both freshness and uniqueness, and expose stable decisions without leaking evidence that should remain secret.

Homework/Exercises to practice the concept

  1. Draw the signed-material grammar for three providers and identify what cannot be generalized.
  2. Design a key-rotation timeline in which old and new keys overlap for four hours.
  3. Classify the response when two identical valid requests arrive concurrently.

Solutions to the homework/exercises

  1. Generalize the callback result and context; leave header grammar, timestamp syntax, encoding, and signed bytes in each adapter.
  2. Activate the new key, retain the old key as verification-only during overlap, then remove it; log only opaque key IDs.
  3. Exactly one atomic claim succeeds; the other receives duplicate_delivery while both remain cryptographically authentic.

3. Project Specification

3.1 What You Will Build

Build a small HTTP gateway with three simulated provider routes. Each adapter verifies a distinct signing format and emits one normalized event envelope. The service offers an audit query command for accepted and rejected deliveries and a downstream test sink that receives only authenticated events.

3.2 Functional Requirements

  1. Preserve bounded raw bodies and reject requests larger than the configured maximum.
  2. Support three provider verifier modules behind one documented behaviour.
  3. Support active-key overlap without exposing secrets in state or logs.
  4. Reject malformed, mismatched, stale, future, and duplicate deliveries with stable codes.
  5. Atomically deduplicate delivery IDs and expire claims by policy.
  6. Decode and normalize only after verification.
  7. Produce redacted audit records for every terminal decision.

3.3 Non-Functional Requirements

  • Verification p95 below 25 ms under the documented local load profile.
  • No signing secret or complete signature appears in logs, telemetry, or exceptions.
  • Memory remains bounded for slow and oversized bodies.
  • Provider modules are independently contract-tested.

3.4 Example Usage / Output

$ gatewayctl verify-fixture stripe_like valid-payment.json
request=req_01JY provider=stripe_like delivery=evt_1042
authentic=true fresh=true duplicate=false key=key_2026_q3
normalized_type=payment.settled forwarded=true

$ gatewayctl verify-fixture github_like replayed-push.json
request=req_01JZ provider=github_like delivery=del_778
authentic=true fresh=true duplicate=true
decision=rejected reason=duplicate_delivery forwarded=false

3.5 Data Formats / Schemas / Protocols

Normalized envelope fields: gateway request ID, provider, provider event type, delivery ID, occurred-at time, payload digest, schema version, and provider-specific data. The audit schema stores decision metadata and never the secret.

3.6 Edge Cases

  • A body delivered in multiple read chunks.
  • A signature header containing several versioned tags.
  • Equal valid signatures from two active rotation keys.
  • Clock skew exactly at the tolerance boundary.
  • Two concurrent attempts for one delivery ID.
  • An authentic body containing invalid JSON.
  • Deduplication persistence failing after authenticity succeeds.

3.7 Real World Outcome

3.7.1 How to Run (Copy/Paste)

$ mix deps.get
$ mix test
$ mix run priv/demo_gateway.exs
$ gatewayctl audit --last 5

3.7.2 Golden Path Demo (Deterministic)

The fixture runner calculates provider signatures from fixed secrets and timestamps anchored to a fake clock. One event is accepted once, its normalized envelope reaches the test sink, and an immediate repeat is rejected as a duplicate.

3.7.3 Exact terminal transcript

$ mix run priv/demo_gateway.exs
gateway listening=127.0.0.1:4101 max_body=1048576 replay_window=300s
POST /hooks/acme status=202 request=req_demo_1 decision=accepted
POST /hooks/acme status=409 request=req_demo_2 decision=duplicate_delivery
POST /hooks/acme status=401 request=req_demo_3 decision=signature_mismatch
sink accepted=1 rejected=2 leaked_secrets=0
DEMO PASS

4. Solution Architecture

4.1 High-Level Design

Provider -> Plug Router -> Raw Body Reader -> Provider Verifier
                                               |       |
                                      Secret Store   Fake/real clock
                                               |
                                    Replay Claim Store
                                               |
                         +---------------------+------------------+
                         v                                        v
                    Audit Writer                         Normalizer -> Sink

4.2 Key Components

Component Responsibility Key Decision
Raw Body Reader Bound and retain exact request bytes Stop reading and reject at limit
Verifier Behaviour Define adapter contract Common result, provider-specific grammar
Secret Resolver Return bounded active key metadata Opaque IDs and rotation intervals
Replay Store Atomically claim delivery IDs Unique claim with expiry
Normalizer Decode accepted bytes into envelope Version normalized schemas
Audit Writer Persist redacted decisions Stable reason taxonomy

4.3 Data Structures (No Full Code)

  • Verification context struct containing provider, headers, raw body, receipt time, and request ID.
  • Verification result tagged as accepted or rejected with a closed reason set.
  • Secret descriptor containing key ID and activation interval, with secret bytes confined to the resolver boundary.
  • Replay claim keyed by {provider, delivery_id}.

4.4 Algorithm Overview

Verification is O(K + B), where K is the small active-key count and B the bounded body size. Replay lookup is expected O(1) with a uniqueness constraint. JSON parsing is absent from rejected traffic by design.

5. Implementation Guide

5.1 Development Environment Setup

Use an actively supported Elixir/OTP pair, Plug, Plug.Crypto, a minimal HTTP adapter, and a persistence option capable of atomic unique claims. Tests require a fake clock and fixture signer.

5.2 Project Structure

webhook_gateway/
├── lib/webhook_gateway/
│   ├── router.ex
│   ├── raw_body_reader.ex
│   ├── verifier.ex
│   ├── providers/{acme,forge,payments}.ex
│   ├── replay_store.ex
│   ├── normalizer.ex
│   └── audit.ex
├── test/contract/
├── test/fixtures/
└── priv/demo_gateway.exs

5.3 The Core Question You’re Answering

“How can an Elixir HTTP boundary prove that a webhook is authentic, fresh, and unique before any business handler trusts it?”

5.4 Concepts You Must Understand First

  1. Raw-body lifecycle — Programming Phoenix 1.4, Chapter 10, “Plug”.
  2. Pattern matching and with — Elixir in Action, 3rd Edition, Chapters 3–4.
  3. MACs and timing-safe verification — Serious Cryptography, 2nd Edition, Chapter 7, “Keyed Hashing”.
  4. Behaviours — Designing for Scalability with Erlang/OTP, Chapter 2, OTP design principles.

5.5 Questions to Guide Your Design

  1. Which bytes are authenticated, and where could they be changed or consumed?
  2. Which rules belong to the gateway versus a provider adapter?
  3. What claim operation remains correct under two simultaneous requests?
  4. Which audit fields help incident response without increasing data exposure?

5.6 Thinking Exercise

Trace an authentic event through these failures: key rotation overlap, replay-store timeout, invalid JSON, and sink outage. Decide the HTTP response, audit reason, retry behavior, and whether the delivery ID remains claimed in each case.

5.7 The Interview Questions They’ll Ask

  1. Why is decoded JSON unsuitable for webhook signature verification?
  2. What does constant-time comparison protect, and what preconditions does it require?
  3. How would you rotate webhook secrets without downtime?
  4. How do you prevent concurrent replays?
  5. Why might an authentic event still be rejected?
  6. What should be included in a safe audit record?

5.8 Hints in Layers

Hint 1: Establish the boundary — Write a body-reader test that proves chunks reassemble into the original fixture bytes.

Hint 2: One provider end-to-end — Implement one signing grammar and a fake signer before adding the behaviour.

Hint 3: Normalize decisions, not algorithms — Provider callbacks should return the same accepted/rejected shape while retaining different header rules.

Hint 4: Race the claim — Release many concurrent tasks with one delivery ID and assert exactly one success.

5.9 Books That Will Help

Topic Book Chapter
Plug boundaries Programming Phoenix 1.4 Ch. 10, Plug
Elixir control flow Elixir in Action, 3rd Edition Ch. 3–4
Message authentication Serious Cryptography, 2nd Edition Ch. 7
Operational failure Release It!, 2nd Edition Ch. 4, Stability Patterns

5.10 Implementation Phases

Phase 1: Byte and Crypto Contract (3-4 hours)

  • Build raw-body fixtures and one verifier.
  • Establish stable error atoms and redacted evidence.

Phase 2: Provider Adapters and Replay State (5-8 hours)

  • Extract the behaviour and add two distinct signing grammars.
  • Add fake-clock freshness and atomic claims.

Phase 3: Gateway Operations (4-6 hours)

  • Add normalization, audit queries, telemetry, load limits, and deterministic demo.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Provider selection Route, header, body Fixed route Avoid trusting unauthenticated body data
Replay response Accept, conflict, service error Conflict for duplicate; service error for store outage Preserve semantics
Audit payload Full, sampled, digest only Digest and metadata by default Reduce sensitive retention
Key rotation Replace instantly, overlap Bounded overlap Prevent delivery loss

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Contract Every adapter obeys common result semantics valid, malformed, mismatched
Cryptographic fixture Match provider signing grammar fixed official-style vectors
Concurrency Prove atomic replay claims 50 simultaneous duplicates
Boundary Bound body and header work chunking, oversized body, huge tag
Redaction Prevent evidence leakage capture logs and telemetry
Integration Prove accepted-only forwarding real Plug request to test sink

6.2 Critical Test Cases

  1. Raw bytes containing harmless JSON whitespace validate only against their own signature.
  2. Valid tag with stale signed timestamp is rejected.
  3. Valid tag and time arriving concurrently 50 times yields one forwarded event.
  4. Wrong-length decoded tag never reaches secure comparison.
  5. Old and new rotation keys both verify during overlap; only new verifies afterward.
  6. Authentic invalid JSON is audited as payload_invalid and never crashes the request process.
  7. Secret-store and replay-store outages fail closed with distinct retry behavior.
  8. Search captured logs for every fixture secret and full signature; find zero matches.

6.3 Test Data

Maintain per-provider valid, tampered-body, tampered-time, malformed-header, rotation, and duplicate fixtures. Use a deterministic clock and fixed non-production secrets.

6.4 Definition of Done

  • Three materially different provider signing grammars pass a shared contract suite.
  • Exact request bytes are authenticated before payload decoding.
  • Timestamp and atomic delivery-ID replay defenses are independently tested.
  • Oversized bodies and malformed signatures are bounded and rejected.
  • Key rotation overlap is observable by opaque key ID.
  • Only accepted events reach the downstream sink.
  • Audit records contain stable reasons and no fixture secret or full signature.
  • Deterministic CLI demo ends with DEMO PASS.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Problem: “Official signatures never match.”

  • Why: The JSON parser consumed or reconstructed the body.
  • Fix: Capture exact bounded bytes in the Plug body-reader boundary.
  • Quick test: Hash captured bytes and fixture bytes; digests must match.

Problem: “Duplicates occasionally forward twice.”

  • Why: Deduplication uses separate lookup and insert operations.
  • Fix: Use one atomic unique claim.
  • Quick test: Start 50 requests behind a barrier and count sink events.

Problem: “Logs expose secrets during errors.”

  • Why: Context or request headers are inspected wholesale.
  • Fix: Construct allow-listed audit fields and custom inspected representations.
  • Quick test: Scan captured output for all fixture credentials.

7.2 Debugging Strategies

  • Log the signed-material digest and length, never the secret or complete signature.
  • Compare fake-signer and verifier grammars byte-by-byte in a local fixture tool.
  • Attach low-cardinality telemetry to decision category, provider, and latency.
  • Inspect body-reader chunk counts when only large requests fail.

7.3 Performance Traps

  • Trying an unbounded historical key set for every request.
  • Parsing large JSON before authentication.
  • Synchronously storing full payloads in the request path.
  • High-cardinality telemetry keyed by delivery ID.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a command that generates redacted local test fixtures.
  • Add configurable future-skew and past-age limits per provider.

8.2 Intermediate Extensions

  • Add asynchronous quarantine for authentic but schema-invalid events.
  • Add secret rotation reports and expiry alerts.
  • Add a canonical CloudEvents-style output adapter.

8.3 Advanced Extensions

  • Back replay claims with a database uniqueness constraint across nodes.
  • Add asymmetric-signature provider verification with certificate rotation.
  • Formally specify the decision state machine and generate adversarial sequences.

9. Real-World Connections

9.1 Industry Applications

Payment processors, Git hosting, identity platforms, and logistics vendors all use webhooks to cross trust boundaries. A shared gateway centralizes verification while allowing teams to consume normalized authenticated events.

  • Plug and Plug.Crypto for request boundaries and safe comparison.
  • Phoenix endpoints for production HTTP integration.
  • Broadway or Oban as possible downstream consumers after authentication.

9.3 Interview Relevance and Scope Boundary

This project demonstrates secure boundary design, binary handling, behaviours, and explicit error modeling. Unlike Project 2, it is not a rate limiter or circuit breaker. Unlike Projects 5 and 13, it does not teach event-flow backpressure or federation. Its success criterion is trustworthy ingress evidence.

10. Resources

10.1 Essential Reading

  • Plug: https://hexdocs.pm/plug/Plug.html
  • Plug.Conn: https://hexdocs.pm/plug/Plug.Conn.html
  • Plug.Crypto: https://hexdocs.pm/plug_crypto/Plug.Crypto.html
  • OTP crypto: https://www.erlang.org/doc/apps/crypto/crypto.html
  • Elixir behaviours: https://hexdocs.pm/elixir/typespecs.html#behaviours

10.2 Standards and Further Study

  • RFC 2104, HMAC: https://www.rfc-editor.org/rfc/rfc2104
  • RFC 8941, Structured Field Values for HTTP: https://www.rfc-editor.org/rfc/rfc8941
  • Serious Cryptography, 2nd Edition, Chapter 7.
  • Release It!, 2nd Edition, Chapters 4 and 13 for failure and observability thinking.