Project 17: Concurrent Link and TLS Expiry Auditor

Audit a finite URL inventory with bounded Task concurrency and turn HTTP, DNS, transport, timeout, and certificate outcomes into deterministic operational evidence.

Quick Reference

Attribute	Value
Difficulty	Level 2 - Intermediate/Developer
Suggested Seniority	Mid-level
Time Estimate	10-16 hours
Main Programming Language	Elixir
Alternative Programming Languages	Erlang
Coolness Level	Level 3 - Genuinely Clever
Business Potential	Level 2 - Micro-SaaS/Pro Tool
Prerequisites	Processes, result tuples, HTTP/TLS basics
Key Topics	Task.async_stream, bounded concurrency, timeouts, deterministic fan-in, Finch, :ssl, :public_key

1. Learning Objectives

By completing this project, you will:

Use Task.async_stream for finite fan-out/fan-in while bounding concurrency and handling every result shape.
Separate per-request deadlines, stream task timeouts, redirect policy, and whole-run budgets.
Normalize HTTP and TLS outcomes into a stable audit result without depending on completion order.
Decode certificate validity and identity metadata through Erlang/OTP interoperability.
Measure throughput and failure behavior while avoiding an accidental denial of service against audited hosts.

2. All Theory Needed (Per-Concept Breakdown)

Bounded Finite Concurrency and Transport Evidence

Fundamentals

Task represents work that can execute in another BEAM process and return a result. Task.async_stream applies that model to a finite enumerable, running only a bounded number of tasks at once and yielding success or exit outcomes. This differs from GenStage: the input inventory is known, work ends, and the problem is fan-out/fan-in rather than a long-lived demand-driven pipeline. Network auditing adds several independent failure boundaries. DNS may fail, TCP connect may fail, TLS negotiation may fail, certificate validation may fail, HTTP may time out, redirects may loop, and a task may exit. A robust auditor preserves these distinctions. It also normalizes completion into deterministic, input-correlated records because concurrent tasks can finish in any order. TLS certificate dates and names are evidence, not proof of a healthy application; the final result should expose transport and HTTP facts separately.

Deep Dive into the concept

Concurrency is a capacity decision, not a synonym for speed. A sequential auditor underuses available network wait time, while one task per URL can exhaust file descriptors, connection pools, memory, DNS capacity, or the target service. Task.async_stream provides a maximum-concurrency control and a stream of outcomes. The correct value depends on the HTTP pool, host distribution, expected latency, local resource limits, and ethical rate policy. Start low, measure, and make the limit visible in the run manifest.

Each inventory item needs a stable input id before fan-out. A worker receives the normalized URL, redirect policy, deadlines, and current observation time. It returns one AuditResult value rather than throwing expected network conditions. The parent consumes every task outcome and pairs exits with the item id. If output is produced in completion order, identical runs differ; sort final records by inventory order or canonical URL.

Timeouts are layered. A connect timeout limits establishing transport. A receive/request timeout limits response progress. Task.async_stream’s timeout limits how long the parent waits for a task. A whole-run budget may stop launching more work. These values should not be collapsed into “timeout” because diagnosis depends on the boundary. The task timeout should normally exceed internal network deadlines enough to allow cleanup and result construction.

Cancellation semantics matter. When the stream timeout is exceeded, configure the desired on-timeout behavior and document whether the worker is terminated. Do not link unsupervised tasks from arbitrary long-lived callers in production code; a Task.Supervisor provides an ownership boundary. For this finite CLI, the application supervision tree can own Finch and a Task.Supervisor, while the command orchestrator consumes results.

HTTP results require explicit redirect policy. Record each hop, status, location, latency, and effective URL. Restrict maximum hops and detect loops. Decide whether HEAD is trustworthy for the inventory; many servers treat it differently, so GET with a bounded body or configurable method may be more representative. Avoid downloading huge payloads when the goal is link status.

TLS has two related but distinct questions: did the connection validate under the configured trust store and hostname rules, and what metadata did the peer certificate contain? Finch or its transport stack may validate HTTPS during the request. Direct “:ssl” or transport hooks can expose the peer certificate, and “:public_key” can decode its ASN.1 structure. Convert notBefore and notAfter into comparable instants, compute days remaining relative to the captured run time, and record subject alternative names, issuer, serial fingerprint, and validation outcome. Avoid parsing human-readable certificate strings when structured decoded fields exist.

Certificate hostname validation must consider subject alternative names; common name fallback is legacy-dependent. The auditor should report the validation performed by the transport rather than invent a second incomplete verifier. If the project includes raw TLS probing for metadata, use OTP verification options and trusted certificates deliberately. Never default to verify-none and label the endpoint healthy.

An HTTPS endpoint may have a valid certificate but return HTTP 500; it may return HTTP 200 with a certificate expiring tomorrow; an HTTP URL has no TLS evidence; a DNS failure has neither. Model independent status dimensions: resolution/transport category, TLS validation and expiry category, HTTP category, redirect chain, timings, and safe diagnostic. A top-level severity policy maps evidence to OK, warning, or failure without discarding detail.

Fairness across hosts is a product concern. A flat inventory with many URLs from one host can monopolize concurrency. A baseline can impose per-host caps or at least sort/round-robin hosts. Connection pooling improves reuse but also means measured TLS handshake data may not occur for every URL. Decide whether certificates are probed once per origin and shared as origin evidence or each URL is independently connected. The former is efficient and conceptually correct when certificate identity is origin-level.

The invariants are: every normalized input id yields exactly one terminal result; concurrency never exceeds the configured bound; redirects never exceed policy; expected network failures become data; validation mode is explicit; certificate expiry uses one captured observation time; result ordering is deterministic; sensitive URL credentials are rejected or redacted. Failure modes include task exits, pool exhaustion, timeout confusion, redirect loops, unbounded bodies, verify-none TLS, malformed certificates, IPv6/IPv4 differences, connection reuse hiding handshake timing, and overloading one host.

How this fits in the project

Task.async_stream coordinates the finite audit, Finch owns pooled HTTP connections, and OTP SSL/public_key provide transport and certificate evidence. The aggregator converts out-of-order completions into one stable report.

Definitions & key terms

Fan-out/fan-in: Dispatching independent work concurrently, then collecting all outcomes.
Maximum concurrency: Upper bound on simultaneously executing tasks.
Task exit outcome: Stream item indicating a worker terminated rather than returned.
Peer certificate: Certificate presented by the remote TLS endpoint.
SAN: Subject Alternative Name entries used for identity validation.
Origin: Scheme, host, and port combination sharing transport policy.

Mental model diagram

urls.txt -> normalize/id -> group origins -> bounded task stream
                                               |
                        +----------------------+------------------+
                        v                      v                  v
                     worker 1               worker 2           worker N
                  DNS/TCP/TLS/HTTP       DNS/TCP/TLS/HTTP   DNS/TCP/TLS/HTTP
                        |                      |                  |
                        +----------- tagged evidence ------------+
                                               |
                                         deterministic fan-in
                                               |
                                report.json + summary + exit status

How it works

Parse and validate URLs, rejecting embedded credentials and unsupported schemes.
Assign stable input ids and capture one observation time.
Optionally probe TLS once per HTTPS origin.
Run URL checks through Task.async_stream with explicit max concurrency and timeout.
Map returned, exited, and timed-out tasks into AuditResult records.
Enforce redirect and body policies inside each worker.
Sort by input id, calculate severities, and render reports.
Assert one result per input and publish run configuration/metrics.

Minimal concrete example

input 17: https://example.invalid/docs

worker outcomes:
  DNS error -> transport=dns_failure, tls=not_attempted, http=not_attempted
  TLS verify error -> transport=tls_failure, tls=invalid_identity
  HTTP 301 -> follow if hops remain; record hop
  HTTP 200 + expiry 12 days -> transport=ok, tls=warning, http=ok

task exit:
  convert to category=worker_exit with safe reason and input id

Common misconceptions

Task.async_stream does not create backpressure semantics equivalent to GenStage.
A successful HTTP response does not imply the certificate has a safe remaining lifetime.
A decoded certificate is not necessarily a validated certificate.
Increasing max concurrency can reduce reliability and throughput.

Check-your-understanding questions

Why assign ids before starting tasks?
Why separate task timeout from HTTP receive timeout?
Why might TLS evidence be cached per origin?

Check-your-understanding answers

Completion order is nondeterministic; stable ids preserve correlation and report order.
They represent different boundaries and require different diagnosis and cleanup.
Certificates and pooled connections are origin-level; repeated probes add cost without new URL-level meaning.

Real-world applications

Documentation link checks, certificate-expiry monitoring, vendor endpoint inventories, migration readiness, uptime prechecks, and CI link validation use bounded concurrent audits.

Where you will apply it

Apply it to URL dispatch, timeout handling, redirect tracking, origin TLS probes, aggregation, and severity policy. Scope distinction: Project 5 is a long-lived GenStage backpressure pipeline; this is a finite Task fan-out/fan-in audit. Project 10 observes a BEAM system; this tool observes external endpoints.

References

Key insight

Useful concurrency is bounded, observable, and lossless: every input receives one explicit result even when the network or worker fails.

Summary

The project teaches process-level concurrency through a real network workload while preserving determinism, ethical capacity limits, layered timeouts, and rigorous TLS/HTTP evidence.

Homework/Exercises

Draw timeout ownership from orchestrator to task to HTTP transport.
Design result shapes for DNS failure, TLS expiry warning, redirect loop, and task exit.
Explain why one host with 5,000 URLs may need a separate cap.

Solutions

Whole-run policy encloses task timeout, which encloses network deadlines; each layer produces a distinct reason.
Keep transport, TLS, HTTP, redirect, timing, and diagnostic fields independent.
A global cap alone can still direct every active task to one host and cause self-inflicted overload.

3. Project Specification

3.1 What You Will Build

Build “url-audit”, a CLI that reads URLs, checks them with bounded concurrency, records redirect chains and latency, and reports TLS validation, subject/issuer, SANs, fingerprint, expiry, and days remaining for HTTPS origins.

Scope boundary: This is a one-shot finite audit. Do not build a scheduler, monitoring daemon, LiveView dashboard, GenStage pipeline, distributed crawler, or general-purpose web spider.

3.2 Functional Requirements

Parse HTTP/HTTPS inventory with stable ids and optional labels.
Configure global concurrency, optional per-host cap, deadlines, redirect maximum, and expiry thresholds.
Return structured categories for DNS, connect, TLS, timeout, HTTP, redirect, and worker failures.
Capture certificate metadata only through validation-aware transport.
Preserve redirect hops and final effective URL.
Emit deterministic JSON/CSV and terminal summaries.

3.3 Non-Functional Requirements

Never exceed configured concurrency.
Reject credential-bearing URLs and redact query strings when configured.
Limit response body bytes.
Publish run time, trust/validation mode, timeout policy, and tool version.
Remain responsive when many endpoints hang.

3.4 Example Usage / Output

$ mix url_audit fixtures/urls.txt --max-concurrency 8 --timeout 5000 --warn-days 30
inputs=12 completed=12 elapsed_ms=1842 max_in_flight=8
ok=7 warnings=2 failures=3
WARN https://docs.example expiry_days=12 status=200 latency_ms=91
FAIL https://old.example category=tls_invalid reason=hostname_mismatch
FAIL https://loop.example category=redirect_loop hops=5
reports=tmp/url-audit.{json,csv}

3.5 Data Formats / Schemas / Protocols

Input records contain id, URL, and optional label. AuditResult contains canonical/effective URLs, transport category, TLS evidence, HTTP status, redirect hops, timings, severity, safe reason, and attempt metadata. Run metadata contains captured time, limits, trust mode, and counts.

3.6 Edge Cases

Duplicate URLs, Internationalized Domain Names, IPv6 literals, nondefault ports.
Redirect from HTTPS to HTTP, loops, missing Location, oversized bodies.
Expired/not-yet-valid certificates, hostname mismatch, incomplete chain.
DNS delay, connect refusal, partial response, task exit, inventory cancellation.
Pooled origin re-use and one host dominating the inventory.

3.7 Real World Outcome

3.7.1 How to Run

Run against deterministic local HTTP/TLS fixtures first, including slow, redirecting, and invalid-certificate endpoints. Use public endpoints only with conservative concurrency and permission.

3.7.2 Golden Path Demo

The fixture inventory produces OK, expiring-soon, 404, redirect-loop, hostname mismatch, connection refusal, and timeout outcomes. Exactly one result exists per line despite out-of-order completion.

3.7.3 Exact Terminal Transcript

$ mix url_audit test/fixtures/urls.txt --max-concurrency 3 --timeout 1000
inputs=7 completed=7 max_in_flight=3
ok=2 warnings=1 failures=4
by_category=http_404:1,tls_expiring:1,tls_invalid:1,timeout:1,redirect_loop:1
$ echo $?
2

4. Solution Architecture

4.1 High-Level Design

CLI -> inventory -> policy -> Task.async_stream -> URL workers
                          |                         |
                          |                   Finch + TLS probe
                          |                         |
                          +------ aggregator <------+
                                      |
                              severity + renderers

4.2 Key Components

Inventory parser/URL normalizer
Audit policy and safe redaction
Supervised HTTP pool and task supervisor
URL worker and redirect state
TLS origin probe/decoder
Deterministic aggregator and report renderers

4.3 Data Structures

Use Input, Policy, RedirectHop, TLSEvidence, AuditResult, Diagnostic, and RunSummary structs. Keep timings monotonic during measurement and convert only report timestamps to wall-clock values.

4.4 Algorithm Overview

Normalize inventory, capture time, construct origin cache, execute bounded checks, normalize all Task outcomes, enforce one-result-per-id, sort, calculate severity, and render atomic reports.

5. Implementation Guide

5.1 Development Environment Setup

Use supported Elixir/OTP, Finch, and local deterministic HTTP/TLS fixtures. Confirm the OS trust store and OTP CA configuration; never weaken verification to make tests pass.

5.2 Project Structure

lib/url_audit/
  inventory
  policy
  worker
  tls_probe
  result
  aggregate
  cli
test/support/servers/

5.3 The Core Question You’re Answering

How do I make finite concurrent network work faster without losing capacity control, failure detail, or deterministic results?

5.4 Concepts You Must Understand First

Understand Task results/exits, async_stream options, supervision ownership, monotonic timing, URL origins, TLS validation versus decoding, and deterministic aggregation.

5.5 Questions to Guide Your Design

What owns and terminates workers?
Which timeout fired, and how is it represented?
Is certificate evidence gathered per URL or per origin?
How do global and per-host limits interact?
Which outcomes make CI fail versus warn?

5.6 Thinking Exercise

Schedule ten URLs with a concurrency of three, where the first task hangs and later tasks finish quickly. Draw active tasks and emitted results over time. Then add six URLs on one host and propose a fairness policy.

5.7 The Interview Questions They’ll Ask

What does Task.async_stream return when a task exits?
How does ordered output differ from completion order?
Why do bounded tasks differ from GenStage backpressure?
How would you distinguish connect, receive, and task timeouts?
What is the difference between decoding and validating a certificate?
How do you prevent a link checker from overloading a target?

5.8 Hints in Layers

Hint 1: Make a sequential worker return one complete AuditResult first.

Hint 2: Wrap the existing worker with async_stream rather than putting aggregation inside tasks.

Hint 3: Convert every “ok” and “exit” stream item to the same result schema.

Hint 4: Instrument in-flight count and per-category latency before tuning concurrency.

5.9 Books That Will Help

Topic	Book	Chapter
Task concurrency	“Elixir in Action” by Saša Jurić	Ch. 9, Isolating Error Effects; Ch. 10, Beyond GenServer
Resilient network calls	“Release It!, 2nd Edition” by Michael Nygard	Ch. 4, Stability Antipatterns; Ch. 5, Stability Patterns
Service boundaries	“Building Microservices, 2nd Edition” by Sam Newman	Ch. 9, Communication Styles; Ch. 12, Resiliency

5.10 Implementation Phases

Phase 1: Foundation (3-5 hours)

Build sequential checks, URL normalization, layered outcomes, local fixtures, and deterministic rendering.

Phase 2: Core Functionality (4-6 hours)

Add supervised Task.async_stream, limits/timeouts, redirect policy, TLS decoding, and origin evidence.

Phase 3: Polish & Edge Cases (3-5 hours)

Add host fairness, redaction, atomic reports, CI exit policy, load measurements, and help documentation.

5.11 Key Implementation Decisions

Document concurrency, host caps, timeout layers, redirect/downgrade policy, HTTP method/body limit, trust store, origin caching, severity thresholds, redaction, and exit codes.

6. Testing Strategy

6.1 Test Categories

URL/inventory unit tests
Deterministic local HTTP/TLS integration tests
Task exit/timeout normalization tests
Concurrency-bound tests using instrumented fixtures
Redirect state-machine tests
Certificate date/identity decoding tests
End-to-end report and exit-code tests

6.2 Critical Test Cases

Active worker count never exceeds the limit.
A hung request times out without blocking later work indefinitely.
Task exit still yields one correlated result.
Redirect loop stops at policy and records hops.
Invalid certificate is not accepted as healthy.
Out-of-order completion produces input-ordered reports.
Query secrets are redacted.

6.3 Test Data

Use local endpoints for 200, 404, redirects, loops, delayed response, dropped connection, expiring certificate, hostname mismatch, and untrusted issuer. Freeze observation time in certificate-policy tests.

6.4 Definition of Done

Every valid input has exactly one terminal result.
Concurrency and timeout limits are measured and enforced.
DNS, transport, TLS, HTTP, redirect, and worker failures are distinct.
Certificate validation mode and observation time are explicit.
Reports are deterministic despite out-of-order execution.
Local fixtures cover slow and invalid endpoints.
P17 is explicitly distinguished from P05’s long-lived GenStage pipeline.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Caller exits: Tasks are linked without an ownership plan. Use the supervised boundary and normalize exits.

Everything says timeout: Inner and outer deadlines share one reason. Tag each boundary.

TLS looks healthy: Verification was disabled or only dates were decoded. Preserve transport validation.

Output changes order: Results were rendered as tasks completed. Sort by stable input id.

7.2 Debugging Strategies

Log safe lifecycle events keyed by input id: queued, started, DNS/connect/TLS/HTTP boundary, finished. Track in-flight and per-host counts. Reproduce certificate problems with local fixtures and structured decoded fields.

7.3 Performance Traps

Unlimited concurrency, one HTTP pool bottleneck, repeated TLS probes per URL, downloading full bodies, serial DNS work, and storing verbose bodies dominate performance. Measure boundary timings before tuning.

8. Extensions & Challenges

8.1 Beginner Extensions

Add allowlisted status codes, CSV labels, Markdown output, and configurable expiry thresholds.

8.2 Intermediate Extensions

Add per-host caps, retry only idempotent transient failures, origin-level certificate caching, and machine-readable CI annotations.

8.3 Advanced Extensions

Add IPv4/IPv6 comparison, OCSP evidence with explicit caveats, distributed inventories, and adaptive concurrency—without turning the baseline into GenStage.

9. Real-World Connections

9.1 Industry Applications

CI documentation checks, certificate renewal inventories, vendor endpoint reviews, incident preflight tools, and migration audits all need bounded external probes.

Finch demonstrates pooled HTTP architecture; OTP SSL/public_key expose transport primitives. Existing link checkers offer report ideas, but the learning target is Elixir concurrency and evidence modeling.

9.3 Interview Relevance

The project provides concrete discussion of Tasks, failure normalization, supervision, timeouts, concurrency limits, TLS boundaries, determinism, and responsible external I/O.