Project 4: Mutual TLS (mTLS) Mesh - Identity at the Wire
Project 4: Mutual TLS (mTLS) Mesh - Identity at the Wire
Build a complete mTLS infrastructure where services authenticate each other cryptographically, eliminating network-based trust entirely.
Quick Reference
| Attribute | Value |
|---|---|
| Difficulty | Level 3: Advanced |
| Time Estimate | 1 Week |
| Language | Go (Alternatives: Python, Rust) |
| Prerequisites | Basic TLS understanding, Go programming, command-line proficiency |
| Key Topics | PKI, X.509 Certificates, Certificate Authorities, TLS Handshake, SPIFFE |
| Knowledge Area | Cryptography / Distributed Systems |
| Tools | OpenSSL, PKI concepts, SPIFFE |
| Main Book | “Zero Trust Networks” by Gilman & Barth |
1. Learning Objectives
By completing this project, you will:
- Master PKI fundamentals: Build and operate your own Certificate Authority from scratch
- Understand X.509 certificates deeply: Know every field and its security implications
- Implement mutual authentication: Both client proves identity to server AND server proves identity to client
- Automate certificate lifecycle: Issue, rotate, and revoke certificates programmatically
- Design for Zero Trust: Eliminate network-based trust through cryptographic identity
- Debug TLS issues: Diagnose certificate chain problems, SAN mismatches, and expiry issues
- Apply SPIFFE concepts: Understand workload identity in modern cloud-native environments
2. Theoretical Foundation
2.1 Why mTLS is the Gold Standard for Zero Trust
In traditional TLS (what you use when visiting https://example.com), only the server proves its identity. The client verifies “I’m talking to the real example.com” but the server has no cryptographic proof of who the client is.
Mutual TLS (mTLS) adds client certificates, creating bidirectional authentication:
┌─────────────────────────────────────────────────────────────────────────┐
│ Standard TLS vs Mutual TLS │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ STANDARD TLS (One-Way): │
│ ┌────────┐ ┌────────┐ │
│ │ Client │ ──── "Who are you?" ───────▶ │ Server │ │
│ │ │ ◀─── Server Certificate ──── │ │ │
│ │ │ (Proves server ID) │ │ │
│ └────────┘ └────────┘ │
│ ✓ Client knows server is legitimate │
│ ✗ Server has NO IDEA who client is │
│ │
│ MUTUAL TLS (Two-Way): │
│ ┌────────┐ ┌────────┐ │
│ │ Client │ ──── "Who are you?" ───────▶ │ Server │ │
│ │ │ ◀─── Server Certificate ──── │ │ │
│ │ │ ──── Client Certificate ───▶ │ │ │
│ │ │ (Proves client ID) │ │ │
│ └────────┘ └────────┘ │
│ ✓ Client knows server is legitimate │
│ ✓ Server knows client is legitimate │
│ ✓ Both identities are CRYPTOGRAPHICALLY proven │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Why this matters for Zero Trust:
- An attacker on the network cannot impersonate either party
- Traffic cannot be sniffed (encrypted with session keys derived from handshake)
- No reliance on network position - identity is in the certificate, not the IP address
- Every connection is authenticated, regardless of source network
2.2 Public Key Infrastructure (PKI) Fundamentals
PKI is the framework that makes certificate-based authentication possible. At its core, it solves the key distribution problem: How do you securely share public keys with entities you’ve never met?
┌─────────────────────────────────────────────────────────────────────────┐
│ PKI Trust Hierarchy │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────┐ │
│ │ Root CA │ │
│ │ (Self-Signed) │ │
│ │ HIGHLY PROTECTED │ │
│ │ (Offline/HSM) │ │
│ └──────────┬──────────┘ │
│ │ │
│ Signs certificates │
│ │ │
│ ┌──────────────────────┼──────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Intermediate CA │ │ Intermediate CA │ │ Intermediate CA │ │
│ │ (Services) │ │ (Users) │ │ (Devices) │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ payment-service │ │ alice@corp.com │ │ laptop-001 │ │
│ │ certificate │ │ certificate │ │ certificate │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
│ Certificate Chain for payment-service: │
│ [payment-service cert] → [Intermediate CA] → [Root CA] │
│ │
│ Verification: "I trust Root CA. Root CA signed Intermediate. │
│ Intermediate signed payment-service. Therefore, │
│ I trust payment-service." │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Key PKI Concepts:
| Concept | Definition | Security Implication |
|---|---|---|
| Root CA | The ultimate trust anchor, self-signed | Compromise = entire PKI compromised |
| Intermediate CA | Signed by Root, signs end-entity certs | Limits blast radius if compromised |
| End-Entity Cert | The actual service/user certificate | What’s presented during handshake |
| Certificate Chain | Path from end-entity to trusted root | Must be complete and valid |
| Trust Store | Collection of trusted root certificates | What CA certs your system trusts |
2.3 X.509 Certificate Structure
X.509 is the standard format for public key certificates. Understanding its structure is essential for debugging TLS issues:
┌─────────────────────────────────────────────────────────────────────────┐
│ X.509 Certificate Structure │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ TBSCertificate (To Be Signed - the actual certificate data) │ │
│ ├───────────────────────────────────────────────────────────────────┤ │
│ │ │ │
│ │ Version: 3 (v3) │ │
│ │ Serial Number: Unique ID (e.g., 17:32:A4:...) │ │
│ │ Signature Algorithm: sha256WithRSAEncryption │ │
│ │ │ │
│ │ Issuer: CN=My Root CA, O=MyOrg │ │
│ │ (WHO signed this cert) │ │
│ │ │ │
│ │ Validity: │ │
│ │ Not Before: 2024-01-01 00:00:00 UTC │ │
│ │ Not After: 2024-01-02 00:00:00 UTC ← SHORT-LIVED! │ │
│ │ │ │
│ │ Subject: CN=payment-service │ │
│ │ (WHO this cert identifies) │ │
│ │ │ │
│ │ Subject Public Key: <RSA 2048-bit public key> │ │
│ │ │ │
│ │ Extensions (X.509 v3): │ │
│ │ ┌─────────────────────────────────────────────────────────────┐│ │
│ │ │ Subject Alternative Name (SAN): ← CRITICAL FOR mTLS ││ │
│ │ │ DNS: payment-service.prod.local ││ │
│ │ │ DNS: payment.internal ││ │
│ │ │ URI: spiffe://trust-domain/ns/prod/sa/payment ││ │
│ │ │ IP: 10.0.1.50 ││ │
│ │ ├─────────────────────────────────────────────────────────────┤│ │
│ │ │ Key Usage: ││ │
│ │ │ digitalSignature, keyEncipherment ││ │
│ │ ├─────────────────────────────────────────────────────────────┤│ │
│ │ │ Extended Key Usage: ││ │
│ │ │ TLS Web Server Authentication (for server certs) ││ │
│ │ │ TLS Web Client Authentication (for client certs) ││ │
│ │ ├─────────────────────────────────────────────────────────────┤│ │
│ │ │ Basic Constraints: ││ │
│ │ │ CA: FALSE (this is an end-entity, not a CA) ││ │
│ │ └─────────────────────────────────────────────────────────────┘│ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │
│ Signature Algorithm: sha256WithRSAEncryption │
│ Signature: <CA's signature over TBSCertificate> │
│ (This proves the CA issued this cert) │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Critical Fields for mTLS:
| Field | Purpose | Common Issues |
|---|---|---|
| Subject Alternative Name (SAN) | Lists all valid identities for this cert | Connection fails if hostname not in SAN |
| Extended Key Usage | Restricts what the cert can be used for | Client auth requires clientAuth EKU |
| Validity Period | When cert is valid | Short-lived certs (hours) reduce breach impact |
| Issuer | Who signed this cert | Must chain to trusted CA |
2.4 Certificate Signing Request (CSR)
A CSR is how an entity requests a certificate from a CA. It contains the public key and requested identity, signed by the corresponding private key (proving ownership):
┌─────────────────────────────────────────────────────────────────────────┐
│ CSR Generation and Signing Flow │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Step 1: Service Generates Key Pair │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ payment-service │ │
│ │ │ │
│ │ Private Key: ████████████████ (NEVER leaves service) │ │
│ │ Public Key: ABCD1234... (Goes in CSR) │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ Step 2: Create and Sign CSR │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Certificate Signing Request (CSR) │ │
│ │ │ │
│ │ Subject: CN=payment-service │ │
│ │ Public Key: ABCD1234... │ │
│ │ Requested SANs: payment-service.prod.local │ │
│ │ Signature: <signed with private key> ← Proves key ownership │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ Step 3: Submit to CA │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Certificate Authority │ │
│ │ │ │
│ │ 1. Verify CSR signature (proves key ownership) │ │
│ │ 2. Validate identity (is this really payment-service?) │ │
│ │ 3. Apply policy (cert duration, allowed SANs, etc.) │ │
│ │ 4. Sign certificate with CA private key │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ Step 4: Return Signed Certificate │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ X.509 Certificate │ │
│ │ │ │
│ │ Subject: CN=payment-service │ │
│ │ Issuer: CN=My CA │ │
│ │ Valid: 2024-01-01 to 2024-01-02 (24 hours) │ │
│ │ SANs: payment-service.prod.local │ │
│ │ CA Signature: <signed by CA> │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
2.5 The TLS Handshake: Standard vs Mutual
Understanding the TLS handshake is critical for debugging connection issues:
┌─────────────────────────────────────────────────────────────────────────┐
│ TLS 1.3 Handshake (Standard) │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Client Server │
│ │ │ │
│ │ ─────────── ClientHello ──────────────────────▶ │ │
│ │ - Supported cipher suites │ │
│ │ - Supported TLS versions │ │
│ │ - Client random (32 bytes) │ │
│ │ - Key share (for key exchange) │ │
│ │ │ │
│ │ ◀─────────── ServerHello ─────────────────────── │ │
│ │ - Chosen cipher suite │ │
│ │ - Server random (32 bytes) │ │
│ │ - Key share (for key exchange) │ │
│ │ │ │
│ │ ◀───── EncryptedExtensions ──────────────────── │ │
│ │ ◀───── Certificate (Server's) ──────────────── │ ← Server │
│ │ ◀───── CertificateVerify ────────────────────── │ proves │
│ │ ◀───── Finished ─────────────────────────────── │ identity │
│ │ │ │
│ │ ─────────── Finished ─────────────────────────▶ │ │
│ │ │ │
│ │ ◀════════════ Application Data ═══════════════▶ │ │
│ │ (Encrypted with session keys) │ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│ TLS 1.3 Handshake (Mutual TLS) │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Client Server │
│ │ │ │
│ │ ─────────── ClientHello ──────────────────────▶ │ │
│ │ │ │
│ │ ◀─────────── ServerHello ─────────────────────── │ │
│ │ ◀───── EncryptedExtensions ──────────────────── │ │
│ │ ◀───── CertificateRequest ──────────────────── │ ← Server │
│ │ (Server asks for client cert) │ requests │
│ │ ◀───── Certificate (Server's) ──────────────── │ client │
│ │ ◀───── CertificateVerify ────────────────────── │ auth │
│ │ ◀───── Finished ─────────────────────────────── │ │
│ │ │ │
│ │ ─────────── Certificate (Client's) ──────────▶ │ ← Client │
│ │ ─────────── CertificateVerify ───────────────▶ │ proves │
│ │ (Signed with client's private key) │ identity │
│ │ ─────────── Finished ─────────────────────────▶ │ │
│ │ │ │
│ │ ◀════════════ Application Data ═══════════════▶ │ │
│ │ │ │
│ RESULT: Both parties have cryptographic proof of identity │
│ │
└─────────────────────────────────────────────────────────────────────────┘
What Happens at Each Step:
| Message | Purpose | What’s Verified |
|---|---|---|
| ClientHello | Initiate handshake, propose parameters | - |
| ServerHello | Accept parameters, provide key share | - |
| CertificateRequest | Server asks client to authenticate | - |
| Certificate (Server) | Server sends its cert chain | Client verifies chain to trusted CA |
| CertificateVerify (Server) | Proves server owns private key | Signature over handshake transcript |
| Certificate (Client) | Client sends its cert chain | Server verifies chain to trusted CA |
| CertificateVerify (Client) | Proves client owns private key | Signature over handshake transcript |
| Finished | Confirms handshake integrity | MAC over entire handshake |
2.6 SPIFFE: Workload Identity for the Cloud
SPIFFE (Secure Production Identity Framework for Everyone) standardizes workload identity:
┌─────────────────────────────────────────────────────────────────────────┐
│ SPIFFE Architecture │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ SPIFFE ID Format: │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ spiffe://trust-domain/path/to/workload │ │
│ │ ────────┬────────── ──────────┬──────── │ │
│ │ Trust Domain Workload Path │ │
│ │ (your org) (identifies specific workload) │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ Examples: │
│ spiffe://example.com/ns/production/sa/payment-service │
│ spiffe://example.com/region/us-west/app/frontend │
│ spiffe://example.com/cluster/k8s-prod/pod/api-server-abc123 │
│ │
│ SVID (SPIFFE Verifiable Identity Document): │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ X.509 Certificate with SPIFFE ID in SAN: │ │
│ │ │ │
│ │ Subject: (can be empty or generic) │ │
│ │ SAN URI: spiffe://example.com/ns/prod/sa/payment │ │
│ │ Valid: Short-lived (1 hour typical) │ │
│ │ Issuer: SPIFFE-compliant CA │ │
│ │ │ │
│ │ The SPIFFE ID in the URI SAN IS the identity. │ │
│ │ Traditional Subject field is not used for identity. │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ SPIRE (SPIFFE Runtime Environment): │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ SPIRE Server│◀── Attestation ───▶│ SPIRE Agent │ │ │
│ │ │ (CA + DB) │ │ (per node) │ │ │
│ │ └─────────────┘ └──────┬──────┘ │ │
│ │ │ │ │
│ │ Unix Socket API │ │
│ │ │ │ │
│ │ ┌───────────────────┼───────────────┐ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ │ Workload │ │ Workload │ │ Workload │ │
│ │ │ A │ │ B │ │ C │ │
│ │ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Why SPIFFE Matters:
- Platform-agnostic identity: Works across Kubernetes, VMs, bare metal
- Automatic rotation: SPIRE handles cert lifecycle
- Attestation-based: Workloads prove identity through multiple signals
- Zero Trust native: No implicit trust based on network
2.7 Certificate Revocation: When Things Go Wrong
What happens when a private key is compromised or a certificate needs to be invalidated before expiry?
┌─────────────────────────────────────────────────────────────────────────┐
│ Certificate Revocation Methods │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ METHOD 1: Certificate Revocation Lists (CRLs) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ CA publishes list of revoked certificate serial numbers: │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
│ │ │ CRL (Certificate Revocation List) │ │ │
│ │ │ │ │ │
│ │ │ Issuer: CN=My CA │ │ │
│ │ │ This Update: 2024-01-15 00:00:00 │ │ │
│ │ │ Next Update: 2024-01-16 00:00:00 │ │ │
│ │ │ │ │ │
│ │ │ Revoked Certificates: │ │ │
│ │ │ Serial: 1234:5678 Revoked: 2024-01-14 Key Compromise │ │
│ │ │ Serial: ABCD:EF01 Revoked: 2024-01-15 Superseded │ │
│ │ │ │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Problems: │ │
│ │ - CRLs grow large over time (linear growth) │ │
│ │ - Clients must download entire CRL │ │
│ │ - Stale data between updates │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ METHOD 2: Online Certificate Status Protocol (OCSP) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Client OCSP Responder CA │ │
│ │ │ │ │ │ │
│ │ │ "Is cert 1234 valid?"│ │ │ │
│ │ │──────────────────────▶ │ │ │
│ │ │ │ (checks status) │ │ │
│ │ │◀────────────────────── │ │ │
│ │ │ "Good" / "Revoked" │ │ │ │
│ │ │ │
│ │ Advantages: Real-time, small response size │ │
│ │ Problems: Privacy (CA knows what certs you're checking) │ │
│ │ Availability (if OCSP down, what to do?) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ METHOD 3: Short-Lived Certificates (Zero Trust Approach) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Issue certificates valid for 1 hour instead of 1 year. │ │
│ │ No revocation needed - just don't renew. │ │
│ │ │ │
│ │ Timeline: │ │
│ │ ├────────────────────────────────────────────────────────┤ │ │
│ │ 0 15m 30m 45m 60m │ │
│ │ │ │ │ │ │ │ │
│ │ ├─────────────── Cert 1 ──────────────────┤ │ │
│ │ ├─────────────── Cert 2 ──────────────────┤ │ │
│ │ (overlap for smooth rotation) │ │
│ │ │ │
│ │ Compromise at 35m? Cert expires at 60m anyway. │ │
│ │ Maximum exposure: remaining validity period. │ │
│ │ │ │
│ │ This is the SPIFFE/Istio/modern cloud-native approach. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
2.8 Identity Bootstrapping: The Chicken-and-Egg Problem
How does a new service prove who it is to get its first certificate?
┌─────────────────────────────────────────────────────────────────────────┐
│ Identity Bootstrapping Methods │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ THE PROBLEM: │
│ New service starts → needs certificate to communicate │
│ But CA asks: "Prove you are payment-service" │
│ Service has no cert yet to prove anything! │
│ │
│ SOLUTION 1: Pre-shared Token/Secret │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ 1. Operator pre-provisions a one-time token │ │
│ │ 2. Service presents token to CA │ │
│ │ 3. CA verifies token, issues certificate │ │
│ │ 4. Token is invalidated │ │
│ │ │ │
│ │ Used by: Vault, manual PKI │ │
│ │ Problem: How do you securely distribute the token? │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ SOLUTION 2: Platform Attestation (SPIRE approach) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Service proves identity by WHERE it's running: │ │
│ │ │ │
│ │ Kubernetes: Pod UID, Service Account, Namespace │ │
│ │ AWS: Instance ID, IAM Role, Security Groups │ │
│ │ GCP: Service Account, Project, Zone │ │
│ │ Linux: Process UID, binary hash, parent process │ │
│ │ │ │
│ │ ┌───────────┐ ┌──────────────┐ ┌────────────┐ │ │
│ │ │ Workload │ ───▶ │ SPIRE Agent │ ───▶ │ Kubernetes │ │ │
│ │ │ │ │ │ │ API │ │ │
│ │ └───────────┘ └──────────────┘ └────────────┘ │ │
│ │ │ │ │ │ │
│ │ │ │ "Pod abc123 in namespace │ │
│ │ │ │ prod with SA payment-svc" │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌──────────────┐ │ │ │
│ │ │ │ SPIRE Server │◀───────────┘ │ │
│ │ │ │ (matches to │ │ │
│ │ │ │ SPIFFE ID) │ │ │
│ │ │ └──────────────┘ │ │
│ │ │ │ │ │
│ │ ◀─── SVID (cert) ────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ SOLUTION 3: Instance Identity Documents (Cloud Provider) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ AWS: Instance Identity Document signed by AWS │ │
│ │ GCP: Instance Identity Token from metadata server │ │
│ │ Azure: Managed Identity tokens │ │
│ │ │ │
│ │ CA trusts cloud provider's attestation. │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
3. Complete Project Specification
3.1 What You Will Build
A complete mTLS mesh system consisting of:
- Mini Certificate Authority (CA): Issues and manages short-lived X.509 certificates
- Two Services: Server and client that communicate exclusively via mTLS
- Automatic Certificate Rotation: Refresh certificates before expiry without connection drops
- Certificate Inspection CLI: Debug and verify certificate chains
3.2 Functional Requirements
FR-1: Mini CA
- Generate and secure a Root CA key pair
- Accept Certificate Signing Requests (CSRs) via API
- Issue certificates with configurable validity (default: 1 hour)
- Include SPIFFE-style URIs in Subject Alternative Names
- Maintain an in-memory list of issued certificates
- Support certificate revocation (mark as revoked)
FR-2: Server Service
- Expose an HTTPS endpoint requiring client certificates
- Validate client certificates against the CA’s root
- Extract and log the client’s identity from the certificate
- Reject connections from unknown/expired/revoked certificates
FR-3: Client Service
- Obtain a certificate from the CA at startup
- Present certificate when connecting to the server
- Validate the server’s certificate against the CA’s root
- Successfully complete an mTLS handshake
FR-4: Certificate Rotation
- Monitor certificate expiry time
- Automatically request new certificate before expiry (e.g., at 50% lifetime)
- Swap certificates without dropping active connections
- Log rotation events
FR-5: Certificate Inspector CLI
- Parse and display certificate details
- Verify certificate chains
- Check certificate validity (dates, signatures)
- Display SPIFFE IDs from SAN URIs
3.3 Non-Functional Requirements
- Security: Private keys never leave their respective components
- Availability: Certificate rotation must not cause connection failures
- Observability: All TLS events must be loggable
- Portability: Run on any system with Go installed
3.4 System Architecture Overview
┌─────────────────────────────────────────────────────────────────────────┐
│ mTLS Mesh Architecture │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Mini CA │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────┐ │ │
│ │ │ Root Key Pair │ │ CSR API │ │ Issued Certs DB │ │ │
│ │ │ (Protected) │ │ /api/sign │ │ (In-Memory) │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────────┘ │ │
│ └────────────────────────────────┬────────────────────────────────┘ │
│ │ │
│ ┌──────────────┴──────────────┐ │
│ │ CA Root Certificate │ │
│ │ (Distributed to all) │ │
│ └──────────────┬──────────────┘ │
│ │ │
│ ┌─────────────────────────┼─────────────────────────┐ │
│ │ │ │ │
│ ▼ │ ▼ │
│ ┌─────────────────┐ │ ┌─────────────────┐ │
│ │ Server Service │◀────── mTLS ──────────────▶│ Client Service │ │
│ │ │ │ │ │ │
│ │ - Server cert │ │ │ - Client cert │ │
│ │ - Server key │ │ │ - Client key │ │
│ │ - CA root (for │ │ │ - CA root (for │ │
│ │ client verify)│ │ │ server verify)│ │
│ │ - Rotation │ │ │ - Rotation │ │
│ │ goroutine │ │ goroutine │ │
│ └─────────────────┘ └─────────────────┘ │
│ │
│ Data Flow: │
│ 1. Service generates key pair locally │
│ 2. Service creates CSR, sends to CA │
│ 3. CA validates and signs, returns certificate │
│ 4. Service uses cert+key for mTLS connections │
│ 5. Rotation monitor refreshes cert before expiry │
│ │
└─────────────────────────────────────────────────────────────────────────┘
4. Real World Outcome
When you complete this project, here’s exactly what you’ll see:
Terminal 1: Start the Mini CA
$ ./mtls-ca --port 8443 --cert-duration 1h
[CA] Generating Root CA key pair (RSA-4096)...
[CA] Root CA certificate created:
Subject: CN=mtls-mesh-root-ca
Serial: 1A:2B:3C:4D:5E:6F:70:81
Valid: 2024-01-15 00:00:00 UTC to 2034-01-15 00:00:00 UTC
Key Usage: Certificate Sign, CRL Sign
[CA] API server listening on :8443
[CA] Endpoints:
POST /api/v1/sign - Submit CSR, receive signed certificate
GET /api/v1/ca - Download CA root certificate
POST /api/v1/revoke - Revoke a certificate
[CA] Ready to issue certificates.
Terminal 2: Start the Server Service
$ ./mtls-server --ca-url http://localhost:8443 --listen :9443
[SERVER] Generating service key pair (ECDSA P-256)...
[SERVER] Creating CSR for identity: spiffe://mesh.local/service/api-server
[SERVER] Requesting certificate from CA at http://localhost:8443/api/v1/sign...
[CA] (on CA side) Received CSR:
Subject: CN=api-server
Requested SAN: spiffe://mesh.local/service/api-server
[SERVER] Certificate issued:
Serial: 2B:3C:4D:5E:6F:70:81:92
Issuer: CN=mtls-mesh-root-ca
Subject: CN=api-server
SAN URI: spiffe://mesh.local/service/api-server
Valid: 2024-01-15 10:00:00 UTC to 2024-01-15 11:00:00 UTC
Expires in: 59m 59s
[SERVER] Starting mTLS server on :9443
[SERVER] Client certificate required: YES
[SERVER] Trusted CA: CN=mtls-mesh-root-ca
[SERVER] Certificate rotation scheduled:
Next rotation: 2024-01-15 10:30:00 UTC (30 minutes)
[SERVER] Ready to accept mTLS connections.
Terminal 3: Start the Client Service
$ ./mtls-client --ca-url http://localhost:8443 --server https://localhost:9443
[CLIENT] Generating service key pair (ECDSA P-256)...
[CLIENT] Creating CSR for identity: spiffe://mesh.local/service/payment-worker
[CLIENT] Requesting certificate from CA...
[CLIENT] Certificate issued:
Serial: 3C:4D:5E:6F:70:81:92:A3
Subject: CN=payment-worker
SAN URI: spiffe://mesh.local/service/payment-worker
Valid: 2024-01-15 10:00:05 UTC to 2024-01-15 11:00:05 UTC
[CLIENT] Initiating mTLS connection to localhost:9443...
[CLIENT] TLS Handshake Details:
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS Version: TLS 1.3
Server Identity: spiffe://mesh.local/service/api-server
Our Identity: spiffe://mesh.local/service/payment-worker
[CLIENT] Connection established! Server verified our identity.
[CLIENT] Sending request: GET /api/status
[CLIENT] Response: 200 OK
Body: {"status":"healthy","caller":"spiffe://mesh.local/service/payment-worker"}
[CLIENT] mTLS mesh communication successful!
Terminal 2 (Server) - Showing Client Connection
[SERVER] New connection from 127.0.0.1:54321
[SERVER] Client certificate presented:
Subject: CN=payment-worker
SAN URI: spiffe://mesh.local/service/payment-worker
Issuer: CN=mtls-mesh-root-ca
Serial: 3C:4D:5E:6F:70:81:92:A3
[SERVER] Client identity verified: spiffe://mesh.local/service/payment-worker
[SERVER] Handling request: GET /api/status
[SERVER] Response sent to payment-worker: 200 OK
Certificate Inspection with OpenSSL
# View client certificate details
$ openssl x509 -in client.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
3c:4d:5e:6f:70:81:92:a3
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = mtls-mesh-root-ca
Validity
Not Before: Jan 15 10:00:05 2024 GMT
Not After : Jan 15 11:00:05 2024 GMT
Subject: CN = payment-worker
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:8a:3b:...
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Subject Alternative Name:
URI:spiffe://mesh.local/service/payment-worker
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
Signature Algorithm: ecdsa-with-SHA256
30:45:02:21:...
Testing with curl
# This succeeds - mTLS with valid client cert
$ curl --cacert ca.crt \
--cert client.crt \
--key client.key \
https://localhost:9443/api/status
{"status":"healthy","caller":"spiffe://mesh.local/service/payment-worker"}
# This fails - no client certificate
$ curl --cacert ca.crt https://localhost:9443/api/status
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
# This fails - wrong CA (server doesn't trust this CA)
$ curl --cacert other-ca.crt \
--cert other-client.crt \
--key other-client.key \
https://localhost:9443/api/status
curl: (60) SSL certificate problem: unable to get local issuer certificate
Certificate Rotation in Action
# 30 minutes later...
[SERVER] Certificate rotation triggered (50% lifetime reached)
[SERVER] Generating new CSR...
[SERVER] New certificate issued:
Serial: 4D:5E:6F:70:81:92:A3:B4
Valid: 2024-01-15 10:30:00 UTC to 2024-01-15 11:30:00 UTC
[SERVER] Hot-swapping certificate...
[SERVER] Active connections: 3 (will use new cert for new connections)
[SERVER] Certificate rotation complete.
[SERVER] Next rotation: 2024-01-15 11:00:00 UTC
The Core Question You’re Answering
“How can services prove their identity to each other cryptographically, eliminating the need to trust the network?”
This question strikes at the heart of Zero Trust networking. In traditional architectures, being “inside the network” grants implicit trust - if a service can reach another service, it’s assumed to be legitimate. mTLS flips this model entirely: every connection must prove identity through cryptographic certificates, regardless of network position.
Your challenge is to build a system where:
- A service cannot communicate without possessing a valid certificate signed by a trusted authority
- Both the caller and the callee verify each other’s identity before exchanging any data
- Network position (IP address, subnet, VPN) provides zero trust - only the certificate matters
- Compromised credentials expire automatically through short-lived certificates
Concepts You Must Understand First
Before diving into implementation, ensure you have a solid grasp of these foundational concepts:
1. X.509 Certificates and PKI (Public Key Infrastructure)
What to know:
- X.509 is the standard format for public key certificates used in TLS
- A certificate binds a public key to an identity (like “payment-service”)
- The binding is attested by a Certificate Authority’s signature
- PKI is the hierarchy of trust: Root CA -> Intermediate CA -> End-Entity Certificates
Key questions to answer:
- What fields in an X.509 certificate are critical for mTLS?
- Why do we use intermediate CAs instead of signing everything with the root?
- What makes a certificate “trusted”?
2. TLS Handshake: Standard vs Mutual
What to know:
- Standard TLS: Only the server proves its identity (what you see with HTTPS)
- Mutual TLS: Both client AND server prove identity via certificates
- The handshake establishes: identity verification, cipher agreement, session key derivation
Key questions to answer:
- At what point in the handshake does the client send its certificate?
- What triggers the client to send a certificate (hint: CertificateRequest)?
- How does each party prove it owns the private key corresponding to its certificate?
3. Certificate Signing Requests (CSR)
What to know:
- A CSR is how you request a certificate from a CA
- It contains your public key and requested identity (Subject, SANs)
- The CSR is signed with your private key, proving you own the key pair
- The CA validates the CSR and creates a signed certificate
Key questions to answer:
- Why does the CSR need to be signed by the requestor?
- What fields from the CSR end up in the final certificate?
- Can the CA add fields not in the CSR? (Yes - like validity period, extensions)
4. SPIFFE/SPIRE Workload Identity
What to know:
- SPIFFE provides a standard identity format:
spiffe://trust-domain/path - SPIFFE IDs are placed in the URI SAN of X.509 certificates (SVIDs)
- SPIRE is the reference implementation: attestation, identity issuance, rotation
Key questions to answer:
- Why use SPIFFE IDs instead of traditional Subject (CN) fields?
- How does a workload prove its identity to get its first certificate?
- What is “attestation” and why is it necessary?
5. Certificate Rotation and Lifecycle
What to know:
- Short-lived certificates (hours, not years) reduce breach impact
- Rotation must happen before expiry to avoid service disruption
- Atomic certificate swap prevents connection failures during rotation
Key questions to answer:
- What’s a good rotation threshold? (Common: 50% of lifetime)
- How do active connections handle certificate rotation?
- What happens if rotation fails? (Retry logic, fallback)
Questions to Guide Your Design
Work through these questions before writing code. They’ll shape your architecture:
Certificate Authority Design
- Where will the CA private key be stored? In memory? On disk? In an HSM? What are the tradeoffs?
- How will you generate serial numbers? Must be unique - what happens if you reuse one?
- What certificate validity period will you use? 1 hour? 24 hours? Why?
- Will you support certificate revocation? If not, why do short-lived certs make this acceptable?
Identity Model
- How will you structure SPIFFE IDs? What path format? (e.g.,
/ns/namespace/sa/serviceaccount) - How will a service prove it should receive a particular identity? (The bootstrapping problem)
- Will you validate requested identities, or trust the requestor?
mTLS Configuration
- What TLS version will you require? TLS 1.2 minimum? TLS 1.3 only?
- What
ClientAuthmode will you use?RequireAndVerifyClientCertis the only truly secure option for mTLS - How will you configure the trust store? RootCAs vs ClientCAs - do you understand the difference?
Rotation Strategy
- When will rotation trigger? Fixed schedule? Percentage of lifetime?
- How will you atomically swap certificates? What synchronization is needed?
- What happens to existing connections during rotation?
Thinking Exercise
Before writing any code, work through this exercise on paper or a whiteboard:
Trace the Complete mTLS Handshake
Draw the message flow between Client and Server for a TLS 1.3 mTLS connection. For each message, note:
- ClientHello
- What does the client send?
- What capabilities is it advertising?
- ServerHello + EncryptedExtensions
- What has the server decided?
- Why are extensions encrypted now?
- CertificateRequest
- What is the server asking for?
- What CAs will the client’s cert need to chain to?
- Certificate (Server) + CertificateVerify
- What’s in the Certificate message?
- What does CertificateVerify prove? (Not just “I have a cert”)
- Certificate (Client) + CertificateVerify
- Why is this step necessary for mTLS?
- What exactly is the client signing?
- Finished (both sides)
- What does Finished prove about the handshake?
Bonus: What would happen if an attacker intercepted all these messages and replayed them later?
Hints in Layers
If you get stuck, reveal hints progressively. Try to solve each phase before looking at hints.
Layer 1: Getting Started
Hint: First step with OpenSSL
Start by generating certificates manually with OpenSSL before writing any Go code. This will help you understand what your CA needs to produce:
# Generate a CA key and self-signed cert
openssl ecparam -name prime256v1 -genkey -noout -out ca.key
openssl req -new -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=My CA"
Once you can generate, sign, and verify certificates manually, translating to Go becomes straightforward.
Layer 2: Go’s crypto/tls Configuration
Hint: The critical tls.Config settings for mTLS
For the server to require client certificates:
tlsConfig := &tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert, // This is the key!
ClientCAs: caCertPool, // CAs that can sign client certs
}
For the client to present its certificate:
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{clientCert}, // Your cert + key
RootCAs: caCertPool, // CAs that can sign server certs
}
Common mistake: Using RootCAs on the server for client verification - that’s wrong! RootCAs is for outgoing connections, ClientCAs is for verifying incoming client certs.
Layer 3: Adding SPIFFE IDs to Certificates
Hint: URI SANs in Go's x509 package
SPIFFE IDs go in the URI Subject Alternative Name. In Go:
import "net/url"
spiffeID, _ := url.Parse("spiffe://mesh.local/service/payment")
template := &x509.Certificate{
// ... other fields ...
URIs: []*url.URL{spiffeID}, // This adds the SPIFFE ID as a URI SAN
}
To extract a SPIFFE ID from a certificate:
for _, uri := range cert.URIs {
if uri.Scheme == "spiffe" {
fmt.Println("SPIFFE ID:", uri.String())
}
}
Layer 4: Certificate Rotation
Hint: Using GetCertificate for dynamic certificates
Instead of setting Certificates directly, use a callback:
tlsConfig := &tls.Config{
GetCertificate: func(chi *tls.ClientHelloInfo) (*tls.Certificate, error) {
cm.mu.RLock()
defer cm.mu.RUnlock()
return cm.currentCert, nil // Return the latest cert
},
}
For client-side rotation, use GetClientCertificate:
tlsConfig := &tls.Config{
GetClientCertificate: func(cri *tls.CertificateRequestInfo) (*tls.Certificate, error) {
cm.mu.RLock()
defer cm.mu.RUnlock()
return cm.currentCert, nil
},
}
The rotation goroutine atomically swaps cm.currentCert - new connections automatically use the new cert.
Layer 5: Debugging Certificate Issues
Hint: Common error messages and their causes
| Error | Likely Cause |
|---|---|
certificate signed by unknown authority |
CA not in trust store (RootCAs or ClientCAs) |
bad certificate |
Client didn’t send cert, or server rejected it |
certificate is valid for X, not Y |
Hostname not in certificate’s SAN |
certificate has expired |
Check NotAfter, also check system clock! |
certificate specifies incompatible key usage |
Missing clientAuth or serverAuth EKU |
Debug with OpenSSL:
# Verbose connection test
openssl s_client -connect localhost:9443 \
-cert client.crt -key client.key \
-CAfile ca.crt -debug
5. Solution Architecture
5.1 Component Design
┌─────────────────────────────────────────────────────────────────────────┐
│ Component Architecture │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ mini-ca/ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────┐ │ │
│ │ │ main.go │ │ ca.go │ │ handlers.go │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ - CLI flags │ │ - GenerateRoot│ │ - POST /sign │ │ │
│ │ │ - Start server│ │ - SignCSR │ │ - GET /ca │ │ │
│ │ │ │ │ - Track issued│ │ - POST /revoke │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ mtls-server/ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────┐ │ │
│ │ │ main.go │ │ certmgr.go │ │ server.go │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ - CLI flags │ │ - GenerateKey │ │ - mTLS config │ │ │
│ │ │ - Bootstrap │ │ - CreateCSR │ │ - RequireClientCert│ │ │
│ │ │ - Start server│ │ - RequestCert │ │ - Identity extract│ │ │
│ │ │ │ │ - Rotate() │ │ │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ mtls-client/ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────┐ │ │
│ │ │ main.go │ │ certmgr.go │ │ client.go │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ - CLI flags │ │ - GenerateKey │ │ - mTLS config │ │ │
│ │ │ - Bootstrap │ │ - CreateCSR │ │ - Dial with cert │ │ │
│ │ │ - Make request│ │ - RequestCert │ │ - Verify server │ │ │
│ │ │ │ │ - Rotate() │ │ │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ shared/ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────┐ │ │
│ │ │ spiffe.go │ │ certutil.go │ │ x509ext.go │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ - ParseID │ │ - PEM encode │ │ - Add SANs │ │ │
│ │ │ - ValidateID │ │ - PEM decode │ │ - Add EKU │ │ │
│ │ │ - FormatID │ │ - Chain verify│ │ - Parse exts │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
5.2 Key Data Structures
// CertificateRequest represents a CSR submission
type CertificateRequest struct {
CSR []byte `json:"csr"` // PEM-encoded CSR
RequestedID string `json:"requested_id"` // SPIFFE ID to include in SAN
TTL Duration `json:"ttl"` // Requested validity duration
}
// CertificateResponse from the CA
type CertificateResponse struct {
Certificate []byte `json:"certificate"` // PEM-encoded cert
Chain [][]byte `json:"chain"` // Intermediate certs (if any)
ExpiresAt time.Time `json:"expires_at"`
Serial string `json:"serial"`
}
// IssuedCertificate tracked by the CA
type IssuedCertificate struct {
Serial *big.Int
Subject string
SpiffeID string
IssuedAt time.Time
ExpiresAt time.Time
Revoked bool
RevokedAt time.Time
}
// CertificateManager handles cert lifecycle
type CertificateManager struct {
mu sync.RWMutex
privateKey crypto.PrivateKey
certificate *tls.Certificate
caRoots *x509.CertPool
spiffeID string
caURL string
rotateAt time.Time
}
5.3 Certificate Rotation Algorithm
┌─────────────────────────────────────────────────────────────────────────┐
│ Certificate Rotation Flow │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Certificate Timeline: │
│ │
│ ├────────────────────────────────────────────────────────────────┤ │
│ │ Issue 25% 50% 75% Expiry │ │
│ ├─────────┼────────┼────────┼────────┼ │ │
│ │ │ │ │ │ │ │
│ │ │ Rotation Window │ │ │ │
│ │ │◀──────────────▶│ │ │ │
│ │ │ │ │ │ │
│ │ │ Get new cert │ │ │ │
│ │ │ before old │ │ │ │
│ │ │ expires │ │ │ │
│ │
│ Rotation Process: │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ 1. Timer fires at 50% of cert lifetime │ │
│ │ │ │
│ │ 2. Generate new key pair (optional - can reuse key) │ │
│ │ │ │
│ │ 3. Create new CSR with same identity │ │
│ │ │ │
│ │ 4. Submit to CA, receive new certificate │ │
│ │ │ │
│ │ 5. Atomically swap certificate in memory: │ │
│ │ │ │
│ │ manager.mu.Lock() │ │
│ │ manager.certificate = newCert │ │
│ │ manager.rotateAt = calculateNextRotation(newCert) │ │
│ │ manager.mu.Unlock() │ │
│ │ │ │
│ │ 6. Existing connections continue with old cert (fine) │ │
│ │ New connections use new cert │ │
│ │ │ │
│ │ 7. Schedule next rotation │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
6. Phased Implementation Guide
Phase 1: Generate Root CA with OpenSSL (Day 1)
Before writing any Go code, understand the OpenSSL commands that your CA will replicate:
Task 1.1: Create the Root CA
# Create directory structure
mkdir -p pki/{root,intermediate,certs,private}
chmod 700 pki/private
# Generate Root CA private key (RSA 4096-bit)
openssl genrsa -out pki/private/root-ca.key 4096
chmod 600 pki/private/root-ca.key
# Create Root CA configuration file
cat > pki/root-ca.conf << 'EOF'
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
prompt = no
[req_distinguished_name]
CN = mtls-mesh-root-ca
O = My Organization
C = US
[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:TRUE, pathlen:1
keyUsage = critical, keyCertSign, cRLSign
EOF
# Generate self-signed Root CA certificate (10 years validity)
openssl req -new -x509 -sha256 \
-key pki/private/root-ca.key \
-out pki/root/root-ca.crt \
-days 3650 \
-config pki/root-ca.conf
# Verify the Root CA certificate
openssl x509 -in pki/root/root-ca.crt -text -noout
Expected Output:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: ... (random)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = mtls-mesh-root-ca, O = My Organization, C = US
Validity
Not Before: Jan 15 00:00:00 2024 GMT
Not After : Jan 15 00:00:00 2034 GMT
Subject: CN = mtls-mesh-root-ca, O = My Organization, C = US
...
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:1
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
Checkpoint: You have a self-signed Root CA certificate with proper CA extensions.
Phase 2: Sign Server and Client Certificates (Day 2)
Task 2.1: Create Server Certificate
# Generate server private key (ECDSA P-256 for efficiency)
openssl ecparam -name prime256v1 -genkey -noout -out pki/private/server.key
# Create server CSR configuration with SAN
cat > pki/server.conf << 'EOF'
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = api-server
[v3_req]
subjectAltName = @alt_names
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
basicConstraints = CA:FALSE
[alt_names]
URI.1 = spiffe://mesh.local/service/api-server
DNS.1 = localhost
DNS.2 = api-server.mesh.local
IP.1 = 127.0.0.1
EOF
# Generate CSR
openssl req -new -sha256 \
-key pki/private/server.key \
-out pki/certs/server.csr \
-config pki/server.conf
# View the CSR
openssl req -in pki/certs/server.csr -text -noout
Task 2.2: Sign the Server Certificate with the CA
# Create signing configuration (for the CA to use)
cat > pki/sign-server.conf << 'EOF'
[v3_server]
subjectAltName = @alt_names
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
basicConstraints = CA:FALSE
[alt_names]
URI.1 = spiffe://mesh.local/service/api-server
DNS.1 = localhost
DNS.2 = api-server.mesh.local
IP.1 = 127.0.0.1
EOF
# Sign the CSR (1 hour validity for short-lived cert demo)
openssl x509 -req \
-in pki/certs/server.csr \
-CA pki/root/root-ca.crt \
-CAkey pki/private/root-ca.key \
-CAcreateserial \
-out pki/certs/server.crt \
-days 0 -hours 1 \
-sha256 \
-extfile pki/sign-server.conf \
-extensions v3_server
# Verify the certificate
openssl x509 -in pki/certs/server.crt -text -noout
# Verify the chain
openssl verify -CAfile pki/root/root-ca.crt pki/certs/server.crt
Task 2.3: Create Client Certificate
# Generate client private key
openssl ecparam -name prime256v1 -genkey -noout -out pki/private/client.key
# Create client CSR configuration
cat > pki/client.conf << 'EOF'
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = payment-worker
[v3_req]
subjectAltName = URI:spiffe://mesh.local/service/payment-worker
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
basicConstraints = CA:FALSE
EOF
# Generate and sign client certificate
openssl req -new -sha256 \
-key pki/private/client.key \
-out pki/certs/client.csr \
-config pki/client.conf
openssl x509 -req \
-in pki/certs/client.csr \
-CA pki/root/root-ca.crt \
-CAkey pki/private/root-ca.key \
-CAcreateserial \
-out pki/certs/client.crt \
-days 0 -hours 1 \
-sha256 \
-extfile pki/client.conf \
-extensions v3_req
Checkpoint: You have CA, server cert, and client cert. All verify against the CA.
Phase 3: Configure Go Server with RequireAndVerifyClientCert (Days 3-4)
Task 3.1: Create the mTLS Server
// server/main.go
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
// Load CA certificate for client verification
caCert, err := ioutil.ReadFile("pki/root/root-ca.crt")
if err != nil {
log.Fatalf("Failed to load CA cert: %v", err)
}
caCertPool := x509.NewCertPool()
if !caCertPool.AppendCertsFromPEM(caCert) {
log.Fatal("Failed to parse CA cert")
}
// Load server certificate and key
serverCert, err := tls.LoadX509KeyPair(
"pki/certs/server.crt",
"pki/private/server.key",
)
if err != nil {
log.Fatalf("Failed to load server cert: %v", err)
}
// Configure TLS with mutual authentication
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientCAs: caCertPool,
ClientAuth: tls.RequireAndVerifyClientCert, // THE KEY SETTING!
MinVersion: tls.VersionTLS13,
}
// Create server with handler that extracts client identity
mux := http.NewServeMux()
mux.HandleFunc("/api/status", func(w http.ResponseWriter, r *http.Request) {
// Extract client identity from verified certificate
if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
clientCert := r.TLS.PeerCertificates[0]
// Extract SPIFFE ID from SAN URIs
spiffeID := "unknown"
for _, uri := range clientCert.URIs {
if uri.Scheme == "spiffe" {
spiffeID = uri.String()
break
}
}
log.Printf("Request from client: %s (CN=%s)",
spiffeID, clientCert.Subject.CommonName)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"status":"healthy","caller":"%s"}`, spiffeID)
}
})
server := &http.Server{
Addr: ":9443",
Handler: mux,
TLSConfig: tlsConfig,
}
log.Println("Starting mTLS server on :9443")
log.Println("Client certificate: REQUIRED")
// ListenAndServeTLS with empty strings because certs are in TLSConfig
log.Fatal(server.ListenAndServeTLS("", ""))
}
Task 3.2: Create the mTLS Client
// client/main.go
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
)
func main() {
// Load CA certificate for server verification
caCert, err := ioutil.ReadFile("pki/root/root-ca.crt")
if err != nil {
log.Fatalf("Failed to load CA cert: %v", err)
}
caCertPool := x509.NewCertPool()
if !caCertPool.AppendCertsFromPEM(caCert) {
log.Fatal("Failed to parse CA cert")
}
// Load client certificate and key
clientCert, err := tls.LoadX509KeyPair(
"pki/certs/client.crt",
"pki/private/client.key",
)
if err != nil {
log.Fatalf("Failed to load client cert: %v", err)
}
// Configure TLS with client certificate
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{clientCert},
RootCAs: caCertPool,
MinVersion: tls.VersionTLS13,
}
// Create HTTP client with mTLS
client := &http.Client{
Transport: &http.Transport{
TLSClientConfig: tlsConfig,
},
}
// Make request
resp, err := client.Get("https://localhost:9443/api/status")
if err != nil {
log.Fatalf("Request failed: %v", err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
fmt.Printf("Response: %s\n", body)
fmt.Printf("TLS Version: %s\n", tlsVersionName(resp.TLS.Version))
fmt.Printf("Cipher Suite: %s\n", tls.CipherSuiteName(resp.TLS.CipherSuite))
}
func tlsVersionName(v uint16) string {
switch v {
case tls.VersionTLS13:
return "TLS 1.3"
case tls.VersionTLS12:
return "TLS 1.2"
default:
return "Unknown"
}
}
Checkpoint: Server requires client certificate. Client presents certificate. Both verify each other.
Phase 4: Programmatic Certificate Generation (Days 5-6)
Task 4.1: Build the Mini CA
// ca/ca.go
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"math/big"
"net/url"
"time"
)
type CA struct {
rootCert *x509.Certificate
rootKey *ecdsa.PrivateKey
certSerial *big.Int
issuedCerts map[string]*IssuedCertificate
}
func NewCA() (*CA, error) {
// Generate CA key pair
rootKey, err := ecdsa.GenerateKey(elliptic.P384(), rand.Reader)
if err != nil {
return nil, err
}
// Generate serial number
serialNumber, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return nil, err
}
// Create CA certificate template
rootTemplate := &x509.Certificate{
SerialNumber: serialNumber,
Subject: pkix.Name{
CommonName: "mtls-mesh-root-ca",
Organization: []string{"My Organization"},
},
NotBefore: time.Now(),
NotAfter: time.Now().AddDate(10, 0, 0), // 10 years
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 1,
}
// Self-sign the CA certificate
rootCertDER, err := x509.CreateCertificate(
rand.Reader, rootTemplate, rootTemplate,
&rootKey.PublicKey, rootKey,
)
if err != nil {
return nil, err
}
rootCert, err := x509.ParseCertificate(rootCertDER)
if err != nil {
return nil, err
}
return &CA{
rootCert: rootCert,
rootKey: rootKey,
certSerial: big.NewInt(1),
issuedCerts: make(map[string]*IssuedCertificate),
}, nil
}
func (ca *CA) SignCSR(csrPEM []byte, spiffeID string, ttl time.Duration) ([]byte, error) {
// Parse CSR
block, _ := pem.Decode(csrPEM)
if block == nil {
return nil, fmt.Errorf("failed to decode CSR PEM")
}
csr, err := x509.ParseCertificateRequest(block.Bytes)
if err != nil {
return nil, fmt.Errorf("failed to parse CSR: %w", err)
}
// Verify CSR signature (proves key ownership)
if err := csr.CheckSignature(); err != nil {
return nil, fmt.Errorf("CSR signature invalid: %w", err)
}
// Generate serial number
ca.certSerial = new(big.Int).Add(ca.certSerial, big.NewInt(1))
// Parse SPIFFE ID
spiffeURI, err := url.Parse(spiffeID)
if err != nil {
return nil, fmt.Errorf("invalid SPIFFE ID: %w", err)
}
// Create certificate template
template := &x509.Certificate{
SerialNumber: ca.certSerial,
Subject: csr.Subject,
NotBefore: time.Now(),
NotAfter: time.Now().Add(ttl),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth},
URIs: []*url.URL{spiffeURI},
DNSNames: []string{"localhost", csr.Subject.CommonName + ".mesh.local"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
}
// Sign the certificate
certDER, err := x509.CreateCertificate(
rand.Reader, template, ca.rootCert,
csr.PublicKey, ca.rootKey,
)
if err != nil {
return nil, fmt.Errorf("failed to sign certificate: %w", err)
}
// Encode to PEM
certPEM := pem.EncodeToMemory(&pem.Block{
Type: "CERTIFICATE",
Bytes: certDER,
})
// Track issued certificate
ca.issuedCerts[ca.certSerial.String()] = &IssuedCertificate{
Serial: ca.certSerial,
Subject: csr.Subject.CommonName,
SpiffeID: spiffeID,
IssuedAt: time.Now(),
ExpiresAt: time.Now().Add(ttl),
}
return certPEM, nil
}
func (ca *CA) RootCertPEM() []byte {
return pem.EncodeToMemory(&pem.Block{
Type: "CERTIFICATE",
Bytes: ca.rootCert.Raw,
})
}
Checkpoint: CA can programmatically generate root cert and sign CSRs with SPIFFE IDs.
Phase 5: Automatic Certificate Rotation (Day 7)
Task 5.1: Implement Certificate Manager with Rotation
// shared/certmgr.go
package shared
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"sync"
"time"
)
type CertificateManager struct {
mu sync.RWMutex
privateKey *ecdsa.PrivateKey
certificate *tls.Certificate
caCert *x509.Certificate
spiffeID string
caURL string
stopCh chan struct{}
}
func NewCertificateManager(caURL, spiffeID string, caCertPEM []byte) (*CertificateManager, error) {
// Parse CA certificate
block, _ := pem.Decode(caCertPEM)
caCert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil, err
}
cm := &CertificateManager{
spiffeID: spiffeID,
caURL: caURL,
caCert: caCert,
stopCh: make(chan struct{}),
}
// Get initial certificate
if err := cm.obtainCertificate(); err != nil {
return nil, err
}
// Start rotation goroutine
go cm.rotationLoop()
return cm, nil
}
func (cm *CertificateManager) obtainCertificate() error {
// Generate new key pair
privateKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return err
}
// Create CSR
csrTemplate := &x509.CertificateRequest{
Subject: pkix.Name{
CommonName: extractCN(cm.spiffeID),
},
}
csrDER, err := x509.CreateCertificateRequest(rand.Reader, csrTemplate, privateKey)
if err != nil {
return err
}
csrPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE REQUEST", Bytes: csrDER})
// Request certificate from CA (simplified - in reality, use HTTP client)
certPEM, err := cm.requestCertFromCA(csrPEM)
if err != nil {
return err
}
// Parse the certificate
block, _ := pem.Decode(certPEM)
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return err
}
// Create tls.Certificate
tlsCert := &tls.Certificate{
Certificate: [][]byte{cert.Raw, cm.caCert.Raw},
PrivateKey: privateKey,
Leaf: cert,
}
// Atomic swap
cm.mu.Lock()
cm.privateKey = privateKey
cm.certificate = tlsCert
cm.mu.Unlock()
log.Printf("[CertMgr] Certificate obtained. Expires: %v", cert.NotAfter)
return nil
}
func (cm *CertificateManager) rotationLoop() {
for {
cm.mu.RLock()
cert := cm.certificate.Leaf
cm.mu.RUnlock()
// Calculate rotation time (50% of lifetime)
lifetime := cert.NotAfter.Sub(cert.NotBefore)
rotateAt := cert.NotBefore.Add(lifetime / 2)
sleepDuration := time.Until(rotateAt)
if sleepDuration <= 0 {
// Already past rotation time, rotate now
sleepDuration = time.Second
}
select {
case <-time.After(sleepDuration):
log.Println("[CertMgr] Rotation triggered")
if err := cm.obtainCertificate(); err != nil {
log.Printf("[CertMgr] Rotation failed: %v. Retrying in 30s", err)
time.Sleep(30 * time.Second)
}
case <-cm.stopCh:
return
}
}
}
// GetCertificate returns the current certificate (called by tls.Config)
func (cm *CertificateManager) GetCertificate(chi *tls.ClientHelloInfo) (*tls.Certificate, error) {
cm.mu.RLock()
defer cm.mu.RUnlock()
return cm.certificate, nil
}
// GetClientCertificate returns the current certificate for client connections
func (cm *CertificateManager) GetClientCertificate(cri *tls.CertificateRequestInfo) (*tls.Certificate, error) {
cm.mu.RLock()
defer cm.mu.RUnlock()
return cm.certificate, nil
}
func (cm *CertificateManager) Stop() {
close(cm.stopCh)
}
Task 5.2: Use CertificateManager in Server
// server/main.go (updated)
func main() {
// ... load CA cert ...
cm, err := shared.NewCertificateManager(
"http://localhost:8443",
"spiffe://mesh.local/service/api-server",
caCertPEM,
)
if err != nil {
log.Fatal(err)
}
defer cm.Stop()
tlsConfig := &tls.Config{
GetCertificate: cm.GetCertificate, // Dynamic certificate!
ClientCAs: caCertPool,
ClientAuth: tls.RequireAndVerifyClientCert,
MinVersion: tls.VersionTLS13,
}
// ... rest of server setup ...
}
Checkpoint: Certificates rotate automatically. New connections use fresh certs.
7. Testing Strategy
7.1 Test Categories
| Category | Purpose | Examples |
|---|---|---|
| Unit Tests | Test certificate parsing, SPIFFE ID validation | ParseSPIFFEID correctly extracts components |
| Integration Tests | Test full mTLS handshake | Client + Server complete handshake successfully |
| Negative Tests | Verify rejection of invalid certs | Expired cert rejected, wrong CA rejected |
| Rotation Tests | Verify seamless rotation | Active connections survive rotation |
| Security Tests | Attack simulation | Man-in-the-middle fails, cert replay fails |
7.2 Critical Test Cases
Test 1: Basic mTLS Handshake
# Start server and client, verify successful communication
./mtls-server &
./mtls-client
# Expected: "status: healthy, caller: spiffe://..."
Test 2: Missing Client Certificate
curl --cacert pki/root/root-ca.crt https://localhost:9443/api/status
# Expected: TLS handshake failure (bad certificate)
Test 3: Expired Certificate
# Create certificate with 1 second validity, wait, try to use
# Expected: Certificate rejected as expired
Test 4: Wrong CA
# Create separate CA, issue cert from it, try to connect
# Expected: "certificate signed by unknown authority"
Test 5: Certificate Rotation Under Load
# Start continuous request loop
while true; do ./mtls-client; sleep 0.1; done &
# Wait for rotation to occur
sleep 1800 # 30 minutes for 1-hour certs at 50% rotation
# Verify no connection failures in logs
Test 6: SAN Mismatch
# Connect to hostname not in certificate's SAN
# Expected: "x509: certificate is valid for X, not Y"
7.3 OpenSSL Verification Commands
# Verify server with mTLS
openssl s_client -connect localhost:9443 \
-CAfile pki/root/root-ca.crt \
-cert pki/certs/client.crt \
-key pki/private/client.key \
-verify_return_error
# Check certificate chain
openssl verify -CAfile pki/root/root-ca.crt pki/certs/server.crt
# View certificate details
openssl x509 -in pki/certs/server.crt -text -noout | grep -A5 "Subject Alternative Name"
# Test TLS handshake timing
time openssl s_client -connect localhost:9443 </dev/null
8. Common Pitfalls and Debugging
8.1 Certificate Chain Issues
| Symptom | Cause | Solution |
|---|---|---|
| “certificate signed by unknown authority” | CA not in trust store | Add CA cert to RootCAs (client) or ClientCAs (server) |
| “x509: certificate has expired” | Certificate past NotAfter | Issue new cert, check system clock |
| “x509: certificate is valid for X, not Y” | Hostname not in SAN | Add correct DNS/IP/URI to certificate |
| “tls: bad certificate” | Client didn’t provide cert | ClientAuth must be RequireAndVerifyClientCert |
| “x509: certificate specifies an incompatible key usage” | Wrong EKU | Ensure serverAuth for server, clientAuth for client |
8.2 Debugging TLS Handshake
// Add to tls.Config for debugging
tlsConfig := &tls.Config{
// ... other settings ...
// Log handshake state (Go 1.16+)
VerifyConnection: func(cs tls.ConnectionState) error {
log.Printf("TLS Version: %x", cs.Version)
log.Printf("Cipher Suite: %s", tls.CipherSuiteName(cs.CipherSuite))
log.Printf("Server Name: %s", cs.ServerName)
for i, cert := range cs.PeerCertificates {
log.Printf("Peer Cert %d: %s", i, cert.Subject)
for _, uri := range cert.URIs {
log.Printf(" SAN URI: %s", uri)
}
}
return nil
},
}
8.3 OpenSSL Debugging
# Verbose TLS handshake debug
openssl s_client -connect localhost:9443 -debug -msg
# Show all certificate extensions
openssl x509 -in cert.pem -text -noout | grep -A100 "X509v3 extensions"
# Verify certificate against CA
openssl verify -verbose -CAfile ca.crt cert.pem
# Check if cert has required EKU
openssl x509 -in cert.pem -purpose -noout
8.4 Common Go Mistakes
// WRONG: Not including intermediate certs in chain
tlsCert := tls.Certificate{
Certificate: [][]byte{leafCert.Raw}, // Missing CA!
PrivateKey: key,
}
// CORRECT: Include full chain
tlsCert := tls.Certificate{
Certificate: [][]byte{leafCert.Raw, intermediateCert.Raw},
PrivateKey: key,
}
// WRONG: Using RootCAs for server's client verification
tlsConfig := &tls.Config{
RootCAs: caCertPool, // This is for OUTGOING connections!
ClientAuth: tls.RequireAndVerifyClientCert,
}
// CORRECT: Use ClientCAs for verifying client certificates
tlsConfig := &tls.Config{
ClientCAs: caCertPool, // This verifies INCOMING client certs
ClientAuth: tls.RequireAndVerifyClientCert,
}
9. Extensions and Challenges
9.1 Beginner Extensions
- Certificate Inspector CLI: Parse and display any X.509 cert with all fields
- JSON Output: Add
--format jsonfor machine-readable CA responses - Certificate Logging: Log all issued certificates to a file for audit
- Graceful Shutdown: Ensure rotation goroutine stops cleanly
9.2 Intermediate Extensions
- Intermediate CA: Add a layer between root and end-entity certs
- Certificate Revocation: Implement revocation API and check during handshake
- OCSP Responder: Build a simple OCSP server for revocation checking
- Multiple SANs: Support DNS, IP, and URI SANs from CSR
- Key Algorithm Choice: Support RSA-2048, RSA-4096, P-256, P-384
9.3 Advanced Extensions
- SPIRE Integration: Replace your CA with SPIRE for production-grade identity
- Hardware Security Module (HSM): Store CA key in simulated HSM (or real PKCS#11)
- Certificate Transparency: Log certificates to a CT log
- mTLS Sidecar Proxy: Build an Envoy-like proxy that handles mTLS for apps
- Workload Attestation: Implement Kubernetes or AWS attestation for bootstrapping
- Policy Engine Integration: Only issue certs based on policy (integrate with P2)
9.4 Challenge Problems
-
Zero-Downtime CA Rotation: How do you rotate the root CA without breaking all services?
-
Cross-Cluster mTLS: Two SPIFFE trust domains need to communicate. Implement federation.
-
Certificate Pinning: Implement cert pinning with graceful rotation support.
-
Audit Trail: Every certificate action (issue, revoke, rotate) must be cryptographically logged.
10. Books That Will Help
| Topic | Book | Chapter/Section |
|---|---|---|
| Zero Trust fundamentals | “Zero Trust Networks” by Gilman & Barth | Chapters 4-5 (Device Trust, Application Trust) |
| mTLS and service mesh | “Zero Trust Networks” by Gilman & Barth | Chapter 6 (Network) |
| TLS protocol deep dive | “Serious Cryptography, 2nd Ed” by Aumasson | Chapter 13 (TLS) |
| X.509 and PKI | “Security in Computing, 5th Ed” by Pfleeger | Chapter 7.2 (Public Key Infrastructure) |
| Applied cryptography | “Cryptography Engineering” by Ferguson, Schneier, Kohno | Chapters 16-18 (Key Management) |
| Go crypto/tls | “Network Programming with Go” by Woodbeck | Chapter 9 (TLS) |
| SPIFFE/SPIRE | “Solving the Bottom Turtle” (free) | spiffe.io documentation |
| Certificate management | “Bulletproof SSL and TLS” by Ristic | Chapters on PKI deployment |
11. Interview Questions
After completing this project, you’ll be prepared for these questions:
Conceptual Questions
- “Explain mutual TLS and why it’s important for zero trust.”
- Expected: Both parties authenticate, not just server. Eliminates network-based trust.
- Bonus: Discuss how it prevents lateral movement after network breach.
- “What happens during a TLS handshake? Walk through mTLS specifically.”
- Expected: ClientHello, ServerHello, Certificates, CertificateVerify, Finished
- Bonus: Explain CertificateRequest for client auth, key derivation
- “How would you implement certificate rotation without downtime?”
- Expected: Use GetCertificate callback, atomic swap, overlap period
- Bonus: Discuss connection draining, monitoring rotation success
- “What is SPIFFE and how does it solve workload identity?”
- Expected: SPIFFE ID format, SVIDs, platform-agnostic identity
- Bonus: Explain SPIRE architecture, attestation plugins
- “Why use short-lived certificates instead of CRLs/OCSP?”
- Expected: Reduces blast radius, eliminates revocation complexity
- Bonus: Discuss 1-hour certs in Istio, SPIRE rotation
Troubleshooting Questions
- “A client is getting ‘certificate signed by unknown authority’. How do you debug?”
- Check if correct CA is in trust store
- Verify certificate chain is complete
- Check for intermediate CA issues
- Use
openssl verifyto trace the problem
- “mTLS works in staging but fails in production. What could be wrong?”
- Clock skew (certificate not yet valid / expired)
- Different CA certificates
- Load balancer terminating TLS
- Missing SNI in client
- “How would you design certificate issuance for a Kubernetes cluster?”
- Use SPIRE with Kubernetes attestation
- Service account identity maps to SPIFFE ID
- Node agent handles attestation
- Short-lived certs with automatic rotation
12. Self-Assessment Checklist
Before considering this project complete, verify:
Understanding
- I can explain the difference between standard TLS and mTLS
- I can draw the TLS handshake with client authentication
- I understand X.509 certificate structure including SANs and EKUs
- I can explain why short-lived certificates are preferred in Zero Trust
- I understand the identity bootstrapping problem and solutions
- I can explain SPIFFE IDs and their purpose
Implementation
- My CA can generate a root certificate with correct extensions
- My CA can sign CSRs and add SPIFFE URIs to SANs
- My server requires and verifies client certificates
- My client presents certificates and verifies server certificates
- Certificate rotation works without connection interruption
- Error handling provides useful debugging information
Security
- Private keys never leave their generating component
- Certificates are short-lived (1 hour default)
- Certificate chain is complete and verifiable
- Invalid/expired/revoked certificates are rejected
- TLS 1.3 is enforced
Operations
- I can debug certificate issues using OpenSSL
- I can verify certificate chains manually
- I understand how to monitor certificate expiry
- I can explain how to rotate the CA key safely
13. Conclusion
This project teaches you the cryptographic foundation of Zero Trust: identity at the wire level. By building your own mTLS mesh, you understand not just how to configure TLS, but why each component exists and how they work together to eliminate network-based trust.
The skills from this project apply directly to:
- Service mesh implementation (Istio, Linkerd, Consul Connect)
- Cloud-native security (SPIFFE/SPIRE deployments)
- Enterprise PKI management
- Secure microservices architecture
- Security engineering interviews
After completing this project, you’ll never again treat TLS as a black box. You’ll understand the handshake, the certificates, the chain of trust, and how to debug when things go wrong.
This guide was expanded from ZERO_TRUST_ARCHITECTURE_DEEP_DIVE.md. For the complete learning path, see the project index.