LEARN HASHICORP VAULT DEEP DIVE
Learn HashiCorp Vault: From Zero to Secrets Master
Goal: Deeply understand HashiCorp Vault—from the cryptographic foundations (Shamir’s Secret Sharing) to building production-grade secrets infrastructure, understanding every layer of how it works behind the scenes.
Why Vault Exists
Before Vault, secrets management was a nightmare:
The Problems Vault Solves
- Secrets Sprawl: Passwords, API keys, and certificates scattered across:
- Environment variables (readable by any process)
- Config files (committed to git “accidentally”)
- CI/CD systems (Jenkins credentials, GitHub secrets)
- Developer laptops (
.envfiles everywhere)
- Static Credentials: Credentials that never change:
- Database passwords from 2019 still in production
- AWS access keys shared across teams
- No way to know who has access to what
- No Audit Trail: When something leaks:
- Who accessed this secret?
- When was it last rotated?
- Where is it being used?
- Manual Rotation: Changing a database password means:
- Update the database
- Update 15 applications
- Hope you didn’t miss any
- Pray nothing breaks at 3 AM
Vault’s Core Value Proposition
┌─────────────────────────────────────────────────────────────┐
│ HashiCorp Vault │
├─────────────────────────────────────────────────────────────┤
│ ✓ Centralized secrets storage with encryption │
│ ✓ Dynamic, short-lived credentials │
│ ✓ Fine-grained access control (who can access what) │
│ ✓ Complete audit log (who accessed when) │
│ ✓ Automatic rotation and revocation │
│ ✓ Encryption as a service │
└─────────────────────────────────────────────────────────────┘
Core Architecture: How Vault Works Behind the Scenes
The Three-Layer Model
┌─────────────────────────────────────────────────────────────┐
│ EXTERNAL ACCESS │
│ (HTTP API / CLI / UI / Agent) │
├─────────────────────────────────────────────────────────────┤
│ ENCRYPTION BARRIER │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │
│ │ │ Auth │ │ Secrets │ │ System │ │ │
│ │ │ Methods │ │ Engines │ │ Backend │ │ │
│ │ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │ │
│ │ │ │ │ │ │
│ │ └─────────────┼─────────────────┘ │ │
│ │ ▼ │ │
│ │ ┌──────────────┐ │ │
│ │ │ CORE │ │ │
│ │ │ (Router + │ │ │
│ │ │ Policies + │ │ │
│ │ │ Tokens) │ │ │
│ │ └──────────────┘ │ │
│ └─────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ STORAGE BACKEND │
│ (Raft / Consul / S3 / etc.) - ALL DATA ENCRYPTED │
└─────────────────────────────────────────────────────────────┘
The Seal/Unseal Mechanism (Shamir’s Secret Sharing)
This is the cryptographic heart of Vault:
INITIALIZATION
│
▼
┌────────────────────────┐
│ Master Key (256b) │
│ (Never stored!) │
└───────────┬────────────┘
│
Shamir's Secret Sharing
(Split into N shares,
require K to reconstruct)
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Share 1 │ │ Share 2 │ │ Share 3 │ ... (N shares)
│ (Key A) │ │ (Key B) │ │ (Key C) │
└─────────┘ └─────────┘ └─────────┘
│ │ │
│ Given to different trusted operators
│
UNSEAL PROCESS
│
Any K shares combine to reconstruct
│
▼
┌────────────────────────┐
│ Master Key │
│ (Reconstructed) │
└───────────┬────────────┘
│
▼
┌────────────────────────┐
│ Decrypts Root Key │
│ (Stored encrypted │
│ in storage backend) │
└───────────┬────────────┘
│
▼
┌────────────────────────┐
│ Root Key decrypts │
│ Encryption Key │
│ (Held in memory only) │
└────────────────────────┘
│
▼
VAULT IS UNSEALED
(Data can be decrypted)
Key Hierarchy
Understanding the key hierarchy is critical:
- Master Key (256-bit): Never stored, only exists when unsealed
- Root Key: Stored encrypted in backend, decrypted by master key
- Encryption Key: Held in memory, encrypts/decrypts all data
- Data Keys: Per-item keys for certain operations
The Encryption Barrier
The barrier is the conceptual wall between Vault’s encrypted storage and the outside world:
- Inside the barrier: Plaintext data, policies, secrets
- Outside the barrier: Encrypted blobs, storage backend
- Crossing the barrier: Requires the encryption key (only in memory when unsealed)
Storage Backend sees: Vault Core sees:
─────────────────── ─────────────────
key: "secret/data/myapp" key: "secret/data/myapp"
val: "AES256:a9f8b2..." val: {"password": "hunter2"}
Project List
Projects are ordered to build understanding from cryptographic foundations to production operations.
Project 1: Build a Mini Secrets Manager (Understand the Problem)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Python
- Alternative Programming Languages: Go, Rust, Node.js
- Coolness Level: Level 3: Genuinely Clever
- Business Potential: 1. The “Resume Gold”
- Difficulty: Level 2: Intermediate
- Knowledge Area: Cryptography / Secrets Management
- Software or Tool: Python cryptography library, SQLite
- Main Book: “Serious Cryptography” by Jean-Philippe Aumasson
What you’ll build: A simple secrets manager from scratch that encrypts secrets at rest, requires a master password to unlock, and provides basic CRUD operations. This is NOT production-ready—it’s designed to teach you what problems Vault solves.
Why it teaches Vault: Before learning Vault, you need to understand the problem space. By building a naive implementation, you’ll discover the hard problems: key management, access control, audit logging, rotation. Then you’ll appreciate why Vault exists.
Core challenges you’ll face:
- Key derivation → maps to how to turn a password into an encryption key
- Storing encrypted data → maps to the encryption barrier concept
- Authentication → maps to why tokens exist
- Access control → maps to why policies exist
Key Concepts:
- Symmetric Encryption: “Serious Cryptography” Chapter 4 - Jean-Philippe Aumasson
- Key Derivation Functions: “Serious Cryptography” Chapter 8 - Jean-Philippe Aumasson
- AEAD Encryption: Cryptography.io Documentation
Difficulty: Intermediate Time estimate: Weekend Prerequisites: Basic Python, understanding of encryption concepts
Real world outcome:
$ python secrets_manager.py init
Master password: ********
Confirm password: ********
Secrets manager initialized.
Encryption key derived from password using Argon2id.
Data will be stored in ./secrets.db (encrypted)
$ python secrets_manager.py set database/password
Master password: ********
Enter secret value: ********
Secret 'database/password' stored successfully.
$ python secrets_manager.py get database/password
Master password: ********
database/password: hunter2
$ python secrets_manager.py list
Master password: ********
Secrets:
- database/password
- api/stripe_key
- ssh/private_key
# Try with wrong password
$ python secrets_manager.py get database/password
Master password: wrongpassword
Error: Decryption failed. Invalid master password.
Implementation Hints:
Architecture of a simple secrets manager:
┌─────────────────────────────────────────┐
│ Master Password │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Key Derivation Function (Argon2id) │
│ - Salt stored in database │
│ - High memory/time cost │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ 256-bit Encryption Key │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ AES-256-GCM Encrypt/Decrypt │
│ - Each secret has unique nonce │
│ - Authentication tag verifies data │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ SQLite Database │
│ - salt: bytes │
│ - secrets: (path, nonce, ciphertext) │
└─────────────────────────────────────────┘
Key questions to ask yourself:
- What happens if someone steals the database file?
- How would you share this with a team?
- How would you rotate the master password?
- How would you know if someone accessed a secret?
- What if you need different access levels?
These questions reveal why Vault is complex—it solves all of them.
Learning milestones:
- Encryption/decryption works → You understand symmetric encryption
- Wrong password fails → You understand authentication tags
- You see the limitations → You understand why Vault exists
- You want audit logs → You understand observability needs
Project 2: Implement Shamir’s Secret Sharing (Understand Seal/Unseal)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Python
- Alternative Programming Languages: Go, Rust, C
- Coolness Level: Level 4: Hardcore Tech Flex
- Business Potential: 1. The “Resume Gold”
- Difficulty: Level 3: Advanced
- Knowledge Area: Cryptography / Threshold Cryptography
- Software or Tool: Python, finite field arithmetic
- Main Book: “Serious Cryptography” by Jean-Philippe Aumasson
What you’ll build: A pure implementation of Shamir’s Secret Sharing algorithm—the cryptographic scheme that Vault uses to split the master key into shares. You’ll be able to split a secret into N shares and reconstruct it from any K shares.
Why it teaches Vault: The seal/unseal mechanism is Vault’s most distinctive feature. By implementing Shamir’s algorithm yourself, you’ll understand exactly how Vault protects its master key and why the threshold scheme provides both security and availability.
Core challenges you’ll face:
- Finite field arithmetic → maps to why GF(2^8) or GF(prime)
- Polynomial interpolation → maps to Lagrange interpolation
- Security properties → maps to information-theoretic security
- Threshold semantics → maps to K-of-N requirements
Key Concepts:
- Shamir’s Original Paper: Adi Shamir, “How to Share a Secret” (1979)
- Finite Field Arithmetic: “Serious Cryptography” Chapter 10
- Lagrange Interpolation: Any cryptography textbook
- Vault’s Implementation: Vault Shamir Package
Difficulty: Advanced Time estimate: 1 week Prerequisites: Basic algebra, polynomial math, Project 1 completed
Real world outcome:
$ python shamir.py split --secret "my-master-key-256-bits" --shares 5 --threshold 3
Splitting secret into 5 shares (threshold: 3)
Share 1: 01-7a3f8c2d1e4b...
Share 2: 02-9b4e7d6c5a3f...
Share 3: 03-2c8f1e7d4b6a...
Share 4: 04-5d9a3c8e2f1b...
Share 5: 05-8e2b4d7f1c9a...
Distribute these shares to different trusted operators.
Any 3 shares can reconstruct the secret.
Fewer than 3 shares reveal NOTHING about the secret.
$ python shamir.py combine
Enter share (or 'done'): 01-7a3f8c2d1e4b...
Enter share (or 'done'): 03-2c8f1e7d4b6a...
Enter share (or 'done'): 05-8e2b4d7f1c9a...
Enter share (or 'done'): done
Reconstructed secret: my-master-key-256-bits
# Verify with wrong shares
$ python shamir.py combine
Enter share (or 'done'): 01-7a3f8c2d1e4b...
Enter share (or 'done'): 03-2c8f1e7d4b6a...
Enter share (or 'done'): done
Error: Insufficient shares. Need 3, got 2.
Implementation Hints:
The core math of Shamir’s scheme:
- Secret as a constant: The secret S is the constant term (a₀) of a polynomial
- Random coefficients: Generate K-1 random coefficients (a₁, a₂, …, aₖ₋₁)
- Polynomial: f(x) = a₀ + a₁x + a₂x² + … + aₖ₋₁xᵏ⁻¹
- Shares: Evaluate f(1), f(2), …, f(N) to get N shares
- Reconstruction: Use Lagrange interpolation to find f(0) = a₀ = S
Polynomial of degree K-1:
f(x) = S + a₁x + a₂x² + ... + aₖ₋₁xᵏ⁻¹
To split secret S into 5 shares with threshold 3:
- Generate random a₁, a₂ (polynomial of degree 2)
- f(x) = S + a₁x + a₂x²
- Share 1 = (1, f(1))
- Share 2 = (2, f(2))
- Share 3 = (3, f(3))
- Share 4 = (4, f(4))
- Share 5 = (5, f(5))
Any 3 points determine a degree-2 polynomial uniquely.
All arithmetic must be done in a finite field (not regular integers) to:
- Keep shares the same size as the secret
- Prevent information leakage
Use GF(256) for byte-wise operations or GF(p) for a large prime p.
Lagrange interpolation to find f(0):
f(0) = Σᵢ yᵢ × ∏ⱼ≠ᵢ (0 - xⱼ)/(xᵢ - xⱼ)
Questions to explore:
- Why is 2-of-3 secure but 1-of-3 would reveal information?
- What happens if you use regular integers instead of finite fields?
- How does Vault handle the x-coordinates (share indices)?
Learning milestones:
- Split/combine works → You understand the algorithm
- You grasp finite fields → You understand the security property
- K-1 shares reveal nothing → You understand information-theoretic security
- You see how Vault uses this → You understand seal/unseal deeply
Project 3: Vault Dev Server Deep Exploration
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Shell/Bash, HCL
- Alternative Programming Languages: Python (hvac client), Go
- Coolness Level: Level 2: Practical but Forgettable
- Business Potential: 1. The “Resume Gold”
- Difficulty: Level 1: Beginner
- Knowledge Area: Vault Operations / CLI
- Software or Tool: Vault CLI, jq
- Main Book: “Running HashiCorp Vault in Production” (HashiCorp Learn)
What you’ll build: An interactive exploration script that demonstrates every major Vault concept using the dev server—authentication, secrets engines, policies, tokens, audit logs—with commentary explaining what’s happening internally.
Why it teaches Vault: The dev server is pre-unsealed and has a root token, making it perfect for learning. This project builds muscle memory with the CLI while explaining the internals at each step.
Core challenges you’ll face:
- Understanding paths → maps to Vault’s path-based routing
- Token mechanics → maps to authentication and authorization
- Secrets engine types → maps to KV, dynamic, transit, PKI
- Policy syntax → maps to HCL and capabilities
Key Concepts:
- Vault CLI: HashiCorp Learn - Getting Started
- Paths and Routing: HashiCorp Docs - Path Routing
- Token Lifecycle: HashiCorp Docs - Tokens
Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic command line, Projects 1-2 helpful but not required
Real world outcome:
$ ./vault-exploration.sh
=== VAULT DEV SERVER EXPLORATION ===
[1/10] Starting Vault dev server...
Vault server started at http://127.0.0.1:8200
Root Token: hvs.root-token-dev-only
[INTERNAL] The dev server:
- Auto-initializes with 1 key share, threshold 1
- Auto-unseals immediately
- Stores data in memory (lost on restart)
- Binds to localhost only
[2/10] Checking seal status...
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false ← Dev server auto-unseals
Total Shares 1 ← Production uses 5
Threshold 1 ← Production uses 3
Version 1.15.4
Storage Type inmem ← Production uses raft/consul
[INTERNAL] In production, you'd see:
Sealed true (until operators provide threshold shares)
[3/10] Exploring the KV secrets engine...
$ vault secrets list
Path Type Description
---- ---- -----------
cubbyhole/ cubbyhole per-token private storage
secret/ kv key-value secrets (v2)
sys/ system system endpoints
[INTERNAL] Paths are like URLs:
- secret/data/myapp → KV v2 secret
- auth/token/create → Create a new token
- sys/health → Health check endpoint
[4/10] Writing and reading a secret...
$ vault kv put secret/database password=hunter2
$ vault kv get secret/database
====== Data ======
Key Value
--- -----
password hunter2
[INTERNAL] What actually happened:
1. Request authenticated via root token
2. Router sent request to 'secret/' engine
3. KV engine encrypted value with encryption key
4. Encrypted blob stored in storage backend
[5/10] Creating a policy...
$ vault policy write readonly-db -<<EOF
path "secret/data/database" {
capabilities = ["read"]
}
EOF
[INTERNAL] Policy capabilities:
- create: Write to non-existing paths
- read: Read secrets
- update: Write to existing paths
- delete: Delete secrets
- list: List keys (not values)
- sudo: Access protected paths
- deny: Explicit deny
[6/10] Creating a token with limited policy...
$ vault token create -policy=readonly-db
Key Value
--- -----
token hvs.limited-token...
token_policies ["default" "readonly-db"]
[7/10] Testing limited token...
$ VAULT_TOKEN=hvs.limited-token vault kv get secret/database
✓ Success: Can read database secret
$ VAULT_TOKEN=hvs.limited-token vault kv put secret/database password=newpass
✗ Error: permission denied (missing 'update' capability)
[8/10] Enabling audit logging...
$ vault audit enable file file_path=/tmp/vault-audit.log
$ vault kv get secret/database
$ cat /tmp/vault-audit.log | jq
{
"type": "request",
"auth": {"token_type": "service", "policies": ["root"]},
"request": {"path": "secret/data/database", "operation": "read"},
"response": {"data": {"data": {"password": "hmac-sha256:..."}}}
}
[INTERNAL] Audit logs show:
- Who (token/identity)
- What (path/operation)
- When (timestamp)
- Response (HMAC'd for privacy)
[9/10] Token TTL and renewal...
$ vault token create -ttl=1m -policy=readonly-db
$ sleep 70
$ VAULT_TOKEN=hvs.expired vault kv get secret/database
Error: token expired
[INTERNAL] Short TTLs are a security feature:
- Compromised tokens expire quickly
- Forces regular re-authentication
- Limits blast radius of breaches
[10/10] Clean up...
Vault dev server stopped.
=== EXPLORATION COMPLETE ===
Implementation Hints:
Create an interactive script that:
- Starts vault dev server in background
- Runs each demonstration with explanatory comments
- Pauses for user to observe
- Shows internal state changes
Key commands to explore:
# Secrets
vault kv put/get/delete/list
# Auth
vault auth list/enable/disable
vault login
vault token create/lookup/revoke
# Policies
vault policy list/read/write/delete
# System
vault status
vault operator init/unseal/seal
vault audit enable/disable/list
# Advanced
vault lease revoke
vault path-help <path>
Questions to explore:
- What’s the difference between
secret/andcubbyhole/? - Why does every token get the
defaultpolicy? - What happens to child tokens when parent is revoked?
- Why are audit logs HMAC’d?
Learning milestones:
- You’re comfortable with the CLI → Basic operations are muscle memory
- You understand path routing → Everything is a path
- You understand token-policy binding → Authorization makes sense
- You can read audit logs → You understand observability
Project 4: Static Secrets with KV Engine (Version 1 vs 2)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Shell/Bash, Python (hvac)
- Alternative Programming Languages: Go, Node.js, Java
- Coolness Level: Level 2: Practical but Forgettable
- Business Potential: 2. The “Micro-SaaS / Pro Tool”
- Difficulty: Level 1: Beginner
- Knowledge Area: Secrets Storage / Versioning
- Software or Tool: Vault KV Secrets Engine
- Main Book: HashiCorp Learn - KV Secrets Engine
What you’ll build: A secrets management workflow using the KV secrets engine, demonstrating versioning, metadata, soft delete, and programmatic access from multiple languages.
Why it teaches Vault: KV is the most commonly used secrets engine. Understanding the difference between v1 (simple) and v2 (versioned), and how to access secrets programmatically, is foundational for any Vault user.
Core challenges you’ll face:
- KV v1 vs v2 paths → maps to understanding the data/ prefix
- Versioning → maps to rollback and history
- Metadata → maps to custom_metadata and check-and-set
- Programmatic access → maps to API vs CLI
Key Concepts:
- KV Secrets Engine: HashiCorp Docs - KV v2
- Python hvac Client: hvac Documentation
- Vault HTTP API: HashiCorp Docs - API
Difficulty: Beginner Time estimate: Weekend Prerequisites: Project 3 completed
Real world outcome:
# Enable KV v2 at a custom path
$ vault secrets enable -path=myapp -version=2 kv
Success! Enabled the kv secrets engine at: myapp/
# Store application secrets
$ vault kv put myapp/config \
db_host=postgres.internal \
db_user=appuser \
db_pass=secretpassword \
api_key=sk-live-abc123
# View with metadata
$ vault kv get -format=json myapp/config | jq
{
"data": {
"data": {
"db_host": "postgres.internal",
"db_user": "appuser",
"db_pass": "secretpassword",
"api_key": "sk-live-abc123"
},
"metadata": {
"created_time": "2025-01-15T10:30:00Z",
"version": 1
}
}
}
# Update a secret (creates version 2)
$ vault kv patch myapp/config db_pass=newpassword
Key Value
--- -----
version 2
# View version history
$ vault kv metadata get myapp/config
...
versions:
1: created_time=2025-01-15T10:30:00Z deletion_time=n/a
2: created_time=2025-01-15T10:35:00Z deletion_time=n/a
# Rollback to version 1
$ vault kv rollback -version=1 myapp/config
Key Value
--- -----
version 3 ← New version with v1's data
# Python programmatic access
$ python app.py
Connected to Vault at http://127.0.0.1:8200
Database config:
Host: postgres.internal
User: appuser
Pass: newpassword
Implementation Hints:
KV v1 vs v2 path differences:
KV v1:
PUT secret/myapp → Store secret
GET secret/myapp → Read secret
KV v2:
PUT secret/data/myapp → Store secret (data/ prefix)
GET secret/data/myapp → Read secret
GET secret/metadata/myapp → Read metadata only
DELETE secret/data/myapp → Soft delete (recoverable)
DELETE secret/metadata/myapp → Permanent delete
Python hvac client example:
import hvac
# Connect to Vault
client = hvac.Client(url='http://127.0.0.1:8200')
client.token = 'hvs.your-token'
# Read secret (KV v2)
secret = client.secrets.kv.v2.read_secret_version(
path='config',
mount_point='myapp'
)
data = secret['data']['data']
print(f"DB Host: {data['db_host']}")
# Write secret
client.secrets.kv.v2.create_or_update_secret(
path='config',
secret={'db_pass': 'rotated-password'},
mount_point='myapp'
)
Check-and-Set (CAS) for safe updates:
# Only update if current version matches
$ vault kv put -cas=2 myapp/config db_pass=evennewerpass
# Fails if version is not 2
Questions to explore:
- When would you use KV v1 vs v2?
- How many versions should you retain?
- How do you handle secret rotation with versioning?
- What’s the performance difference between v1 and v2?
Learning milestones:
- CRUD operations work → You understand basic KV
- Versioning makes sense → You understand v2 value
- API access works → You can integrate with applications
- CAS prevents races → You understand concurrent access
Project 5: Dynamic Database Credentials
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: SQL, Shell, Python
- Alternative Programming Languages: Go, Java, Node.js
- Coolness Level: Level 4: Hardcore Tech Flex
- Business Potential: 4. The “Open Core” Infrastructure
- Difficulty: Level 3: Advanced
- Knowledge Area: Dynamic Secrets / Database Security
- Software or Tool: Vault Database Secrets Engine, PostgreSQL
- Main Book: HashiCorp Learn - Dynamic Secrets
What you’ll build: A dynamic credentials system where Vault generates unique, short-lived PostgreSQL credentials for each application request. No more shared database passwords!
Why it teaches Vault: Dynamic secrets are Vault’s killer feature—they eliminate the concept of static, long-lived credentials. This project shows you how Vault talks to databases to create/revoke credentials on demand.
Core challenges you’ll face:
- Database connection configuration → maps to root credentials management
- Role definitions → maps to SQL templates for users
- TTL and leases → maps to automatic revocation
- Credential rotation → maps to security lifecycle
Key Concepts:
- Dynamic Secrets: HashiCorp Learn - Database Secrets Engine
- Leases and Renewal: HashiCorp Docs - Lease
- PostgreSQL Integration: HashiCorp Docs - PostgreSQL
Difficulty: Advanced Time estimate: 1 week Prerequisites: Basic SQL, PostgreSQL, Projects 1-4 completed
Real world outcome:
# Start PostgreSQL (Docker)
$ docker run -d --name postgres \
-e POSTGRES_PASSWORD=rootpass \
-p 5432:5432 postgres:15
# Enable database secrets engine
$ vault secrets enable database
# Configure PostgreSQL connection
$ vault write database/config/mydb \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@localhost:5432/postgres?sslmode=disable" \
allowed_roles="readonly,readwrite" \
username="postgres" \
password="rootpass"
# Create a readonly role
$ vault write database/roles/readonly \
db_name=mydb \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Request dynamic credentials
$ vault read database/creds/readonly
Key Value
--- -----
lease_id database/creds/readonly/abc123...
lease_duration 1h
username v-token-readonly-xyz789
password A1b2C3d4E5f6G7h8
# Verify in PostgreSQL
$ psql -U v-token-readonly-xyz789 -h localhost postgres
Password: A1b2C3d4E5f6G7h8
postgres=> SELECT * FROM users; -- Works!
postgres=> DELETE FROM users; -- Fails! (readonly)
# Watch automatic revocation
$ vault lease revoke database/creds/readonly/abc123...
All revocation operations queued successfully!
$ psql -U v-token-readonly-xyz789 -h localhost postgres
FATAL: role "v-token-readonly-xyz789" does not exist
# User automatically deleted!
# Application integration
$ python app.py
[10:00:00] Requesting database credentials from Vault...
[10:00:01] Got credentials: v-token-readonly-abc123 (TTL: 1h)
[10:00:01] Connected to PostgreSQL
[10:00:01] Fetched 100 rows from users table
[10:30:00] Credentials expiring in 30m, renewing lease...
[10:30:01] Lease renewed for another 1h
[11:00:00] Application shutdown, revoking credentials...
[11:00:01] Credentials revoked successfully
Implementation Hints:
How dynamic secrets work internally:
Application Vault PostgreSQL
│ │ │
│ GET /database/creds/ │ │
│ readonly │ │
├────────────────────────►│ │
│ │ CREATE ROLE "v-xxx" │
│ │ WITH PASSWORD 'yyy' │
│ ├───────────────────────────►│
│ │ OK │
│ │◄───────────────────────────┤
│ { username, password, │ │
│ lease_id, ttl } │ │
│◄────────────────────────┤ │
│ │ │
│ │ [After TTL expires] │
│ │ │
│ │ DROP ROLE "v-xxx" │
│ ├───────────────────────────►│
│ │ OK │
│ │◄───────────────────────────┤
Key configuration considerations:
- Root rotation: Vault can rotate its own root credentials
- TTL strategy: Balance security (short) vs performance (long)
- Max connections: Each credential is a DB connection
- Role templates: SQL must be idempotent for retries
Application pattern for dynamic credentials:
import hvac
import psycopg2
import time
import threading
class VaultDBClient:
def __init__(self, vault_url, token, role):
self.client = hvac.Client(url=vault_url, token=token)
self.role = role
self.credentials = None
self.lease_id = None
self._get_new_credentials()
self._start_renewal_thread()
def _get_new_credentials(self):
response = self.client.secrets.database.generate_credentials(
name=self.role
)
self.credentials = response['data']
self.lease_id = response['lease_id']
self.ttl = response['lease_duration']
def _renew_lease(self):
self.client.sys.renew_lease(
lease_id=self.lease_id,
increment=self.ttl
)
def get_connection(self):
return psycopg2.connect(
host='localhost',
user=self.credentials['username'],
password=self.credentials['password'],
dbname='postgres'
)
Questions to explore:
- What happens if Vault can’t reach the database during revocation?
- How do you handle connection pooling with dynamic credentials?
- What’s the right TTL for your use case?
- How do you test with dynamic credentials in development?
Learning milestones:
- Credentials are generated → You understand dynamic secrets
- Revocation works → You understand leases
- Application integrates → You can use this in production
- You see the security benefit → No more shared passwords
Project 6: Transit Secrets Engine (Encryption as a Service)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Python, Shell
- Alternative Programming Languages: Go, Java, Node.js
- Coolness Level: Level 4: Hardcore Tech Flex
- Business Potential: 4. The “Open Core” Infrastructure
- Difficulty: Level 3: Advanced
- Knowledge Area: Cryptography / Encryption Services
- Software or Tool: Vault Transit Secrets Engine
- Main Book: “Serious Cryptography” by Jean-Philippe Aumasson
What you’ll build: An encryption-as-a-service system where applications encrypt/decrypt data without ever handling encryption keys directly. The keys never leave Vault.
Why it teaches Vault: Transit solves the “key management problem”—how do you encrypt data in your applications without managing keys? This project shows Vault as a cryptographic service, not just a secrets store.
Core challenges you’ll face:
- Key types and algorithms → maps to AES-GCM, RSA, ECDSA
- Key versioning and rotation → maps to re-wrapping without re-encrypting data
- Convergent encryption → maps to searchable encrypted data
- Performance → maps to batch operations and caching
Key Concepts:
- Transit Engine: HashiCorp Docs - Transit
- Key Rotation: HashiCorp Learn - Encryption Key Rotation
- Convergent Encryption: HashiCorp Docs - Convergent Mode
Difficulty: Advanced Time estimate: 1 week Prerequisites: Basic cryptography, Projects 1-5 completed
Real world outcome:
# Enable transit secrets engine
$ vault secrets enable transit
# Create an encryption key
$ vault write -f transit/keys/myapp-key
Success! Data written to: transit/keys/myapp-key
# Encrypt data (base64 encoded plaintext)
$ vault write transit/encrypt/myapp-key \
plaintext=$(echo "credit-card: 4111-1111-1111-1111" | base64)
Key Value
--- -----
ciphertext vault:v1:8SDd3WHDOjf7mq69CyCqy...
# Decrypt data
$ vault write transit/decrypt/myapp-key \
ciphertext="vault:v1:8SDd3WHDOjf7mq69CyCqy..."
Key Value
--- -----
plaintext Y3JlZGl0LWNhcmQ6IDQxMTEtMTExMS0xMTExLTExMTE=
$ echo "Y3JlZGl0LWNhcmQ6IDQxMTEtMTExMS0xMTExLTExMTE=" | base64 -d
credit-card: 4111-1111-1111-1111
# Rotate the key (old ciphertexts still work!)
$ vault write -f transit/keys/myapp-key/rotate
$ vault read transit/keys/myapp-key
Key Value
--- -----
latest_version 2
min_decryption_version 1
min_encryption_version 2
# Re-wrap: Update ciphertext to use new key version
$ vault write transit/rewrap/myapp-key \
ciphertext="vault:v1:8SDd3WHDOjf7mq69CyCqy..."
Key Value
--- -----
ciphertext vault:v2:NewCiphertextWithV2Key...
# Application example
$ python encrypt_app.py
Enter PII to store: SSN: 123-45-6789
Encrypting with Vault Transit...
Stored in database: {
"id": 1,
"ssn_encrypted": "vault:v2:abc123...",
"created_at": "2025-01-15"
}
Enter search SSN: 123-45-6789
Searching... (using convergent encryption)
Found record ID: 1
Decrypting with Vault Transit...
SSN: 123-45-6789
Implementation Hints:
How Transit encryption works:
Application Vault Transit Database
│ │ │
│ POST /transit/encrypt │ │
│ {plaintext: base64(data)} │ │
├─────────────────────────────►│ │
│ │ │
│ ┌────────┴────────┐ │
│ │ 1. Get latest │ │
│ │ key version │ │
│ │ 2. Generate IV │ │
│ │ 3. AES-GCM │ │
│ │ encrypt │ │
│ │ 4. Prepend │ │
│ │ "vault:v1:" │ │
│ └────────┬────────┘ │
│ │ │
│ {ciphertext: vault:v1:...} │ │
│◄─────────────────────────────┤ │
│ │ │
│ INSERT (ciphertext) │ │
├──────────────────────────────┼──────────────────────────────►│
│ │ │
Key version in ciphertext format:
vault:v1:base64-ciphertext
│ │
│ └─ Key version used for encryption
└────── Prefix identifying Vault ciphertext
Convergent encryption for searchable encrypted data:
# Same plaintext + same context = same ciphertext
$ vault write transit/encrypt/myapp-key \
plaintext=$(echo "SSN:123-45-6789" | base64) \
context=$(echo "user-search-key" | base64)
ciphertext: vault:v1:abc123... # Deterministic!
# You can now search: WHERE ssn_encrypted = 'vault:v1:abc123...'
Python application pattern:
class VaultEncryption:
def __init__(self, vault_client, key_name):
self.client = vault_client
self.key_name = key_name
def encrypt(self, plaintext: str) -> str:
response = self.client.secrets.transit.encrypt_data(
name=self.key_name,
plaintext=base64.b64encode(plaintext.encode()).decode()
)
return response['data']['ciphertext']
def decrypt(self, ciphertext: str) -> str:
response = self.client.secrets.transit.decrypt_data(
name=self.key_name,
ciphertext=ciphertext
)
return base64.b64decode(response['data']['plaintext']).decode()
def rotate_key(self):
self.client.secrets.transit.rotate_key(name=self.key_name)
def rewrap(self, ciphertext: str) -> str:
response = self.client.secrets.transit.rewrap_data(
name=self.key_name,
ciphertext=ciphertext
)
return response['data']['ciphertext']
Questions to explore:
- Why is key rotation “free” with Transit?
- What’s the performance overhead of calling Vault for every encrypt?
- When should you use convergent vs non-convergent encryption?
- How do you batch encrypt for performance?
Learning milestones:
- Encrypt/decrypt works → You understand basic Transit
- Key rotation doesn’t break old data → You understand versioning
- Convergent encryption enables search → You understand advanced patterns
- Application integrates cleanly → You can use this in production
Project 7: PKI Secrets Engine (Certificate Authority)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Shell, OpenSSL
- Alternative Programming Languages: Go, Python (cryptography)
- Coolness Level: Level 4: Hardcore Tech Flex
- Business Potential: 4. The “Open Core” Infrastructure
- Difficulty: Level 4: Expert
- Knowledge Area: PKI / TLS / Certificate Management
- Software or Tool: Vault PKI Secrets Engine, OpenSSL
- Main Book: “Bulletproof TLS and PKI” by Ivan Ristić
What you’ll build: A complete internal Certificate Authority using Vault, with root CA, intermediate CA, and the ability to issue short-lived TLS certificates for services.
Why it teaches Vault: Manual certificate management is a nightmare (hello, expired certs at 2 AM). This project shows how Vault automates the entire PKI lifecycle—issuance, rotation, and revocation.
Core challenges you’ll face:
- CA hierarchy → maps to root vs intermediate CAs
- Certificate templates → maps to roles and allowed domains
- Short-lived certs → maps to TTL and automatic rotation
- Revocation → maps to CRL and OCSP
Key Concepts:
- PKI Fundamentals: “Bulletproof TLS and PKI” by Ivan Ristić
- Vault PKI Engine: HashiCorp Docs - PKI Secrets Engine
- Certificate Lifecycle: HashiCorp Learn - Build Your Own CA
Difficulty: Expert Time estimate: 2 weeks Prerequisites: TLS/PKI basics, Projects 1-6 completed
Real world outcome:
# Create Root CA (kept offline in production)
$ vault secrets enable -path=pki_root pki
$ vault secrets tune -max-lease-ttl=87600h pki_root # 10 years
$ vault write pki_root/root/generate/internal \
common_name="My Organization Root CA" \
ttl=87600h
Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----...
issuing_ca -----BEGIN CERTIFICATE-----...
serial_number 7a:2b:...
# Create Intermediate CA (used for daily operations)
$ vault secrets enable -path=pki_int pki
$ vault secrets tune -max-lease-ttl=43800h pki_int # 5 years
$ vault write pki_int/intermediate/generate/internal \
common_name="My Organization Intermediate CA"
Key Value
--- -----
csr -----BEGIN CERTIFICATE REQUEST-----...
# Sign intermediate with root
$ vault write pki_root/root/sign-intermediate \
csr=@intermediate.csr \
format=pem_bundle \
ttl=43800h
$ vault write pki_int/intermediate/set-signed \
certificate=@signed_intermediate.pem
# Create a role for issuing certificates
$ vault write pki_int/roles/web-servers \
allowed_domains="internal.company.com,company.local" \
allow_subdomains=true \
max_ttl=720h \
key_type="ec" \
key_bits=256
# Issue a certificate for a web server
$ vault write pki_int/issue/web-servers \
common_name="api.internal.company.com" \
ttl=24h
Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----...
issuing_ca -----BEGIN CERTIFICATE-----...
private_key -----BEGIN EC PRIVATE KEY-----...
serial_number 3f:8c:...
expiration 1705420800
# Verify the certificate
$ echo "$CERT" | openssl x509 -text -noout
Certificate:
Subject: CN = api.internal.company.com
Issuer: CN = My Organization Intermediate CA
Validity:
Not Before: Jan 15 10:00:00 2025 GMT
Not After : Jan 16 10:00:00 2025 GMT
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
# Use with nginx
$ vault write pki_int/issue/web-servers \
common_name="web.internal.company.com" \
-format=json | jq -r '.data.certificate' > /etc/nginx/ssl/cert.pem
$ vault write pki_int/issue/web-servers \
common_name="web.internal.company.com" \
-format=json | jq -r '.data.private_key' > /etc/nginx/ssl/key.pem
$ curl https://web.internal.company.com
Hello from TLS-secured server with Vault-issued certificate!
Implementation Hints:
PKI hierarchy with Vault:
┌─────────────────────────────────────┐
│ ROOT CA │
│ (pki_root, 10 year TTL) │
│ OFFLINE - Only signs intermediates │
└─────────────────┬───────────────────┘
│ Signs
▼
┌─────────────────────────────────────┐
│ INTERMEDIATE CA │
│ (pki_int, 5 year TTL) │
│ ONLINE - Issues end-entity certs │
└─────────────────┬───────────────────┘
│ Issues
▼
┌─────────────────────────────────────┐
│ END-ENTITY CERTS │
│ (Short-lived, hours to days) │
│ - api.internal.company.com │
│ - web.internal.company.com │
│ - db.internal.company.com │
└─────────────────────────────────────┘
Why short-lived certificates:
- No revocation needed: Certs expire before damage can be done
- Automatic rotation: Forces good hygiene
- Reduced blast radius: Compromised cert is useless in 24 hours
Role configuration options:
vault write pki_int/roles/web-servers \
allowed_domains="company.com" # Domain restrictions
allow_subdomains=true # *.company.com allowed
allow_bare_domains=false # company.com not allowed
allow_glob_domains=false # No wildcards
max_ttl=720h # Max 30 days
key_type="ec" # ECDSA keys
key_bits=256 # P-256 curve
require_cn=true # CN must be provided
server_flag=true # TLS server cert
client_flag=false # Not a client cert
Questions to explore:
- Why use an intermediate CA instead of issuing directly from root?
- How do you distribute the CA certificate to clients?
- What’s the right TTL for certificates?
- How do you handle certificate revocation?
Learning milestones:
- CA chain works → You understand PKI hierarchy
- Certificates validate → You understand trust chains
- Services use Vault certs → You can integrate with applications
- Short-lived certs auto-rotate → You understand modern PKI
Project 8: Authentication Methods Deep Dive
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Shell, Python, Go
- Alternative Programming Languages: Java, Node.js
- Coolness Level: Level 3: Genuinely Clever
- Business Potential: 3. The “Service & Support” Model
- Difficulty: Level 3: Advanced
- Knowledge Area: Authentication / Identity / Zero Trust
- Software or Tool: Vault Auth Methods (AppRole, JWT, LDAP, Kubernetes)
- Main Book: HashiCorp Learn - Authentication
What you’ll build: A comprehensive authentication setup with multiple methods: AppRole for applications, JWT/OIDC for humans via SSO, and Kubernetes auth for pods.
Why it teaches Vault: Authentication is the front door to Vault. Understanding how different auth methods work—and when to use each—is critical for production deployments.
Core challenges you’ll face:
- AppRole security model → maps to role-id vs secret-id
- JWT validation → maps to JWKS and claims mapping
- Kubernetes auth → maps to service account tokens
- Policy binding → maps to auth method to policy mapping
Key Concepts:
- Auth Methods Overview: HashiCorp Docs - Auth Methods
- AppRole: HashiCorp Learn - AppRole
- JWT/OIDC: HashiCorp Docs - JWT Auth
- Kubernetes Auth: HashiCorp Docs - Kubernetes
Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Projects 1-7 completed, basic Kubernetes helpful
Real world outcome:
# ============ AppRole (Machine Authentication) ============
# Enable AppRole
$ vault auth enable approle
# Create a role for the "webapp" application
$ vault write auth/approle/role/webapp \
token_policies="webapp-policy" \
token_ttl=1h \
token_max_ttl=4h \
secret_id_ttl=24h \
secret_id_num_uses=0
# Get role-id (public, can be baked into image)
$ vault read auth/approle/role/webapp/role-id
role_id: 675a50e7-...
# Get secret-id (private, delivered securely at runtime)
$ vault write -f auth/approle/role/webapp/secret-id
secret_id: 7d97e71d-...
secret_id_accessor: 1e77c8e2-...
# Application authenticates
$ vault write auth/approle/login \
role_id=675a50e7-... \
secret_id=7d97e71d-...
Key Value
--- -----
token hvs.webapp-token-...
token_policies ["default" "webapp-policy"]
token_ttl 1h
# ============ JWT/OIDC (Human Authentication) ============
# Enable JWT auth (for SSO via Okta, Auth0, etc.)
$ vault auth enable -path=oidc oidc
# Configure with your IdP
$ vault write auth/oidc/config \
oidc_discovery_url="https://your-tenant.okta.com" \
oidc_client_id="vault-client-id" \
oidc_client_secret="your-secret" \
default_role="developer"
# Create role mapping claims to policies
$ vault write auth/oidc/role/developer \
user_claim="email" \
allowed_redirect_uris="http://localhost:8250/oidc/callback" \
token_policies="developer-policy" \
bound_claims='{"groups": ["engineering"]}'
# Human logs in (opens browser)
$ vault login -method=oidc role=developer
Complete the login via your OIDC provider...
Success! You are now authenticated.
# ============ Kubernetes (Pod Authentication) ============
# Enable Kubernetes auth
$ vault auth enable kubernetes
# Configure with cluster info
$ vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes.default.svc" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# Create role for pods in "app" namespace
$ vault write auth/kubernetes/role/webapp \
bound_service_account_names="webapp-sa" \
bound_service_account_namespaces="production" \
policies="webapp-policy" \
ttl=1h
# From inside a Kubernetes pod:
$ curl -s --request POST \
--data '{"role": "webapp", "jwt": "'$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)'"}' \
http://vault:8200/v1/auth/kubernetes/login | jq
{
"auth": {
"client_token": "hvs.k8s-token-...",
"policies": ["default", "webapp-policy"]
}
}
Implementation Hints:
AppRole security model:
┌─────────────────────────────────────────────────────────────┐
│ AppRole Login │
├─────────────────────────────────────────────────────────────┤
│ │
│ role-id secret-id │
│ (public) (private) │
│ ───────── ─────────── │
│ • Baked into Docker image • Delivered at runtime │
│ • Stored in CI/CD • Single use (optionally) │
│ • Identifies "who" • Proves "authorization" │
│ │
│ Together they authenticate: │
│ role-id + secret-id = Token with attached policies │
│ │
└─────────────────────────────────────────────────────────────┘
Secret-ID delivery patterns:
- Push mode: Orchestrator writes secret-id, app reads it
- Pull mode: Trusted system fetches secret-id for app
- Response wrapping: Secret-id wrapped in single-use token
Kubernetes auth flow:
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Pod │ │ Vault │ │ Kubernetes │
│ │ │ │ │ API │
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘
│ │ │
│ 1. Login with │ │
│ SA token │ │
├─────────────────►│ │
│ │ 2. Validate │
│ │ SA token │
│ ├─────────────────►│
│ │ │
│ │ 3. Token valid │
│ │ for sa/ns │
│ │◄─────────────────┤
│ │ │
│ 4. Vault token │ │
│ with policies │ │
│◄─────────────────┤ │
Questions to explore:
- How do you bootstrap the first secret-id delivery?
- What’s the difference between bound_claims and claim_mappings?
- How do you handle service mesh (Envoy/Istio) with Vault auth?
- When should you use token auth vs method-specific auth?
Learning milestones:
- AppRole works → You understand machine auth
- OIDC works → You understand human auth with SSO
- Kubernetes auth works → You understand pod identity
- You choose the right method → You understand trade-offs
Project 9: Policy System and Sentinel (Authorization)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: HCL (Policy), Sentinel (Enterprise)
- Alternative Programming Languages: N/A
- Coolness Level: Level 3: Genuinely Clever
- Business Potential: 3. The “Service & Support” Model
- Difficulty: Level 3: Advanced
- Knowledge Area: Authorization / Access Control / Compliance
- Software or Tool: Vault Policies, Sentinel (Enterprise)
- Main Book: HashiCorp Docs - Policies
What you’ll build: A comprehensive policy system with least-privilege access, templated policies for multi-tenancy, and (if using Enterprise) Sentinel policies for advanced governance.
Why it teaches Vault: Policies are how you express “who can do what.” Understanding capabilities, path templating, and policy composition is essential for secure Vault deployments.
Core challenges you’ll face:
- Capability model → maps to create, read, update, delete, list, sudo, deny
- Path patterns → maps to wildcards and regex
- Templated policies → maps to identity-based paths
- Policy composition → maps to multiple policies on one token
Key Concepts:
- ACL Policies: HashiCorp Docs - Policies
- Policy Templating: HashiCorp Docs - Policy Templating
- Sentinel (Enterprise): HashiCorp Docs - Sentinel
Difficulty: Advanced Time estimate: 1 week Prerequisites: Projects 1-8 completed
Real world outcome:
# ============ Basic Policy ============
# File: webapp-policy.hcl
# Read own secrets
path "secret/data/webapp/*" {
capabilities = ["read", "list"]
}
# Create/update own secrets
path "secret/data/webapp/*" {
capabilities = ["create", "update"]
}
# Deny access to other apps' secrets
path "secret/data/+/*" {
capabilities = ["deny"]
}
# ============ Templated Policy ============
# File: user-workspace.hcl
# Each user gets their own workspace
# User's personal secrets
path "secret/data/users/{{identity.entity.name}}/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Team secrets (read-only)
path "secret/data/teams/{{identity.entity.metadata.team}}/*" {
capabilities = ["read", "list"]
}
# ============ Admin Policy ============
# File: admin-policy.hcl
# Full access to KV
path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Manage auth methods
path "auth/*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Manage policies (but not root policy)
path "sys/policies/acl/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Cannot modify root policy
path "sys/policies/acl/root" {
capabilities = ["deny"]
}
# Apply policies
$ vault policy write webapp webapp-policy.hcl
$ vault policy write user-workspace user-workspace.hcl
$ vault policy write admin admin-policy.hcl
# Test policy with limited token
$ vault token create -policy=webapp
$ VAULT_TOKEN=hvs.webapp-token
# Can read webapp secrets
$ vault kv get secret/webapp/config
✓ Success
# Cannot read other app secrets
$ vault kv get secret/otherapp/config
Error: permission denied
# Cannot access sys paths
$ vault auth list
Error: permission denied
# ============ Sentinel Policy (Enterprise) ============
# File: require-mfa.sentinel
import "mfa"
import "strings"
# Require MFA for production secrets
precond = rule {
strings.has_prefix(request.path, "secret/data/prod/")
}
main = rule when precond {
mfa.methods.totp.valid
}
# Apply Sentinel policy
$ vault write sys/policies/egp/require-mfa-prod \
policy=@require-mfa.sentinel \
paths="secret/data/prod/*" \
enforcement_level="hard-mandatory"
Implementation Hints:
Policy capability reference:
| Capability | HTTP Verbs | Use Case |
|---|---|---|
| create | POST/PUT to new path | Write new secrets |
| read | GET | Read secrets |
| update | POST/PUT to existing | Modify secrets |
| delete | DELETE | Remove secrets |
| list | LIST | Enumerate paths |
| sudo | N/A | Access protected paths |
| deny | N/A | Explicit deny (overrides allow) |
Path pattern syntax:
# Exact match
path "secret/data/webapp/config" { ... }
# Wildcard (any single segment)
path "secret/data/+/config" { ... } # Matches secret/data/app1/config
# Glob (any path suffix)
path "secret/data/webapp/*" { ... } # Matches secret/data/webapp/foo/bar
# Parameter (capture for templating)
path "secret/data/{{identity.entity.name}}/*" { ... }
Policy composition (additive):
Token has policies: [default, webapp, readonly-db]
Effective capabilities = union of all policies
If ANY policy denies, access is denied (deny is absolute)
Questions to explore:
- What’s the difference between missing capability and explicit deny?
- How do you test policies before deployment?
- What’s the performance impact of complex policies?
- How do you audit policy changes?
Learning milestones:
- Basic policies work → You understand capabilities
- Templated policies work → You understand multi-tenancy
- Policy composition works → You understand privilege management
- You design least-privilege → You understand security principles
Project 10: Vault Agent and Auto-Auth
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: HCL (Agent config), Shell
- Alternative Programming Languages: N/A
- Coolness Level: Level 3: Genuinely Clever
- Business Potential: 3. The “Service & Support” Model
- Difficulty: Level 3: Advanced
- Knowledge Area: Sidecar Patterns / Secret Injection
- Software or Tool: Vault Agent
- Main Book: HashiCorp Docs - Vault Agent
What you’ll build: A Vault Agent deployment that automatically authenticates, fetches secrets, and renders them into config files or environment variables for applications.
Why it teaches Vault: Most applications can’t integrate with Vault directly. Vault Agent acts as a sidecar/daemon that handles authentication, token renewal, and secret delivery, making Vault integration seamless for any application.
Core challenges you’ll face:
- Auto-auth configuration → maps to authentication without human interaction
- Template rendering → maps to secrets as files/env vars
- Token management → maps to sinks and caching
- Secret refresh → maps to TTL and re-rendering
Key Concepts:
- Vault Agent: HashiCorp Docs - Agent
- Template Rendering: HashiCorp Docs - Agent Templates
- Auto-Auth: HashiCorp Docs - Auto-Auth
Difficulty: Advanced Time estimate: 1 week Prerequisites: Projects 1-8 completed
Real world outcome:
# vault-agent-config.hcl
pid_file = "/tmp/vault-agent.pid"
vault {
address = "http://vault:8200"
}
auto_auth {
method "approle" {
config = {
role_id_file_path = "/etc/vault/role-id"
secret_id_file_path = "/etc/vault/secret-id"
remove_secret_id_file_after_reading = true
}
}
sink "file" {
config = {
path = "/tmp/vault-token"
}
}
}
cache {
use_auto_auth_token = true
}
listener "tcp" {
address = "127.0.0.1:8100"
tls_disable = true
}
template {
source = "/etc/vault/templates/app.conf.tpl"
destination = "/app/config/app.conf"
command = "pkill -HUP myapp" # Reload app on change
}
template {
source = "/etc/vault/templates/env.tpl"
destination = "/app/config/.env"
perms = 0600
}
# Template file: /etc/vault/templates/app.conf.tpl
{{ with secret "secret/data/webapp/config" }}
[database]
host = {{ .Data.data.db_host }}
port = {{ .Data.data.db_port }}
user = {{ .Data.data.db_user }}
password = {{ .Data.data.db_pass }}
[api]
key = {{ .Data.data.api_key }}
{{ end }}
# Dynamic database credentials
{{ with secret "database/creds/readonly" }}
[database_dynamic]
user = {{ .Data.username }}
password = {{ .Data.password }}
# Note: These credentials refresh automatically!
{{ end }}
# Template file: /etc/vault/templates/env.tpl
{{ with secret "secret/data/webapp/config" }}
export DB_HOST={{ .Data.data.db_host }}
export DB_PASS={{ .Data.data.db_pass }}
export API_KEY={{ .Data.data.api_key }}
{{ end }}
# Start Vault Agent
$ vault agent -config=/etc/vault/agent-config.hcl
==> Vault agent started! Log data will stream in below:
Auto-auth: starting method "approle"
Auto-auth: successfully authenticated
Template: rendered template at /app/config/app.conf
Template: rendered template at /app/config/.env
# Verify rendered config
$ cat /app/config/app.conf
[database]
host = postgres.internal
port = 5432
user = webapp
password = secretpassword
[api]
key = sk-live-abc123
[database_dynamic]
user = v-approle-readonly-abc123
password = A1b2C3d4E5f6G7h8
# Application just reads the file - no Vault SDK needed!
$ ./myapp --config=/app/config/app.conf
App started with database config loaded from file
Implementation Hints:
Vault Agent architecture:
┌─────────────────────────────────────────────────────────────┐
│ Application Pod │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────┐ ┌───────────────────────────┐ │
│ │ │ │ │ │
│ │ Vault │ │ Application │ │
│ │ Agent │ │ Container │ │
│ │ │ │ │ │
│ │ • Auto-auth │ │ Reads: │ │
│ │ • Template │───────►│ • /app/config/app.conf │ │
│ │ • Cache │ │ • /app/config/.env │ │
│ │ │ │ │ │
│ └───────┬───────┘ │ No Vault SDK needed! │ │
│ │ │ │ │
│ │ └───────────────────────────┘ │
│ │ │
└──────────┼─────────────────────────────────────────────────┘
│
▼
┌──────────────┐
│ Vault │
│ Server │
└──────────────┘
Template functions:
# Read a secret
{{ with secret "secret/data/app" }}
{{ .Data.data.password }}
{{ end }}
# Dynamic credentials with lease refresh
{{ with secret "database/creds/readonly" }}
username={{ .Data.username }}
password={{ .Data.password }}
{{ end }}
# Conditional rendering
{{ if .Data.data.feature_enabled }}
FEATURE=true
{{ end }}
# Looping
{{ range secrets "secret/metadata/apps/" }}
{{ with secret (printf "secret/data/apps/%s" .) }}
{{ .Data.data.name }}={{ .Data.data.value }}
{{ end }}
{{ end }}
Agent modes:
- File sink: Write token to file for app to read
- Template mode: Render secrets into config files
- Cache mode: Act as local Vault proxy with caching
Questions to explore:
- How does Agent handle secret rotation?
- What’s the security of the token sink file?
- How do you handle Agent failures?
- What’s the difference between Agent and CSI Driver?
Learning milestones:
- Auto-auth works → You understand machine authentication
- Templates render → You understand secret injection
- Dynamic creds refresh → You understand lease management
- Apps work without SDK → You understand the sidecar pattern
Project 11: High Availability with Raft Storage
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: HCL, Shell
- Alternative Programming Languages: N/A
- Coolness Level: Level 3: Genuinely Clever
- Business Potential: 4. The “Open Core” Infrastructure
- Difficulty: Level 4: Expert
- Knowledge Area: Distributed Systems / HA / Consensus
- Software or Tool: Vault with Integrated Storage (Raft)
- Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann
What you’ll build: A highly available Vault cluster using integrated Raft storage with 3-5 nodes, demonstrating leader election, failover, and data replication.
Why it teaches Vault: Production Vault requires high availability. This project teaches you about Raft consensus, leader election, and how Vault maintains consistency across nodes—critical knowledge for operating Vault at scale.
Core challenges you’ll face:
- Raft consensus → maps to leader election and log replication
- Cluster initialization → maps to join vs bootstrap
- Failover behavior → maps to what happens when leader dies
- Network partitions → maps to split-brain prevention
Key Concepts:
- Raft Consensus: “Designing Data-Intensive Applications” Chapter 9 - Martin Kleppmann
- Vault Integrated Storage: HashiCorp Docs - Integrated Storage
- HA Reference Architecture: HashiCorp Learn - Raft Reference Architecture
Difficulty: Expert Time estimate: 2 weeks Prerequisites: Docker/Kubernetes, Projects 1-10 completed
Real world outcome:
# Start a 3-node Vault cluster
$ docker-compose up -d
Creating vault-1 ... done
Creating vault-2 ... done
Creating vault-3 ... done
# Initialize the first node
$ vault operator init -key-shares=5 -key-threshold=3
Unseal Key 1: xxxx
Unseal Key 2: xxxx
Unseal Key 3: xxxx
Unseal Key 4: xxxx
Unseal Key 5: xxxx
Initial Root Token: hvs.xxxx
# Unseal node 1
$ vault operator unseal # Repeat 3 times with different keys
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
HA Enabled true
HA Cluster https://vault-1:8201
HA Mode active ← This is the leader
# Join other nodes to the cluster
$ VAULT_ADDR=http://vault-2:8200 vault operator raft join http://vault-1:8200
Key Value
--- -----
Joined true
$ VAULT_ADDR=http://vault-3:8200 vault operator raft join http://vault-1:8200
Key Value
--- -----
Joined true
# Unseal the other nodes
$ VAULT_ADDR=http://vault-2:8200 vault operator unseal
$ VAULT_ADDR=http://vault-3:8200 vault operator unseal
# Check cluster status
$ vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-1 vault-1:8201 leader true
vault-2 vault-2:8201 follower true
vault-3 vault-3:8201 follower true
# Simulate leader failure
$ docker stop vault-1
vault-1
# New leader is elected automatically
$ VAULT_ADDR=http://vault-2:8200 vault status
HA Mode active ← vault-2 is now leader
$ vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-2 vault-2:8201 leader true ← New leader
vault-3 vault-3:8201 follower true
vault-1 vault-1:8201 (offline) true
# When vault-1 comes back, it rejoins as follower
$ docker start vault-1
$ # (unseal vault-1)
$ vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-2 vault-2:8201 leader true
vault-3 vault-3:8201 follower true
vault-1 vault-1:8201 follower true ← Rejoined
Implementation Hints:
Raft consensus in Vault:
┌─────────────────────────────────────────────────────────────┐
│ Raft Cluster │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ vault-1 │ │ vault-2 │ │ vault-3 │ │
│ │ LEADER │ │ FOLLOWER │ │ FOLLOWER │ │
│ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
│ │ │ │ │
│ │ Replicate │ Replicate │ │
│ ├───────────────►├───────────────►│ │
│ │ │ │ │
│ │ ◄──────────── │ ◄──────────── │ │
│ │ Ack │ Ack │ │
│ │ │ │ │
│ ┌─────┴─────┐ ┌─────┴─────┐ ┌─────┴─────┐ │
│ │ Raft │ │ Raft │ │ Raft │ │
│ │ Log │ │ Log │ │ Log │ │
│ └───────────┘ └───────────┘ └───────────┘ │
│ │
│ All writes go to leader, replicated to majority │
│ before commit. Reads can go to any node. │
│ │
└─────────────────────────────────────────────────────────────┘
Node configuration:
# vault-1.hcl
storage "raft" {
path = "/vault/data"
node_id = "vault-1"
retry_join {
leader_api_addr = "http://vault-2:8200"
}
retry_join {
leader_api_addr = "http://vault-3:8200"
}
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true # Use TLS in production!
}
cluster_addr = "http://vault-1:8201"
api_addr = "http://vault-1:8200"
Why 3 or 5 nodes:
- Raft requires majority quorum
- 3 nodes: can lose 1 (majority = 2)
- 5 nodes: can lose 2 (majority = 3)
- Even numbers don’t help (no tie-breaker)
Questions to explore:
- What happens during a network partition?
- How do you perform a planned failover?
- What’s the performance impact of more nodes?
- How do you backup a Raft cluster?
Learning milestones:
- Cluster forms → You understand Raft basics
- Failover works → You understand leader election
- Data is consistent → You understand replication
- Recovery works → You understand operations
Project 12: Kubernetes Integration (Vault Secrets Operator)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: YAML, HCL
- Alternative Programming Languages: Helm, Kustomize
- Coolness Level: Level 4: Hardcore Tech Flex
- Business Potential: 4. The “Open Core” Infrastructure
- Difficulty: Level 4: Expert
- Knowledge Area: Kubernetes / Secrets Injection / GitOps
- Software or Tool: Vault Secrets Operator, Kubernetes
- Main Book: “Kubernetes in Action” by Marko Lukša
What you’ll build: A complete Kubernetes integration where pods automatically receive secrets from Vault via the Vault Secrets Operator (VSO) or CSI Driver, with no application changes needed.
Why it teaches Vault: Kubernetes is where most modern workloads run. Understanding the different integration patterns (Agent Sidecar, CSI Driver, Secrets Operator) and when to use each is essential for platform teams.
Core challenges you’ll face:
- VSO vs CSI vs Agent → maps to choosing the right pattern
- VaultAuth custom resource → maps to Kubernetes service account auth
- VaultStaticSecret/VaultDynamicSecret → maps to secret synchronization
- Secret rotation → maps to how updates propagate to pods
Key Concepts:
- Vault Secrets Operator: HashiCorp Docs - VSO
- Vault CSI Provider: HashiCorp Docs - CSI
- Agent Sidecar Injector: HashiCorp Docs - Agent Injector
Difficulty: Expert Time estimate: 2 weeks Prerequisites: Kubernetes proficiency, Projects 1-11 completed
Real world outcome:
# ============ Vault Auth Configuration ============
# vaultauth.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: vault-auth
namespace: app
spec:
method: kubernetes
mount: kubernetes
kubernetes:
role: webapp
serviceAccount: webapp-sa
audiences:
- vault
---
# ============ Static Secret ============
# static-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: webapp-config
namespace: app
spec:
type: kv-v2
mount: secret
path: webapp/config
destination:
name: webapp-config # Creates K8s Secret with this name
create: true
refreshAfter: 30s
vaultAuthRef: vault-auth
---
# ============ Dynamic Secret (Database Creds) ============
# dynamic-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
name: db-creds
namespace: app
spec:
mount: database
path: creds/readonly
destination:
name: db-creds
create: true
renewalPercent: 67 # Renew when 67% of TTL elapsed
vaultAuthRef: vault-auth
---
# ============ Application Deployment ============
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
namespace: app
spec:
replicas: 3
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
serviceAccountName: webapp-sa
containers:
- name: webapp
image: myapp:latest
envFrom:
- secretRef:
name: webapp-config # Static secrets as env vars
- secretRef:
name: db-creds # Dynamic creds as env vars
# No Vault SDK needed - just reads env vars!
# Install Vault Secrets Operator
$ helm install vault-secrets-operator hashicorp/vault-secrets-operator \
-n vault-secrets-operator-system --create-namespace
# Apply configurations
$ kubectl apply -f vaultauth.yaml
$ kubectl apply -f static-secret.yaml
$ kubectl apply -f dynamic-secret.yaml
$ kubectl apply -f deployment.yaml
# Verify secrets are synced
$ kubectl get secrets -n app
NAME TYPE DATA AGE
webapp-config Opaque 3 30s
db-creds Opaque 2 30s
$ kubectl get secret webapp-config -n app -o jsonpath='{.data.db_host}' | base64 -d
postgres.internal
# Watch dynamic credential rotation
$ kubectl get vaultdynamicsecret db-creds -n app -w
NAME VAULT PATH STATUS AGE
db-creds database/creds/ro Valid 1m
db-creds database/creds/ro Valid 31m # Renewed!
db-creds database/creds/ro Valid 61m # Renewed again!
# Pod sees fresh credentials
$ kubectl exec -n app deploy/webapp -- env | grep DB_
DB_USERNAME=v-k8s-readonly-xyz123
DB_PASSWORD=NewRotatedPassword
Implementation Hints:
Comparison of Kubernetes integration patterns:
| Pattern | Pros | Cons | Best For |
|---|---|---|---|
| VSO | Native K8s CRDs, no sidecars, automatic sync | Requires operator deployment | Most use cases |
| CSI Driver | Secrets as files, works with existing apps | Mounted at pod start only | File-based secrets |
| Agent Injector | Flexible templating, works with init | Sidecar overhead per pod | Complex transformations |
VSO workflow:
┌─────────────────────────────────────────────────────────────┐
│ Vault Secrets Operator │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. Watches VaultStaticSecret/VaultDynamicSecret CRDs │
│ │
│ 2. Authenticates to Vault using VaultAuth │
│ (Kubernetes Service Account) │
│ │
│ 3. Fetches secrets from Vault │
│ │
│ 4. Creates/updates Kubernetes Secrets │
│ │
│ 5. Pods mount/env the K8s Secrets (native K8s) │
│ │
│ 6. Periodically syncs changes (refreshAfter) │
│ │
└─────────────────────────────────────────────────────────────┘
Rollout on secret change:
# Add this annotation to trigger pod restart on secret change
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject-secret-checksum: "{{ .Values.secretChecksum }}"
Questions to explore:
- How do pods get updated when secrets change?
- What’s the security model of VSO vs Agent?
- How do you handle secrets across namespaces?
- What’s the backup strategy for Vault-managed K8s secrets?
Learning milestones:
- VSO syncs secrets → You understand the operator pattern
- Dynamic creds rotate → You understand lease management in K8s
- Apps work without changes → You understand the value proposition
- You choose the right pattern → You understand trade-offs
Project 13: Building a Custom Secrets Engine (Go)
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: Go
- Alternative Programming Languages: N/A (plugins must be Go)
- Coolness Level: Level 5: Pure Magic
- Business Potential: 5. The “Industry Disruptor”
- Difficulty: Level 5: Master
- Knowledge Area: SDK Development / Plugin Architecture
- Software or Tool: Vault Plugin SDK, Go
- Main Book: HashiCorp Docs - Building Plugins
What you’ll build: A custom secrets engine that generates dynamic credentials for an internal service or third-party API not natively supported by Vault.
Why it teaches Vault: This is the deepest understanding of Vault possible. By building a secrets engine, you’ll understand the plugin architecture, credential lifecycle, lease management, and how Vault is really just a framework for managing secrets of any kind.
Core challenges you’ll face:
- Plugin framework → maps to backend, paths, operations
- Credential lifecycle → maps to create, renew, revoke
- State management → maps to storage and encryption
- Testing plugins → maps to Docker-based testing
Key Concepts:
- Plugin Development: HashiCorp Docs - Custom Secrets Engine
- Vault Plugin SDK: GitHub - vault-plugin-sdk
- Go Programming: “The Go Programming Language” by Donovan & Kernighan
Difficulty: Master Time estimate: 1 month+ Prerequisites: Go proficiency, deep Vault knowledge, all previous projects
Real world outcome:
# Your custom plugin managing API keys for an internal service
$ vault secrets enable -path=myservice myservice-secrets-plugin
# Configure the backend
$ vault write myservice/config \
api_endpoint="https://internal-api.company.com" \
admin_token="admin-secret-token"
# Create a role
$ vault write myservice/roles/readonly \
permissions="read" \
ttl="1h" \
max_ttl="24h"
# Generate dynamic credentials
$ vault read myservice/creds/readonly
Key Value
--- -----
lease_id myservice/creds/readonly/abc123...
lease_duration 1h
api_key myservice-v1-readonly-xyz789
permissions read
# The plugin called your internal API to create this key!
# When the lease expires, Vault calls the revoke endpoint.
# Revoke early
$ vault lease revoke myservice/creds/readonly/abc123...
All revocation operations queued successfully!
# Plugin deleted the API key from your internal service
Implementation Hints:
Plugin structure:
// backend.go
package myservice
import (
"context"
"github.com/hashicorp/vault/sdk/framework"
"github.com/hashicorp/vault/sdk/logical"
)
type backend struct {
*framework.Backend
}
func Factory(ctx context.Context, conf *logical.BackendConfig) (logical.Backend, error) {
b := &backend{}
b.Backend = &framework.Backend{
BackendType: logical.TypeLogical,
Paths: []*framework.Path{
pathConfig(b),
pathRoles(b),
pathCreds(b),
},
Secrets: []*framework.Secret{
secretApiKey(b),
},
}
return b, nil
}
// path_creds.go - Generate credentials
func pathCreds(b *backend) *framework.Path {
return &framework.Path{
Pattern: "creds/" + framework.GenericNameRegex("name"),
Fields: map[string]*framework.FieldSchema{
"name": {Type: framework.TypeString},
},
Operations: map[logical.Operation]framework.OperationHandler{
logical.ReadOperation: &framework.PathOperation{
Callback: b.pathCredsRead,
},
},
}
}
func (b *backend) pathCredsRead(ctx context.Context, req *logical.Request, d *framework.FieldData) (*logical.Response, error) {
roleName := d.Get("name").(string)
// 1. Get role config from storage
role, err := b.getRole(ctx, req.Storage, roleName)
// 2. Call your internal API to create credentials
apiKey, err := b.createApiKey(role.Permissions)
// 3. Return as a secret with lease
resp := b.Secret(secretApiKeyType).Response(map[string]interface{}{
"api_key": apiKey.Key,
"permissions": apiKey.Permissions,
}, map[string]interface{}{
"api_key_id": apiKey.ID, // Internal data for revocation
})
resp.Secret.TTL = role.TTL
resp.Secret.MaxTTL = role.MaxTTL
return resp, nil
}
// secret_api_key.go - Handle revocation
func secretApiKey(b *backend) *framework.Secret {
return &framework.Secret{
Type: secretApiKeyType,
Revoke: b.secretApiKeyRevoke,
Renew: b.secretApiKeyRenew,
}
}
func (b *backend) secretApiKeyRevoke(ctx context.Context, req *logical.Request, d *framework.FieldData) (*logical.Response, error) {
apiKeyID := req.Secret.InternalData["api_key_id"].(string)
// Call your internal API to delete the key
err := b.deleteApiKey(apiKeyID)
return nil, err
}
Questions to explore:
- How do you handle the backend losing connection to the external API?
- What’s the right granularity for a secrets engine?
- How do you version and upgrade plugins?
- How do you test plugins in CI/CD?
Learning milestones:
- Plugin compiles and loads → You understand the framework
- Credentials generate → You understand the lifecycle
- Revocation works → You understand lease management
- Plugin is production-ready → You understand operations
Project Comparison Table
| Project | Difficulty | Time | Depth of Understanding | Fun Factor |
|---|---|---|---|---|
| 1. Mini Secrets Manager | Intermediate | Weekend | ⭐⭐⭐⭐ | ⭐⭐⭐ |
| 2. Shamir’s Secret Sharing | Advanced | 1 week | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| 3. Dev Server Exploration | Beginner | Weekend | ⭐⭐⭐ | ⭐⭐⭐ |
| 4. KV Secrets Engine | Beginner | Weekend | ⭐⭐⭐ | ⭐⭐ |
| 5. Dynamic Database Creds | Advanced | 1 week | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| 6. Transit (Encryption) | Advanced | 1 week | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| 7. PKI (Certificates) | Expert | 2 weeks | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| 8. Authentication Methods | Advanced | 1-2 weeks | ⭐⭐⭐⭐ | ⭐⭐⭐ |
| 9. Policy System | Advanced | 1 week | ⭐⭐⭐⭐ | ⭐⭐⭐ |
| 10. Vault Agent | Advanced | 1 week | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| 11. HA with Raft | Expert | 2 weeks | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| 12. Kubernetes Integration | Expert | 2 weeks | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| 13. Custom Secrets Engine | Master | 1 month+ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
Recommendation
If you want to understand the internals (your goal):
Start with Project 1 (Mini Secrets Manager) → Project 2 (Shamir’s Secret Sharing)
These projects teach you the cryptographic foundations. You’ll understand WHY Vault works the way it does before you use it.
If you need to use Vault quickly for work:
Start with Project 3 (Dev Server) → Project 4 (KV) → Project 5 (Dynamic Secrets)
This path gets you productive fastest while building foundational knowledge.
If you’re building a platform:
Focus on Project 8 (Auth Methods) → Project 10 (Vault Agent) → Project 12 (Kubernetes)
Platform engineers need to understand how applications integrate with Vault without code changes.
If you want to be a Vault expert:
Complete all projects, especially:
- Project 2 (Shamir’s) — Understand the cryptographic heart
- Project 11 (HA Raft) — Understand distributed operations
- Project 13 (Custom Engine) — The ultimate deep dive
Final Overall Project: Enterprise Secrets Platform
- File: LEARN_HASHICORP_VAULT_DEEP_DIVE.md
- Main Programming Language: HCL, Go, YAML
- Alternative Programming Languages: Python (automation), Terraform
- Coolness Level: Level 5: Pure Magic
- Business Potential: 5. The “Industry Disruptor”
- Difficulty: Level 5: Master
- Knowledge Area: Enterprise Security / Platform Engineering
- Software or Tool: Vault Enterprise (or OSS), Kubernetes, Terraform
- Main Book: HashiCorp Sentinel Guide
What you’ll build: A complete enterprise secrets platform including:
- Multi-cluster Vault with performance replication
- Disaster recovery configuration
- Namespace-based multi-tenancy
- Complete auth method coverage (OIDC for humans, AppRole/K8s for machines)
- Dynamic credentials for databases, cloud providers, PKI
- Vault Agent/VSO for Kubernetes workloads
- Terraform-managed configuration (GitOps)
- Sentinel policies for compliance
- Full audit logging to SIEM
- Operational runbooks and DR procedures
Why this is the ultimate test: This synthesizes everything: cryptography, distributed systems, Kubernetes, security policies, operations. It’s what enterprise Vault looks like.
Core challenges you’ll face:
- Multi-cluster coordination → Performance vs DR replication
- Namespace design → Team boundaries and delegation
- Auth method matrix → Right method for each use case
- Compliance requirements → SOC2, PCI, HIPAA mapping
- DR testing → Regular failover drills
- Operational excellence → Monitoring, alerting, runbooks
Real world outcome:
Enterprise Vault Platform:
├── Infrastructure (Terraform)
│ ├── vault-cluster-primary/
│ ├── vault-cluster-dr/
│ └── vault-cluster-performance/
├── Configuration (Terraform Vault Provider)
│ ├── auth-methods/
│ │ ├── oidc-humans/
│ │ ├── approle-apps/
│ │ └── kubernetes/
│ ├── secrets-engines/
│ │ ├── kv/
│ │ ├── database/
│ │ ├── pki/
│ │ └── transit/
│ ├── policies/
│ │ ├── team-a/
│ │ ├── team-b/
│ │ └── platform/
│ └── namespaces/
│ ├── team-a/
│ └── team-b/
├── Kubernetes (VSO/Agent)
│ ├── vault-secrets-operator/
│ └── vault-auth-configs/
├── Monitoring
│ ├── prometheus-rules/
│ └── grafana-dashboards/
└── Runbooks
├── unsealing.md
├── dr-failover.md
├── secret-rotation.md
└── incident-response.md
# Verify platform health
$ vault status
Sealed: false
HA Enabled: true
HA Mode: active
Replication: performance primary, dr primary
$ vault namespace list
team-a/
team-b/
platform/
$ terraform plan
No changes. Infrastructure is up to date.
Learning milestones:
- Platform deploys end-to-end → You understand architecture
- Teams can self-service → You understand delegation
- DR failover works → You understand resilience
- Compliance audits pass → You understand governance
Resources Summary
Essential Books
- “Serious Cryptography” by Jean-Philippe Aumasson — Understand the crypto behind Vault
- “Designing Data-Intensive Applications” by Martin Kleppmann — Understand Raft and distributed systems
- “Bulletproof TLS and PKI” by Ivan Ristić — Understand PKI for Project 7
Official Resources
- HashiCorp Vault Documentation — Authoritative reference
- HashiCorp Learn - Vault — Hands-on tutorials
- Vault Architecture Internals — Deep dive into how it works
Community Resources
- Vault GitHub — Source code, especially
/shamirand/sdk - Vault Certification Study Guide — Structured learning path
Summary
| # | Project | Main Language |
|---|---|---|
| 1 | Build a Mini Secrets Manager | Python |
| 2 | Implement Shamir’s Secret Sharing | Python |
| 3 | Vault Dev Server Deep Exploration | Shell/Bash, HCL |
| 4 | Static Secrets with KV Engine | Shell, Python |
| 5 | Dynamic Database Credentials | SQL, Shell, Python |
| 6 | Transit Secrets Engine | Python, Shell |
| 7 | PKI Secrets Engine | Shell, OpenSSL |
| 8 | Authentication Methods Deep Dive | Shell, Python, Go |
| 9 | Policy System and Sentinel | HCL |
| 10 | Vault Agent and Auto-Auth | HCL, Shell |
| 11 | High Availability with Raft Storage | HCL, Shell |
| 12 | Kubernetes Integration | YAML, HCL |
| 13 | Building a Custom Secrets Engine | Go |
| Final | Enterprise Secrets Platform | HCL, Go, YAML |