Project 2: Payment Tokenization Vault
Project 2: Payment Tokenization Vault
Project Overview
| Attribute | Value |
|---|---|
| Difficulty | Level 3: Advanced |
| Time Estimate | 1-2 weeks |
| Programming Language | C |
| Knowledge Area | Payment Security / Cryptography |
| Key Technologies | AES-256, Tokenization, Format-Preserving Encryption |
| Coolness Level | Level 3: Genuinely Clever |
| Business Potential | 4. The “Open Core” Infrastructure |
Learning Objectives
By completing this project, you will:
- Understand tokenization architecture - Learn why tokenization is THE core technique in modern payment security
- Implement AES-256 encryption - Master symmetric encryption for data at rest
- Design format-preserving tokens - Create tokens that pass Luhn validation for legacy system compatibility
- Build secure key management - Understand key lifecycle, rotation, and storage
- Create vault architecture - Design bidirectional token-to-PAN mapping with O(1) lookup
- Understand PCI scope reduction - Learn how tokenization removes systems from compliance requirements
The Core Question You’re Answering
“How can merchants process payments without ever seeing the actual card number?”
This is the central problem tokenization solves. Consider this scenario:
- Customer enters card
4532015112830366on your website - Your server receives…
tok_9f8g7h6j5k4l3m2n(a token) - You store, process, and reference the token everywhere
- Only the tokenization vault knows the mapping back to the real card
- Your entire system never touches the real PAN
Why this matters:
┌──────────────────────────────────────────────────────────────────────────┐
│ PCI SCOPE: WITH vs. WITHOUT TOKENIZATION │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ WITHOUT TOKENIZATION WITH TOKENIZATION │
│ ═══════════════════ ═════════════════ │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Web Server │ ← PCI Scope │ Web Server │ │
│ │ (handles PAN) │ │ (handles token) │ ← OUT of PCI │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │
│ ┌────────▼────────┐ ┌────────▼────────┐ │
│ │ App Server │ ← PCI Scope │ App Server │ ← OUT of PCI │
│ │ (handles PAN) │ │ (handles token) │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │
│ ┌────────▼────────┐ ┌────────▼────────┐ │
│ │ Database │ ← PCI Scope │ Token Vault │ ← PCI Scope │
│ │ (stores PAN) │ │ (stores PAN) │ (ONLY this!)│
│ └─────────────────┘ └─────────────────┘ │
│ │
│ PCI Scope: 3 systems PCI Scope: 1 system │
│ Audit cost: $$$$$ Audit cost: $$ │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Tokenization reduces PCI scope from your entire infrastructure to a single, hardened vault.
Deep Theoretical Foundation
1. What is Tokenization?
Tokenization replaces sensitive data (PAN) with a non-sensitive substitute (token) that has no exploitable value outside the tokenization system.
Key properties of tokens:
- Non-reversible externally: You cannot derive the PAN from the token without access to the vault
- Unique mapping: Each PAN maps to one (or more) tokens
- Useless if stolen: Unlike encrypted data, a stolen token database reveals nothing
- Format-compatible: Can optionally match PAN format for legacy systems
Tokenization vs. Encryption:
| Aspect | Encryption | Tokenization |
|---|---|---|
| Reversibility | Key holder can decrypt | Only vault can map back |
| If data stolen | Attacker can try to crack | Token is useless outside vault |
| Format preservation | Ciphertext is different format | Token can match original format |
| Key management | Keys must be protected everywhere | Keys only in vault |
| PCI scope | All systems with keys are in scope | Only vault is in scope |
2. Token Types
┌─────────────────────────────────────────────────────────────────────────┐
│ TOKEN TYPES │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ RANDOM TOKEN │
│ ════════════ │
│ PAN: 4532015112830366 │
│ Token: tok_9f8g7h6j5k4l3m2n │
│ │
│ • No format relationship to PAN │
│ • Highest security │
│ • Requires system changes to accept new format │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ FORMAT-PRESERVING TOKEN │
│ ═══════════════════════ │
│ PAN: 4532015112830366 │
│ Token: 4532019876543210 ← Same length, passes Luhn! │
│ │
│ • Looks like a card number │
│ • Works with legacy systems expecting 16 digits │
│ • BIN may or may not be preserved │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ PARTIAL TOKEN (BIN-preserving) │
│ ══════════════════════════════ │
│ PAN: 4532015112830366 │
│ Token: 4532019876543210 │
│ ^^^^^^ │
│ BIN preserved (for routing) │
│ │
│ • Allows network routing without detokenization │
│ • Common in payment processing │
│ │
└─────────────────────────────────────────────────────────────────────────┘
3. AES-256 Encryption
AES (Advanced Encryption Standard) with 256-bit keys is the standard for protecting PANs at rest.
AES Block Cipher Basics:
- Block size: 128 bits (16 bytes)
- Key sizes: 128, 192, or 256 bits
- Mode of operation: CBC, GCM, CTR (each with different properties)
Why AES-256 for payment data:
- Symmetric: Same key encrypts and decrypts (fast)
- 256-bit: Quantum-resistant (AES-128 less so)
- Well-analyzed: 20+ years of cryptanalysis, no practical attacks
- Hardware acceleration: Modern CPUs have AES-NI instructions
Mode Selection for Token Vault:
| Mode | Pros | Cons | Use Case |
|---|---|---|---|
| ECB | Simple | Patterns visible, NEVER use | Never |
| CBC | Hides patterns | Needs IV, padding oracle attacks | Legacy |
| GCM | Auth + encryption | Nonce reuse catastrophic | Recommended |
| CTR | Streaming, parallelizable | No authentication | With HMAC |
Recommended: AES-256-GCM (authenticated encryption)
4. Format-Preserving Encryption (FPE)
FPE encrypts data while preserving its format (length, character set).
NIST Standards:
- FF1: Based on Feistel cipher, NIST SP 800-38G
- FF3: Alternate algorithm, also NIST approved
How FPE works (conceptual):
┌─────────────────────────────────────────────────────────────────────────┐
│ FORMAT-PRESERVING ENCRYPTION │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ Regular AES: │
│ Input: "4532015112830366" (16 chars) │
│ Output: "a7f3b2c1d4e5f6g7h8i9j0k1l2m3n4o5p6" (32+ hex chars) │
│ │
│ Format-Preserving (FF1): │
│ Input: "4532015112830366" (16 digits) │
│ Output: "9876543210123456" (16 digits) │
│ │
│ Algorithm (Feistel-based): │
│ ┌─────────────────────────────────────────────┐ │
│ │ 1. Split input into two halves: L, R │ │
│ │ 2. For i = 0 to rounds: │ │
│ │ L' = R │ │
│ │ R' = L XOR F(R, key, tweak, i) │ │
│ │ 3. Return L' || R' │ │
│ └─────────────────────────────────────────────┘ │
│ │
│ F() = AES-based round function that outputs │
│ values in the same alphabet as input │
│ │
└─────────────────────────────────────────────────────────────────────────┘
For this project: Start with random tokens. FPE is complex—implement it as an extension.
5. Key Management
Keys are the critical element. A stolen vault with encrypted data is useless without keys.
Key Hierarchy:
┌─────────────────────────────────────────────────────────────────────────┐
│ KEY HIERARCHY │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────┐ │
│ │ Master Key (MK) │ ← Stored in HSM or │
│ │ │ key management service │
│ └─────────┬──────────┘ │
│ │ │
│ ┌───────────────┼───────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Key Encryption │ │ Data Encryption │ │ Token Generation│ │
│ │ Key (KEK) │ │ Key (DEK) │ │ Key (TGK) │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Encrypts other Encrypts PANs Generates tokens │
│ keys at rest in the vault (HMAC-based) │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Key Rotation:
- Generate new DEK
- Re-encrypt all PANs with new DEK
- Encrypt new DEK with KEK
- Destroy old DEK
For this project: Simulate key management. Production would use HSM.
6. Vault Architecture
Bidirectional Mapping:
┌─────────────────────────────────────────────────────────────────────────┐
│ TOKEN VAULT DATA MODEL │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ TOKENIZE: PAN → Token │
│ ═══════════════════════ │
│ │
│ 1. Receive PAN: 4532015112830366 │
│ 2. Check if PAN already tokenized (hash lookup) │
│ 3. If exists: return existing token │
│ 4. If new: │
│ a. Generate random token │
│ b. Encrypt PAN with DEK │
│ c. Store mapping: token → encrypted_pan │
│ d. Store hash index: hash(PAN) → token │
│ e. Return token │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ DETOKENIZE: Token → PAN │
│ ═══════════════════════ │
│ │
│ 1. Receive token: tok_9f8g7h6j5k4l3m2n │
│ 2. Lookup: token → encrypted_pan │
│ 3. Decrypt encrypted_pan with DEK │
│ 4. Return PAN: 4532015112830366 │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ DATABASE SCHEMA: │
│ ═══════════════ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ tokens │ │
│ │ ├── token_id (PK): varchar(32) │ │
│ │ ├── encrypted_pan: varbinary(64) │ │
│ │ ├── pan_hash: char(64) ← For dedup, salted SHA-256 │ │
│ │ ├── created_at: timestamp │ │
│ │ ├── last_used: timestamp │ │
│ │ └── metadata: json ← BIN, card type, etc. │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ INDEX on pan_hash for O(1) duplicate detection │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Project Specification
What You’ll Build
A secure tokenization service that:
- Accepts card numbers via API
- Stores them encrypted in a vault
- Returns format-preserving (or random) tokens
- Provides authenticated detokenization
- Supports key rotation
Expected Output
# Tokenize a card
$ curl -X POST http://localhost:8080/tokenize \
-H "Content-Type: application/json" \
-d '{"pan": "4532015112830366"}'
{
"success": true,
"token": "tok_a1b2c3d4e5f6g7h8",
"metadata": {
"network": "Visa",
"last_four": "0366",
"bin": "453201"
}
}
# Detokenize (requires authentication)
$ curl -X POST http://localhost:8080/detokenize \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <api_key>" \
-d '{"token": "tok_a1b2c3d4e5f6g7h8"}'
{
"success": true,
"pan": "4532015112830366"
}
# Vault statistics (admin only)
$ curl http://localhost:8080/admin/stats \
-H "Authorization: Bearer <admin_key>"
{
"total_tokens": 15234,
"unique_pans": 12891,
"tokens_by_network": {
"Visa": 8234,
"Mastercard": 4521,
"Amex": 1479
},
"created_today": 523
}
Project Structure
tokenization_vault/
├── src/
│ ├── main.c # HTTP server entry point
│ ├── vault.c # Core vault logic
│ ├── vault.h
│ ├── crypto.c # AES encryption wrapper
│ ├── crypto.h
│ ├── token_gen.c # Token generation
│ ├── token_gen.h
│ ├── key_manager.c # Key lifecycle
│ ├── key_manager.h
│ ├── storage.c # SQLite persistence
│ ├── storage.h
│ ├── http_server.c # Minimal HTTP handling
│ └── http_server.h
├── include/
│ └── config.h # Build configuration
├── tests/
│ ├── test_vault.c
│ ├── test_crypto.c
│ ├── test_token_gen.c
│ └── test_storage.c
├── scripts/
│ ├── key_ceremony.sh # Key generation script
│ └── rotate_keys.sh # Key rotation script
├── data/
│ └── vault.db # SQLite database
├── Makefile
└── README.md
Core API Design
// vault.h
#ifndef VAULT_H
#define VAULT_H
#include <stdbool.h>
#include <stdint.h>
typedef struct {
char token[33]; // 32 chars + null
char network[32]; // Card network
char last_four[5]; // Last 4 digits (safe to store)
char bin[9]; // First 6-8 digits
bool new_token; // True if newly created
} TokenizeResult;
typedef struct {
char pan[20]; // Decrypted PAN
bool success;
char error[128]; // Error message if failed
} DetokenizeResult;
// Primary operations
TokenizeResult vault_tokenize(const char* pan);
DetokenizeResult vault_detokenize(const char* token, const char* api_key);
// Vault management
bool vault_init(const char* db_path, const char* master_key_path);
void vault_shutdown(void);
// Key management
bool vault_rotate_keys(void);
int vault_get_key_version(void);
// Statistics (for admin dashboard)
typedef struct {
int64_t total_tokens;
int64_t unique_pans;
int64_t tokens_today;
} VaultStats;
VaultStats vault_get_stats(void);
#endif
Solution Architecture
System Design
┌─────────────────────────────────────────────────────────────────────────┐
│ TOKENIZATION VAULT ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────┐ │
│ │ API Gateway / │ │
│ │ HTTP Server │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌───────────────┼───────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ /tokenize │ │ /detokenize │ │ /admin/stats │ │
│ │ │ │ │ │ │ │
│ │ • Validate │ │ • Auth check│ │ • Auth check│ │
│ │ • Luhn check│ │ • Rate limit│ │ • Read only │ │
│ │ • Dedupe │ │ • Audit log │ │ │ │
│ └──────┬───────┘ └──────┬───────┘ └──────────────┘ │
│ │ │ │
│ └────────┬────────┘ │
│ ▼ │
│ ┌───────────────────────────────────────────────┐ │
│ │ VAULT CORE │ │
│ │ │ │
│ │ ┌────────────────┐ ┌────────────────────┐ │ │
│ │ │ Token Generator│ │ Crypto Engine │ │ │
│ │ │ │ │ │ │ │
│ │ │ • CSPRNG │ │ • AES-256-GCM │ │ │
│ │ │ • Format-pres │ │ • Key derivation │ │ │
│ │ │ • Uniqueness │ │ • Key wrapping │ │ │
│ │ └────────────────┘ └────────────────────┘ │ │
│ │ │ │
│ └───────────────────────┬───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────┐ │
│ │ KEY MANAGER │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Master Key │ │ DEK Cache │ │ │
│ │ │ (env/file) │ │ (in memory)│ │ │
│ │ └────────────┘ └────────────┘ │ │
│ │ │ │
│ └───────────────────────┬───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────┐ │
│ │ STORAGE LAYER │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ SQLite Database │ │ │
│ │ │ │ │ │
│ │ │ tokens: token_id, encrypted_pan, │ │ │
│ │ │ pan_hash, metadata │ │ │
│ │ │ │ │ │
│ │ │ keys: key_version, encrypted_dek, │ │ │
│ │ │ created_at, active │ │ │
│ │ │ │ │ │
│ │ │ audit_log: timestamp, action, │ │ │
│ │ │ token_id, actor │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ │ │
│ └───────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Tokenization Flow
┌─────────────────────────────────────────────────────────────────────────┐
│ TOKENIZATION FLOW │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ INPUT: PAN "4532015112830366" │
│ │
│ Step 1: VALIDATE │
│ ════════════════ │
│ ┌─────────────────────────────────────────┐ │
│ │ • Check Luhn validity │ │
│ │ • Verify length (13-19 digits) │ │
│ │ • Identify network from BIN │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Step 2: DEDUPLICATE │
│ ═══════════════════ │
│ ┌─────────────────────────────────────────┐ │
│ │ pan_hash = HMAC-SHA256(PAN, hash_key) │ │
│ │ existing = SELECT token WHERE hash=? │ │
│ │ IF existing: return existing token │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Step 3: GENERATE TOKEN │
│ ══════════════════════ │
│ ┌─────────────────────────────────────────┐ │
│ │ token = "tok_" + random_base62(24) │ │
│ │ Verify uniqueness in database │ │
│ │ (or use UUID for guaranteed uniqueness) │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Step 4: ENCRYPT PAN │
│ ═══════════════════ │
│ ┌─────────────────────────────────────────┐ │
│ │ nonce = random(12 bytes) │ │
│ │ encrypted = AES-256-GCM(PAN, DEK, nonce) │ │
│ │ ciphertext = nonce || encrypted || tag │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Step 5: STORE │
│ ═════════════ │
│ ┌─────────────────────────────────────────┐ │
│ │ INSERT INTO tokens ( │ │
│ │ token_id, encrypted_pan, pan_hash, │ │
│ │ metadata, key_version, created_at │ │
│ │ ) │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ OUTPUT: token "tok_a1b2c3d4e5f6g7h8" │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Security Design Decisions
- Never log PANs: Even for debugging, only log tokens and metadata
- Immediate memory clearing: Zero out PAN buffers after use
- Authenticated encryption: AES-GCM prevents tampering
- Salted PAN hashes: Hash key is separate from encryption key
- Key versioning: Support old keys during rotation
- Audit logging: Every detokenization is logged with actor ID
Implementation Guide
Phase 1: Crypto Foundation
Goal: Implement AES-256-GCM encryption wrapper.
Using OpenSSL:
#include <openssl/evp.h>
#include <openssl/rand.h>
typedef struct {
unsigned char nonce[12];
unsigned char tag[16];
unsigned char* ciphertext;
size_t ciphertext_len;
} EncryptedData;
EncryptedData* encrypt_pan(const char* pan, const unsigned char* key);
char* decrypt_pan(const EncryptedData* data, const unsigned char* key);
Key tasks:
- Generate cryptographically secure random nonces
- Implement AES-256-GCM encryption
- Implement decryption with tag verification
- Zero memory after use
Phase 2: Token Generation
Goal: Generate unique, random tokens.
Random token approach:
// Generate 24 random bytes, base62 encode
// Result: "tok_" + 32 base62 characters
char* generate_token(void) {
unsigned char random_bytes[24];
RAND_bytes(random_bytes, 24);
char* token = malloc(37); // "tok_" + 32 + null
strcpy(token, "tok_");
base62_encode(random_bytes, 24, token + 4);
return token;
}
Format-preserving token (extension):
// Generate token that looks like a card number
// Preserves BIN, generates random middle, calculates valid Luhn check digit
char* generate_fp_token(const char* original_pan) {
char* token = malloc(20);
// Copy BIN (first 6 digits)
strncpy(token, original_pan, 6);
// Generate random middle
for (int i = 6; i < 15; i++) {
token[i] = '0' + (rand() % 10);
}
// Calculate valid Luhn check digit
token[15] = calculate_luhn_check(token);
token[16] = '\0';
return token;
}
Phase 3: Storage Layer
Goal: Persist tokens and encrypted PANs in SQLite.
Schema:
CREATE TABLE tokens (
token_id TEXT PRIMARY KEY,
encrypted_pan BLOB NOT NULL,
pan_hash TEXT NOT NULL UNIQUE,
bin TEXT NOT NULL,
last_four TEXT NOT NULL,
network TEXT NOT NULL,
key_version INTEGER NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_accessed TIMESTAMP
);
CREATE INDEX idx_pan_hash ON tokens(pan_hash);
CREATE TABLE keys (
key_version INTEGER PRIMARY KEY,
encrypted_dek BLOB NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
active BOOLEAN DEFAULT 1
);
CREATE TABLE audit_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
action TEXT NOT NULL, -- 'tokenize', 'detokenize'
token_id TEXT,
actor_id TEXT,
ip_address TEXT
);
Phase 4: Vault Core
Goal: Implement tokenize/detokenize operations.
Key functions:
TokenizeResult vault_tokenize(const char* pan) {
TokenizeResult result = {0};
// 1. Validate
if (!luhn_check(pan)) {
// Return error
}
// 2. Check for existing token
char* hash = hmac_sha256(pan, get_hash_key());
char* existing = storage_find_by_hash(hash);
if (existing) {
strcpy(result.token, existing);
result.new_token = false;
return result;
}
// 3. Generate new token
char* token = generate_token();
// 4. Encrypt PAN
EncryptedData* encrypted = encrypt_pan(pan, get_current_dek());
// 5. Store
storage_insert_token(token, encrypted, hash, extract_metadata(pan));
// 6. Build result
strcpy(result.token, token);
result.new_token = true;
// ... copy metadata
// 7. Clean up sensitive data
secure_zero(pan_copy, strlen(pan_copy));
return result;
}
Phase 5: HTTP Server
Goal: Expose vault via REST API.
Minimal HTTP server options:
- libmicrohttpd: Small, embeddable HTTP server
- mongoose: Single-file HTTP library
- Custom: Parse HTTP manually (educational but tedious)
Recommended: Use mongoose for simplicity.
Phase 6: Key Management
Goal: Implement key loading and rotation.
Key storage options:
- Environment variable:
VAULT_MASTER_KEY(development) - File: Key file with strict permissions (simple production)
- HSM: Hardware Security Module (real production)
Key rotation process:
bool vault_rotate_keys(void) {
// 1. Generate new DEK
unsigned char new_dek[32];
RAND_bytes(new_dek, 32);
// 2. Encrypt new DEK with master key
EncryptedData* wrapped_dek = encrypt_dek(new_dek, get_master_key());
// 3. Store new key with incremented version
int new_version = get_key_version() + 1;
storage_insert_key(new_version, wrapped_dek);
// 4. Re-encrypt all PANs (background job)
storage_reencrypt_all(get_dek(get_key_version() - 1), new_dek, new_version);
// 5. Mark old key inactive (but keep for disaster recovery)
storage_deactivate_key(new_version - 1);
return true;
}
Testing Strategy
Unit Tests
// test_crypto.c
void test_encrypt_decrypt_roundtrip() {
const char* pan = "4111111111111111";
unsigned char key[32];
RAND_bytes(key, 32);
EncryptedData* encrypted = encrypt_pan(pan, key);
char* decrypted = decrypt_pan(encrypted, key);
assert(strcmp(pan, decrypted) == 0);
free_encrypted(encrypted);
free(decrypted);
}
void test_wrong_key_fails() {
const char* pan = "4111111111111111";
unsigned char key1[32], key2[32];
RAND_bytes(key1, 32);
RAND_bytes(key2, 32);
EncryptedData* encrypted = encrypt_pan(pan, key1);
char* result = decrypt_pan(encrypted, key2); // Wrong key
assert(result == NULL); // Decryption should fail
}
// test_vault.c
void test_tokenize_same_pan_returns_same_token() {
const char* pan = "4111111111111111";
TokenizeResult r1 = vault_tokenize(pan);
TokenizeResult r2 = vault_tokenize(pan);
assert(strcmp(r1.token, r2.token) == 0);
assert(r1.new_token == true);
assert(r2.new_token == false);
}
Integration Tests
#!/bin/bash
# test_api.sh
# Start server
./vault_server &
PID=$!
sleep 1
# Test tokenize
RESPONSE=$(curl -s -X POST http://localhost:8080/tokenize \
-H "Content-Type: application/json" \
-d '{"pan": "4111111111111111"}')
TOKEN=$(echo $RESPONSE | jq -r '.token')
assert_starts_with "tok_" "$TOKEN"
# Test detokenize
RESPONSE=$(curl -s -X POST http://localhost:8080/detokenize \
-H "Content-Type: application/json" \
-H "Authorization: Bearer test_api_key" \
-d "{\"token\": \"$TOKEN\"}")
PAN=$(echo $RESPONSE | jq -r '.pan')
assert_equals "4111111111111111" "$PAN"
# Cleanup
kill $PID
Security Tests
// test_security.c
void test_pan_not_in_logs() {
// Redirect stderr to capture logs
// Tokenize a PAN
// Verify PAN doesn't appear in log output
}
void test_memory_cleared_after_use() {
// This is hard to test directly
// Use Valgrind or AddressSanitizer to check for leaks
}
void test_detokenize_requires_auth() {
// Call detokenize without auth header
// Verify 401 response
}
Common Pitfalls & Debugging
Pitfall 1: Nonce Reuse
Symptom: Security completely broken (same plaintext + key + nonce = same ciphertext).
Cause: Using static or predictable nonce.
Fix: Always use RAND_bytes() for nonce generation.
Pitfall 2: Not Verifying GCM Tag
Symptom: Accepting tampered ciphertext.
Cause: Ignoring authentication tag verification.
Fix: Always check EVP_DecryptFinal_ex() return value.
Pitfall 3: Timing Attacks on Hash Comparison
Symptom: Attackers can determine token validity by timing responses.
Cause: Using strcmp() for hash comparison (early exit on mismatch).
Fix: Use constant-time comparison:
int secure_compare(const char* a, const char* b, size_t len) {
unsigned char result = 0;
for (size_t i = 0; i < len; i++) {
result |= a[i] ^ b[i];
}
return result == 0;
}
Pitfall 4: Key in Memory After Use
Symptom: Key extractable from memory dump.
Cause: Not zeroing key buffers.
Fix: Use explicit_bzero() or volatile-write loop:
void secure_zero(void* ptr, size_t len) {
volatile unsigned char* p = ptr;
while (len--) *p++ = 0;
}
Extensions & Challenges
Extension 1: Format-Preserving Encryption
Implement FF1 algorithm from NIST SP 800-38G for true format-preserving tokens.
Extension 2: Multi-Tenant Vault
Add tenant isolation—each merchant has their own token space.
Extension 3: HSM Integration
Integrate with SoftHSM2 for realistic key management simulation.
Extension 4: Token Scoping
Implement scoped tokens: same PAN gets different tokens for different merchants/channels.
Extension 5: Expiring Tokens
Add token expiration for temporary/session tokens.
Interview Questions This Prepares You For
- “Why use tokenization instead of encryption?”
- Scope reduction, no key management outside vault, tokens useless if stolen
- “How do you ensure unique tokens?”
- CSPRNG + collision check, or UUID-based
- “What’s the difference between FF1 and random tokenization?”
- FF1 preserves format (length, character set), random is simpler but changes format
- “How do you handle key rotation?”
- Re-encrypt data, version keys, keep old keys temporarily
- “What’s in PCI scope with a tokenization vault?”
- Only the vault itself; systems handling only tokens are out of scope
Resources
Books
| Topic | Book | Chapter |
|---|---|---|
| AES & Block Ciphers | Serious Cryptography (Aumasson) | Ch. 4-5 |
| Key Management | Security in Computing (Pfleeger) | Key management chapter |
| Format-Preserving Encryption | NIST SP 800-38G | Full document |
Specifications
- PCI Token Guidelines
- NIST SP 800-38G (FPE)
- NIST SP 800-108 (Key Derivation)
Libraries
- OpenSSL (libcrypto): AES, RAND, HMAC
- libsodium: Modern crypto primitives
- SQLite: Storage
Self-Assessment Checklist
- AES-256-GCM encryption works correctly
- Tokens are unique and non-predictable
- Same PAN always returns same token
- Detokenization requires authentication
- PANs never appear in logs
- Memory is cleared after PAN handling
- Key rotation re-encrypts all data
- Audit log captures all detokenization
- Can explain why tokenization reduces PCI scope
What’s Next?
After this project, you understand how to protect card data at rest. Move to Project 3: P2PE Simulator to learn how data is protected in transit from the card reader to the processor.