Project 6: Full HFT Trading System
The capstone project: integrate all previous components into a complete electronic trading ecosystem - exchange simulator, market data feed, trading gateway, strategy engine, risk management, and monitoring dashboard.
Quick Reference
| Attribute | Value |
|---|---|
| Difficulty | Expert |
| Time Estimate | 2-3 months |
| Language | C++ or Rust |
| Prerequisites | Projects 1-5 completed |
| Key Topics | System integration, thread orchestration, end-to-end latency, operational concerns |
| Coolness | Pure Magic (Super Cool) |
| Portfolio Value | Unicorn-level Resume Gold |
1. Learning Objectives
By completing this capstone project, you will:
- Integrate multiple low-latency components into a coherent system where data flows seamlessly from market data through strategy to order execution
- Master thread orchestration including CPU pinning, NUMA awareness, and coordinating multiple specialized threads without contention
- Measure and optimize end-to-end latency across the entire trading pipeline, understanding where every microsecond goes
- Implement production-grade operational concerns including monitoring, alerting, configuration management, graceful shutdown, and failure recovery
- Design for reliability and correctness ensuring the system handles edge cases, network failures, and market anomalies without losing money
- Build real-time observability with dashboards showing positions, P&L, latency distributions, and system health
- Understand the complete trading lifecycle from market data arrival to trade confirmation and risk update
2. Theoretical Foundation
2.1 Core Concepts
System Architecture Patterns
Building a complete HFT system requires understanding how components communicate and how to minimize the overhead of that communication.
FULL HFT TRADING SYSTEM ARCHITECTURE
┌────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐ │
│ │ EXCHANGE SIMULATOR (P3) │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ TCP/UDP │───►│ Matching │───►│ Market │ │ │
│ │ │ Listener │ │ Engine │ │ Data Pub │ │ │
│ │ └─────────────┘ └─────────────┘ └──────┬──────┘ │ │
│ └───────────────────────────────────────────────│──────────────────────────────┘ │
│ ▲ │ │
│ │ Orders │ Market Data │
│ │ ▼ │
│ ┌────────────────────────┴─────────────────────────────────────────────────────┐ │
│ │ TRADING APPLICATION │ │
│ │ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ │
│ │ │ Gateway │◄───────►│ Strategy │◄───────►│ Risk │ │ │
│ │ │ (Orders) │ │ Engine │ │ Manager │ │ │
│ │ │ │ │ (P4 Logic) │ │ │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────┘ │ │
│ │ ▲ ▲ ▲ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌───────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ LOCK-FREE MESSAGE BUS (P2) │ │ │
│ │ │ Market Data ──► Strategy ──► Orders ──► Gateway ──► Exchange │ │ │
│ │ │ ◄──────────── Fills/Acks ◄────────────────── │ │ │
│ │ └───────────────────────────────────────────────────────────────────────┘ │ │
│ │ ▲ │ │
│ │ │ │ │
│ │ ┌───────────────┐ ┌─────────┴─────┐ ┌───────────────┐ │ │
│ │ │ Order Book │ │ Position │ │ Monitoring │ │ │
│ │ │ Cache │ │ Tracker │ │ Dashboard │ │ │
│ │ │ (P1) │ │ │ │ (HTTP/WS) │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────┘ │ │
│ │ │ │
│ │ Memory Layer: Custom Allocators (P5) throughout all components │ │
│ └───────────────────────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────────────────┘
Component Communication Patterns:
| Pattern | Use Case | Latency | Complexity |
|---|---|---|---|
| Shared Memory | Same process, different threads | ~10-50ns | Medium |
| Lock-Free Queue | Producer/consumer within process | ~50-100ns | High |
| Unix Domain Socket | Same machine, different processes | ~1-5us | Low |
| TCP Loopback | Same machine, network stack | ~10-30us | Low |
| TCP Network | Different machines | ~50-200us+ | Low |
Thread Orchestration
HFT systems dedicate specific CPU cores to specific tasks. This eliminates context switching and cache pollution.
CPU CORE ALLOCATION STRATEGY
┌────────────────────────────────────────────────────────────────────────┐
│ │
│ NUMA Node 0 NUMA Node 1 │
│ ┌────────────────────────┐ ┌────────────────────────┐ │
│ │ Core 0: Market Data │ │ Core 8: Network I/O │ │
│ │ Receiver │ │ (Gateway) │ │
│ ├────────────────────────┤ ├────────────────────────┤ │
│ │ Core 1: Order Book │ │ Core 9: Logging │ │
│ │ Updates │ │ (Non-critical) │ │
│ ├────────────────────────┤ ├────────────────────────┤ │
│ │ Core 2: Strategy │ │ Core 10: Monitoring │ │
│ │ Engine │ │ Dashboard │ │
│ ├────────────────────────┤ ├────────────────────────┤ │
│ │ Core 3: Risk Manager │ │ Core 11: Spare / │ │
│ │ │ │ Housekeeping │ │
│ └────────────────────────┘ └────────────────────────┘ │
│ │
│ Memory: Allocate strategy data on Node 0 │
│ Allocate network buffers on Node 1 │
│ │
└────────────────────────────────────────────────────────────────────────┘
Key Threading Principles:
- Dedicated Cores: Pin each critical thread to its own physical core
- Isolcpus: Remove cores from the OS scheduler for true isolation
- NUMA Awareness: Keep data close to the cores that use it
- Avoid Hyperthreading: Use only physical cores for latency-critical work
- Spin Instead of Sleep: Busy-wait loops for lowest latency (burns CPU but saves microseconds)
End-to-End Latency Analysis
Understanding where latency comes from is crucial for optimization.
END-TO-END LATENCY BREAKDOWN
┌──────────────────────────────────────────────────────────────────────┐
│ │
│ Market Data Arrives │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Network Receive (NIC → Kernel → Userspace) ~2-10 μs │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Parse Market Data Message ~0.1-0.5 μs│ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Update Order Book ~0.3-1 μs │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Strategy Signal Generation ~0.5-5 μs │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Risk Checks (position limits, P&L limits) ~0.1-0.5 μs│ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Order Creation + Serialization ~0.2-0.5 μs│ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Network Send (Userspace → Kernel → NIC) ~2-10 μs │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Order Sent to Exchange │
│ │
│ ═══════════════════════════════════════════════════════════════════ │
│ TOTAL INTERNAL LATENCY: ~5-30 μs │
│ (Excludes network round-trip to exchange) │
│ │
└──────────────────────────────────────────────────────────────────────┘
Operational Concerns
Production systems need more than just fast code.
OPERATIONAL REQUIREMENTS
┌──────────────────────────────────────────────────────────────────────┐
│ │
│ CONFIGURATION MONITORING │
│ ┌────────────────────┐ ┌────────────────────┐ │
│ │ • Symbol universe │ │ • Orders/second │ │
│ │ • Position limits │ │ • Latency p50/p99 │ │
│ │ • P&L limits │ │ • Queue depths │ │
│ │ • Strategy params │ │ • Position by sym │ │
│ │ • Network endpoints│ │ • Realized P&L │ │
│ │ • Thread affinity │ │ • Unrealized P&L │ │
│ └────────────────────┘ └────────────────────┘ │
│ │
│ ALERTING RECOVERY │
│ ┌────────────────────┐ ┌────────────────────┐ │
│ │ • Latency spike │ │ • Reconnection │ │
│ │ • Position breach │ │ • State rebuild │ │
│ │ • P&L limit hit │ │ • Order reconcile │ │
│ │ • Connection lost │ │ • Position sync │ │
│ │ • Error rate high │ │ • Graceful shutdown│ │
│ └────────────────────┘ └────────────────────┘ │
│ │
│ LOGGING TESTING │
│ ┌────────────────────┐ ┌────────────────────┐ │
│ │ • Trade log │ │ • Unit tests │ │
│ │ • Latency log │ │ • Integration tests│ │
│ │ • Error log │ │ • Chaos engineering│ │
│ │ • Audit trail │ │ • Replay testing │ │
│ │ • Debug log (off) │ │ • Load testing │ │
│ └────────────────────┘ └────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────┘
2.2 Why This Matters
The Integration Challenge:
Building individual components is one thing. Making them work together at nanosecond precision is another entirely. The capstone reveals:
- Hidden Latencies: The overhead of moving data between components that seemed trivial in isolation
- Emergent Complexity: Bugs that only appear when components interact under load
- Operational Reality: The gap between “it runs” and “it runs in production without losing money”
- System Thinking: Understanding that optimizing one component can hurt another
Industry Relevance:
Every HFT firm runs systems exactly like this. The architecture patterns, the threading model, the monitoring requirements - these are universal. Building one yourself transforms you from someone who “understands trading systems” to someone who “has built a trading system.”
2.3 Historical Context
The evolution of electronic trading systems:
EVOLUTION OF TRADING SYSTEMS
┌──────────────────────────────────────────────────────────────────────┐
│ │
│ 1980s: Floor Trading │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Human traders shouting on exchange floors │ │
│ │ Latency: Minutes to hours │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 1990s: Electronic Markets Begin │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ NASDAQ goes electronic, ECNs emerge │ │
│ │ Latency: Seconds │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2000s: Algorithmic Trading │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ VWAP, TWAP algorithms, co-location begins │ │
│ │ Latency: Milliseconds │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2010s: HFT Arms Race │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Kernel bypass, FPGAs, microwave networks │ │
│ │ Latency: Microseconds to nanoseconds │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2020s: Mature HFT + Crypto │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Consolidation, regulatory pressure, crypto HFT │ │
│ │ Latency: Sub-microsecond (with kernel bypass) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────┘
2.4 Common Misconceptions
Misconception 1: “Faster is always better” Reality: Speed matters only up to a point. A system that trades at 10us but makes bad decisions loses money faster than one at 50us with better signals. Risk management and correctness trump raw speed.
Misconception 2: “You need kernel bypass and FPGAs to build HFT” Reality: Many successful trading systems run in userspace with standard TCP. Kernel bypass and FPGAs are optimizations for when you’re already competitive - they’re not the starting point.
Misconception 3: “The strategy is the hard part” Reality: The infrastructure is often 80% of the work. A simple strategy on robust infrastructure beats a complex strategy on brittle infrastructure.
Misconception 4: “More threads = faster” Reality: More threads often means more contention. The fastest HFT systems are often single-threaded per component, carefully orchestrated.
Misconception 5: “Testing in production is fine” Reality: Paper trading and simulation are mandatory. The first real trade should be boring because you’ve already seen it work thousands of times.
3. Project Specification
3.1 What You Will Build
A complete electronic trading ecosystem consisting of seven integrated components:
┌────────────────────────────────────────────────────────────────────────┐
│ COMPLETE TRADING ECOSYSTEM │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. EXCHANGE SIMULATOR (from P3) │
│ • Matching engine with price-time priority │
│ • Market data publication (BBO, trades) │
│ • Order acknowledgment and fill reporting │
│ • Support for limit/market orders │
│ │
│ 2. MARKET DATA HANDLER (from P2) │
│ • Lock-free ingestion of market data │
│ • Order book reconstruction │
│ • Distribution to strategy via SPSC queue │
│ │
│ 3. TRADING GATEWAY (NEW) │
│ • Order submission to exchange │
│ • Fill/ack reception and routing │
│ • Sequence number management │
│ • Connection management and heartbeats │
│ │
│ 4. STRATEGY ENGINE (adapted from P4) │
│ • Signal generation from market data │
│ • Order decision logic │
│ • Position-aware trading │
│ • At least one working strategy (e.g., mean reversion) │
│ │
│ 5. RISK MANAGER (NEW) │
│ • Pre-trade risk checks │
│ • Position limit enforcement │
│ • P&L limit enforcement │
│ • Order rate limiting │
│ │
│ 6. POSITION TRACKER (NEW) │
│ • Real-time position by symbol │
│ • Realized and unrealized P&L │
│ • Trade blotter │
│ • Cost basis tracking │
│ │
│ 7. MONITORING DASHBOARD (NEW) │
│ • Web-based real-time display │
│ • Latency histograms │
│ • Position and P&L display │
│ • System health indicators │
│ │
│ INFRASTRUCTURE (from P5, throughout) │
│ • Custom allocators for all trading-path allocations │
│ • Lock-free queues for inter-component communication │
│ • Thread pinning and NUMA-aware allocation │
│ │
└────────────────────────────────────────────────────────────────────────┘
3.2 Functional Requirements
FR1: Exchange Simulation
- Accept TCP connections from trading applications
- Process limit and market orders
- Maintain order book state with FIFO matching at each price level
- Publish real-time market data (best bid/offer, last trade)
- Send acknowledgments and fills back to clients
- Support multiple symbols (minimum 5)
- Handle 10,000+ orders per second
FR2: Market Data Processing
- Receive market data from exchange simulator
- Parse and validate incoming messages with zero copy where possible
- Reconstruct and maintain local order book view
- Distribute updates to strategy engine via lock-free queue
- Handle gaps and reconnection gracefully
FR3: Order Management
- Submit orders to exchange with unique sequence numbers
- Track order state (pending, accepted, filled, cancelled)
- Handle partial fills correctly
- Receive and process acknowledgments and fills
- Support order cancellation
FR4: Strategy Execution
- React to market data updates with trading signals
- Generate orders based on strategy logic
- Implement at least one complete strategy (e.g., mean reversion on spread)
- Respect position limits from risk manager
- Support enabling/disabling strategies at runtime
FR5: Risk Management
- Enforce maximum position per symbol (e.g., 1000 shares)
- Enforce maximum total P&L loss (e.g., -$10,000 daily stop)
- Limit order rate (e.g., 100 orders/second)
- Block orders that would violate limits
- Allow runtime adjustment of limits
FR6: Position and P&L Tracking
- Track position per symbol in real-time
- Calculate realized P&L on each fill
- Calculate unrealized P&L on market data updates
- Maintain trade history (blotter)
- Support end-of-day position reporting
FR7: Monitoring and Observability
- Expose HTTP/WebSocket endpoint for dashboard
- Report latency percentiles (p50, p95, p99)
- Show current positions and P&L
- Display order rate and fill rate
- Alert on error conditions
3.3 Non-Functional Requirements
NFR1: Latency
- Market data to order decision: < 20us (p99)
- Order decision to wire: < 10us (p99)
- End-to-end (data arrival to order sent): < 50us (p99)
NFR2: Throughput
- Handle 100,000 market data updates per second
- Process 10,000 order decisions per second
- Dashboard updates at 10 Hz without affecting trading path
NFR3: Reliability
- Zero data loss on graceful shutdown
- Automatic reconnection on network failure
- State recovery after restart
- No crashes under load
NFR4: Maintainability
- Modular design allowing component replacement
- Clear interfaces between components
- Comprehensive logging (with minimal hot-path impact)
- Configuration via file (YAML/JSON/TOML)
3.4 Example Usage / Output
$ ./trading_system --config config.yaml
================================================================================
HFT TRADING SYSTEM v1.0
================================================================================
[2024-01-15 09:29:55.000] [SYSTEM] Loading configuration from config.yaml
[2024-01-15 09:29:55.001] [SYSTEM] Initializing components...
[2024-01-15 09:29:55.002] [ALLOCATOR] Memory pools initialized:
- Order pool: 10000 orders x 128 bytes = 1.2 MB
- Message pool: 50000 messages x 64 bytes = 3.1 MB
- Total pre-allocated: 4.3 MB
[2024-01-15 09:29:55.003] [THREADS] Pinning threads to cores:
- Market Data Handler: Core 0 (isolated)
- Strategy Engine: Core 1 (isolated)
- Gateway Network: Core 2 (isolated)
- Risk Manager: Core 3 (shared)
- Monitoring: Core 4 (shared)
[2024-01-15 09:29:55.010] [EXCHANGE] Exchange simulator started
- Listening on 127.0.0.1:9000
- Symbols: AAPL, MSFT, GOOGL, AMZN, META
[2024-01-15 09:29:55.015] [MKTDATA] Connecting to exchange...
[2024-01-15 09:29:55.016] [MKTDATA] Connected. Receiving market data.
[2024-01-15 09:29:55.020] [GATEWAY] Connecting to exchange for orders...
[2024-01-15 09:29:55.021] [GATEWAY] Connected. Order session established.
[2024-01-15 09:29:55.025] [STRATEGY] Loading strategy: MeanReversion
- Lookback period: 100 ticks
- Entry threshold: 2.0 std deviations
- Position size: 100 shares
- Max position: 500 shares
[2024-01-15 09:29:55.026] [RISK] Risk limits configured:
- Max position per symbol: 1000 shares
- Max daily loss: $10,000
- Max order rate: 100/second
[2024-01-15 09:29:55.030] [MONITOR] Dashboard available at http://localhost:8080
[2024-01-15 09:29:55.031] [SYSTEM] All components initialized. Trading enabled.
================================================================================
LIVE TRADING LOG
================================================================================
[09:30:00.000123] [MKTDATA] AAPL BID 150.25 x 500 | ASK 150.26 x 300
[09:30:00.000145] [STRATEGY] AAPL: Price 150.255 | MA 150.22 | Zscore -1.2
[09:30:00.001456] [MKTDATA] AAPL BID 150.24 x 800 | ASK 150.26 x 300
[09:30:00.001478] [STRATEGY] AAPL: Price 150.25 | MA 150.22 | Zscore -2.1 -> SIGNAL: BUY
[09:30:00.001485] [STRATEGY] Signal: BUY 100 AAPL @ MARKET
[09:30:00.001490] [RISK] Pre-trade check: PASS (position: 0, limit: 1000)
[09:30:00.001495] [GATEWAY] Order sent: id=1001 BUY 100 AAPL MKT
[09:30:00.001520] [GATEWAY] Order ACK: id=1001 ACCEPTED
[09:30:00.001845] [GATEWAY] FILL: id=1001 100@150.26
[09:30:00.001850] [POSITION] AAPL: +100 @ 150.26 | Cost: $15,026.00
[09:30:00.001855] [RISK] Position updated: AAPL +100 (limit: 1000)
[09:30:00.005234] [MKTDATA] AAPL BID 150.28 x 400 | ASK 150.29 x 200
[09:30:00.005256] [POSITION] AAPL: Unrealized P&L: +$2.00 (150.285 mid)
... [trading continues] ...
[09:30:15.123456] [STRATEGY] Signal: SELL 100 AAPL @ MARKET (mean reversion exit)
[09:30:15.123461] [RISK] Pre-trade check: PASS
[09:30:15.123466] [GATEWAY] Order sent: id=1047 SELL 100 AAPL MKT
[09:30:15.123491] [GATEWAY] Order ACK: id=1047 ACCEPTED
[09:30:15.123756] [GATEWAY] FILL: id=1047 100@150.35
[09:30:15.123761] [POSITION] AAPL: 0 shares | Realized P&L: +$9.00
================================================================================
PERIODIC STATUS (every 10 seconds)
================================================================================
[09:30:10.000] [STATUS]
Positions: AAPL +100, MSFT +0, GOOGL -50
Unrealized P&L: +$15.50
Realized P&L: +$45.00
Orders sent: 47
Fills received: 42
Latency (last 10s):
Market data -> Strategy: p50=8us p95=15us p99=22us
Strategy -> Gateway: p50=3us p95=6us p99=12us
End-to-end internal: p50=12us p95=25us p99=38us
Throughput:
Market data messages: 12,456/s
Orders submitted: 4.7/s
Fills processed: 4.2/s
================================================================================
SHUTDOWN
================================================================================
$ kill -SIGTERM <pid>
[09:45:00.000] [SYSTEM] Received shutdown signal
[09:45:00.001] [STRATEGY] Strategy disabled. No new orders.
[09:45:00.002] [GATEWAY] Cancelling 3 open orders...
[09:45:00.015] [GATEWAY] All orders cancelled or filled.
[09:45:00.016] [POSITION] Final positions:
AAPL: 0 shares
MSFT: 0 shares
GOOGL: 0 shares
[09:45:00.017] [POSITION] Session P&L:
Realized: +$127.50
Unrealized: $0.00
Total: +$127.50
[09:45:00.018] [SYSTEM] Writing state to checkpoint file...
[09:45:00.020] [SYSTEM] Disconnecting from exchange...
[09:45:00.025] [SYSTEM] Shutdown complete.
3.5 Real World Outcome
Upon completion, you will have:
- A Working Trading System that you can run and observe making simulated trades
- A Real-Time Dashboard showing positions, P&L, and latency metrics
- Demonstrable Latency Numbers proving your system meets HFT-grade performance
- Production-Like Features including risk management, reconnection, and graceful shutdown
- Portfolio-Ready Artifact that demonstrates systems programming expertise
4. Solution Architecture
4.1 High-Level Design
SYSTEM COMPONENT DIAGRAM
┌────────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ EXTERNAL CONNECTIONS │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Exchange │◄────TCP─────►│ Dashboard │ │ │
│ │ │ Simulator │ │ Browser │ │ │
│ │ │ (Port 9000) │ │ (Port 8080) │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ │ ▲ ▲ ▲ │ │
│ └─────────│──│───────────────────────────│─────────────────────────────────┘ │
│ │ │ │ │
│ ┌─────────│──│───────────────────────────│─────────────────────────────────┐ │
│ │ │ │ │ TRADING APPLICATION PROCESS │ │
│ │ │ │ │ │ │
│ │ ┌────┴──┴────┐ ┌───────┴───────┐ │ │
│ │ │ Network │ │ Monitoring │ │ │
│ │ │ I/O │ │ HTTP/WS │ │ │
│ │ │ Thread │ │ Server │ │ │
│ │ └─────┬──────┘ └───────────────┘ │ │
│ │ │ ▲ │ │
│ │ │ (raw bytes) │ (metrics) │ │
│ │ ▼ │ │ │
│ │ ┌─────────────┐ │ │ │
│ │ │ Protocol │ │ │ │
│ │ │ Parser │ │ │ │
│ │ └──────┬──────┘ │ │ │
│ │ │ │ │ │
│ │ │ (typed messages) │ │ │
│ │ ┌──────┴──────────────────┐ │ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ │ │ │
│ │ ┌───────────┐ ┌───────────┐ │ │ │
│ │ │ Market │ │ Gateway │ │ │ │
│ │ │ Data │ │ Order │ │ │ │
│ │ │ Handler │ │ Handler │ │ │ │
│ │ └─────┬─────┘ └─────┬─────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ SPSC Queue │ SPSC Q │ │ │
│ │ ▼ ▼ │ │ │
│ │ ┌─────────────────────────────────────┼───────┐ │ │
│ │ │ │ │ │ │
│ │ │ ┌───────────┐ ┌───────────┐ ┌──┴────┐ │ │ │
│ │ │ │ Order │ │ Strategy │ │ Risk │ │ │ │
│ │ │ │ Book │◄──│ Engine │──►│Manager│ │ │ │
│ │ │ │ (P1) │ │ (P4) │ │ │ │ │ │
│ │ │ └───────────┘ └─────┬─────┘ └───────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ │ ┌─────────────────────┴─────────────────┐ │ │ │
│ │ │ │ Position Tracker │───┼─────────────────────────┘ │
│ │ │ │ (positions, P&L) │ │ │
│ │ │ └───────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ │ ┌───────────────────────────────────────┐ │ │
│ │ │ │ Memory Pools (P5 Allocators) │ │ │
│ │ │ │ • Order Pool • Message Pool │ │ │
│ │ │ │ • Book Pool • String Pool │ │ │
│ │ │ └───────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ │ STRATEGY CORE (Pinned Cores) │ │
│ │ └──────────────────────────────────────────────┘ │
│ │ │
│ └───────────────────────────────────────────────────────────────────────────────┘
│ │
└───────────────────────────────────────────────────────────────────────────────────┘
4.2 Key Components
Component 1: Exchange Simulator
Responsibility: Simulate a real exchange with order matching, market data publication, and order acknowledgment.
Interfaces:
- TCP listener for order connections
- TCP publisher for market data (or multicast UDP)
- Binary protocol for orders and fills
Key Decisions:
- Single-threaded matching to ensure determinism
- Pre-allocated order book data structures
- Configurable latency injection for testing
Component 2: Market Data Handler
Responsibility: Receive, parse, and distribute market data with minimal latency.
Interfaces:
- TCP/UDP receiver from exchange
- SPSC queue output to strategy
- Order book state queries
Key Decisions:
- Lock-free SPSC queue for distribution
- Zero-copy parsing where possible
- Separate thread for network I/O
Component 3: Trading Gateway
Responsibility: Manage the order lifecycle from submission to fill.
Interfaces:
- Order submission from strategy
- Fill/ack distribution to position tracker
- TCP connection to exchange
Key Decisions:
- Sequence number generation for orders
- Pending order cache for fill matching
- Connection management and heartbeats
Component 4: Strategy Engine
Responsibility: Generate trading signals based on market data.
Interfaces:
- Market data input from handler
- Order output to gateway (via risk)
- Position queries from tracker
Key Decisions:
- Event-driven (react to each update)
- Stateful (maintains signal state)
- Configurable parameters
Component 5: Risk Manager
Responsibility: Enforce pre-trade risk limits.
Interfaces:
- Order requests from strategy
- Position state from tracker
- Approved/rejected orders to gateway
Key Decisions:
- Synchronous checks (blocks order flow)
- Fail-closed (reject on uncertainty)
- Runtime-adjustable limits
Component 6: Position Tracker
Responsibility: Track all positions and P&L in real-time.
Interfaces:
- Fill updates from gateway
- Market data for mark-to-market
- Position queries from strategy/risk
Key Decisions:
- Average cost basis tracking
- Separate realized vs unrealized P&L
- Trade blotter for audit
Component 7: Monitoring Dashboard
Responsibility: Provide real-time visibility into system state.
Interfaces:
- HTTP server for dashboard page
- WebSocket for real-time updates
- Metric collection from all components
Key Decisions:
- Non-blocking metric collection
- Separate thread (not latency critical)
- Simple web stack (no heavy frameworks)
4.3 Data Structures
CORE DATA STRUCTURES
┌────────────────────────────────────────────────────────────────────────┐
│ │
│ ORDER │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ struct Order { │ │
│ │ uint64_t order_id; // Unique sequence number │ │
│ │ uint32_t symbol_id; // Index into symbol table │ │
│ │ Side side; // BUY or SELL │ │
│ │ OrderType type; // LIMIT, MARKET │ │
│ │ int32_t quantity; // Shares (signed for direction) │ │
│ │ int64_t price; // Price in fixed-point (x10000) │ │
│ │ OrderStatus status; // PENDING, ACCEPTED, FILLED, etc. │ │
│ │ uint64_t timestamp_ns; // Submission time (nanoseconds) │ │
│ │ }; │ │
│ │ // Size: 48 bytes (fits in cache line with padding) │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
│ FILL │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ struct Fill { │ │
│ │ uint64_t order_id; // Which order was filled │ │
│ │ uint64_t fill_id; // Unique fill identifier │ │
│ │ uint32_t symbol_id; // Symbol │ │
│ │ int32_t quantity; // Filled quantity │ │
│ │ int64_t price; // Fill price │ │
│ │ uint64_t timestamp_ns; // Fill time │ │
│ │ }; │ │
│ │ // Size: 40 bytes │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
│ POSITION │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ struct Position { │ │
│ │ uint32_t symbol_id; │ │
│ │ int32_t quantity; // Net shares (positive=long) │ │
│ │ int64_t avg_cost; // Average cost in fixed-point │ │
│ │ int64_t realized_pnl; // Cumulative realized P&L │ │
│ │ int64_t unrealized_pnl; // Current mark-to-market P&L │ │
│ │ }; │ │
│ │ // Size: 32 bytes │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
│ MARKET DATA UPDATE │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ struct MarketDataUpdate { │ │
│ │ uint32_t symbol_id; │ │
│ │ UpdateType type; // BBO, TRADE, DEPTH │ │
│ │ int64_t bid_price; │ │
│ │ int32_t bid_qty; │ │
│ │ int64_t ask_price; │ │
│ │ int32_t ask_qty; │ │
│ │ int64_t last_price; // Last trade price │ │
│ │ int32_t last_qty; // Last trade quantity │ │
│ │ uint64_t exchange_ts; // Exchange timestamp │ │
│ │ uint64_t receive_ts; // Our receive timestamp │ │
│ │ }; │ │
│ │ // Size: 64 bytes (one cache line) │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
│ LATENCY SAMPLE │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ struct LatencySample { │ │
│ │ uint64_t t_market_data_receive; │ │
│ │ uint64_t t_orderbook_update; │ │
│ │ uint64_t t_strategy_decision; │ │
│ │ uint64_t t_risk_check; │ │
│ │ uint64_t t_order_serialize; │ │
│ │ uint64_t t_wire_send; │ │
│ │ }; │ │
│ │ // Used for per-event latency tracking │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
4.4 Algorithm Overview
Main Event Loop (Strategy Thread)
WHILE system.is_running():
// Non-blocking poll for market data
update = market_data_queue.try_pop()
IF update IS NOT NULL:
// Update local order book view
order_book.apply_update(update)
// Record timing
sample.t_orderbook_update = now()
// Run strategy logic
signal = strategy.on_market_data(update, order_book)
sample.t_strategy_decision = now()
IF signal.should_trade:
// Build order
order = create_order(signal)
// Risk check (synchronous)
IF risk_manager.check(order):
sample.t_risk_check = now()
// Submit order
gateway.submit(order)
sample.t_wire_send = now()
// Record latency
latency_tracker.record(sample)
ELSE:
log("Order rejected by risk manager")
// Process any fills from gateway
fill = fill_queue.try_pop()
IF fill IS NOT NULL:
position_tracker.apply_fill(fill)
strategy.on_fill(fill)
risk_manager.update_position(fill)
Mean Reversion Strategy Logic
FUNCTION on_market_data(update, order_book):
// Update rolling statistics
mid_price = (update.bid_price + update.ask_price) / 2
price_history.push(mid_price)
IF price_history.size() < LOOKBACK:
RETURN no_signal
// Calculate z-score
mean = price_history.mean()
std = price_history.stddev()
zscore = (mid_price - mean) / std
current_position = position_tracker.get(update.symbol_id)
// Entry signals
IF zscore < -ENTRY_THRESHOLD AND current_position <= 0:
RETURN Signal(BUY, POSITION_SIZE)
IF zscore > ENTRY_THRESHOLD AND current_position >= 0:
RETURN Signal(SELL, POSITION_SIZE)
// Exit signals (mean reversion)
IF current_position > 0 AND zscore > 0:
RETURN Signal(SELL, current_position)
IF current_position < 0 AND zscore < 0:
RETURN Signal(BUY, -current_position)
RETURN no_signal
5. Implementation Guide
5.1 Development Environment Setup
# Required packages (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install -y \
build-essential \
cmake \
ninja-build \
clang \
lld \
libboost-all-dev \
libbenchmark-dev \
perf-tools-unstable \
linux-tools-generic \
numactl \
hwloc
# For Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default stable
# Verify tools
perf --version
numactl --show
hwloc-ls
# Create project structure
mkdir -p hft_system/{exchange,market_data,gateway,strategy,risk,position,monitor,common,config,scripts,tests}
cd hft_system
5.2 Project Structure
trading_system/
├── CMakeLists.txt # Top-level CMake (or Cargo.toml for Rust)
├── config/
│ ├── config.yaml # Main configuration
│ ├── symbols.yaml # Symbol definitions
│ └── strategies/
│ └── mean_reversion.yaml # Strategy parameters
│
├── exchange/ # Exchange Simulator (from P3)
│ ├── matching_engine.hpp/cpp
│ ├── order_book.hpp/cpp
│ ├── market_data_publisher.hpp/cpp
│ ├── exchange_server.hpp/cpp
│ └── tests/
│
├── market_data/ # Market Data Handler (from P2)
│ ├── md_receiver.hpp/cpp
│ ├── md_parser.hpp/cpp
│ ├── orderbook_cache.hpp/cpp
│ └── tests/
│
├── gateway/ # Order Gateway (new)
│ ├── order_gateway.hpp/cpp
│ ├── order_protocol.hpp/cpp
│ ├── sequence_generator.hpp/cpp
│ ├── pending_orders.hpp/cpp
│ └── tests/
│
├── strategy/ # Strategy Engine (adapted from P4)
│ ├── strategy_interface.hpp
│ ├── mean_reversion.hpp/cpp
│ ├── signal.hpp
│ ├── price_history.hpp/cpp
│ └── tests/
│
├── risk/ # Risk Manager (new)
│ ├── risk_manager.hpp/cpp
│ ├── position_limits.hpp/cpp
│ ├── pnl_limits.hpp/cpp
│ ├── rate_limiter.hpp/cpp
│ └── tests/
│
├── position/ # Position Tracker (new)
│ ├── position_tracker.hpp/cpp
│ ├── trade_blotter.hpp/cpp
│ ├── pnl_calculator.hpp/cpp
│ └── tests/
│
├── monitor/ # Monitoring Dashboard (new)
│ ├── http_server.hpp/cpp
│ ├── websocket_handler.hpp/cpp
│ ├── metrics_collector.hpp/cpp
│ ├── static/ # HTML/CSS/JS for dashboard
│ │ ├── index.html
│ │ ├── dashboard.js
│ │ └── style.css
│ └── tests/
│
├── common/ # Shared Infrastructure (from P5)
│ ├── allocators/
│ │ ├── pool_allocator.hpp/cpp
│ │ ├── arena_allocator.hpp/cpp
│ │ └── memory_pool.hpp/cpp
│ ├── lockfree/
│ │ ├── spsc_queue.hpp
│ │ └── mpsc_queue.hpp
│ ├── threading/
│ │ ├── thread_utils.hpp/cpp # CPU pinning, affinity
│ │ └── spin_lock.hpp
│ ├── protocol/
│ │ ├── binary_codec.hpp/cpp
│ │ └── message_types.hpp
│ ├── time/
│ │ ├── clock.hpp/cpp # High-resolution timing
│ │ └── timestamp.hpp
│ ├── logging/
│ │ ├── logger.hpp/cpp
│ │ └── log_levels.hpp
│ └── types/
│ ├── order.hpp
│ ├── fill.hpp
│ ├── position.hpp
│ └── market_data.hpp
│
├── app/ # Main Application
│ ├── main.cpp
│ ├── application.hpp/cpp
│ ├── config_loader.hpp/cpp
│ └── signal_handler.hpp/cpp
│
├── scripts/
│ ├── run_system.sh
│ ├── run_benchmarks.sh
│ ├── isolate_cpus.sh # CPU isolation setup
│ └── generate_load.py # Load testing
│
└── tests/
├── unit/
├── integration/
└── performance/
5.3 The Core Question You’re Answering
“How do you build a complete trading system where every component works together with minimal latency, maximum reliability, and full operational visibility?”
This question forces you to confront:
- The overhead of integration (components that are fast alone may be slow together)
- The complexity of state management (positions, orders, risk limits must be consistent)
- The importance of observability (you can’t improve what you can’t measure)
- The reality of failure modes (networks fail, processes crash, data corrupts)
5.4 Concepts You Must Understand First
| Concept | Why It Matters | Where to Learn |
|---|---|---|
| Lock-free programming | Inter-component communication without blocking | Project 2, “Rust Atomics and Locks” |
| Event-driven architecture | Reacting to market data efficiently | “Building Low Latency Applications with C++” |
| TCP/IP networking | Connection management, buffering, Nagle | “TCP/IP Illustrated” Vol. 1 |
| CPU cache hierarchy | Data layout affects latency | CS:APP Ch. 6 |
| Thread affinity and NUMA | Isolating critical threads | Linux perf documentation |
| Binary protocols | Fast serialization/deserialization | Project 3 protocol work |
| Financial concepts | Orders, fills, positions, P&L | Any trading fundamentals book |
5.5 Questions to Guide Your Design
Before writing code, answer these on paper:
Architecture Questions:
- How will market data flow from network to strategy with minimal latency?
- Where will you introduce queues vs direct function calls?
- How will you handle the case where the strategy generates orders faster than the gateway can send?
- What happens if the exchange connection drops during trading?
Threading Questions:
- Which components need dedicated threads? Which can share?
- How will you prevent threads from contending on shared data?
- What is the optimal core assignment for your specific machine?
- How will you handle the monitoring thread without affecting trading?
State Management Questions:
- How will position state stay consistent across fills and risk checks?
- What is the source of truth for order state?
- How will you reconcile position after a restart?
- What state needs to survive a process restart?
Operational Questions:
- How will you know if the system is healthy?
- What metrics are essential to track in real-time?
- How will you handle configuration changes?
- What is the shutdown sequence to avoid losing orders?
5.6 Thinking Exercise
Before building the full system, trace through this scenario by hand:
Scenario: At T=0, market data arrives showing AAPL bid at 150.25, ask at 150.26. Your strategy decides to buy 100 shares. Trace the flow:
- T=0: Market data packet arrives at network interface
- What happens in the kernel?
- How does it reach your application?
- T=?: Market data handler processes the message
- How is the message parsed?
- How is it placed in the queue to strategy?
- T=?: Strategy receives the update
- How does it query current position?
- How does it decide to trade?
- T=?: Strategy requests an order
- How does risk manager check limits?
- What if position limit is already at max?
- T=?: Gateway sends the order
- How is the order serialized?
- How is it sent to the exchange?
- T=?: Fill comes back
- How does the fill get routed?
- How does position tracker update?
Draw the sequence diagram with estimated latencies at each step. Your goal is to understand where every microsecond goes.
5.7 Hints in Layers
Hint 1: Getting Started - Component Order
Build components in this order:
- Common infrastructure first: Types, allocators, queues, logging
- Exchange simulator second: You need something to trade against
- Market data handler third: Feed data to your system
- Position tracker fourth: Track state before trading
- Risk manager fifth: Gate before orders go out
- Gateway sixth: Actually send orders
- Strategy seventh: Make trading decisions
- Monitoring last: Observe the working system
This order ensures each component has what it depends on.
Hint 2: Integration Strategy
Start with synchronous integration, then optimize:
- Phase 1: Single-threaded, all components in one loop
- Easier to debug
- Proves the logic works
- Will be slow but correct
- Phase 2: Add queues between components
- Market data -> Strategy queue
- Strategy -> Gateway queue
- Gateway -> Position tracker queue
- Phase 3: Move components to dedicated threads
- Pin threads to cores
- Verify latency improvements
- Profile for contention
- Phase 4: Optimize hot paths
- Profile with perf
- Optimize based on data
Hint 3: Lock-Free Queue Sizing
Queue size matters:
- Too small: Producer blocks, latency spikes
- Too large: Memory waste, cache pollution
Rule of thumb:
- Market data queue: 10,000 messages (bursty input)
- Order queue: 1,000 orders (rate-limited output)
- Fill queue: 1,000 fills (matches order rate)
Power-of-two sizes enable fast modulo via bit masking.
Hint 4: Latency Measurement
Use clock_gettime(CLOCK_MONOTONIC, ...) or rdtsc for timing:
uint64_t rdtsc() {
unsigned int lo, hi;
__asm__ __volatile__ ("rdtsc" : "=a" (lo), "=d" (hi));
return ((uint64_t)hi << 32) | lo;
}
Stamp at each stage:
t1: Market data receivedt2: Order book updatedt3: Strategy decision madet4: Risk check passedt5: Order sent
Compute deltas and build histograms.
Hint 5: Configuration Management
Use YAML for configuration:
system:
log_level: info
checkpoint_interval_sec: 60
threads:
market_data_core: 0
strategy_core: 1
gateway_core: 2
exchange:
host: "127.0.0.1"
port: 9000
risk:
max_position_per_symbol: 1000
max_daily_loss: 10000
max_orders_per_second: 100
strategies:
- name: mean_reversion
enabled: true
symbols: [AAPL, MSFT]
lookback: 100
entry_threshold: 2.0
Load at startup, support runtime reload for some parameters.
Hint 6: Dashboard Implementation
Keep the dashboard simple:
- Use a lightweight HTTP library (cpp-httplib, tiny_http)
- Serve static HTML/JS/CSS
- Use WebSocket for real-time updates
- Push metrics from a dedicated thread every 100ms
Dashboard thread should never block trading threads:
- Use a lock-free queue for metric updates
- Atomic reads for counters
- Copy-on-read for complex state
5.8 The Interview Questions They’ll Ask
After completing this project, you’ll be ready for:
- “Walk me through your trading system architecture”
- They want: Clear component separation, data flow, threading model
- Bonus: Latency numbers, tradeoffs you made
- “How do you handle failures in a trading system?”
- They want: Reconnection logic, state recovery, graceful degradation
- Discuss: Order state reconciliation, position sync
- “How do you ensure thread safety without locks?”
- They want: SPSC queues, atomics, careful data ownership
- Discuss: Memory ordering, cache line considerations
- “What’s your latency budget and how do you measure it?”
- They want: End-to-end breakdown, measurement methodology
- Discuss: P99 vs P50, why percentiles matter
- “How does your risk management work?”
- They want: Pre-trade checks, position limits, kill switches
- Discuss: Fail-closed design, regulatory requirements
- “How would you scale this to multiple strategies/symbols?”
- They want: Horizontal scaling, sharding approaches
- Discuss: State partitioning, reducing contention
- “What happens when you get a partial fill?”
- They want: Order state machine, position update logic
- Discuss: Average cost calculation, remaining quantity tracking
- “How do you test a trading system?”
- They want: Unit tests, integration tests, replay testing
- Discuss: Chaos engineering, paper trading
5.9 Books That Will Help
| Topic | Book | Specific Chapters |
|---|---|---|
| Complete trading system design | Building Low Latency Applications with C++ (Ghosh) | Entire book (it builds exactly this) |
| Multi-language HFT perspective | Developing High-Frequency Trading Systems (Donadio, Ghosh, Rossier) | Ch. 4-8 |
| Lock-free programming | Rust Atomics and Locks (Bos) | Ch. 5-6 (Channels and Queues) |
| Networking fundamentals | TCP/IP Illustrated, Vol. 1 (Stevens) | Ch. 17-18 (TCP) |
| CPU and memory | Computer Systems: A Programmer’s Perspective (Bryant & O’Hallaron) | Ch. 6 (Memory), Ch. 12 (Concurrency) |
| Event-driven design | Designing Data-Intensive Applications (Kleppmann) | Ch. 11 (Stream Processing) |
| Operational excellence | Site Reliability Engineering (Google) | Ch. 4-6 (SLOs, Toil, Monitoring) |
| Financial fundamentals | Trading and Exchanges (Harris) | Ch. 1-5 (Market structure) |
5.10 Implementation Phases
Phase 1: Foundation and Exchange (Weeks 1-2)
Goals:
- Set up project structure and build system
- Implement common types and allocators
- Get exchange simulator running standalone
Tasks:
- Create project structure with CMake/Cargo
- Implement common types (Order, Fill, MarketData, Position)
- Port or reference P5 allocators (pool, arena)
- Port or reference P2 SPSC queue
- Set up exchange simulator from P3
- Test exchange with telnet/nc commands
Checkpoint: Exchange accepts orders via TCP and returns fills.
Phase 2: Market Data and Gateway (Weeks 3-4)
Goals:
- Receive and parse market data
- Submit and track orders
- Basic network I/O working
Tasks:
- Implement market data receiver (TCP client)
- Implement binary protocol parser
- Implement order gateway (TCP client)
- Implement order serialization
- Test: Send market data -> Receive in handler
- Test: Send order -> Receive ack and fill
Checkpoint: Can receive market data and submit orders programmatically.
Phase 3: Strategy and Position (Weeks 5-6)
Goals:
- Implement position tracking
- Implement P&L calculation
- Get basic strategy running
Tasks:
- Implement position tracker with fill handling
- Implement realized/unrealized P&L calculation
- Port strategy logic from P4 (or implement mean reversion)
- Connect: Market data -> Strategy -> Order generation
- Connect: Fills -> Position tracker
- Test: Full loop working (data -> trade -> fill -> position)
Checkpoint: Strategy generates trades based on market data, positions update on fills.
Phase 4: Risk and Control (Weeks 7-8)
Goals:
- Implement risk manager
- Add graceful shutdown
- Add configuration loading
Tasks:
- Implement position limit checks
- Implement P&L limit (kill switch)
- Implement order rate limiting
- Implement YAML config loading
- Implement signal handling (SIGTERM, SIGINT)
- Implement graceful shutdown sequence
- Test: Risk blocks orders beyond limits
Checkpoint: Risk manager prevents bad trades, system shuts down cleanly.
Phase 5: Monitoring and Dashboard (Weeks 9-10)
Goals:
- Implement latency tracking
- Create monitoring dashboard
- Add real-time metrics
Tasks:
- Instrument all components with timestamps
- Implement latency histogram collection
- Implement HTTP server for dashboard
- Create dashboard HTML/JS
- Implement WebSocket for real-time updates
- Display: positions, P&L, latency, throughput
- Test: Dashboard shows live data
Checkpoint: Browser dashboard shows real-time trading activity.
Phase 6: Optimization and Testing (Weeks 11-12)
Goals:
- Profile and optimize hot paths
- Implement comprehensive testing
- Achieve latency targets
Tasks:
- Profile with perf/flamegraph
- Identify and fix hot spots
- Implement thread pinning
- Tune queue sizes and memory pools
- Write integration tests
- Implement load testing script
- Document latency characteristics
Checkpoint: System meets latency targets under load, tests pass.
5.11 Key Implementation Decisions
| Decision | Options | Recommendation | Rationale |
|---|---|---|---|
| Single process vs multiple | Single | Single process | Simpler, lower latency between components |
| TCP vs UDP for market data | Both viable | TCP | Simpler, reliable, UDP only if multicast needed |
| Sync vs async network I/O | Async (epoll/io_uring) | Async | Don’t block strategy on network |
| Strategy thread model | Busy-wait vs event-driven | Busy-wait SPSC pop | Lowest latency, acceptable CPU cost |
| Price representation | Double vs fixed-point | Fixed-point int64 | Deterministic, no floating-point issues |
| Order ID generation | UUID vs sequence | Sequence (uint64_t) | Faster, ordered, debuggable |
| Config format | JSON vs YAML vs TOML | YAML | Human-readable, widely supported |
| Dashboard framework | React/Vue vs plain JS | Plain JS | Simpler, fewer dependencies |
| Logging | Sync vs async | Async with queue | Don’t block trading on disk I/O |
6. Testing Strategy
6.1 Testing Pyramid
TESTING PYRAMID
┌────────────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────┐ │
│ / \ │
│ / E2E Tests \ │
│ / (Few, Slow) \ │
│ / \ │
│ /────────────────────\ │
│ / \ │
│ / Integration Tests \ │
│ / (More, Medium) \ │
│ / \ │
│ /──────────────────────────────\ │
│ / \ │
│ / Unit Tests \ │
│ / (Many, Fast) \ │
│ /──────────────────────────────────────\ │
│ │
│ Unit Tests: Test each component in isolation │
│ Integration: Test component interactions │
│ E2E: Test complete trading scenarios │
│ │
└────────────────────────────────────────────────────────────────────┘
6.2 Unit Test Categories
| Component | Key Unit Tests |
|---|---|
| Order Book | Add/cancel/match orders, price level management |
| SPSC Queue | Concurrent push/pop, wraparound, empty/full |
| Position Tracker | Fill processing, P&L calculation, cost basis |
| Risk Manager | Limit checks (pass and fail cases) |
| Strategy | Signal generation for various market conditions |
| Protocol | Serialization/deserialization roundtrip |
| Allocator | Alloc/free patterns, pool exhaustion |
6.3 Integration Tests
INTEGRATION TEST SCENARIOS
┌────────────────────────────────────────────────────────────────────┐
│ │
│ Scenario 1: Market Data Flow │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 1. Exchange publishes market data │ │
│ │ 2. Market data handler receives and parses │ │
│ │ 3. Order book cache updates │ │
│ │ 4. Strategy receives update via queue │ │
│ │ Verify: All data arrives, order book state correct │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Scenario 2: Order Lifecycle │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 1. Strategy generates buy order │ │
│ │ 2. Risk manager approves │ │
│ │ 3. Gateway sends to exchange │ │
│ │ 4. Exchange accepts and fills │ │
│ │ 5. Gateway receives fill │ │
│ │ 6. Position tracker updates │ │
│ │ Verify: Order state correct at each step, position updated │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Scenario 3: Risk Rejection │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 1. Set position limit to 100 │ │
│ │ 2. Fill 100 shares │ │
│ │ 3. Strategy generates another buy │ │
│ │ 4. Risk manager rejects │ │
│ │ Verify: Order not sent, position unchanged │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Scenario 4: Reconnection │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 1. System connected and trading │ │
│ │ 2. Kill exchange connection │ │
│ │ 3. Gateway detects disconnect │ │
│ │ 4. Restart exchange │ │
│ │ 5. Gateway reconnects │ │
│ │ Verify: Trading resumes, no lost orders │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────┘
6.4 Chaos Engineering
Deliberately inject failures to test resilience:
| Failure Type | How to Inject | Expected Behavior |
|---|---|---|
| Network delay | tc qdisc add delay | Latency increases, no crashes |
| Packet loss | tc qdisc add loss | Reconnection, state recovery |
| Exchange slow | Sleep in matching engine | Queue backup, no crash |
| Disk full | Fill /tmp | Logging degrades gracefully |
| OOM condition | Lower cgroup memory limit | Graceful shutdown |
| Signal during trade | SIGTERM mid-order | Order completes or cancels |
6.5 Load Testing
# Generate synthetic load
python scripts/generate_load.py \
--symbols 5 \
--updates-per-second 100000 \
--duration-seconds 60 \
--output-file /tmp/load_test.log
# Run system and measure
./trading_system --config config.yaml &
sleep 5 # Let system warm up
python scripts/send_load.py /tmp/load_test.log
# Analyze results
python scripts/analyze_latency.py /tmp/latency_samples.log
6.6 Replay Testing
Use recorded market data to test strategy consistency:
REPLAY TESTING FLOW
┌────────────────────────────────────────────────────────────────────┐
│ │
│ Historical Data │
│ ┌─────────────────┐ │
│ │ 2024-01-15.pcap │ │
│ │ Market data │ │
│ │ with timestamps │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Replay Engine │────►│ Trading System │ │
│ │ (Paces data) │ │ (Normal mode) │ │
│ └─────────────────┘ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Trade Log │ │
│ │ (Today's run) │ │
│ └────────┬────────┘ │
│ │ │
│ ┌───────────────────────┴───────────────────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐│
│ │ Expected │ │ Actual ││
│ │ Trades │ DIFF │ Trades ││
│ │ (Baseline) │◄────────────────────────────│ (This run) ││
│ └─────────────────┘ └─────────────────┘│
│ │
│ If diff is non-empty: Something changed (bug or intentional?) │
│ │
└────────────────────────────────────────────────────────────────────┘
7. Common Pitfalls & Debugging
7.1 Integration Pitfalls
| Pitfall | Symptom | Root Cause | Solution |
|---|---|---|---|
| Race condition on position | Position shows wrong value occasionally | Unsynchronized access from multiple threads | Use atomic or lock-free update |
| Queue overflow | Messages dropped, latency spikes | Queue too small for burst | Increase queue size, add back-pressure |
| Order/fill mismatch | Fill arrives for unknown order | Gateway didn’t track pending order | Fix order tracking, add sequence validation |
| Stale order book | Strategy trades on old prices | Queue backup or slow processing | Profile and optimize, add staleness check |
| Thread starvation | One component never runs | Poor priority or core allocation | Review threading, add monitoring |
7.2 Latency Debugging
LATENCY DEBUGGING DECISION TREE
┌────────────────────────────────────────────────────────────────────┐
│ │
│ Latency too high? │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Step 1: Identify which stage is slow │ │
│ │ Use per-stage timestamps │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ├──► Network slow? ──► Check kernel bypass options │
│ │ Tune socket buffers │
│ │ │
│ ├──► Parsing slow? ──► Profile parser │
│ │ Use zero-copy where possible │
│ │ │
│ ├──► Strategy slow? ─► Profile strategy code │
│ │ Check for allocations │
│ │ Simplify calculations │
│ │ │
│ ├──► Queue slow? ────► Check for false sharing │
│ │ Verify queue sizing │
│ │ │
│ └──► Inconsistent? ──► Check for GC (if using managed lang) │
│ Check for page faults │
│ Check for context switches │
│ │
│ Step 2: Profile with perf │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ perf record -g ./trading_system --config config.yaml │ │
│ │ perf report │ │
│ │ Look for hot functions, cache misses, branch mispredictions │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Step 3: Flamegraph for visualization │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg│ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────┘
7.3 State Debugging
When state seems wrong:
- Add tracing: Log every state transition with timestamps
- Reproduce minimally: Find smallest sequence that causes issue
- Check race conditions: Add assertions for thread safety
- Verify message ordering: Ensure fills match orders
- Check for duplicates: Ensure idempotent handling
7.4 Performance Traps
| Trap | Why It Happens | How to Avoid |
|---|---|---|
| Logging in hot path | printf/cout is slow | Use async logging, log to buffer |
| Allocation in hot path | malloc is non-deterministic | Pre-allocate all memory |
| Virtual function calls | Indirect branch overhead | Use templates or direct calls |
| String operations | Allocation, copying | Use fixed buffers, string_view |
| Exception handling | Stack unwinding overhead | Don’t throw in hot path |
| Mutex contention | Threads waiting for each other | Use lock-free or partition data |
| Cache thrashing | Poor data locality | Co-locate related data |
8. Extensions & Challenges
8.1 Beginner Extensions
- Multiple symbol support: Trade 10+ symbols simultaneously
- Order types: Add stop orders, IOC, FOK
- Better dashboard: Add charts for price and P&L over time
- Logging levels: Add runtime log level adjustment
8.2 Intermediate Extensions
- Paper trading with real data: Connect to free market data feed (IEX, Alpaca)
- Multiple strategies: Run two strategies side by side
- FIX protocol: Implement basic FIX for orders (industry standard)
- Persistence: Save and restore state across restarts
- Replay from recording: Record live session, replay for testing
8.3 Advanced Extensions
- Kernel bypass networking: Use DPDK or Solarflare OpenOnload
- FPGA integration: Offload parsing or risk checks to FPGA
- Co-location simulation: Add configurable network latency
- Multi-process architecture: Separate exchange and trading app
- Cloud deployment: Run on AWS with monitoring (CloudWatch)
- Regulatory compliance: Add pre-trade risk controls per MiFID II
8.4 Stretch Goals
- Smart order routing: Route to multiple exchanges
- Market making strategy: Provide liquidity, manage inventory
- Machine learning signals: Add simple ML model for signals
- Cross-asset trading: Trade multiple asset classes
9. Real-World Connections
9.1 Industry Comparison
| Aspect | Your Project | Real HFT Firm |
|---|---|---|
| Latency | 10-50 microseconds | 1-10 microseconds (with kernel bypass) |
| Throughput | 100K messages/sec | 1M+ messages/sec |
| Strategies | 1 simple strategy | Dozens of complex strategies |
| Risk | Basic position limits | Real-time Greeks, VaR, stress testing |
| Connectivity | TCP to simulator | Direct market access, co-location |
| Redundancy | None | Active-passive failover |
| Compliance | None | Full audit trail, regulatory reporting |
9.2 Career Relevance
This project prepares you for roles at:
- HFT Firms: Jane Street, Two Sigma, Citadel, Jump Trading, Virtu
- Exchanges: NYSE, NASDAQ, CME, ICE
- Banks: Trading technology at Goldman, Morgan Stanley, JP Morgan
- Crypto: Trading infrastructure at Coinbase, Binance, FTX-successors
- Fintech: Robinhood, Alpaca, Interactive Brokers
9.3 Related Open Source
- QuickFIX: C++ FIX protocol implementation
- Aeron: High-performance messaging for HFT
- Chronicle Queue: Persisted low-latency queue
- LMAX Disruptor: Lock-free ring buffer (Java, concepts apply)
- Libtrading: Trading tools and protocol implementations
10. Resources
10.1 Essential Reading
| Resource | What You’ll Learn |
|---|---|
| “Building Low Latency Applications with C++” (Ghosh) | Step-by-step trading system construction |
| “Developing High-Frequency Trading Systems” (Donadio et al.) | Multi-language perspective on HFT |
| “Rust Atomics and Locks” (Bos) | Lock-free programming fundamentals |
| “TCP/IP Illustrated Vol. 1” (Stevens) | Network programming essentials |
10.2 Video Resources
- CppCon talks on low-latency (search YouTube)
- Trading Tech Talks (various exchanges host these)
- Jane Street tech talks (quantitative finance perspective)
10.3 Online References
- awesome-lockfree - Curated lock-free resources
- Linux kernel bypass networking - Cloudflare’s experience
- LMAX Architecture - Martin Fowler on Disruptor
10.4 Tools
| Tool | Purpose |
|---|---|
| perf | CPU profiling |
| flamegraph | Visualization of perf output |
| Valgrind | Memory error detection (slow, for testing) |
| AddressSanitizer | Fast memory error detection |
| ThreadSanitizer | Race condition detection |
| numactl | NUMA topology and binding |
| hwloc | Hardware topology visualization |
| turbostat | CPU frequency monitoring |
11. Self-Assessment Checklist
Understanding
- I can explain the complete data flow from market data to order execution
- I can describe why each component exists and what it does
- I understand the threading model and why specific cores are assigned
- I can explain the tradeoffs between latency, throughput, and reliability
- I understand how risk management protects against trading errors
Implementation
- All components communicate correctly
- System handles 100K+ market data updates per second
- End-to-end latency meets targets (< 50us p99)
- Risk manager correctly blocks orders beyond limits
- Position and P&L tracking is accurate
- Dashboard displays real-time data
- Graceful shutdown works correctly
- System recovers from network disconnection
Operations
- I can start the system and see it trade
- I can adjust risk limits at runtime
- I can interpret the latency metrics
- I can diagnose why a trade did or didn’t happen
- I can identify performance bottlenecks using perf
Growth
- I have debugged at least one difficult integration issue
- I understand concepts I didn’t understand when I started
- I can discuss HFT architecture in a technical interview
- I can identify areas for further optimization
12. Submission / Completion Criteria
Minimum Viable Completion
- Exchange simulator runs and matches orders
- Market data flows from exchange to strategy
- Strategy generates at least one trade
- Position updates on fill
- Basic console output shows trading activity
Full Completion
- All 7 components working and integrated
- Risk manager enforces position and P&L limits
- Dashboard displays real-time positions, P&L, latency
- Latency meets target (< 50us p99 end-to-end internal)
- Graceful shutdown with no lost orders
- Configuration via YAML file
- Basic integration tests pass
Excellence (Going Above & Beyond)
- Multiple strategies running simultaneously
- FIX protocol support
- Replay testing infrastructure
- Chaos engineering test suite
- Cloud deployment with monitoring
- Sub-20us end-to-end latency
- Connection to real market data feed (paper trading)
Conclusion
This capstone project is the culmination of everything you’ve learned. Building individual components taught you specific skills - lock-free queues, order books, matching engines. Integrating them teaches you something more valuable: systems thinking.
The challenges you’ll face - race conditions that only appear under load, latency spikes from unexpected interactions, state inconsistencies from timing windows - these are the real problems that HFT engineers solve daily.
When you complete this project, you won’t just have a working trading system. You’ll have:
- Deep understanding of low-latency architecture
- Practical experience with lock-free programming
- Knowledge of operational concerns in critical systems
- A portfolio piece that demonstrates senior-level skills
The goal isn’t to compete with Jane Street. The goal is to understand what Jane Street builds.
Now go build your trading system.
This guide was expanded from HIGH_FREQUENCY_TRADING_CPP_RUST_LEARNING_PROJECTS.md. For the complete HFT learning path, see the project index.