Learning Virtualization, Hypervisors & Hyperconvergence

Goal: After completing these projects, you’ll deeply understand how virtualization works at every layer—from hardware-assisted CPU virtualization (Intel VT-x/AMD-V) to memory management with extended page tables, I/O virtualization with paravirtualized drivers, and distributed storage in hyperconverged systems. You’ll know why containers are “lightweight VMs” (and why that’s misleading), how hypervisors trap and emulate privileged instructions, and how enterprises build cloud platforms that survive hardware failures while live-migrating workloads. Most importantly, you’ll have built working systems that demonstrate these principles, giving you the same understanding that VMware/AWS/Nutanix engineers have.

Why Virtualization Matters: The Technology That Built the Cloud

The Revolution Nobody Sees

Virtualization is the invisible foundation of modern computing. When you spin up an EC2 instance, stream a video, or use any cloud service, you’re benefiting from technology that makes one physical server look like hundreds of independent computers. This isn’t just clever software—it required fundamental changes to CPU architectures, memory management, and how we think about computing resources.

By The Numbers (2024-2025)

The impact is staggering and measurable:

The Historical Arc: From Hardware Waste to Software-Defined Infrastructure

1960s - The Mainframe Era: IBM creates CP-40, the first hypervisor, to allow multiple users to time-share expensive mainframes. Virtualization was born from economic necessity—computers cost millions, so you needed to share them.

1999 - VMware ESX Server: Brings virtualization to commodity x86 hardware. Problem: x86 wasn’t designed for virtualization—it had 17 non-virtualizable instructions. VMware’s solution: binary translation (rewriting guest code on the fly).

2005-2006 - Hardware Takes Over: Intel releases VT-x, AMD releases AMD-V. CPUs now have dedicated virtualization support, making hypervisors faster and simpler. This changes everything.

2008 - Linux KVM Merged: Virtualization becomes a core Linux feature. Suddenly, open-source hypervisors can match proprietary solutions.

2013 - Docker Releases: Containers explode in popularity. They’re not VMs (no hypervisor), but they feel similar—isolated processes with their own filesystem and networking.

2018-Present - Hyperconvergence Goes Mainstream: Storage, compute, and networking merge into software-defined infrastructure. Companies like Nutanix, VMware vSAN, and Scale Computing let you build clouds from commodity hardware.

The Evolution: From Physical to Virtual to Containers to Functions

Evolution of Compute Abstraction:

1990s: Physical Servers              2000s: Virtual Machines           2010s: Containers                2020s: Serverless Functions
┌─────────────────────┐             ┌─────────────────────┐          ┌─────────────────────┐         ┌─────────────────────┐
│  App A │ App B       │             │ ┌────┐   ┌────┐    │          │ ┌────┐ ┌────┐       │         │ ┌──────────────────┐ │
│  ───── │ ─────       │             │ │AppA│   │AppB│    │          │ │AppA│ │AppB│       │         │ │  Function Calls  │ │
│  OS (Linux)          │             │ │OS  │   │OS  │    │          │ │Bins│ │Bins│       │         │ │  (milliseconds)  │ │
│  Hardware            │             │ └────┘   └────┘    │          │ └────┘ └────┘       │         │ └──────────────────┘ │
└─────────────────────┘             │   Hypervisor        │          │  Container Engine    │         │   Function Runtime   │
                                     │   Hardware          │          │  OS                  │         │   Container/VM       │
                                     └─────────────────────┘          │  Hardware            │         │   Hypervisor         │
                                                                      └─────────────────────┘         │   Hardware           │
Waste: 70-90% idle                  Efficiency: 10-15 VMs/server    Density: 100s/server            Scale: 1000s/server     │
Boot time: Minutes                  Boot time: 30-60 seconds        Boot time: 1-3 seconds          Boot time: <100ms       │
Isolation: Perfect                  Isolation: Strong (HW)          Isolation: Good (OS)            Isolation: Shared kernel │
                                                                                                                              │
                                     └────────────────────────────────────────────────────────────────────────────────────────┘
                                                     Each layer trades stronger isolation for higher density

Why This Knowledge Is Career-Critical

For Infrastructure Engineers: You can’t troubleshoot cloud platforms without understanding virtualization. When a VM’s performance tanks, is it noisy neighbors? Memory ballooning? Shadow page table overhead? You need to know.

For Security Professionals: VM escapes, container breakouts, and Spectre/Meltdown all exploit virtualization boundaries. Understanding the isolation mechanisms is essential for security architecture.

For Software Architects: Choosing between VMs, containers, or bare metal requires understanding the trade-offs: boot time, isolation strength, resource overhead, and operational complexity.

For DevOps/SRE: Kubernetes, Docker, Terraform—all built on virtualization primitives. You’re orchestrating virtual resources; understanding what’s underneath makes you significantly more effective.

The Interview Reality

When Google, AWS, or VMware interview infrastructure engineers, they ask:

  • “How does a hypervisor handle a guest executing a privileged instruction?”
  • “What’s the difference between Type 1 and Type 2 hypervisors, and why does it matter for performance?”
  • “Explain shadow page tables vs. extended page tables.”
  • “Why can containers start in seconds while VMs take minutes?”

These aren’t trivia questions—they reveal whether you understand the systems you’re operating.

Prerequisites & Background Knowledge

Essential Prerequisites (Must Have)

✅ Solid C Programming: You’ll write code that interacts with KVM APIs, manages memory regions, and handles VM exits ✅ Basic x86 Assembly: Need to understand instruction formats, registers (RAX, RBX, CR3), and protected/real mode ✅ Linux Systems Programming: Experience with syscalls, file descriptors, memory mapping (mmap), and /dev interfaces ✅ Operating Systems Fundamentals: Understand processes, virtual memory, page tables, and interrupts ✅ Networking Basics: Know TCP/IP, routing, bridging, and how virtual networks differ from physical ones

Helpful But Not Required (You’ll Learn These)

📚 CPU Architecture Deep Dives: How VT-x works, VMCS structure, EPT internals—you’ll learn this in the projects 📚 Distributed Systems: Consensus, replication, split-brain—you’ll encounter this in the HCI project 📚 Storage Systems: RAID, Ceph, thin provisioning—covered as you build 📚 Advanced Networking: VXLAN, SR-IOV, OVS—you’ll implement these concepts

Self-Assessment Questions

Before starting, honestly assess whether you can answer:

  1. Memory Management: Can you explain what a page table does and how virtual addresses map to physical addresses?
  2. Privileged Instructions: Do you know the difference between user mode (ring 3) and kernel mode (ring 0) on x86?
  3. C Programming: Can you allocate aligned memory with posix_memalign() and map files with mmap()?
  4. Linux Debugging: Can you use strace to see syscalls and gdb to debug segfaults?
  5. Process Isolation: Do you understand how Linux isolates processes (memory spaces, file descriptors, etc.)?

If you answered “no” to more than 2: Consider completing a CS:APP-based systems programming course first. These projects assume systems-level knowledge.

Development Environment Setup

Required Tools:

  • Linux machine with KVM support: Check with egrep -c '(vmx|svm)' /proc/cpuinfo (should be > 0)
  • Compiler toolchain: gcc, make, nasm (for assembly)
  • Virtualization tools: qemu-kvm, libvirt, virt-manager
  • Debugging tools: gdb, strace, perf
  • Networking tools: bridge-utils, iproute2, tcpdump

For HCI Projects (Project 4):

  • 3+ physical machines or VMs: Each with 4+ CPU cores, 8GB+ RAM, 50GB+ disk
  • Network switch: For cluster networking (can be virtual)

For Cloud Platform (Project 6):

  • Multiple VMs or bare-metal hosts: To simulate multi-node infrastructure

Time Investment (Realistic Estimates)

Project Learning Time Building Time Total
Project 1: Toy KVM Hypervisor 1 week (reading Intel SDM, KVM docs) 2-3 weeks 3-4 weeks
Project 2: Vagrant Clone 2-3 days (libvirt API docs) 1 week 1.5 weeks
Project 3: Container Runtime 3-4 days (namespaces/cgroups research) 1 week 1.5 weeks
Project 4: HCI Home Lab 1 week (Ceph/Proxmox docs) 2-3 weeks 3-4 weeks
Project 5: Full VMM (VT-x) 2-3 weeks (Intel SDM deep dive) 4-6 weeks 6-9 weeks
Project 6: Mini Cloud Platform 2 weeks (distributed systems) 6-8 weeks 8-10 weeks

Complete all 6 projects: 6-9 months part-time (10-15 hours/week)

Important Reality Check

⚠️ These are hard projects. You will:

  • Spend hours reading Intel’s 5000-page CPU manual
  • Debug VM exits where the guest crashes without error messages
  • Deal with memory corruption bugs that crash your entire system
  • Configure networks that mysteriously don’t work

This is normal. Virtualization is complex because it sits at the intersection of hardware, operating systems, and distributed systems. The learning curve is steep, but the payoff is massive—you’ll understand infrastructure at a level most engineers never reach.

Core Concept Analysis

Virtualization breaks down into these fundamental building blocks:

Layer Core Concepts
CPU Virtualization Trap-and-emulate, hardware-assisted (VT-x/AMD-V), ring compression, binary translation
Memory Virtualization Shadow page tables, Extended Page Tables (EPT/NPT), memory overcommit, ballooning
I/O Virtualization Device emulation, paravirtualization (virtio), passthrough (SR-IOV), vSwitches
Storage Virtualization Virtual disks, thin provisioning, snapshots, distributed storage
Hypervisor Architecture Type 1 (bare-metal) vs Type 2 (hosted), microkernel vs monolithic
Hyperconvergence Unified compute/storage/network, distributed systems, software-defined everything

Deep Dive: How CPU Virtualization Actually Works

The fundamental challenge: x86 CPUs weren’t designed for virtualization. They had instructions that behaved differently in user mode vs. kernel mode, but didn’t trap when executed—making classic “trap-and-emulate” impossible.

The Virtualization Problem (Pre-VT-x):

Guest OS executes privileged instruction (e.g., "CLI" - Clear Interrupts)
          │
          ├─── If guest is in ring 0 (it thinks it is): Executes directly → Affects HOST!
          │
          └─── If guest is in ring 3 (where hypervisor puts it): Silently fails (no trap!)

VMware's Solution (1999-2006): Binary Translation
┌─────────────────────────────────────────────────┐
│  Guest executes code                            │
│         ↓                                       │
│  Hypervisor scans code for privileged instrs   │
│         ↓                                       │
│  Rewrites them to safe equivalents              │
│         ↓                                       │
│  Executes rewritten code                        │
│         ↓                                       │
│  Emulates original behavior                     │
└─────────────────────────────────────────────────┘

Intel VT-x Solution (2006+): Hardware Support
┌─────────────────────────────────────────────────┐
│  CPU has two modes: VMX root & VMX non-root     │
│                                                 │
│  Host runs in: VMX root mode (full privileges)  │
│  Guest runs in: VMX non-root mode               │
│                                                 │
│  Privileged instruction in non-root?            │
│         ↓                                       │
│  AUTOMATIC VM EXIT → Hypervisor handles it      │
└─────────────────────────────────────────────────┘

Key Insight: Modern hypervisors don’t emulate CPUs—they use hardware to trap privileged operations and emulate only those specific actions.

Deep Dive: Memory Virtualization’s Double Translation

Virtualizing memory means three address spaces:

  1. Guest Virtual Address (GVA): What the application sees
  2. Guest Physical Address (GPA): What the guest OS sees as “physical” RAM
  3. Host Physical Address (HPA): Actual RAM chips
Memory Translation Journey:

Application in VM:  GVA (0x4000_0000)
                     ↓ (guest page table)
Guest OS sees:      GPA (0x1000_0000) ← "Physical" address in VM's view
                     ↓ (shadow page table OR EPT)
Hypervisor maps:    HPA (0x8FFA_0000) ← Actual RAM location

Old Way - Shadow Page Tables (Pre-EPT):
┌──────────────────────────────────────────┐
│ Hypervisor maintains SECOND page table   │
│ Maps GVA → HPA directly                  │
│ Must synchronize when guest changes      │
│ its page tables (expensive!)             │
└──────────────────────────────────────────┘

Modern Way - Extended Page Tables (EPT/NPT):
┌──────────────────────────────────────────┐
│ Hardware walks TWO page tables:          │
│   1. Guest's GVA → GPA                   │
│   2. EPT's GPA → HPA                     │
│ No synchronization needed!               │
│ Much faster, much simpler                │
└──────────────────────────────────────────┘

Deep Dive: Containers vs. VMs—Not What You Think

Containers are NOT lightweight VMs. They’re fundamentally different:

Virtual Machine Architecture:              Container Architecture:
┌─────────────────────────┐               ┌─────────────────────────┐
│  App                    │               │  App                    │
│  ───                    │               │  ───                    │
│  libc, bins             │               │  libc, bins (optional)  │
│  ───────────────────    │               │  ─────────────────────  │
│  Guest Kernel (Linux)   │               │                         │
│  ───────────────────    │               │  [No guest kernel!]     │
│  Virtual Hardware       │               │                         │
│  ───────────────────    │               │  ─────────────────────  │
│  Hypervisor             │               │  Host Kernel (Linux)    │
│  ───────────────────    │               │  • Namespaces (PID/NET) │
│  Host Kernel            │               │  • cgroups (limits)     │
│  ───────────────────    │               │  • capabilities         │
│  Hardware (CPU w/ VT-x) │               │  ─────────────────────  │
└─────────────────────────┘               │  Hardware (any CPU)     │
                                          └─────────────────────────┘

Boot Process:
VM: Boot guest kernel → init → services    Container: Fork process → exec → done
Time: 30-60 seconds                        Time: <1 second

Isolation Mechanism:
VM: Hardware enforced (different CPU mode)  Container: OS enforced (kernel features)

Overhead:
VM: Full kernel, virtual devices            Container: Shared kernel, minimal overhead

Security:
VM: Strong (HW boundary, different kernel)  Container: Weaker (same kernel, breakouts possible)

When to Use:
VM: Different OS, strong isolation needed   Container: Same OS, fast iteration, density

Concept Summary: What You Must Internalize

Concept What You Must Understand Why It Matters Validated By
Trap-and-Emulate Why privileged instructions must trap to the hypervisor Without this, guests could affect the host Project 1, 5
VT-x/AMD-V How hardware assists virtualization (VMX modes, VM exits) Makes modern hypervisors possible Project 1, 5
Shadow Page Tables How hypervisors virtualized memory before EPT Understanding the performance cost Reading + Project 5
Extended Page Tables (EPT) Hardware-assisted memory virtualization Why modern VMs are fast Project 1, 5
VM Exits What causes guests to trap to hypervisor (I/O, CPUID, HLT) The performance bottleneck in virtualization Project 1, 5
Device Emulation Software emulates hardware (serial, disk, network) How guests see “hardware” that doesn’t exist Project 1, 2
Paravirtualization Guest knows it’s virtualized, uses special drivers (virtio) Much faster than full emulation Project 2, 4
SR-IOV Hardware allows sharing a single device across VMs How to get near-native I/O performance Reading + Project 4
Namespaces Linux isolates process views (PID, NET, MNT, etc.) How containers achieve isolation Project 3
cgroups Linux limits resources per process group How to prevent containers from hogging resources Project 3
Distributed Storage Data replicated across nodes (Ceph, vSAN) How clusters survive disk failures Project 4
Live Migration Moving running VMs between hosts with zero downtime The “magic” behind cloud maintenance Project 4, 6
Hyperconvergence Compute + storage + network in one software-defined platform Why companies pay millions for Nutanix Project 4, 6
Software-Defined Networking (SDN) Virtual networks, overlays (VXLAN), programmable switches How cloud platforms create isolated networks Project 6

Deep Dive Reading by Concept

Concept Book / Resource Specific Chapters Why This Matters
Memory Virtualization “Computer Systems: A Programmer’s Perspective” (Bryant & O’Hallaron) Chapter 9: Virtual Memory Foundational understanding of page tables
Hardware-Assisted Virtualization “Intel 64 and IA-32 Architectures SDM, Volume 3C” Chapters 23-33: VMX THE definitive reference for VT-x
KVM Internals Linux Kernel Documentation /Documentation/virt/kvm/api.rst How to use KVM APIs
Container Internals “The Linux Programming Interface” (Kerrisk) Chapter 28: Namespaces How Linux implements isolation
cgroups “How Linux Works, 3rd Edition” (Ward) Chapter 8: Process Management Resource limits and control
Distributed Storage (Ceph) “Learning Ceph, 2nd Edition” (Singh) Chapters 1-4: RADOS fundamentals How distributed storage works
Distributed Systems Consensus “Designing Data-Intensive Applications” (Kleppmann) Chapter 9: Consistency and Consensus Required for understanding HCI
Hyperconverged Infrastructure “Nutanix Bible” (Free PDF) All sections Real-world HCI architecture
Networking Fundamentals “Computer Networks, 5th Edition” (Tanenbaum) Chapter 5: Network Layer Routing, bridging, switching
Software-Defined Networking “Software Defined Networks” (Nadeau & Gray) Chapters 1-3 Overlay networks, VXLAN
Cloud Architecture “Cloud Architecture Patterns” (Wilder) Chapters on scaling and availability How to design resilient systems

Quick Start: First 48 Hours for the Overwhelmed

Feeling intimidated? Start here:

Day 1 (4-6 hours): Understand What You’re Building

  1. Read “Why Virtualization Matters” section above (15 min)
  2. Watch a VM boot in real time: Install VirtualBox, create Ubuntu VM, watch it boot (1 hour)
  3. Experiment with strace: Run strace ls and see system calls—this is what a hypervisor sees (30 min)
  4. Read Intel SDM Chapter 23 Introduction (skim, don’t deep dive) (1 hour)
  5. Explore /dev/kvm: Run ls -l /dev/kvm and lsmod | grep kvm to see it’s real (15 min)
  6. Play with containers: docker run -it alpine sh, then run ps aux inside and outside—see isolation (1 hour)

Goal: Build intuition for the concepts before diving into code.

Day 2 (4-6 hours): Write Tiny Code to Feel It

  1. Minimal mmap example: Allocate memory, write to it, understand virtual addressing (1 hour)
  2. Open /dev/kvm: Write a C program that opens /dev/kvm and reads the API version (1 hour)
  3. Namespaces experiment: Use unshare to create isolated namespaces, see PID 1 (1 hour)
  4. Read Project 3 (Container Runtime) fully—it’s the most approachable (1 hour)
  5. Start Project 3—implement PID namespace isolation (2 hours)

Goal: Get your hands dirty with code that touches virtualization primitives.

After 48 hours, you’ll have a sense of whether this is for you. If you’re hooked, continue with Project 3, then Project 1.

Path 1: “I Want to Understand the Whole Stack” (Comprehensive)

Best for: Infrastructure engineers, those aiming for FAANG-level understanding

  1. Project 3: Container Runtime (2 weeks) — Start with OS-level virtualization
  2. Project 1: Toy KVM Hypervisor (4 weeks) — Learn hardware virtualization
  3. Project 2: Vagrant Clone (2 weeks) — Understand orchestration
  4. Project 4: HCI Home Lab (4 weeks) — See how it scales
  5. Project 5: Full VMM (8 weeks) — Go deep into VT-x
  6. Project 6: Mini Cloud (10 weeks) — Build the whole platform

Total time: 7-9 months part-time Outcome: FAANG-ready infrastructure knowledge

Path 2: “I Need Practical Skills Fast” (Career-Focused)

Best for: DevOps engineers, SREs, those needing immediate job skills

  1. Project 3: Container Runtime (2 weeks) — Docker/Kubernetes foundation
  2. Project 2: Vagrant Clone (2 weeks) — IaC and automation
  3. Project 4: HCI Home Lab (4 weeks) — Production-like infrastructure
  4. Done — You can now ace DevOps interviews

Total time: 2 months part-time Outcome: Production-ready skills

Path 3: “I’m a Security Researcher” (Security-Focused)

Best for: Security engineers, those interested in VM escapes/container breakouts

  1. Project 1: Toy KVM Hypervisor (4 weeks) — Understand the boundary
  2. Project 5: Full VMM (8 weeks) — Deep dive into isolation mechanisms
  3. Project 3: Container Runtime (2 weeks) — Understand container boundaries
  4. Study: Spectre/Meltdown mitigations, VM escape techniques
  5. Done

Total time: 4 months part-time Outcome: Deep security understanding

Path 4: “I’m Building a Startup Cloud Platform” (Entrepreneurial)

Best for: Founders, senior engineers at infrastructure companies

  1. Project 2: Vagrant Clone (2 weeks) — Understand VM lifecycle management
  2. Project 4: HCI Home Lab (4 weeks) — Distributed systems fundamentals
  3. Project 6: Mini Cloud (10 weeks) — Build the whole thing
  4. Scale it — Add autoscaling, billing, monitoring
  5. Done

Total time: 4-5 months part-time Outcome: MVP cloud platform

Virtualization Stack

Hypervisors virtualize CPU, memory, and devices. Understanding how a VMM intercepts privileged instructions is essential to building reliable systems.

VM Isolation and Resource Control

Isolation depends on page tables, traps, and scheduling. You will learn the boundary between host and guest and how resources are enforced.

Storage and Networking Virtualization

Virtual disks and virtual NICs are abstractions with performance trade-offs. You will explore how they map to host resources.

Concept Summary Table

Concept Cluster What You Need to Internalize
Hypervisor model Type-1 vs Type-2 and trap-and-emulate.
Isolation Page tables, traps, and scheduling.
Virtual devices Disk and network virtualization layers.

Deep Dive Reading by Concept

Concept Book & Chapter
Virtualization basics Operating Systems: Three Easy Pieces — Virtualization chapters
Hypervisors Virtualization Essentials — Ch. 1-3
I/O virtualization Modern Operating Systems — Ch. 7

Project 1: “Toy Type-2 Hypervisor Using KVM” — Virtualization / Systems Programming

View Detailed Guide

Attribute Value
File VIRTUALIZATION_HYPERVISORS_HYPERCONVERGENCE.md
Programming Language C
Coolness Level Level 4: Hardcore Tech Flex
Business Potential 1. The “Resume Gold” (Educational/Personal Brand)
Difficulty Level 4: Expert (The Systems Architect)
Knowledge Area Virtualization / Systems Programming
Software or Tool Hypervisor
Main Book “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron

What you’ll build: A minimal hypervisor in C that uses Linux KVM to boot a tiny guest OS and execute instructions in a virtual CPU.

Why it teaches virtualization: KVM exposes the raw virtualization primitives through /dev/kvm. You’ll directly interact with virtual CPUs, memory regions, and I/O—seeing exactly how the hardware assists virtualization. This strips away all abstraction and shows you what QEMU/VirtualBox do under the hood.

Core challenges you’ll face:

  • Setting up VM memory regions and mapping guest physical addresses (maps to: memory virtualization)
  • Handling VM exits—when the guest does something requiring hypervisor intervention (maps to: trap-and-emulate)
  • Emulating basic I/O devices like a serial port (maps to: device emulation)
  • Understanding the x86 boot process and real mode (maps to: low-level systems)

Key Concepts: | Concept | Resource | |———|———-| | Hardware-assisted virtualization (VT-x) | “Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C” Ch. 23-33 - Intel | | KVM internals | “The Definitive KVM API Documentation” - kernel.org /Documentation/virt/kvm/api.rst | | Memory virtualization | “Computer Systems: A Programmer’s Perspective” Ch. 9 - Bryant & O’Hallaron | | x86 boot process | “Writing a Simple Operating System from Scratch” - Nick Blundell (Free PDF) |

Difficulty: Advanced Time estimate: 2-4 weeks Prerequisites: C programming, basic x86 assembly, Linux systems programming

Real world outcome:

  • Your hypervisor will boot a minimal “kernel” (even just a few instructions)
  • You’ll see output on a virtual serial console: “Hello from VM!”
  • You’ll be able to step through VM exits and watch the guest execute instruction-by-instruction

Learning milestones:

  1. First milestone: Create a VM, allocate memory, and load a tiny guest binary—understand memory mapping
  2. Second milestone: Handle your first VM exit (I/O or HLT instruction)—grasp the trap-and-emulate model
  3. Third milestone: Implement serial port emulation and see “Hello from VM!”—understand device emulation
  4. Final milestone: Add multiple vCPUs—understand SMP virtualization challenges

Real World Outcome (Detailed CLI Output)

When your hypervisor works, here’s EXACTLY what you’ll see:

$ gcc -o kvmhv hypervisor.c
$ ./kvmhv guest.bin

[KVM Hypervisor v0.1]
Opening /dev/kvm... OK (API version: 12)
Creating VM... OK (fd=4)
Allocating guest memory: 128MB at 0x7f8a40000000
Setting up memory region: gpa=0x0, size=134217728... OK
Creating vCPU 0... OK (fd=5)
Loading guest binary 'guest.bin' (512 bytes) at 0x1000
Setting initial registers:
  RIP = 0x1000 (entry point)
  RSP = 0x8000 (stack pointer)
  RFLAGS = 0x2 (interrupts disabled)

Running VM...

[VM Exit #1] Reason: HLT instruction at RIP=0x1004
  Guest executed: HLT (pause until interrupt)
  Action: Resuming execution

[VM Exit #2] Reason: I/O instruction at RIP=0x1008
  Port: 0x3F8 (serial COM1)
  Direction: OUT
  Data: 0x48 ('H')

[VM Exit #3] Reason: I/O instruction at RIP=0x100C
  Port: 0x3F8
  Data: 0x65 ('e')

[Serial Output]: He

... (continues for each character) ...

[Serial Output]: Hello from VM!

[VM Exit #25] Reason: HLT instruction
  Guest finished. Exiting.

Total VM exits: 25
Total instructions executed: ~1,247
Runtime: 0.003 seconds

This proves you’ve built a working hypervisor—the guest code runs in a separate CPU context, traps on privileged operations, and you’re emulating hardware devices.

The Core Question You’re Answering

“How does a hypervisor use hardware support (VT-x) to run guest code safely while maintaining control?”

Specifically:

  • How do you configure the CPU to enter VMX non-root mode?
  • What happens when the guest executes a privileged instruction?
  • How do you map guest physical addresses to host virtual addresses?
  • How do you make a guest think it’s talking to hardware when it’s just your code?

Concepts You Must Understand First

Before starting this project, you need solid understanding of:

Concept Why It’s Required Book Reference
Virtual Memory & Page Tables You’ll map guest memory into your process’s address space “CS:APP” Ch. 9, pages 787-825
x86 Privilege Levels (Rings) Understanding why guests can’t run in ring 0 Intel SDM Vol. 3A, Section 5.5
File Descriptors & ioctl() KVM is controlled via /dev/kvm and ioctl calls “The Linux Programming Interface” Ch. 13
Memory Mapping (mmap) Allocating memory for the guest “CS:APP” Ch. 9.8, pages 846-852
x86 Boot Process (Real Mode) Guests boot in 16-bit real mode initially “Writing a Simple OS from Scratch” - Nick Blundell
Basic Assembly (x86) Reading/writing simple guest code “CS:APP” Ch. 3

Questions to Guide Your Design

Ask yourself these while implementing:

  1. Memory Setup: How large should the guest’s RAM be? Where in my process’s address space will it live?
  2. Guest Code Format: Should I load a raw binary, or support ELF? Where does execution start?
  3. Register Initialization: What values should RIP, RSP, and RFLAGS have when the guest starts?
  4. VM Exit Handling: Which VM exits are critical (I/O, HLT) vs. optional (CPUID, MSR access)?
  5. Device Emulation: Do I emulate devices (slow but flexible) or use virtio (fast but complex)?
  6. Error Handling: What happens if the guest accesses invalid memory or executes bad instructions?

Thinking Exercise (Do This BEFORE Coding)

Mental Model Building Exercise:

Draw this flow on paper and explain each step:

[Your Program (Hypervisor)]
         ↓
    Open /dev/kvm
         ↓
    Create VM (ioctl KVM_CREATE_VM)
         ↓
    Allocate memory (mmap)
         ↓
    Map it to guest (ioctl KVM_SET_USER_MEMORY_REGION)
         ↓
    Create vCPU (ioctl KVM_CREATE_VCPU)
         ↓
    Set registers (ioctl KVM_SET_REGS)
         ↓
    Run guest (ioctl KVM_RUN) ← CPU enters VMX non-root mode
         ↓
    [Guest code executes]
         ↓
    VM Exit occurs (HLT / I/O / etc.)
         ↓
    Check exit reason
         ↓
    Emulate operation
         ↓
    Resume guest (ioctl KVM_RUN)

Question: At which point does the CPU switch from VMX root to VMX non-root mode? (Answer: During KVM_RUN ioctl)

The Interview Questions They’ll Ask

Completing this project prepares you for:

  1. “Explain the difference between Type 1 and Type 2 hypervisors. Where does KVM fit?”
    • Answer: Type 1 runs on bare metal (ESXi, Xen), Type 2 runs on a host OS (VirtualBox, VMware Workstation). KVM is unique—it’s a kernel module that turns Linux into a Type 1 hypervisor.
  2. “What’s a VM exit? Give three examples of what causes them.”
    • Answer: When the guest does something requiring hypervisor intervention. Examples: I/O instruction (IN/OUT), HLT instruction, access to control registers (CR3), CPUID instruction.
  3. “How does EPT improve performance over shadow page tables?”
    • Answer: Shadow page tables required synchronization whenever the guest modified its page tables. EPT lets hardware do two-level translation (GVA→GPA, GPA→HPA) without hypervisor intervention.
  4. “Your VM’s performance is terrible. How do you debug it?”
    • Answer: Count VM exits (perf kvm stat), identify hot paths (too many I/O exits?), consider paravirtualization (virtio), check if EPT is enabled.
  5. “Can a guest escape the hypervisor? How would you prevent it?”
    • Answer: Yes, via bugs (VM escape vulnerabilities like CVE-2019-14821). Prevention: validate all guest inputs, use hardware isolation (IOMMU), keep hypervisor code minimal.
  6. “How would you implement live migration?”
    • Answer: Iteratively copy memory pages while VM runs, pause VM, copy final dirty pages, transfer CPU state, resume on destination. Requires shared storage or storage migration.

Hints in Layers (Use When Stuck)

Level 1 Hint - Getting Started:

// Opening KVM and creating a VM
int kvm_fd = open("/dev/kvm", O_RDWR);
if (kvm_fd < 0) {
    perror("Failed to open /dev/kvm");
    return 1;
}

int vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0);
if (vm_fd < 0) {
    perror("Failed to create VM");
    return 1;
}

printf("KVM opened successfully, VM created\n");

Level 2 Hint - Allocating Guest Memory:

// Allocate 128MB for the guest
size_t mem_size = 128 * 1024 * 1024;
void *mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

// Tell KVM about this memory region
struct kvm_userspace_memory_region region = {
    .slot = 0,
    .flags = 0,
    .guest_phys_addr = 0,      // Guest sees this at physical address 0
    .memory_size = mem_size,
    .userspace_addr = (uint64_t)mem  // Pointer in our process
};

ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region);

Level 3 Hint - Creating vCPU and Setting Registers:

int vcpu_fd = ioctl(vm_fd, KVM_CREATE_VCPU, 0);  // CPU ID = 0

// Get the kvm_run structure (shared memory for CPU state)
size_t run_size = ioctl(kvm_fd, KVM_GET_VCPU_MMAP_SIZE, 0);
struct kvm_run *run = mmap(NULL, run_size, PROT_READ | PROT_WRITE,
                            MAP_SHARED, vcpu_fd, 0);

// Set initial registers
struct kvm_regs regs;
memset(&regs, 0, sizeof(regs));
regs.rip = 0x1000;  // Start execution at guest address 0x1000
regs.rsp = 0x8000;  // Stack at 0x8000
regs.rflags = 0x2;  // Interrupts disabled
ioctl(vcpu_fd, KVM_SET_REGS, &regs);

Level 4 Hint - Main VM Loop and Exit Handling:

while (1) {
    // Run the guest
    ioctl(vcpu_fd, KVM_RUN, 0);

    // Guest exited - check why
    switch (run->exit_reason) {
        case KVM_EXIT_HLT:
            printf("Guest halted\n");
            return 0;  // Guest finished

        case KVM_EXIT_IO:
            // I/O instruction
            if (run->io.direction == KVM_EXIT_IO_OUT &&
                run->io.port == 0x3F8) {  // Serial port
                // Get the data from the shared kvm_run structure
                uint8_t *data = (uint8_t *)run + run->io.data_offset;
                printf("%c", *data);  // Print character
                fflush(stdout);
            }
            break;

        default:
            fprintf(stderr, "Unhandled exit reason: %d\n", run->exit_reason);
            return 1;
    }
}

Books That Will Help

Topic Book Specific Chapters Why Read This
KVM API Linux Kernel Docs Documentation/virt/kvm/api.rst Only complete reference for KVM ioctl calls
VT-x Internals Intel SDM Volume 3C Chapters 23-33 Understand what KVM does under the hood
Memory Management “CS:APP” 3rd Ed Chapter 9 (Virtual Memory) Essential for understanding memory mapping
x86 Boot Process “Writing a Simple OS from Scratch” (Free PDF) Chapters 1-3 Understand real mode and why guests boot differently
Systems Programming “The Linux Programming Interface” Ch. 13 (File I/O), Ch. 49 (Memory Mappings) Master mmap, ioctl, and file descriptors
Virtualization Theory “Computer Organization and Design” (Patterson & Hennessy) Chapter 5.6 (Virtual Machines) High-level understanding of virtualization

Common Pitfalls & Debugging

Problem 1: “KVM_CREATE_VM fails with ENOENT or EACCES”

  • Why: Your CPU doesn’t support VT-x/AMD-V, or it’s disabled in BIOS, or /dev/kvm has wrong permissions
  • Fix: Check egrep -c '(vmx|svm)' /proc/cpuinfo (should be > 0), enable in BIOS, run sudo chmod 666 /dev/kvm
  • Quick test: ls -l /dev/kvm should show crw-rw-rw-

Problem 2: “Guest immediately crashes / triple faults”

  • Why: Wrong register initialization (RIP points to invalid address, or stack not set up)
  • Fix: Ensure RIP points to where you loaded guest code, RSP points to valid stack area
  • Quick test: Print registers before first KVM_RUN to verify values

Problem 3: “KVM_RUN returns but nothing happens”

  • Why: You’re not handling the VM exit reason
  • Fix: Add printf to print run->exit_reason and see what the guest is doing
  • Quick test: printf("Exit reason: %d\n", run->exit_reason); before switch statement

Problem 4: “Serial port output doesn’t show”

  • Why: Buffering, or you’re not reading from the correct offset in kvm_run
  • Fix: Add fflush(stdout) after printf, verify run->io.data_offset points to valid data
  • Quick test: Print the data in hex: printf("Data: 0x%x\n", *data);

Problem 5: “Memory corruption / segfaults in guest”

  • Why: Guest is accessing memory outside its allocated region, or you didn’t zero the memory
  • Fix: Use memset(mem, 0, mem_size) after mmap, validate guest memory accesses
  • Quick test: Run under valgrind to catch invalid memory accesses

Problem 6: “How do I create the guest binary?”

  • Why: Confusion about what guest code looks like
  • Fix: Start with simple assembly: 
    ; guest.asm
mov al, 'H'       ; Load 'H' into AL register
mov dx, 0x3F8     ; Serial port address
out dx, al        ; Output to serial port
hlt               ; Halt

    Assemble with: nasm -f bin guest.asm -o guest.bin

  • Quick test: hexdump -C guest.bin should show opcodes

Project 2: “Build Your Own Vagrant Clone” — DevOps / Infrastructure as Code

View Detailed Guide

Attribute Value
File VIRTUALIZATION_HYPERVISORS_HYPERCONVERGENCE.md
Programming Language Python or Go
Coolness Level Level 3: Genuinely Clever
Business Potential 2. The “Micro-SaaS / Pro Tool” (Solo-Preneur Potential)
Difficulty Level 2: Intermediate (The Developer)
Knowledge Area DevOps / Infrastructure as Code
Software or Tool VM Orchestrator
Main Book “Infrastructure as Code” by Kief Morris

What you’ll build: A CLI tool that reads a configuration file, provisions VMs (using libvirt/QEMU or VirtualBox), configures networking, and runs provisioning scripts—like a simplified Vagrant.

Why it teaches virtualization: You’ll learn the VM lifecycle (create, start, stop, destroy), networking (NAT, bridged, host-only), storage management (base images, copy-on-write), and configuration management. This is how real infrastructure tools work.

Core challenges you’ll face:

  • Interfacing with hypervisor APIs (libvirt, VirtualBox SDK) programmatically (maps to: hypervisor management)
  • Managing virtual disk images and copy-on-write overlays (maps to: storage virtualization)
  • Configuring virtual networks and SSH access (maps to: network virtualization)
  • Implementing idempotent provisioning (maps to: infrastructure-as-code principles)

Key Concepts: | Concept | Resource | |———|———-| | libvirt architecture | “libvirt Application Development Guide Using Python” - libvirt.org documentation | | QCOW2 disk format | “QEMU QCOW2 Specification” - QEMU documentation | | Virtual networking | “Linux Bridge-Networking Tutorial” - kernel.org bridge documentation | | SSH automation | “The Secure Shell: The Definitive Guide” Ch. 2-3 - Barrett & Silverman |

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Python or Go, basic Linux administration, familiarity with VMs as a user

Real world outcome:

  • Run ./mybox up and watch it create a VM, configure networking, SSH in, and run your provisioning scripts
  • Run ./mybox destroy to tear it down
  • You’ll have a working dev environment tool you can actually use daily

Learning milestones:

  1. First milestone: Create and start a VM from a base image—understand VM lifecycle
  2. Second milestone: Configure networking and SSH access—understand virtual networking
  3. Third milestone: Implement snapshotting and rollback—understand copy-on-write storage
  4. Final milestone: Add multi-VM support with private networking—understand software-defined networking

Real World Outcome (Detailed CLI Output)

When your Vagrant clone works, here’s EXACTLY what you’ll see:

$ cat Myboxfile
vm:
  name: "dev-ubuntu"
  image: "ubuntu-22.04-base.qcow2"
  memory: 2048
  cpus: 2
  network:
    type: "nat"
    forward_port: "8080:80"
  provision:
    - type: "shell"
      script: "./setup.sh"

$ ./mybox up

[Mybox] Reading configuration from Myboxfile... OK
[Mybox] Checking for libvirt connection... Connected to qemu:///system
[Mybox] Creating VM 'dev-ubuntu' from base image 'ubuntu-22.04-base.qcow2'...
  → Creating copy-on-write overlay disk... done (2s)
  → Overlay: /var/lib/libvirt/images/dev-ubuntu-overlay.qcow2 (547MB)
[Mybox] Configuring VM resources...
  → CPUs: 2 cores
  → Memory: 2048 MB
  → Network: NAT with port forward 8080→80
[Mybox] Starting VM... done (8s)
[Mybox] Waiting for VM to boot and get IP address...
  → IP assigned: 192.168.122.45 (12s)
[Mybox] Configuring SSH access...
  → Injecting SSH keys... done
  → Testing connection... connected!
[Mybox] Running provisioning scripts...
  → Executing 'setup.sh'...
    ✓ apt-get update
    ✓ Installing nginx
    ✓ Configuring firewall
  → Provisioning complete (42s)

[Mybox] VM 'dev-ubuntu' is ready!
  SSH: ssh -p 22 -i .mybox/private_key vagrant@192.168.122.45
  Web: http://localhost:8080 (forwarded to VM port 80)

Total setup time: 64 seconds

$ ./mybox status

NAME          STATE    IP              UPTIME   MEMORY    CPU
dev-ubuntu    running  192.168.122.45  2m 14s   341/2048  12%

$ ./mybox snapshot create baseline

[Mybox] Creating snapshot 'baseline' for VM 'dev-ubuntu'...
  → Pausing VM... done
  → Creating memory snapshot... done (1.2s)
  → Creating disk snapshot... done (0.3s)
  → Resuming VM... done
[Mybox] Snapshot 'baseline' created successfully

$ ./mybox destroy

[Mybox] Destroying VM 'dev-ubuntu'...
  → Stopping VM... done (3s)
  → Removing overlay disk... done
  → Removing VM definition... done
[Mybox] VM 'dev-ubuntu' destroyed

This proves you’ve built a working VM orchestration tool—you’re managing the full VM lifecycle programmatically with network and storage configuration.

The Core Question You’re Answering

“How do infrastructure-as-code tools manage VM lifecycle, storage, and networking across different hypervisors?”

Specifically:

  • How do you abstract hypervisor differences (QEMU, VirtualBox, VMware)?
  • What’s the minimum state needed to recreate a VM (idempotency)?
  • How do you make networking “just work” (NAT, port forwarding, host-only)?
  • How do you handle copy-on-write disks to avoid duplicating base images?

Concepts You Must Understand First

Before starting this project, you need solid understanding of:

Concept Why It’s Required Book Reference
VM Lifecycle Management You’ll create, start, stop, and destroy VMs programmatically “Infrastructure as Code” Ch. 6, pages 89-112 - Kief Morris
libvirt/VirtualBox APIs These are the hypervisor control interfaces libvirt.org documentation / VirtualBox SDK reference
QCOW2 Disk Format Understanding copy-on-write and backing files QEMU documentation “QCOW2 Image Format”
Virtual Networking Modes NAT vs bridged vs host-only “Computer Networks, 5th Edition” Ch. 5 - Tanenbaum
SSH Key Management Automating secure access to VMs “SSH: The Secure Shell” Ch. 6 - Barrett & Silverman
YAML/Config Parsing Reading user configuration files Language-specific documentation

Questions to Guide Your Design

Ask yourself these while implementing:

  1. Configuration Format: YAML or custom format? What’s the minimum config to define a VM?
  2. State Management: Where do I track VM state (running/stopped, IP addresses, snapshots)?
  3. Image Storage: Where do base images live vs. VM-specific overlays?
  4. Networking Strategy: Do I create networks on-demand or use existing ones?
  5. Provisioning: Shell scripts only, or support Ansible/Puppet too?
  6. Error Recovery: What if the VM fails to start? How do I clean up partial failures?
  7. Multi-VM: How do I let VMs talk to each other on private networks?

Thinking Exercise (Do This BEFORE Coding)

Mental Model Building Exercise:

Trace this flow on paper:

User runs: ./mybox up
     ↓
Parse Myboxfile
     ↓
Check if VM already exists (idempotency)
     ↓
If not, create from base image:
  - Create QCOW2 overlay (backing_file = base image)
  - Define VM XML (libvirt) or config (VBox)
     ↓
Attach virtual NICs
     ↓
Start VM
     ↓
Wait for DHCP lease (how do you detect IP?)
     ↓
Inject SSH keys (how? cloud-init? mount disk?)
     ↓
SSH in and run provision scripts
     ↓
Report success

Question: What happens if the user runs ./mybox up twice? (Answer: Should be idempotent—detect existing VM and skip creation)

The Interview Questions They’ll Ask

Completing this project prepares you for:

  1. “How does Vagrant achieve provider independence (VirtualBox, VMware, AWS)?”
    • Answer: Plugin architecture with a common interface. Each provider implements create/destroy/start/stop methods. Core Vagrant code calls these generic methods.
  2. “Explain copy-on-write disk images. Why use them?”
    • Answer: QCOW2 overlays reference a read-only base image. Writes go to the overlay. Saves disk space (1 base + many small overlays vs. many full copies). Faster VM creation.
  3. “How would you implement live VM snapshots?”
    • Answer: Pause VM, snapshot disk (QCOW2 internal snapshot or external), snapshot memory state, resume. For external: create new QCOW2 with current as backing file.
  4. “Your VMs can’t reach the internet. How do you debug?”
    • Answer: Check VM’s network config (ip a), verify NAT is configured on host (iptables -t nat -L), test DNS, check libvirt network is active (virsh net-list).
  5. “How would you implement multi-VM coordination (e.g., web + database)?”
    • Answer: Create a private network, assign static IPs or use DNS, ensure dependency ordering (DB starts before web), pass environment variables to provision scripts.
  6. “What’s the difference between NAT, bridged, and host-only networking?”
    • Answer: NAT: VMs share host’s IP, can reach internet. Bridged: VMs get IPs on physical network. Host-only: VMs can talk to host and each other, but not internet.

Hints in Layers (Use When Stuck)

Level 1 Hint - Connecting to libvirt:

import libvirt

# Connect to local QEMU/KVM
conn = libvirt.open('qemu:///system')
if conn is None:
    print("Failed to connect to hypervisor")
    sys.exit(1)

print(f"Connected to: {conn.getType()}")

# List all VMs
for vm_id in conn.listDomainsID():
    dom = conn.lookupByID(vm_id)
    print(f"VM: {dom.name()} (state: {dom.state()[0]})")

Level 2 Hint - Creating QCOW2 Overlay:

import subprocess

base_image = "/var/lib/libvirt/images/ubuntu-22.04-base.qcow2"
overlay_image = "/var/lib/libvirt/images/myvm-overlay.qcow2"

# Create copy-on-write overlay
cmd = [
    "qemu-img", "create",
    "-f", "qcow2",
    "-F", "qcow2",  # Format of backing file
    "-b", base_image,  # Backing file
    overlay_image
]
subprocess.run(cmd, check=True)
print(f"Created overlay: {overlay_image}")

Level 3 Hint - Defining VM with libvirt:

vm_xml = f"""
<domain type='kvm'>
  <name>myvm</name>
  <memory unit='MiB'>2048</memory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='hd'/>
  </os>
  <devices>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='{overlay_image}'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <graphics type='vnc' port='-1'/>
  </devices>
</domain>
"""

dom = conn.defineXML(vm_xml)
dom.create()  # Start the VM
print(f"VM {dom.name()} started")

Level 4 Hint - Getting VM IP Address:

import time
import re

def get_vm_ip(dom, timeout=60):
    """Wait for VM to get DHCP lease and return its IP"""
    start = time.time()
    while time.time() - start < timeout:
        try:
            # Get DHCP leases from the network
            ifaces = dom.interfaceAddresses(
                libvirt.VIR_DOMAIN_INTERFACE_ADDRESSES_SRC_LEASE
            )
            for iface, data in ifaces.items():
                if 'addrs' in data:
                    for addr in data['addrs']:
                        if addr['type'] == 0:  # IPv4
                            return addr['addr']
        except:
            pass
        time.sleep(2)
    return None

ip = get_vm_ip(dom)
print(f"VM IP: {ip}")

Books That Will Help

Topic Book Specific Chapters Why Read This
Infrastructure as Code “Infrastructure as Code” 2nd Ed - Kief Morris Ch. 6 (VM Management), Ch. 14 (Patterns) Understand IaC principles and VM lifecycle
libvirt Programming libvirt.org documentation Python bindings guide, Domain XML format Only comprehensive reference for libvirt API
QCOW2 Internals QEMU documentation “QCOW2 Image Format” specification Understand backing files and snapshots
Virtual Networking “Computer Networks, 5th Edition” - Tanenbaum Ch. 5.6 (Virtual Networks) NAT, bridging, routing fundamentals
SSH Automation “SSH: The Secure Shell” - Barrett & Silverman Ch. 6 (Key Management), Ch. 11 (Automation) Programmatic SSH and key injection
Python Systems Programming “Python Cookbook” 3rd Ed - Beazley & Jones Ch. 13 (System Admin/Scripting) Subprocess management, file operations

Common Pitfalls & Debugging

Problem 1: “libvirt.open() returns None or permission denied”

  • Why: User not in libvirt/kvm group, or qemu:///system requires root
  • Fix: Add user to group: sudo usermod -aG libvirt $USER and re-login, or use qemu:///session for user VMs
  • Quick test: groups should show libvirt, or try virsh -c qemu:///system list

Problem 2: “VM starts but immediately crashes”

  • Why: Overlay disk’s backing file path is wrong or inaccessible
  • Fix: Check with qemu-img info overlay.qcow2—verify backing file exists at that path
  • Quick test: qemu-img rebase -b /correct/path/base.qcow2 overlay.qcow2

Problem 3: “Can’t get VM’s IP address”

  • Why: VM not using DHCP, or network not started, or timing issue
  • Fix: Ensure libvirt network is active (virsh net-start default), check VM has virtio NIC
  • Quick test: virsh domifaddr <vmname> or check network DHCP leases

Problem 4: “SSH connection refused or times out”

  • Why: VM firewall blocking port 22, sshd not running, or wrong IP
  • Fix: Access VM console (virsh console <vmname>), check systemctl status sshd, verify firewall
  • Quick test: From host: nc -zv <vm-ip> 22 (should connect)

Problem 5: “Provisioning script fails but no clear error”

  • Why: Script errors not captured, or SSH non-zero exit ignored
  • Fix: Log SSH command output, check return codes, run script manually to debug
  • Quick test: ssh user@vm 'bash -x /path/to/script.sh' for verbose debugging

Problem 6: “Snapshots don’t work or VM won’t revert”

  • Why: External vs internal snapshot confusion, or disk format doesn’t support snapshots
  • Fix: For libvirt managed snapshots: virsh snapshot-create <vmname>, ensure QCOW2 format
  • Quick test: virsh snapshot-list <vmname> to see snapshots

Project 3: “Container Runtime from Scratch” — Containers / Linux Internals

View Detailed Guide

Attribute Value
File VIRTUALIZATION_HYPERVISORS_HYPERCONVERGENCE.md
Programming Language C or Go
Coolness Level Level 4: Hardcore Tech Flex
Business Potential 1. The “Resume Gold” (Educational/Personal Brand)
Difficulty Level 3: Advanced (The Engineer)
Knowledge Area Containers / Linux Internals
Software or Tool Container Runtime
Main Book “The Linux Programming Interface” by Michael Kerrisk

What you’ll build: A minimal container runtime in C or Go that uses Linux namespaces, cgroups, and chroot to isolate processes—like a simplified runc.

Why it teaches virtualization: Containers are “lightweight virtualization.” Building one exposes the Linux primitives that make isolation possible WITHOUT a hypervisor. Understanding this contrast deepens your grasp of what VMs provide vs. what OS-level virtualization provides.

Core challenges you’ll face:

  • Using namespaces (PID, NET, MNT, UTS, IPC, USER) to isolate the container (maps to: OS-level isolation)
  • Implementing cgroups to limit CPU/memory (maps to: resource management)
  • Setting up a root filesystem with pivot_root (maps to: filesystem isolation)
  • Creating virtual network interfaces (maps to: container networking)

Key Concepts: | Concept | Resource | |———|———-| | Linux namespaces | “The Linux Programming Interface” Ch. 28 - Michael Kerrisk | | cgroups | “How Linux Works, 3rd Edition” Ch. 8 - Brian Ward | | Container security | “Container Security” Ch. 1-5 - Liz Rice | | Filesystem isolation | “Linux System Programming” Ch. 2 - Robert Love |

Difficulty: Intermediate-Advanced Time estimate: 1-2 weeks Prerequisites: C or Go, Linux systems programming basics, understanding of processes

Real world outcome:

  • Run ./mycontainer run /bin/sh and get an isolated shell with its own PID namespace (PID 1)
  • Container has limited resources (512MB RAM max) and separate network stack
  • You’ll see how Docker works underneath and can explain it to anyone

Learning milestones:

  1. First milestone: Create a PID namespace—process sees itself as PID 1
  2. Second milestone: Add filesystem isolation with chroot/pivot_root—container has its own root
  3. Third milestone: Implement cgroup limits—container can only use 512MB RAM
  4. Final milestone: Set up virtual networking with veth pairs—container can communicate out

Real World Outcome (Detailed CLI Output)

When your container runtime works, here’s EXACTLY what you’ll see:

$ ./mycontainer run alpine /bin/sh

[MyContainer] Creating container 'alpine-c4a2f1'
[MyContainer] Setting up namespaces...
  ✓ PID namespace (isolated process tree)
  ✓ Mount namespace (isolated filesystem)
  ✓ Network namespace (isolated network stack)
  ✓ UTS namespace (isolated hostname)
  ✓ IPC namespace (isolated IPC resources)
[MyContainer] Setting up root filesystem...
  → Extracting alpine rootfs to /var/lib/mycontainer/alpine-c4a2f1/rootfs
  → Mounting proc, sys, dev
  → Pivot root to container rootfs
[MyContainer] Configuring cgroups...
  → Memory limit: 512 MB
  → CPU limit: 50% of 1 core
  → Creating cgroup: /sys/fs/cgroup/mycontainer/alpine-c4a2f1
[MyContainer] Setting up networking...
  → Creating veth pair: veth0-c4a2f1 <-> veth1-c4a2f1
  → Moving veth1-c4a2f1 into container namespace
  → Assigning IP: 172.16.0.2/24
  → Setting up NAT on host
[MyContainer] Container ready! (boot time: 1.2s)

/ # ps aux
PID   USER     COMMAND
    1 root     /bin/sh          ← We're PID 1! This is a container!
    5 root     ps aux

/ # cat /proc/1/cgroup
12:memory:/mycontainer/alpine-c4a2f1      ← We're in a cgroup
11:cpu:/mycontainer/alpine-c4a2f1

/ # hostname
alpine-c4a2f1                              ← Custom hostname (UTS namespace)

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
    inet 127.0.0.1/8 scope host lo
3: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
    inet 172.16.0.2/24 scope global eth0  ← Container has its own IP

/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=12.3 ms  ← Internet works!

/ # dd if=/dev/zero of=/tmp/test bs=1M count=600
dd: writing '/tmp/test': No space left on device  ← Memory limit enforced!
524+0 records in
523+0 records out

/ # exit

[MyContainer] Cleaning up container 'alpine-c4a2f1'...
  → Removing veth pair
  → Deleting cgroup
  → Unmounting filesystems
  → Removing rootfs
[MyContainer] Container stopped (uptime: 2m 14s)

$ ./mycontainer stats

CONTAINER     CPU    MEM USAGE / LIMIT     NET I/O       PIDS
alpine-c4a2f1 12.4%  145MB / 512MB        1.2KB / 850B  3

This proves you’ve built a real container runtime—the process is completely isolated with its own PID namespace, filesystem, network stack, and resource limits.

The Core Question You’re Answering

“How does Docker/podman achieve process isolation and resource limits WITHOUT a hypervisor?”

Specifically:

  • How do Linux namespaces create isolated views of system resources?
  • What’s the difference between a container and a chroot jail?
  • How do cgroups enforce CPU/memory limits on process groups?
  • How do containers get their own network stack while sharing the kernel?

Concepts You Must Understand First

Before starting this project, you need solid understanding of:

Concept Why It’s Required Book Reference
Linux Namespaces Core isolation mechanism—PID, NET, MNT, UTS, IPC, USER “The Linux Programming Interface” Ch. 28 - Kerrisk
cgroups (Control Groups) Resource limitation and accounting “How Linux Works, 3rd Edition” Ch. 8 - Ward
clone() system call Creating processes with namespace isolation “The Linux Programming Interface” Ch. 28.2 - Kerrisk
pivot_root vs chroot Changing root filesystem safely Linux man pages: pivot_root(2), chroot(2)
Virtual Network Devices veth pairs, bridges, network namespaces “Computer Networks, 5th Edition” Ch. 5 - Tanenbaum
Linux Capabilities Fine-grained privilege control “The Linux Programming Interface” Ch. 39 - Kerrisk

Questions to Guide Your Design

Ask yourself these while implementing:

  1. Namespace Order: Which namespaces do I create first? (PID, MNT, NET, UTS, IPC, USER)
  2. Filesystem Isolation: Do I use chroot or pivot_root? What’s the security difference?
  3. Rootfs Source: Do I extract a tarball, or use existing directory, or download from registry?
  4. cgroup Hierarchy: Where do I mount cgroups v2? How do I enforce limits?
  5. Networking Strategy: Bridge networking or macvlan? How do containers reach internet (NAT)?
  6. Security: Do I drop capabilities? Run as non-root inside container?
  7. Process Reaping: Who reaps zombie processes? (PID 1’s responsibility!)

Thinking Exercise (Do This BEFORE Coding)

Mental Model Building Exercise:

Trace what happens when you run ./mycontainer run alpine /bin/sh:

1. Main process (on host):
   ↓
2. clone() with CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | ...
   ↓
3. Child process enters new namespaces:
   - Inside PID namespace: child is PID 1
   - Inside MOUNT namespace: has private mount table
   - Inside NET namespace: has empty network stack
   ↓
4. Set up root filesystem:
   - Extract alpine rootfs to temporary directory
   - Mount /proc, /sys, /dev inside it
   - pivot_root to switch root
   ↓
5. Configure cgroups (from PARENT process):
   - Write limits to /sys/fs/cgroup/.../memory.max
   - Add child PID to cgroup.procs
   ↓
6. Set up networking (from PARENT process):
   - Create veth pair
   - Move one end into child's network namespace
   - Configure IP address on both ends
   - Set up NAT rules
   ↓
7. exec("/bin/sh") inside child:
   - Replaces container init with /bin/sh
   - User now has interactive shell

Question: Why must cgroup and network setup happen from the PARENT process, not the child? (Answer: Because cgroups and network manipulation require privileges the child may not have)

The Interview Questions They’ll Ask

Completing this project prepares you for:

  1. “Explain the difference between VMs and containers at the kernel level.”
    • Answer: VMs run separate kernels via hypervisor. Containers share the host kernel but use namespaces for isolation. VMs use hardware-assisted virtualization (VT-x), containers use kernel features (namespaces/cgroups).
  2. “What are Linux namespaces? Name at least 5 types.”
    • Answer: Namespaces provide isolated views of system resources. Types: PID (process IDs), MNT (mount points), NET (network stack), UTS (hostname), IPC (shared memory), USER (user/group IDs), Cgroup (cgroup hierarchy).
  3. “How would you prevent a container from using more than 512MB of RAM?”
    • Answer: Use cgroups. Write 536870912 (512MB in bytes) to /sys/fs/cgroup/<container>/memory.max, add container’s PID to cgroup.procs.
  4. “What’s the difference between chroot and pivot_root?”
    • Answer: chroot changes root for current process but old root still accessible via ... pivot_root swaps root mount, making old root inaccessible—more secure for containers.
  5. “How do containers get their own IP address?”
    • Answer: Create network namespace, create veth pair (virtual ethernet), move one end into namespace, assign IP to each end, configure routing/NAT on host.
  6. “Why can’t containers run different kernels?”
    • Answer: Containers share the host kernel. Namespaces isolate resources, not the kernel itself. VMs virtualize hardware, allowing different kernels.
  7. “What’s a zombie process and why do containers care?”
    • Answer: Zombie = terminated but not reaped. In containers, process PID 1 must reap zombies. If not, they accumulate and waste PIDs.

Hints in Layers (Use When Stuck)

Level 1 Hint - Creating PID Namespace (C):

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

#define STACK_SIZE (1024 * 1024)

static char child_stack[STACK_SIZE];

static int child_func(void *arg) {
    printf("Child PID: %d\n", getpid());  // Will be 1!
    printf("Parent PID: %d\n", getppid()); // Will be 0!

    execl("/bin/sh", "sh", NULL);
    return 0;
}

int main() {
    pid_t pid = clone(child_func,
                      child_stack + STACK_SIZE,
                      CLONE_NEWPID | SIGCHLD,
                      NULL);

    waitpid(pid, NULL, 0);
    return 0;
}

Level 2 Hint - Setting Up Filesystem with pivot_root:

// After entering mount namespace
// 1. Mount new root as bind mount
mount("./rootfs", "./rootfs", NULL, MS_BIND | MS_REC, NULL);

// 2. Create directory for old root
mkdir("./rootfs/oldroot", 0755);

// 3. Pivot root
syscall(SYS_pivot_root, "./rootfs", "./rootfs/oldroot");

// 4. Change to new root
chdir("/");

// 5. Unmount old root
umount2("/oldroot", MNT_DETACH);
rmdir("/oldroot");

// 6. Mount proc, sys, dev
mount("proc", "/proc", "proc", 0, NULL);
mount("sysfs", "/sys", "sysfs", 0, NULL);
mount("tmpfs", "/dev", "tmpfs", MS_NOSUID | MS_STRICTATIME, "mode=755");

Level 3 Hint - Setting Up cgroups (from parent):

#include <sys/stat.h>
#include <fcntl.h>

void setup_cgroups(pid_t child_pid) {
    // Create cgroup directory
    char cgroup_path[256];
    snprintf(cgroup_path, sizeof(cgroup_path),
             "/sys/fs/cgroup/mycontainer-%d", child_pid);
    mkdir(cgroup_path, 0755);

    // Set memory limit (512MB)
    char mem_limit_path[512];
    snprintf(mem_limit_path, sizeof(mem_limit_path),
             "%s/memory.max", cgroup_path);
    int fd = open(mem_limit_path, O_WRONLY);
    write(fd, "536870912", 9);  // 512MB in bytes
    close(fd);

    // Add process to cgroup
    char procs_path[512];
    snprintf(procs_path, sizeof(procs_path),
             "%s/cgroup.procs", cgroup_path);
    fd = open(procs_path, O_WRONLY);
    char pid_str[32];
    snprintf(pid_str, sizeof(pid_str), "%d", child_pid);
    write(fd, pid_str, strlen(pid_str));
    close(fd);
}

Level 4 Hint - Creating veth Pair (Python/shell):

import subprocess

def setup_network(container_pid):
    """Set up veth pair for container networking"""

    # Create veth pair
    subprocess.run([
        "ip", "link", "add", "veth0",
        "type", "veth", "peer", "name", "veth1"
    ], check=True)

    # Move veth1 into container's network namespace
    subprocess.run([
        "ip", "link", "set", "veth1",
        "netns", str(container_pid)
    ], check=True)

    # Configure host side (veth0)
    subprocess.run(["ip", "addr", "add", "172.16.0.1/24", "dev", "veth0"], check=True)
    subprocess.run(["ip", "link", "set", "veth0", "up"], check=True)

    # Configure container side (need nsenter)
    subprocess.run([
        "nsenter", "-t", str(container_pid), "-n",
        "ip", "addr", "add", "172.16.0.2/24", "dev", "veth1"
    ], check=True)

    subprocess.run([
        "nsenter", "-t", str(container_pid), "-n",
        "ip", "link", "set", "veth1", "up"
    ], check=True)

    # Set up NAT for internet access
    subprocess.run([
        "iptables", "-t", "nat", "-A", "POSTROUTING",
        "-s", "172.16.0.0/24", "-j", "MASQUERADE"
    ], check=True)

Books That Will Help

Topic Book Specific Chapters Why Read This
Linux Namespaces “The Linux Programming Interface” - Kerrisk Ch. 28 (entire chapter on namespaces) THE authoritative guide to namespaces
cgroups “How Linux Works, 3rd Edition” - Ward Ch. 8.4 (cgroups), pages 192-197 Practical cgroup usage
Container Security “Container Security” - Liz Rice Ch. 1-5 (Linux security fundamentals) Security implications of containers
Network Namespaces “Linux Kernel Networking” - Rami Rosen Ch. 12 (Network Namespaces) Deep dive into network isolation
clone() and Processes “The Linux Programming Interface” - Kerrisk Ch. 24 (Process Creation), Ch. 28.2 How to use clone() with flags
Filesystem Isolation “Linux System Programming” - Love Ch. 2.9 (chroot), kernel docs for pivot_root chroot vs pivot_root

Common Pitfalls & Debugging

Problem 1: “clone() fails with EINVAL”

  • Why: Missing _GNU_SOURCE define, or invalid namespace flags
  • Fix: Add #define _GNU_SOURCE at top of file, ensure flags are OR’d correctly: CLONE_NEWPID | CLONE_NEWNS | SIGCHLD
  • Quick test: strace your program to see exact error

Problem 2: “Container can’t see /proc or /sys”

  • Why: Forgot to mount them after pivot_root
  • Fix: Mount after changing root: mount("proc", "/proc", "proc", 0, NULL);
  • Quick test: Inside container, run ls /proc—should see processes

Problem 3: “Permission denied when writing to cgroup files”

  • Why: Need root privileges, or cgroup v1 vs v2 mismatch
  • Fix: Run as root, check cgroup version: mount | grep cgroup
  • Quick test: cat /proc/cgroups to see available controllers

Problem 4: “Container has no network connectivity”

  • Why: veth not configured, or NAT not set up, or routing missing
  • Fix: Check both ends of veth pair have IPs and are UP, verify iptables NAT rule exists
  • Quick test: From container: ip addr (should see eth0), ip route (should see default via 172.16.0.1)

Problem 5: “Memory limit not enforced”

  • Why: Wrong cgroup path, or cgroup controller not enabled
  • Fix: Verify cgroup exists: ls /sys/fs/cgroup/mycontainer-*, check memory.max file contains your limit
  • Quick test: cat /sys/fs/cgroup/<container>/memory.current to see usage

Problem 6: “Zombie processes accumulating in container”

  • Why: PID 1 not reaping children (must call wait())
  • Fix: Container’s PID 1 must set up SIGCHLD handler or periodically call wait()
  • Quick test: ps aux inside container—look for <defunct> processes

Problem 7: “pivot_root fails with EBUSY”

  • Why: Current directory not in new root, or old root still has mounts
  • Fix: Ensure chdir() into new root first, check /proc/mounts for lingering mounts
  • Quick test: Use MNT_DETACH flag with umount2() to force unmount

Project 4: “Hyperconverged Home Lab with Distributed Storage” — Infrastructure / Distributed Systems

View Detailed Guide

Attribute Value
File VIRTUALIZATION_HYPERVISORS_HYPERCONVERGENCE.md
Programming Language N/A (Infrastructure/Configuration)
Coolness Level Level 3: Genuinely Clever
Business Potential 3. The “Service & Support” Model (B2B Utility)
Difficulty Level 2: Intermediate (The Developer)
Knowledge Area Infrastructure / Distributed Systems
Software or Tool HCI Cluster
Main Book “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A 3-node cluster running Proxmox VE with Ceph storage, demonstrating VM high availability, live migration, and distributed storage—a mini enterprise HCI setup.

Why it teaches hyperconvergence: Hyperconvergence is about unifying compute, storage, and networking in a software-defined manner. By building a cluster that can survive node failures and migrate VMs live, you’ll understand why companies pay millions for Nutanix/vSAN.

Core challenges you’ll face:

  • Setting up a Ceph cluster for distributed storage (maps to: software-defined storage)
  • Configuring HA and fencing for automatic VM failover (maps to: cluster management)
  • Implementing live migration and understanding its constraints (maps to: stateful workload mobility)
  • Network design with VLANs and bonding (maps to: software-defined networking)

Key Concepts: | Concept | Resource | |———|———-| | Distributed storage (RADOS/Ceph) | “Learning Ceph, 2nd Edition” Ch. 1-4 - Karan Singh | | Cluster consensus | “Designing Data-Intensive Applications” Ch. 9 - Martin Kleppmann | | Live migration | Proxmox documentation on live migration requirements | | Software-defined networking | “Computer Networks, Fifth Edition” Ch. 5 - Tanenbaum & Wetherall |

Difficulty: Intermediate (but hardware-intensive) Time estimate: 1-2 weeks (plus hardware acquisition) Prerequisites: Basic Linux administration, networking fundamentals, 3 machines (can be VMs for learning, but physical preferred)

Real world outcome:

  • VMs running on your cluster with shared storage
  • Pull the power on one node—watch the VMs automatically restart on surviving nodes
  • Trigger a live migration—watch a running VM move between hosts with zero downtime
  • This is enterprise infrastructure at home

Learning milestones:

  1. First milestone: Build 3-node Ceph cluster with replicated storage—understand distributed storage
  2. Second milestone: Deploy VMs on shared storage—understand storage-compute separation
  3. Third milestone: Perform live migration—understand memory and state transfer
  4. Final milestone: Simulate node failure and watch HA recover—understand fencing and failover

Real World Outcome (Detailed CLI Output)

When your hyperconverged cluster works, here’s EXACTLY what you’ll see:

# On Node 1 (pve-node1):
$ pvecm status
Cluster information
───────────────────
Name:             homelab-hci
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
──────────────────
Date:             Sat Dec 28 14:32:18 2024
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.8
Quorate:          Yes             ← Cluster has quorum!

Membership information
──────────────────────
    Nodeid      Votes Name
0x00000001          1 pve-node1 (local)
0x00000002          1 pve-node2
0x00000003          1 pve-node3

$ ceph status
  cluster:
    id:     a1b2c3d4-e5f6-7890-abcd-ef1234567890
    health: HEALTH_OK             ← All good!

  services:
    mon: 3 daemons, quorum pve-node1,pve-node2,pve-node3 (age 2h)
    mgr: pve-node1(active, since 2h), standbys: pve-node2, pve-node3
    osd: 9 osds: 9 up (since 2h), 9 in (since 2h)

  data:
    pools:   3 pools, 256 pgs
    objects: 1.24k objects, 4.8 GiB
    usage:   14.4 GiB used, 885.6 GiB / 900 GiB avail
    pgs:     256 active+clean        ← All data is healthy and replicated!

$ pvesm status
Name             Type     Status           Total            Used       Available        %
ceph-pool        rbd      active      900.00 GiB       14.40 GiB      885.60 GiB    1.60%
local            dir      active      100.00 GiB       25.30 GiB       74.70 GiB   25.30%

# Create VM on shared storage
$ qm create 100 --name web-server --memory 2048 --cores 2 \
    --scsi0 ceph-pool:32,discard=on --net0 virtio,bridge=vmbr0 --boot c

$ qm start 100
Starting VM 100... done

$ qm status 100
status: running
ha-state: started
ha-managed: 1                      ← HA is managing this VM!
node: pve-node1
pid: 12345
uptime: 42

# Perform live migration
$ qm migrate 100 pve-node2 --online

[Migration] Starting online migration of VM 100 to pve-node2
  → Precopy phase: iteratively copying memory pages
    Pass 1: 2048 MB @ 1.2 GB/s (1.7s)
    Pass 2: 145 MB @ 980 MB/s (0.15s) ← Dirty pages from pass 1
    Pass 3: 12 MB @ 850 MB/s (0.01s)  ← Converging!
  → Switching VM execution to pve-node2
    Stop VM on pve-node1... done (10ms)
    Transfer final state (CPU, devices)... done (35ms)
    Start VM on pve-node2... done (22ms)
  → Cleanup on pve-node1... done

Migration completed successfully!
Downtime: 67ms                     ← VM was unreachable for only 67ms!

$ qm status 100
status: running
node: pve-node2                    ← Now running on node2!
uptime: 1m 54s (migration was seamless)

# Now simulate node failure
$ ssh pve-node2
$ sudo systemctl stop pxe-ha-lrm   # Simulate crash (or pull power cable)

# Back on pve-node1 (30 seconds later):
$ pvecm status
...
Nodes:            2               ← Only 2 nodes responding!
Quorate:          Yes             ← Still have quorum (majority: 2/3)

$ journalctl -u pve-ha-lrm -f
[HA Manager] Node pve-node2 not responding (timeout)
[HA Manager] Fencing node pve-node2...
[HA Manager] VM 100 marked for recovery
[HA Manager] Starting VM 100 on pve-node1... done
[HA Manager] VM 100 recovered successfully

$ qm status 100
status: running
node: pve-node1                    ← Automatically restarted on node1!
uptime: 45s (recovered from failure)

$ ceph status
  cluster:
    health: HEALTH_WARN            ← Warning because of degraded data
            9 osds: 6 up, 6 in     ← 3 OSDs down (from node2)
            Degraded data redundancy: 256/768 objects degraded

  data:
    pgs:     128 active+clean
             128 active+clean+degraded  ← Data still accessible!

# Bring node2 back online
$ ssh pve-node2
$ sudo systemctl start pve-cluster

# 5 minutes later:
$ ceph status
  cluster:
    health: HEALTH_OK              ← Cluster recovered!

  services:
    osd: 9 osds: 9 up, 9 in        ← All OSDs back

  data:
    pgs:     256 active+clean      ← Data fully replicated again!

This proves you’ve built enterprise-grade hyperconverged infrastructure—VMs run on shared storage, can live-migrate with minimal downtime, and automatically recover from node failures.

The Core Question You’re Answering

“How do hyperconverged systems unify compute and storage while surviving hardware failures?”

Specifically:

  • How does distributed storage (Ceph) keep data available when disks/nodes fail?
  • What’s the mechanism for live migration (how do you move RAM and CPU state)?
  • How does high availability detect failures and restart VMs elsewhere?
  • Why does quorum matter in distributed systems?

Concepts You Must Understand First

Before starting this project, you need solid understanding of:

Concept Why It’s Required Book Reference
Distributed Consensus Understanding quorum, split-brain, fencing “Designing Data-Intensive Applications” Ch. 9 - Kleppmann
Replication Strategies How Ceph replicates data across nodes “Designing Data-Intensive Applications” Ch. 5 - Kleppmann
RADOS (Ceph) Ceph’s core distributed object store “Learning Ceph, 2nd Edition” Ch. 1-4 - Singh
Live Migration Memory transfer, pre-copy, post-copy Proxmox/KVM documentation on migration
Cluster Networking VLANs, bonding, MTU, latency requirements “Computer Networks, 5th Edition” Ch. 5 - Tanenbaum
Fencing/STONITH Preventing split-brain in HA clusters Proxmox HA documentation

Questions to Guide Your Design

Ask yourself these while implementing:

  1. Node Count: Why minimum 3 nodes? (Answer: Quorum needs majority)
  2. Network Design: Separate networks for management, storage (Ceph), VM traffic?
  3. Replication Factor: Ceph replica count—2 or 3? Trade-off?
  4. Fencing Mechanism: How to safely kill a non-responsive node?
  5. Storage Performance: SSD for journals/metadata, HDD for data?
  6. HA Policy: Automatic restart, or manual intervention?
  7. Failure Scenarios: What if 2 nodes fail? Cluster becomes read-only!

Thinking Exercise (Do This BEFORE Building)

Mental Model Building Exercise:

Draw the data path for a VM disk write:

VM writes to virtual disk
     ↓
QEMU/KVM on Node1
     ↓
RBD client library (Ceph)
     ↓
Calculates object placement (CRUSH algorithm)
     ↓
Network: sends to 3 OSDs on different nodes
     ↓
OSD on Node1: writes to disk
OSD on Node2: writes to disk (replica 1)
OSD on Node3: writes to disk (replica 2)
     ↓
All 3 ACK → Write confirmed to VM

Question: What happens if Node2’s OSD is down when the write occurs? (Answer: Ceph marks it degraded, still writes to Node1 and Node3—data safe, will re-replicate later)

The Interview Questions They’ll Ask

Completing this project prepares you for:

  1. “Explain how Ceph ensures data durability with replica count 3.”
    • Answer: Each object written to 3 different OSDs (on different nodes/failure domains). Write succeeds only after all 3 ACK. CRUSH algorithm deterministically places replicas.
  2. “What’s quorum and why does a 3-node cluster need 2 nodes minimum?”
    • Answer: Quorum prevents split-brain. With 3 nodes, majority is 2. If network splits 2 vs 1, only the 2-node side has quorum and can make decisions.
  3. “How does live migration work? How much downtime?”
    • Answer: Pre-copy migration: iteratively copy RAM while VM runs. Final switchover: pause VM, copy last dirty pages + CPU state, resume on destination. Downtime: typically 50-200ms.
  4. “What’s fencing and why is it critical for HA?”
    • Answer: Fencing = forcibly powering off a non-responsive node. Critical to prevent two nodes running the same VM (split-brain). Uses IPMI/iLO or network-based STONITH.
  5. “Your Ceph cluster shows HEALTH_WARN. What do you check?”
    • Answer: ceph health detail for specifics. Common: degraded PGs (replicating), slow OSDs (network/disk issue), clock skew, mon quorum issues.
  6. “Can you do live migration without shared storage?”
    • Answer: Yes, but you must migrate storage too (block migration). Much slower—copy entire disk over network while copying RAM.

Hints in Layers (Use When Stuck)

Level 1 Hint - Installing Proxmox Cluster:

# On all nodes: Install Proxmox VE
# Node 1 (first node):
$ pvecm create homelab-hci

# Node 2 and 3:
$ pvecm add 192.168.1.10  # IP of node1

Level 2 Hint - Creating Ceph Cluster:

# On Node 1:
$ pveceph install
$ pveceph init --network 10.0.0.0/24  # Ceph cluster network

# On all nodes:
$ pveceph mon create

# On each node, for each disk:
$ pveceph osd create /dev/sdb  # Repeat for sdc, sdd, etc.

# Create pool for VMs:
$ pveceph pool create vm-pool --add_storages

Level 3 Hint - Configuring HA:

# Enable HA service on all nodes:
$ systemctl enable pve-ha-lrm
$ systemctl enable pve-ha-crm

# Add VM to HA management:
$ ha-manager add vm:100
$ ha-manager set vm:100 --state started
$ ha-manager set vm:100 --max_restart 2

Level 4 Hint - Testing Live Migration:

# Ensure VM is on shared storage (Ceph)
$ qm config 100 | grep scsi0
scsi0: ceph-pool:vm-100-disk-0,size=32G

# Migrate with monitoring:
$ qm migrate 100 pve-node2 --online --verbose

# Watch migration progress in another terminal:
$ watch -n 0.5 'qm status 100'

Books That Will Help

Topic Book Specific Chapters Why Read This
Distributed Storage “Learning Ceph, 2nd Edition” - Singh Ch. 1-4 (RADOS, OSDs, Pools) Understanding Ceph architecture
Distributed Consensus “Designing Data-Intensive Applications” - Kleppmann Ch. 9 (Consistency and Consensus) Quorum, split-brain, CAP theorem
Cluster Networking “Computer Networks, 5th Edition” - Tanenbaum Ch. 5 (Network Layer), Ch. 6 (Link Layer) VLANs, bonding, performance tuning
High Availability Proxmox HA Documentation Full HA guide Practical HA configuration
Replication “Designing Data-Intensive Applications” - Kleppmann Ch. 5 (Replication) Sync vs async, failure handling

Common Pitfalls & Debugging

Problem 1: “Cluster won’t form quorum”

  • Why: Network connectivity issues, or time skew, or wrong cluster network
  • Fix: Ensure all nodes can ping each other, sync clocks (NTP), check firewall (ports 5404-5405, 3128)
  • Quick test: pvecm status on each node, verify all nodes listed

Problem 2: “Ceph OSDs won’t start or are down”

  • Why: Disk permissions, SELinux, or disk already has data/partitions
  • Fix: Wipe disk first: wipefs -a /dev/sdb, check journalctl -u ceph-osd@*
  • Quick test: ceph osd tree to see OSD status

Problem 3: “Live migration fails with ‘storage not available’“

  • Why: VM disk is on local storage, not shared (Ceph)
  • Fix: Migrate disk to Ceph first, or use --with-local-disks for block migration
  • Quick test: qm config <vmid> | grep scsi0 should show ceph-pool:

Problem 4: “HA doesn’t restart VM after node failure”

  • Why: Fencing not configured, or VM not added to HA management
  • Fix: Configure fencing (IPMI), ensure ha-manager status shows VM as managed
  • Quick test: ha-manager status should list VM with state “started”

Problem 5: “Ceph performance is terrible”

  • Why: No SSD for journal/DB, or network latency, or wrong replica size
  • Fix: Use SSDs for WAL/DB, dedicated 10GbE network, tune pg_num
  • Quick test: ceph osd perf to see OSD latency, iperf3 between nodes for network

Problem 6: “Split-brain—two nodes think they have quorum”

  • Why: Network partition without proper fencing
  • Fix: This is catastrophic—prevent with fencing. If it happens, stop cluster, manually reconcile
  • Quick test: pvecm status on each partition—should only ONE have Quorate: Yes

Project 5: “Write a Simple Virtual Machine Monitor (VMM)” — Virtualization / CPU Architecture

View Detailed Guide

Attribute Value
File VIRTUALIZATION_HYPERVISORS_HYPERCONVERGENCE.md
Programming Language C
Coolness Level Level 5: Pure Magic (Super Cool)
Business Potential 1. The “Resume Gold” (Educational/Personal Brand)
Difficulty Level 5: Master (The First-Principles Wizard)
Knowledge Area Virtualization / CPU Architecture
Software or Tool Virtual Machine Monitor
Main Book “Intel SDM Volume 3C, Chapters 23-33” by Intel

What you’ll build: A user-space VMM that uses Intel VT-x directly (via /dev/kvm or raw VMXON) to run guest code, handle VM exits, and emulate a minimal set of devices.

Why it teaches hypervisors: This goes deeper than Project 1—you’ll understand VMCS (Virtual Machine Control Structure), EPT (Extended Page Tables), and the full VM lifecycle at the hardware level. This is what engineers at VMware/AWS work on.

Core challenges you’ll face:

  • Programming the VMCS fields correctly for guest/host state (maps to: VT-x internals)
  • Implementing EPT for memory virtualization (maps to: hardware-assisted memory virtualization)
  • Handling complex VM exits (CPUID, MSR access, I/O) (maps to: instruction emulation)
  • Making the guest actually boot a real kernel (maps to: full system emulation)

Key Concepts: | Concept | Resource | |———|———-| | VT-x architecture | “Intel SDM Volume 3C, Chapter 23-33” - Intel (the definitive reference) | | VMCS programming | “Hypervisor From Scratch” tutorial series - Sina Karvandi | | Extended Page Tables | “Understanding the Linux Kernel” Hardware-Assisted Virtualization chapter - Bovet & Cesati | | x86 system programming | “Write Great Code, Volume 2” - Randall Hyde |

Difficulty: Expert Time estimate: 1+ month Prerequisites: Strong C, x86 assembly, OS internals, patience for hardware documentation

Real world outcome:

  • Boot a minimal Linux kernel in your VMM and see it print to the console
  • You’ll have built something comparable to the core of QEMU/Firecracker
  • A genuine hypervisor that you built from scratch

Learning milestones:

  1. First milestone: Enter VMX operation and create a basic VMCS—understand virtualization root mode
  2. Second milestone: Run guest code and handle first VM exit—understand VM entry/exit flow
  3. Third milestone: Implement EPT and boot to protected mode—understand hardware memory virtualization
  4. Final milestone: Boot a real kernel and handle complex devices—you’ve built a hypervisor

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor Hardware Needed
Toy KVM Hypervisor Advanced 2-4 weeks ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ Linux with KVM
Vagrant Clone Intermediate 1-2 weeks ⭐⭐⭐ ⭐⭐⭐⭐ Any with VMs
Container Runtime Intermediate-Advanced 1-2 weeks ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ Linux only
HCI Home Lab Intermediate 1-2 weeks ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ 3+ machines
Full VMM (VT-x) Expert 1+ month ⭐⭐⭐⭐⭐ ⭐⭐⭐ Linux + Intel CPU

Recommendation

Based on learning depth while remaining achievable:

Start with Project 3 (Container Runtime) if you want immediate, satisfying results. It teaches isolation primitives in 1-2 weeks and gives you deep Linux systems knowledge that transfers directly to understanding VMs.

Then do Project 1 (Toy KVM Hypervisor) to understand what containers don’t provide and why hardware virtualization exists. This builds on the Linux systems knowledge from Project 3.

Follow up with Project 4 (HCI Home Lab) to see how these technologies scale in production—this connects the low-level understanding to real enterprise infrastructure.

This progression takes you from “how does process isolation work?” → “how does hardware virtualization work?” → “how do we build reliable infrastructure with these?”

Project 6: “Build a Mini Cloud Platform” — Cloud Infrastructure / Distributed Systems

View Detailed Guide

Attribute Value
File VIRTUALIZATION_HYPERVISORS_HYPERCONVERGENCE.md
Programming Language Python/Go + C
Coolness Level Level 5: Pure Magic (Super Cool)
Business Potential 4. The “Open Core” Infrastructure (Enterprise Scale)
Difficulty Level 5: Master (The First-Principles Wizard)
Knowledge Area Cloud Infrastructure / Distributed Systems
Software or Tool Cloud Platform
Main Book “Designing Data-Intensive Applications” by Martin Kleppmann

What you’ll build: A complete “cloud-in-a-box” system that combines everything: a custom orchestrator that manages VMs across multiple hypervisor nodes, with distributed storage, an API for VM lifecycle management, live migration, and a web dashboard—essentially a minimal OpenStack/Proxmox you built yourself.

Why it teaches everything: This project forces you to integrate CPU virtualization, memory management, storage virtualization, networking, distributed systems, and API design. You can’t fake understanding when you have to make all these pieces work together.

Core challenges you’ll face:

  • Designing a multi-node architecture with a control plane (maps to: distributed systems)
  • Implementing VM scheduling and placement (maps to: resource management)
  • Building distributed storage or integrating Ceph (maps to: software-defined storage)
  • Creating overlay networks for VM connectivity across hosts (maps to: SDN)
  • Implementing live migration with storage and memory transfer (maps to: stateful mobility)
  • Building a REST API and dashboard (maps to: cloud API design)

Key Concepts: | Concept | Resource | |———|———-| | Cloud architecture | “Cloud Architecture Patterns” - Bill Wilder (O’Reilly) | | Distributed systems | “Designing Data-Intensive Applications” Ch. 5, 8, 9 - Martin Kleppmann | | OpenStack architecture | “OpenStack Operations Guide” - OpenStack Foundation | | SDN fundamentals | “Software Defined Networks” - Nadeau & Gray (O’Reilly) | | API design | “REST API Design Rulebook” - Mark Massé (O’Reilly) | | KVM/QEMU integration | “Mastering KVM Virtualization” Ch. 1-6 - Humble Devassy |

Difficulty: Expert Time estimate: 2-3 months Prerequisites: Completed at least 2-3 projects above, strong programming skills, networking knowledge

Real world outcome:

  • curl -X POST /api/vms -d '{"name": "web01", "cpu": 2, "ram": 4096}' creates a VM
  • The scheduler places it on the least-loaded host
  • The VM gets an IP on your overlay network
  • You can live-migrate it between hosts via API
  • A dashboard shows all VMs, resource usage, and cluster health

This is what you’d build as a founding engineer at a cloud startup. Completing this means you genuinely understand virtualization infrastructure.

Learning milestones:

  1. First milestone: Build control plane that tracks hosts and VMs—understand distributed state
  2. Second milestone: Implement VM lifecycle API (create/start/stop/destroy)—understand orchestration
  3. Third milestone: Add networked storage for VM images—understand shared storage requirements
  4. Fourth milestone: Implement basic scheduling—understand resource bin-packing
  5. Fifth milestone: Add overlay networking with VXLAN—understand network virtualization
  6. Sixth milestone: Implement live migration—understand stateful workload mobility
  7. Final milestone: Build dashboard—see your cloud working visually

Summary: Your Journey from Virtualization Novice to Infrastructure Expert

After completing these projects, you will have built:

  1. A working KVM hypervisor that boots guest OSs using hardware virtualization (Project 1)
  2. A VM orchestration tool like Vagrant for infrastructure-as-code (Project 2)
  3. A container runtime from scratch using Linux namespaces and cgroups (Project 3)
  4. A hyperconverged cluster with distributed storage and high availability (Project 4)
  5. A full VMM with VT-x support and EPT memory virtualization (Project 5)
  6. A mini cloud platform with orchestration, networking, and live migration (Project 6)

Complete Project List

# Project Name Main Language Difficulty Time Key Concepts
1 Toy KVM Hypervisor C Expert 3-4 weeks VT-x, VM exits, memory mapping
2 Vagrant Clone Python/Go Intermediate 1.5 weeks libvirt, QCOW2, virtual networking
3 Container Runtime C/Go Advanced 1.5 weeks Namespaces, cgroups, pivot_root
4 HCI Home Lab Infrastructure Intermediate 3-4 weeks Ceph, quorum, live migration
5 Full VMM (VT-x) C Master 6-9 weeks VMCS, EPT, instruction emulation
6 Mini Cloud Platform Python/Go + C Master 8-10 weeks Distributed systems, SDN, orchestration

Skills You Will Have Mastered

Hardware Virtualization:

  • Intel VT-x/AMD-V architecture and VMCS programming
  • Extended Page Tables (EPT) for memory virtualization
  • VM entry/exit handling and instruction emulation
  • Device emulation and paravirtualization (virtio)

Operating System Concepts:

  • Linux namespaces (PID, NET, MNT, UTS, IPC, USER)
  • cgroups for resource limitation
  • Process isolation mechanisms
  • Virtual networking with veth pairs and bridges

Distributed Systems:

  • Consensus and quorum (Raft/Paxos principles)
  • Data replication strategies (Ceph RADOS)
  • Split-brain prevention and fencing
  • Live migration and state transfer

Software-Defined Infrastructure:

  • Virtual networking (VXLAN, SDN, overlay networks)
  • Software-defined storage (Ceph, thin provisioning)
  • Hyperconverged architecture
  • Cloud API design and orchestration

Interview Readiness

You’ll be able to confidently answer questions like:

  • “Explain how a hypervisor traps privileged instructions”
  • “What’s the difference between containers and VMs at the kernel level?”
  • “How does live migration achieve sub-second downtime?”
  • “Explain shadow page tables vs. Extended Page Tables”
  • “How does Ceph ensure data durability across node failures?”
  • “What’s quorum and why does it matter in distributed systems?”
  • “Walk me through what happens when you execute docker run”

Career Paths This Enables

Infrastructure Engineering at Scale:

  • Cloud platforms (AWS, Google Cloud, Azure)
  • Virtualization companies (VMware, Nutanix, Red Hat)
  • Container platforms (Docker, Kubernetes teams)

Systems Programming:

  • Hypervisor development
  • Container runtime engineering
  • Kernel/low-level systems work

Site Reliability Engineering (SRE):

  • Deep understanding of infrastructure for debugging production issues
  • Ability to optimize virtualization/container performance
  • Understanding of failure modes in distributed systems

Security Engineering:

  • VM escape and container breakout analysis
  • Isolation mechanism auditing
  • Security boundaries in multi-tenant environments

Beyond These Projects

Once you’ve completed these, you’re ready for:

  • Contributing to open source: KVM, QEMU, containerd, Kubernetes
  • Building production systems: You understand the primitives well enough to architect cloud platforms
  • Advanced topics: Confidential computing (AMD SEV, Intel TDX), unikernels, serverless infrastructure
  • Research: Virtualization performance, security, novel isolation mechanisms

Sources and Further Reading

Market Research and Statistics

The global hyperconverged infrastructure market is experiencing explosive growth:

  • The HCI market reached USD 12.52 billion in 2024 and is expected to reach USD 51.22 billion by 2030, growing at a CAGR of 25.09% (Industry ARC Report)

  • Virtual machine market projected to reach USD 43.81 billion by 2034, growing at 14.71% CAGR (Precedence Research)

  • 96% of organizations use containers in production or development environments (Aqua Security)

  • Global cloud spending forecast at $723.4 billion in 2025, up from $595.7 billion in 2024 (Technavio)

Performance Tuning Resources

Security Comparisons

You’ve reached the end of this learning path. The journey from understanding trap-and-emulate to building a full cloud platform is long, but every step builds on the previous one. Start with Project 3 (containers) for quick wins, then tackle Project 1 (KVM hypervisor) for deep understanding. Good luck!