Project 13: Mini-QEMU Clone
Build a user-space emulator like QEMU that combines a CPU emulator, memory manager, and device emulators to run a simple operating system.
Quick Reference
| Attribute | Value |
|---|---|
| Difficulty | Expert (Level 4: The Systems Architect) |
| Time Estimate | 6-8 weeks |
| Language | C (alternatives: Rust) |
| Prerequisites | Projects 2 (RISC-V Emulator), 6 (Memory Mapper), 7-9 (Device Emulators) |
| Key Topics | Full System Emulation, CPU Emulation, Device Models, Interrupt Controllers, Boot Process |
1. Learning Objectives
By completing this project, you will:
- Understand how QEMU and similar full-system emulators work internally
- Learn to integrate CPU emulation, memory management, and device models into a cohesive system
- Implement a main emulation loop that coordinates all components
- Build an interrupt controller that routes device interrupts to the CPU
- Understand the PC boot process from power-on to OS execution
- Master the architecture of production-quality system emulators
2. Theoretical Foundation
2.1 Core Concepts
What is Full System Emulation?
Full system emulation creates a complete virtual computer in software:
┌────────────────────────────────────────────────────────────────────────┐
│ Real Computer vs. Emulated Computer │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ REAL COMPUTER EMULATED COMPUTER (QEMU-style) │
│ ┌─────────────────────┐ ┌─────────────────────────────┐ │
│ │ Application │ │ Application │ │
│ ├─────────────────────┤ ├─────────────────────────────┤ │
│ │ Guest OS │ │ Guest OS │ │
│ ├─────────────────────┤ ├─────────────────────────────┤ │
│ │ Hardware: │ │ Virtual Hardware: │ │
│ │ ┌─────────────┐ │ │ ┌─────────────────────┐ │ │
│ │ │ CPU │ │ │ │ CPU Emulator │ │ │
│ │ │ RAM │ │ │ │ Memory Manager │ │ │
│ │ │ Disk │ │ │ │ Block Device Emu │ │ │
│ │ │ Network │ │ │ │ Network Device Emu │ │ │
│ │ │ Graphics │ │ │ │ Display Emulator │ │ │
│ │ │ Serial │ │ │ │ UART Emulator │ │ │
│ │ │ Interrupt │ │ │ │ PIC/APIC Emulator │ │ │
│ │ └─────────────┘ │ │ └─────────────────────┘ │ │
│ └─────────────────────┘ ├─────────────────────────────┤ │
│ │ │ Emulator Core: │ │
│ │ Direct HW access │ ┌─────────────────────┐ │ │
│ ▼ │ │ Main Loop │ │ │
│ ┌─────────────────────┐ │ │ Device Dispatch │ │ │
│ │ Physical Hardware │ │ │ Event Handling │ │ │
│ └─────────────────────┘ │ └─────────────────────┘ │ │
│ ├─────────────────────────────┤ │
│ │ Host OS │ │
│ ├─────────────────────────────┤ │
│ │ Physical Hardware │ │
│ └─────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
QEMU Architecture Overview
┌────────────────────────────────────────────────────────────────────────┐
│ QEMU Architecture │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Main Thread │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ Main Loop │ │ │
│ │ │ │ │ │
│ │ │ while (running) { │ │ │
│ │ │ poll_events(); // Check I/O, timers │ │ │
│ │ │ process_timers(); // Handle expired timers │ │ │
│ │ │ run_vcpu(); // Execute guest code │ │ │
│ │ │ handle_io(); // Process device I/O │ │ │
│ │ │ } │ │ │
│ │ │ │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────┐ ┌──────────────────────┐ │
│ │ CPU Emulation │ │ Memory Subsystem │ │
│ │ │ │ │ │
│ │ ┌──────────────────┐ │ │ ┌──────────────────┐ │ │
│ │ │ TCG (Tiny Code │ │ │ │ Address Space │ │ │
│ │ │ Generator) │ │ │ │ Manager │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ - Translate │ │ │ │ - RAM regions │ │ │
│ │ │ guest → host │ │ │ │ - ROM regions │ │ │
│ │ │ - Block cache │ │ │ │ - MMIO dispatch │ │ │
│ │ │ - Optimization │ │ │ │ - DMA handling │ │ │
│ │ └──────────────────┘ │ │ └──────────────────┘ │ │
│ │ OR │ │ │ │
│ │ ┌──────────────────┐ │ │ │ │
│ │ │ KVM Acceleration │ │ │ │ │
│ │ │ (Hardware VMX) │ │ │ │ │
│ │ └──────────────────┘ │ │ │ │
│ └──────────────────────┘ └──────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Device Layer │ │
│ │ │ │
│ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │
│ │ │ PIC │ │ Timer │ │ UART │ │ IDE │ │ NIC │ │ │
│ │ │ 8259A │ │ PIT │ │ 16550 │ │ (disk) │ │ e1000 │ │ │
│ │ └────────┘ └────────┘ └────────┘ └────────┘ └────────┘ │ │
│ │ │ │ │ │ │ │ │
│ │ └──────────┴──────────┴──────────┴──────────┘ │ │
│ │ │ │ │
│ │ ┌─────────▼─────────┐ │ │
│ │ │ Interrupt │ │ │
│ │ │ Controller │ │ │
│ │ │ (PIC/APIC) │ │ │
│ │ └─────────┬─────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ CPU IRQ input │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Backend Layer │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Disk Image │ │ TAP Device │ │ PTY │ │ │
│ │ │ (.qcow2) │ │ (network) │ │ (serial) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
The Main Emulation Loop
The heart of any system emulator is the main loop:
┌────────────────────────────────────────────────────────────────────────┐
│ Main Emulation Loop │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ INITIALIZE: │ │
│ │ - Load BIOS/firmware │ │
│ │ - Initialize CPU state │ │
│ │ - Create devices │ │
│ │ - Map memory regions │ │
│ │ │ │
│ │ MAIN LOOP: │ │
│ │ ┌────────────────────────────────────────────────────────────┐ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ 1. CHECK EVENTS │ │ │ │
│ │ │ │ - Poll file descriptors (network, serial, disk) │ │ │ │
│ │ │ │ - Check for user input (keyboard, mouse) │ │ │ │
│ │ │ │ - Process host signals │ │ │ │
│ │ │ └─────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ 2. PROCESS TIMERS │ │ │ │
│ │ │ │ - Check if any timers expired │ │ │ │
│ │ │ │ - Fire timer callbacks (PIT, RTC, watchdog) │ │ │ │
│ │ │ │ - Potentially inject timer interrupt │ │ │ │
│ │ │ └─────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ 3. CHECK INTERRUPTS │ │ │ │
│ │ │ │ - Query interrupt controller (PIC/APIC) │ │ │ │
│ │ │ │ - If interrupt pending and CPU accepts: │ │ │ │
│ │ │ │ * Inject interrupt into CPU │ │ │ │
│ │ │ └─────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ 4. EXECUTE CPU │ │ │ │
│ │ │ │ - Run N instructions (quantum) │ │ │ │
│ │ │ │ - Stop early if: │ │ │ │
│ │ │ │ * I/O instruction (needs device) │ │ │ │
│ │ │ │ * Halt instruction (wait for interrupt) │ │ │ │
│ │ │ │ * Interrupt pending │ │ │ │
│ │ │ └─────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ 5. HANDLE I/O │ │ │ │
│ │ │ │ - If CPU stopped for I/O: │ │ │ │
│ │ │ │ * Dispatch to appropriate device │ │ │ │
│ │ │ │ * Device processes request │ │ │ │
│ │ │ │ * May trigger interrupt │ │ │ │
│ │ │ └─────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ 6. UPDATE DEVICES │ │ │ │
│ │ │ │ - Tick device state machines │ │ │ │
│ │ │ │ - Process async I/O completions │ │ │ │
│ │ │ │ - Update display if changed │ │ │ │
│ │ │ └─────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ └─────────► (back to step 1) │ │ │
│ │ │ │ │ │
│ │ └────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ SHUTDOWN: │ │
│ │ - Stop CPU │ │
│ │ - Flush device state │ │
│ │ - Close backends │ │
│ │ - Free resources │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Memory-Mapped I/O Dispatch
┌────────────────────────────────────────────────────────────────────────┐
│ MMIO Address Space Layout │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ Guest Physical Address Space (typical x86 PC layout): │
│ │
│ 0x00000000 ┌───────────────────────────────────────────┐ │
│ │ Conventional Memory (640KB) │ │
│ │ RAM - direct access │ │
│ 0x000A0000 ├───────────────────────────────────────────┤ │
│ │ VGA Frame Buffer (128KB) │ │
│ │ MMIO - dispatch to VGA device │ │
│ 0x000C0000 ├───────────────────────────────────────────┤ │
│ │ Video BIOS ROM (32KB) │ │
│ │ ROM - read only │ │
│ 0x000C8000 ├───────────────────────────────────────────┤ │
│ │ Option ROMs (160KB) │ │
│ │ ROM/empty │ │
│ 0x000F0000 ├───────────────────────────────────────────┤ │
│ │ System BIOS ROM (64KB) │ │
│ │ ROM - read only │ │
│ 0x00100000 ├───────────────────────────────────────────┤ │
│ │ Extended Memory (RAM) │ │
│ │ RAM - direct access │ │
│ │ ... (depends on configured memory size) │ │
│ │ │ │
│ 0xFEC00000 ├───────────────────────────────────────────┤ │
│ │ I/O APIC │ │
│ │ MMIO - dispatch to APIC device │ │
│ 0xFEE00000 ├───────────────────────────────────────────┤ │
│ │ Local APIC │ │
│ │ MMIO - dispatch to APIC device │ │
│ 0xFFFFFFFF └───────────────────────────────────────────┘ │
│ │
│ I/O Port Space (x86 specific): │
│ │
│ Port Range │ Device │
│ ──────────────┼───────────────────────── │
│ 0x0000-0x001F │ DMA Controller │
│ 0x0020-0x0021 │ PIC (8259A) Master │
│ 0x0040-0x0043 │ PIT (8254 Timer) │
│ 0x0060 │ Keyboard Controller │
│ 0x0070-0x0071 │ RTC/CMOS │
│ 0x00A0-0x00A1 │ PIC (8259A) Slave │
│ 0x01F0-0x01F7 │ IDE Primary │
│ 0x02F8-0x02FF │ COM2 │
│ 0x03F8-0x03FF │ COM1 (Serial) │
│ 0x03D4-0x03D5 │ VGA CRTC │
│ │
└────────────────────────────────────────────────────────────────────────┘
2.2 Why This Matters
Building a full system emulator teaches you:
How Operating Systems Actually Boot:
- BIOS → bootloader → kernel sequence
- Real mode → protected mode → long mode transitions
- Device initialization and enumeration
How Hardware/Software Interact:
- Interrupt-driven I/O model
- DMA and memory-mapped I/O
- Device driver architecture
How QEMU Works:
- QEMU is one of the most important open-source projects
- Used by libvirt, Docker (on Mac/Windows), Android emulator
- Understanding QEMU opens many career opportunities
Real-World Applications:
- Security research (malware analysis in isolation)
- Embedded development (test without hardware)
- OS development (test OS without rebooting)
- Cloud infrastructure (KVM backend)
2.3 Historical Context
1999: Fabrice Bellard starts QEMU as a personal project
- Initially just a fast x86 emulator using dynamic translation
- Novel approach: generate host code at runtime
2003: QEMU 0.1 released publicly
- Supported Linux guest on Linux host
- Introduced the emulation approach still used today
2006: KVM (Kernel-based Virtual Machine) created
- Used QEMU for device models
- QEMU + KVM became the dominant open-source virtualization solution
Today: QEMU supports dozens of architectures
- Over 1 million lines of code
- Used by AWS, Google, countless organizations
- Still actively developed
2.4 Common Misconceptions
Misconception 1: “QEMU is a hypervisor”
- Reality: QEMU alone is an emulator (software CPU). With KVM, the combination is a hypervisor. QEMU provides device models; KVM provides hardware virtualization.
Misconception 2: “Emulators are always slow”
- Reality: With JIT compilation (TCG), QEMU achieves good performance. With KVM acceleration, performance is near-native.
Misconception 3: “You need to implement every device”
- Reality: You only need devices the guest actually uses. A minimal system needs: CPU, RAM, interrupt controller, timer, serial port, and boot device.
Misconception 4: “The hard part is the CPU emulator”
- Reality: CPU emulation is well-understood. The hard parts are: accurate device timing, interrupt handling, and the complex interactions between components.
3. Project Specification
3.1 What You Will Build
A user-space system emulator that:
- Emulates a simple CPU (RISC-V RV32I or simplified x86)
- Manages guest memory with RAM and MMIO regions
- Implements essential devices (interrupt controller, timer, UART, block device)
- Runs a main loop coordinating all components
- Boots and runs a simple operating system (e.g., xv6, FreeDOS, or a custom kernel)
3.2 Functional Requirements
- CPU Emulation: Fetch-decode-execute loop for chosen ISA
- Memory System: RAM allocation, MMIO dispatch, address space management
- Interrupt Controller: Route device interrupts to CPU
- Timer: Periodic interrupts for OS scheduling
- UART: Serial I/O for console
- Block Device: Read/write sectors from disk image
- Boot Process: Load and execute bootloader/kernel
3.3 Non-Functional Requirements
- Correctness: Guest OS must boot and run correctly
- Debuggability: CPU state dump, memory inspection, device tracing
- Modularity: Clean separation between CPU, memory, and devices
- Performance: Reasonable speed for interactive use (not real-time)
3.4 Example Usage / Output
$ ./miniemu --ram=64M --disk=freedos.img --serial=stdio
[MINIEMU] Mini-QEMU Clone v0.1
[MINIEMU] ═══════════════════════════════════════════════════════════════
[MINIEMU] Configuration:
CPU: Software RISC-V RV32I Emulator
RAM: 64MB at base 0x80000000
Disk: freedos.img (512MB, VirtIO-blk)
Serial: Connected to stdio
[MINIEMU] Memory Map:
┌────────────────────────────────────────────────────────────────────┐
│ 0x00001000 - 0x00001FFF: Boot ROM (4KB) │
│ 0x0C000000 - 0x0C00FFFF: PLIC (Platform-Level Interrupt Controller)│
│ 0x10000000 - 0x10000FFF: UART 16550 (Serial Port) │
│ 0x10001000 - 0x10001FFF: VirtIO-blk (Block Device) │
│ 0x80000000 - 0x83FFFFFF: RAM (64MB) │
└────────────────────────────────────────────────────────────────────┘
[MINIEMU] Devices:
- PLIC: Initialized with 32 interrupt sources
- UART: 16550 at 0x10000000, IRQ 1
- VirtIO-blk: At 0x10001000, IRQ 2, backing file: freedos.img
[MINIEMU] Loading bootloader from sector 0...
[MINIEMU] Read 512 bytes from disk
[MINIEMU] Bootloader loaded at 0x80000000
[MINIEMU] Starting CPU at 0x80000000...
[MINIEMU] ═══════════════════════════════════════════════════════════════
[CPU] Cycle 0: PC=0x80000000
[CPU] Executing: lui t0, 0x10000 ; Load UART base address
[CPU] Registers: t0=0x10000000
[CPU] Cycle 1: PC=0x80000004
[CPU] Executing: addi t1, zero, 'B' ; Character to print
[CPU] Registers: t1=0x42
[CPU] Cycle 2: PC=0x80000008
[CPU] Executing: sb t1, 0(t0) ; Write to UART THR
[MMIO] Write to 0x10000000 (UART THR): 0x42 ('B')
[UART] TX: 'B'
[... bootloader continues ...]
[MINIEMU] ═══════════════════════════════════════════════════════════════
[MINIEMU] Boot Messages:
═══════════════════════════════════════════════════════════════════════════
Bootloader v1.0
Loading kernel...
Kernel loaded at 0x80100000
Initializing hardware...
UART: OK
Timer: OK
Disk: VirtIO-blk, 1048576 sectors
Starting kernel...
FreeDOS kernel loading...
FreeDOS 1.3 [build 2043]
HIMEM driver installed
JEMM386 driver installed
C:\> dir
Volume in drive C is FREEDOS
Directory of C:\
KERNEL SYS 45,945 03-01-2022 12:00p
COMMAND COM 66,945 03-01-2022 12:00p
AUTOEXEC BAT 125 03-01-2022 12:00p
CONFIG SYS 89 03-01-2022 12:00p
4 file(s) 113,104 bytes
463,211,520 bytes free
C:\> echo Hello from Mini-QEMU!
Hello from Mini-QEMU!
C:\> type autoexec.bat
@echo off
echo Welcome to FreeDOS on Mini-QEMU!
═══════════════════════════════════════════════════════════════════════════
[MINIEMU] Session Statistics:
Total CPU cycles: 847,293,412
Instructions executed: 312,456,789
MIPS: ~52 (on host Intel i7-9700K)
Interrupts:
Timer: 15,234
UART RX: 47
Disk: 1,234
I/O Operations:
UART TX: 12,456 bytes
UART RX: 47 bytes
Disk reads: 1,178 sectors
Disk writes: 56 sectors
Memory:
Peak guest memory usage: 8.2MB
Host memory used: 68MB
[MINIEMU] Shutdown complete.
4. Solution Architecture
4.1 High-Level Design
┌────────────────────────────────────────────────────────────────────────┐
│ Mini-QEMU Clone Architecture │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ main.c │ │
│ │ ┌────────────────────────────────────────────────────────────┐ │ │
│ │ │ main() │ │ │
│ │ │ - Parse command line │ │ │
│ │ │ - Initialize emulator │ │ │
│ │ │ - Run main loop │ │ │
│ │ │ - Cleanup │ │ │
│ │ └────────────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ emu.c (Core Emulator) │ │
│ │ ┌────────────────────────────────────────────────────────────┐ │ │
│ │ │ struct emulator │ │ │
│ │ │ - CPU state │ │ │
│ │ │ - Memory manager │ │ │
│ │ │ - Device list │ │ │
│ │ │ - Interrupt controller │ │ │
│ │ └────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌────────────────────────────────────────────────────────────┐ │ │
│ │ │ emu_run() - Main Loop │ │ │
│ │ │ │ │ │
│ │ │ while (running) { │ │ │
│ │ │ // 1. Poll I/O events │ │ │
│ │ │ poll_io_events(); │ │ │
│ │ │ │ │ │
│ │ │ // 2. Handle timers │ │ │
│ │ │ process_timers(); │ │ │
│ │ │ │ │ │
│ │ │ // 3. Check and inject interrupts │ │ │
│ │ │ if (plic_has_pending() && cpu_irq_enabled()) │ │ │
│ │ │ cpu_inject_interrupt(plic_claim()); │ │ │
│ │ │ │ │ │
│ │ │ // 4. Execute CPU instructions │ │ │
│ │ │ for (int i = 0; i < 1000; i++) │ │ │
│ │ │ cpu_step(); │ │ │
│ │ │ │ │ │
│ │ │ // 5. Tick devices │ │ │
│ │ │ uart_tick(); │ │ │
│ │ │ virtio_blk_tick(); │ │ │
│ │ │ } │ │ │
│ │ └────────────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────┼─────────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ cpu.c │ │ memory.c │ │ devices/*.c │ │
│ │ │ │ │ │ │ │
│ │ - Registers │ │ - RAM alloc │ │ - UART │ │
│ │ - Fetch │ │ - MMIO map │ │ - Timer │ │
│ │ - Decode │◄───────►│ - Read/Write │◄───────►│ - Block │ │
│ │ - Execute │ │ - Dispatch │ │ - PLIC │ │
│ │ - Interrupt │ │ │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ ▲ │ │
│ │ │ │ │
│ │ │ │ │
│ └──────────── memory_read/write() ─────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
4.2 Key Components
1. CPU Emulator (cpu.c)
- Maintains register file (32 registers for RISC-V)
- Implements fetch-decode-execute cycle
- Handles memory access via memory subsystem
- Supports interrupt injection
2. Memory Subsystem (memory.c)
- Allocates and manages guest RAM
- Maintains address space mapping
- Dispatches MMIO to appropriate devices
- Provides read/write interface for CPU
3. Device Layer (devices/*.c)
Each device implements a common interface:
struct device {
const char *name;
uint64_t base_addr;
uint64_t size;
uint32_t (*read)(struct device *dev, uint64_t offset, int size);
void (*write)(struct device *dev, uint64_t offset, uint32_t val, int size);
void (*tick)(struct device *dev);
int irq;
};
4. Interrupt Controller (plic.c)
- Tracks pending interrupts from devices
- Prioritizes and delivers to CPU
- Implements PLIC (RISC-V) or PIC (x86) interface
5. Main Loop (emu.c)
- Coordinates all components
- Handles timing and synchronization
- Processes external I/O events
4.3 Data Structures
/* Main emulator state */
struct emulator {
/* CPU */
struct cpu_state cpu;
/* Memory */
struct memory_region *regions;
int num_regions;
void *ram; /* Pointer to guest RAM */
size_t ram_size;
/* Devices */
struct device *devices;
int num_devices;
/* Interrupt controller */
struct plic plic;
/* Event loop */
int epoll_fd; /* For I/O multiplexing */
bool running;
/* Statistics */
uint64_t total_cycles;
uint64_t total_instructions;
};
/* CPU state (RISC-V RV32I) */
struct cpu_state {
uint32_t regs[32]; /* x0-x31 */
uint32_t pc; /* Program counter */
/* CSRs (Control and Status Registers) */
uint32_t mstatus; /* Machine status */
uint32_t mie; /* Machine interrupt enable */
uint32_t mip; /* Machine interrupt pending */
uint32_t mtvec; /* Machine trap vector */
uint32_t mepc; /* Machine exception PC */
uint32_t mcause; /* Machine cause */
/* Internal state */
bool halted; /* Waiting for interrupt */
bool interrupt_pending; /* Interrupt to inject */
int pending_irq; /* IRQ number */
};
/* Memory region */
struct memory_region {
uint64_t base;
uint64_t size;
enum region_type type; /* RAM, ROM, MMIO */
void *data; /* For RAM/ROM: pointer to data */
struct device *device; /* For MMIO: device to dispatch to */
int flags; /* Read, Write, Execute permissions */
};
/* Interrupt controller (simplified PLIC) */
struct plic {
uint32_t pending; /* Bitmap of pending interrupts */
uint32_t enabled; /* Bitmap of enabled interrupts */
uint32_t priority[32]; /* Priority per source */
uint32_t threshold; /* Priority threshold */
uint32_t claimed; /* Currently serviced interrupt */
};
4.4 Algorithm Overview
Main Emulation Loop
ALGORITHM: emu_run()
1. INITIALIZE:
a. Parse command line options
b. Allocate guest RAM
c. Create and map devices
d. Load BIOS/firmware into ROM
e. Load disk image
f. Set CPU to initial state (PC = reset vector)
2. MAIN LOOP (while running):
a. POLL I/O:
- Use epoll/select to check file descriptors
- If UART input available, queue to UART RX FIFO
- If disk I/O complete, update virtio status
b. PROCESS TIMERS:
- Check wall-clock time
- If timer expired:
* Reload timer
* Set timer interrupt pending
c. CHECK INTERRUPTS:
- Query PLIC for highest-priority pending interrupt
- If interrupt pending AND CPU accepts interrupts:
* Save CPU state (PC to mepc)
* Set mcause to interrupt code
* Jump to mtvec (interrupt handler)
d. EXECUTE CPU:
- For N instructions (N = quantum, e.g., 1000):
* Fetch instruction from memory[PC]
* Decode instruction
* Execute (may access memory, MMIO)
* Update PC (PC+4 or branch target)
* If ECALL, EBREAK, or HLT: break early
e. TICK DEVICES:
- UART: Check TX FIFO, update status registers
- Disk: Process virtqueue, complete I/O
- Timer: Decrement counter
3. SHUTDOWN:
a. Save disk image if modified
b. Free all resources
c. Print statistics
Memory Access
ALGORITHM: memory_read(addr, size)
1. Find region containing addr:
FOR each region:
IF addr >= region.base AND addr < region.base + region.size:
BREAK
2. IF no region found:
RETURN bus error / exception
3. SWITCH region.type:
CASE RAM:
RETURN read from ram[addr - region.base]
CASE ROM:
RETURN read from rom[addr - region.base]
CASE MMIO:
offset = addr - region.base
RETURN region.device->read(device, offset, size)
5. Implementation Guide
5.1 Development Environment Setup
# Create project directory
mkdir miniemu && cd miniemu
# Create directory structure
mkdir -p src/{cpu,memory,devices} include tests guest
# Install dependencies
# For Linux:
sudo apt-get install build-essential
# For cross-compilation (if emulating RISC-V):
sudo apt-get install gcc-riscv64-linux-gnu
# Create basic Makefile
cat > Makefile << 'EOF'
CC = gcc
CFLAGS = -Wall -Wextra -O2 -g -Iinclude
LDFLAGS =
SRCS = src/main.c src/emu.c src/cpu.c src/memory.c \
src/devices/plic.c src/devices/uart.c \
src/devices/virtio_blk.c src/devices/timer.c
OBJS = $(SRCS:.c=.o)
TARGET = miniemu
all: $(TARGET)
$(TARGET): $(OBJS)
$(CC) $(LDFLAGS) -o $@ $^
clean:
rm -f $(OBJS) $(TARGET)
.PHONY: all clean
EOF
5.2 Project Structure
miniemu/
├── Makefile
├── include/
│ ├── emu.h # Emulator core definitions
│ ├── cpu.h # CPU state and operations
│ ├── memory.h # Memory subsystem
│ └── devices/
│ ├── device.h # Device interface
│ ├── plic.h # Interrupt controller
│ ├── uart.h # Serial port
│ ├── virtio.h # VirtIO common
│ └── timer.h # Timer device
├── src/
│ ├── main.c # Entry point, argument parsing
│ ├── emu.c # Main loop, coordination
│ ├── cpu.c # CPU emulation
│ ├── memory.c # Memory management
│ └── devices/
│ ├── plic.c # PLIC implementation
│ ├── uart.c # 16550 UART
│ ├── virtio_blk.c# VirtIO block device
│ └── timer.c # Timer/RTC
├── tests/
│ ├── test_cpu.c # CPU unit tests
│ └── test_devices.c # Device unit tests
└── guest/
├── bootloader.S # Simple bootloader
└── Makefile # Build guest code
5.3 The Core Question You’re Answering
“How does a system emulator coordinate the execution of guest software with virtual hardware devices, maintaining the illusion of a complete computer system?”
The answer involves:
- Time slicing: CPU runs for a quantum, then devices get a chance to update
- Event-driven I/O: Interrupts notify CPU of device events
- Memory-mapped I/O: CPU memory accesses are routed to devices
- State machines: Each device maintains its own state, updated in sync with CPU
5.4 Concepts You Must Understand First
CPU Emulation:
- How does fetch-decode-execute work?
- How are interrupts delivered to a CPU?
- What triggers a CPU to stop executing (I/O, halt, interrupt)?
Memory Management:
- What is the difference between physical and virtual addresses?
- How does MMIO dispatch work?
- What is memory-mapped I/O vs. port I/O?
Devices:
- How does interrupt-driven I/O work?
- What is the role of device status registers?
- How do FIFOs (queues) buffer data?
Book References:
- “Computer Systems: A Programmer’s Perspective” Chapter 1, 6, 9
- “Operating Systems: Three Easy Pieces” Chapters 35-37 (I/O)
5.5 Questions to Guide Your Design
Main Loop:
- How often should devices be ticked relative to CPU instructions?
- What determines the execution quantum (instructions per iteration)?
- How do you handle blocking I/O without stalling the emulator?
CPU:
- How do you detect when the CPU needs to stop for I/O?
- How do you inject an interrupt into the CPU?
- How do you handle privileged instructions?
Memory:
- How do you efficiently dispatch MMIO accesses?
- What happens on access to unmapped memory?
- How do you handle different access sizes (byte, half, word)?
Devices:
- How do devices signal interrupts?
- When do devices update their state?
- How do you connect device backends (file, socket, PTY)?
5.6 Thinking Exercise
Trace this scenario through your design:
Scenario: Guest OS reads a character from serial port
- Guest executes:
lw a0, 0(t0)where t0 = UART base address - CPU emulator:
- Fetches instruction from memory[PC]
- Decodes: lw (load word)
- Calculates address: t0 + 0 = 0x10000000
- Calls: memory_read(0x10000000, 4)
- Memory subsystem:
- Finds region containing 0x10000000 → UART (MMIO)
- Calls: uart_read(uart, 0, 4)
- UART device:
- Offset 0 = RBR (Receive Buffer Register)
- If RX FIFO has data: return next byte
- If RX FIFO empty: return 0 or block
- Back to CPU:
- Stores result in a0
- Advances PC
Questions to consider:
- What if the UART has no data available?
- How did data get into the UART RX FIFO in the first place?
- What if the guest enables UART RX interrupts?
5.7 Hints in Layers
Hint 1 - Starting Point (Conceptual Direction)
Build incrementally:
- First: CPU that can execute instructions and access memory
- Then: Add UART for output (guest can print)
- Then: Add interrupt controller
- Then: Add timer (guest can schedule)
- Finally: Add disk (guest can boot fully)
Hint 2 - Next Level (More Specific Guidance)
Start with the simplest possible main loop:
void emu_run(struct emulator *emu) {
while (emu->running) {
// Just run CPU forever at first
for (int i = 0; i < 1000; i++) {
cpu_step(&emu->cpu, emu);
}
}
}
Then add complexity:
- Add I/O polling
- Add interrupt checking
- Add device ticking
Hint 3 - Technical Details (Approach/Pseudocode)
/* Simplified RISC-V instruction execution */
void cpu_step(struct cpu_state *cpu, struct emulator *emu) {
/* Fetch */
uint32_t inst = memory_read(emu, cpu->pc, 4);
/* Decode opcode */
uint32_t opcode = inst & 0x7F;
switch (opcode) {
case 0x37: /* LUI */
{
int rd = (inst >> 7) & 0x1F;
uint32_t imm = inst & 0xFFFFF000;
cpu->regs[rd] = imm;
cpu->pc += 4;
}
break;
case 0x03: /* LOAD (LB, LH, LW) */
{
int rd = (inst >> 7) & 0x1F;
int rs1 = (inst >> 15) & 0x1F;
int32_t imm = (int32_t)inst >> 20; /* Sign-extend */
uint32_t addr = cpu->regs[rs1] + imm;
cpu->regs[rd] = memory_read(emu, addr, 4);
cpu->pc += 4;
}
break;
case 0x23: /* STORE (SB, SH, SW) */
{
int rs1 = (inst >> 15) & 0x1F;
int rs2 = (inst >> 20) & 0x1F;
int32_t imm = ((inst >> 7) & 0x1F) | (((inst >> 25) & 0x7F) << 5);
imm = (imm << 20) >> 20; /* Sign-extend */
uint32_t addr = cpu->regs[rs1] + imm;
memory_write(emu, addr, cpu->regs[rs2], 4);
cpu->pc += 4;
}
break;
/* ... more opcodes ... */
default:
printf("Unknown opcode: 0x%02x\n", opcode);
cpu->halted = true;
}
/* x0 is always zero */
cpu->regs[0] = 0;
}
Hint 4 - Tools/Debugging (Verification Methods)
Debug with extensive tracing:
/* Add trace mode */
if (emu->trace) {
printf("[CPU] PC=0x%08x INST=0x%08x ", cpu->pc, inst);
disassemble(inst);
printf("\n");
}
Test with simple guest programs:
- Hello world (test UART output)
- Countdown (test timer interrupt)
- Disk read (test block device)
5.8 The Interview Questions They’ll Ask
- “Explain the main loop of a system emulator.”
- Poll for I/O events
- Process timers
- Check for interrupts
- Execute CPU for a quantum
- Tick devices
- Repeat
- “How does MMIO work in an emulator?”
- CPU issues memory access to device address range
- Memory subsystem recognizes as MMIO (not RAM)
- Dispatches to device read/write handler
- Device returns data or processes write
- “How do you handle device interrupts?”
- Device sets pending bit in interrupt controller
- Main loop checks interrupt controller before CPU runs
- If interrupt pending and CPU accepts, inject interrupt
- CPU jumps to interrupt handler
- “Why use a quantum-based execution model?”
- Allows devices to update state between CPU runs
- Provides opportunities to check for external events
- Balances responsiveness with efficiency
- Simulates time-sharing of real hardware
- “What are the performance considerations?”
- JIT compilation (translate guest to host code)
- Block caching (don’t re-translate same code)
- Lazy device updates (only when needed)
- Efficient MMIO dispatch (avoid linear search)
5.9 Books That Will Help
| Topic | Book | Chapter |
|---|---|---|
| System Overview | CS:APP | Chapter 1 |
| I/O | OS:TEP | Chapters 35-37 |
| Interrupts | OS:TEP | Chapter 33 |
| RISC-V ISA | RISC-V Spec | Vol 1 (Unprivileged) |
| RISC-V Privileged | RISC-V Spec | Vol 2 (Privileged) |
| Device Programming | Linux Device Drivers | Chapters 9-10 |
| x86 Boot | How Computers Really Work | Chapter 10 |
5.10 Implementation Phases
Phase 1: Minimal CPU (Weeks 1-2)
Goal: Execute basic RISC-V instructions
Implement a subset of RV32I:
- Arithmetic: ADD, SUB, AND, OR, XOR, SLT
- Immediates: ADDI, ANDI, ORI, XORI, SLTI, LUI, AUIPC
- Load/Store: LW, SW, LB, SB
- Control: JAL, JALR, BEQ, BNE, BLT, BGE
Validation:
- Run simple test programs
- Verify register state after each instruction
- Test branches and jumps
Phase 2: Memory System (Week 3)
Goal: RAM and MMIO dispatch
struct memory_region regions[] = {
{ .base = 0x80000000, .size = 64*1024*1024, .type = RAM, .data = ram },
{ .base = 0x10000000, .size = 0x1000, .type = MMIO, .device = &uart },
};
Validation:
- RAM read/write works
- MMIO dispatches to device
- Out-of-bounds access returns error
Phase 3: UART (Week 4)
Goal: Serial output from guest
Implement 16550 UART:
- THR (Transmit Holding Register) - write to output
- LSR (Line Status Register) - check if TX ready
- Connect to stdout initially
Validation:
- Guest can print “Hello”
- LSR correctly indicates TX empty
Phase 4: Interrupt Controller (Week 5)
Goal: Devices can interrupt CPU
Implement simplified PLIC:
- Set pending interrupt
- Query highest-priority pending
- Claim and complete interrupts
Validation:
- Device can set pending interrupt
- CPU receives interrupt when enabled
Phase 5: Timer (Week 6)
Goal: Periodic timer interrupts
Implement CLINT timer:
- mtime register (current time)
- mtimecmp register (compare value)
- Timer interrupt when mtime >= mtimecmp
Validation:
- Timer interrupt fires at correct time
- Guest can use timer for delays
Phase 6: Block Device (Weeks 7-8)
Goal: Read/write disk sectors
Implement virtio-blk:
- Virtqueue for requests
- Read sector from file
- Write sector to file
Validation:
- Guest can read boot sector
- Guest can boot from disk image
- Writes persist to disk file
5.11 Key Implementation Decisions
Decision 1: RISC-V vs. x86
RISC-V (Recommended)
- Clean, modern ISA
- Well-documented
- ~50 instructions to implement
- Good toolchain (GCC, QEMU for comparison)
x86
- Complex instruction encoding
- Many instructions needed for real OS
- But: real DOS/FreeDOS available
Recommendation: RISC-V for learning, x86 for practical use
Decision 2: Timing Model
Instruction-Accurate
- Count cycles per instruction
- More accurate but complex
- Needed for cycle-exact emulation
Instruction-Count-Based (Recommended)
- Execute N instructions, then tick devices
- Simple and sufficient for OS emulation
- What QEMU TCG does
Decision 3: Device Backends
Direct (stdout, files)
- Simple to implement
- Good for development
Host Devices (PTY, TAP)
- More realistic
- Allows external tools to interact
- More complex setup
Recommendation: Start with direct, add host devices as extension
6. Testing Strategy
6.1 Unit Tests
/* Test CPU instruction execution */
void test_cpu_add(void) {
struct emulator emu = {0};
struct cpu_state *cpu = &emu.cpu;
/* Program: add x1, x2, x3 */
/* x2 = 5, x3 = 3, result in x1 should be 8 */
cpu->regs[2] = 5;
cpu->regs[3] = 3;
/* Encode: add x1, x2, x3 (R-type) */
uint32_t inst = 0x003100B3; /* add x1, x2, x3 */
store_instruction(&emu, 0x80000000, inst);
cpu->pc = 0x80000000;
cpu_step(cpu, &emu);
assert(cpu->regs[1] == 8);
assert(cpu->pc == 0x80000004);
printf("TEST: ADD - PASS\n");
}
/* Test MMIO dispatch */
void test_mmio_dispatch(void) {
struct emulator emu = {0};
init_emulator(&emu);
/* Write to UART THR */
memory_write(&emu, UART_BASE + 0, 'X', 1);
/* Verify UART received it */
assert(uart_tx_char == 'X');
printf("TEST: MMIO dispatch - PASS\n");
}
6.2 Integration Tests
#!/bin/bash
# test_emulator.sh
# Test 1: Hello World
echo "Test 1: Hello World..."
./miniemu --guest=tests/hello.bin 2>&1 | grep -q "Hello" \
&& echo "PASS" || echo "FAIL"
# Test 2: Timer interrupt
echo "Test 2: Timer interrupt..."
./miniemu --guest=tests/timer.bin 2>&1 | grep -q "Timer fired" \
&& echo "PASS" || echo "FAIL"
# Test 3: Disk read
echo "Test 3: Disk read..."
./miniemu --guest=tests/disk_read.bin --disk=tests/test.img 2>&1 | \
grep -q "Sector 0 read OK" && echo "PASS" || echo "FAIL"
# Test 4: Full boot
echo "Test 4: Boot OS..."
timeout 10 ./miniemu --disk=freedos.img 2>&1 | \
grep -q "FreeDOS" && echo "PASS" || echo "FAIL"
6.3 Guest Test Programs
hello.S - Test UART output:
.globl _start
_start:
li t0, 0x10000000 # UART base
# Print "Hello\n"
li t1, 'H'
sb t1, 0(t0)
li t1, 'e'
sb t1, 0(t0)
li t1, 'l'
sb t1, 0(t0)
li t1, 'l'
sb t1, 0(t0)
li t1, 'o'
sb t1, 0(t0)
li t1, '\n'
sb t1, 0(t0)
# Halt
wfi
7. Common Pitfalls & Debugging
| Problem | Root Cause | Fix | Verification |
|---|---|---|---|
| Guest hangs at start | PC not set correctly | Set PC to correct reset vector | Add trace, check first instructions |
| UART output garbled | Endianness mismatch | Check byte order in load/store | Print bytes as hex before ASCII |
| Interrupts not firing | mstatus.MIE not set | Guest must enable interrupts | Check CSR values |
| Disk read fails | VirtIO queue not initialized | Follow VirtIO spec for init | Add VirtIO trace logging |
| Timer never fires | mtime not advancing | Tick timer in main loop | Print mtime value periodically |
| Guest crashes randomly | Memory access bug | Check bounds, MMIO dispatch | Add memory access tracing |
Debugging Tips
/* Add comprehensive tracing */
void trace_instruction(struct cpu_state *cpu, uint32_t inst) {
printf("[CPU] PC=%08x INST=%08x ", cpu->pc, inst);
/* Print disassembly */
printf("%s\n", disassemble(inst));
/* Print modified registers */
for (int i = 0; i < 32; i++) {
if (cpu->regs[i] != last_regs[i]) {
printf(" x%d: %08x -> %08x\n", i, last_regs[i], cpu->regs[i]);
last_regs[i] = cpu->regs[i];
}
}
}
/* Add memory access tracing */
uint32_t memory_read_traced(struct emulator *emu, uint32_t addr, int size) {
uint32_t val = memory_read_internal(emu, addr, size);
if (emu->trace)
printf("[MEM] READ %08x (%d bytes) = %08x\n", addr, size, val);
return val;
}
8. Extensions & Challenges
8.1 JIT Compilation
Convert interpreter to JIT:
- Translate basic blocks to host machine code
- Cache translated blocks
- Achieve 10-100x speedup
8.2 Multi-core Support
Emulate multiple CPUs:
- Multiple threads for vCPUs
- Inter-processor interrupts
- Memory consistency model
8.3 Graphics Support
Add VGA/framebuffer emulation:
- Map VGA memory at 0xA0000
- Render to SDL window
- Support text and graphics modes
8.4 Network Device
Add virtio-net:
- Connect to TAP device
- Enable guest networking
- NAT or bridged mode
8.5 Snapshot and Restore
Save/restore VM state:
- Serialize CPU registers
- Save memory to file
- Restore device state
9. Real-World Connections
How QEMU Compares
Your emulator is a simplified version of QEMU:
| Component | Your Emulator | QEMU |
|---|---|---|
| CPU | Interpreter | TCG (JIT) or KVM |
| Memory | Simple dispatch | MemoryRegion API with hierarchy |
| Devices | Basic | Hundreds of devices |
| Disk | Simple file | qcow2, thin provisioning, snapshots |
| Network | None/basic | Multiple backends, VLAN support |
| Display | None | SDL, VNC, SPICE |
How This Enables…
OS Development:
- Test kernel without rebooting
- Debug with full visibility
- Faster iteration
Security Research:
- Analyze malware safely
- Snapshot before/after execution
- Inspect all memory and registers
Embedded Development:
- Develop without hardware
- Test hardware abstraction layers
- Automate testing
10. Resources
Primary References
- QEMU Internals Blog - Deep dive into QEMU
- RISC-V Specification - ISA reference
- Virtual I/O Device (VIRTIO) - VirtIO specification
Code References
- QEMU Source - Full emulator
- xv6-riscv - Simple OS to run
- mini-rv32ima - Minimal RISC-V emulator
Tutorials
11. Self-Assessment Checklist
CPU
- Implements core RISC-V instructions (arithmetic, memory, control)
- Handles interrupts correctly (save state, jump to handler)
- Correctly implements privileged instructions (CSR access)
Memory
- RAM allocation and access works
- MMIO dispatch works correctly
- Different access sizes (byte, half, word) work
Devices
- UART TX works (guest can print)
- UART RX works (guest can receive input)
- Timer interrupts fire correctly
- Block device reads work
- Block device writes work
Integration
- Main loop coordinates all components
- Interrupts delivered to CPU correctly
- Guest OS can boot
- Guest OS runs stably
12. Submission / Completion Criteria
Your emulator is complete when you can demonstrate:
- Boot Sequence
- Show emulator loading guest code
- Show CPU starting at correct address
- Show UART initialization messages
- Serial I/O
- Guest prints messages to console
- Guest can receive keyboard input
- Interactive shell works
- Timer
- Timer interrupts fire at regular intervals
- Guest can measure time delays
- Disk Access
- Guest can read files from disk
- Guest can write files to disk
- Full OS Boot
- OS boots to prompt (xv6, FreeDOS, or similar)
- OS can run programs
- System is stable
Bonus Points:
- JIT compilation implemented
- Graphics output
- Network connectivity
- Snapshot/restore
After completing this project, you’ll have built a complete virtual computer in software. You’ll understand how QEMU, VirtualBox, and similar tools work internally. This is the foundation for understanding cloud computing, where every virtual instance is ultimately running on an emulator or hypervisor very similar to what you’ve built.