Project 2: VMX Capability Explorer

Build a capability scanner that reports VMX/ SVM support, control bits, and EPT features in human-readable form.

Quick Reference

Attribute Value
Difficulty Level 3: Intermediate
Time Estimate 1 week
Main Programming Language C (Alternatives: Rust)
Alternative Programming Languages Rust
Coolness Level Level 3: Hardware Archaeology
Business Potential Level 2: Kernel Skills
Prerequisites CPUID basics, MSR concepts
Key Topics VMX capability MSRs, control validation

1. Learning Objectives

By completing this project, you will:

  1. Interpret VMX capability MSRs and CPUID flags safely.
  2. Produce a feature matrix usable by a hypervisor.
  3. Explain which VMX controls are mandatory vs optional.
  4. Validate assumptions before attempting VM entry.

2. All Theory Needed (Per-Concept Breakdown)

2.1 VMX Capability Discovery and Control Fields

Fundamentals VMX capability discovery is the process of determining which virtualization features the CPU supports and which control bits are legal. This is done by reading CPUID feature flags and specific MSRs that encode allowed 0 and allowed 1 settings for VMX controls. A hypervisor cannot choose arbitrary control values; it must compute a valid configuration that satisfies the CPU’s constraints. This is why capability discovery is the first step before any VM entry attempt: incorrect control bits cause immediate VM entry failure. Understanding this process is essential for safe, portable hypervisor development.

A practical VMX control set is often built by starting from the allowed-0/allowed-1 masks, then enabling only the minimum needed features. This reduces the surface area for bugs and makes exit behavior more predictable. As you add features (like EPT or unrestricted guest), you must revisit the dependencies and make sure each control bit is still legal. Many entry failures are simply caused by invalid combinations. A solid capability report helps you avoid trial-and-error debugging and lets you reason about why a feature is unavailable on a given CPU.Deep Dive into the concept VMX exposes its capabilities through a set of feature flags and MSRs. CPUID indicates whether VMX is supported at all, but the fine-grained feature set is encoded in control MSRs. Each control MSR is a 64-bit value where the lower 32 bits indicate which control bits must be 0, and the upper 32 bits indicate which control bits are allowed to be 1. The hypervisor must compute its desired control settings and then mask them against these constraints. A common error is to set a control bit that the CPU does not permit, which results in VM entry failure that can be hard to debug without a capability report.

The controls are grouped into categories: pin-based controls, primary processor-based controls, secondary processor-based controls, entry controls, and exit controls. Each category governs a different aspect of virtualization: which instructions trap, which interrupts are delivered to the guest, whether EPT or VPID is enabled, and how state is saved and restored. The capability explorer’s job is to present these constraints clearly so that a later VMM can construct a valid control word. This also supports portability: a control configuration that works on one CPU may fail on another if its capability MSRs differ.

Capability discovery is also critical for features like unrestricted guest mode, which allows real-mode guests without awkward workarounds. Some features require a cascade of control bits, so the explorer must highlight dependencies. For example, enabling EPT may require specific secondary controls, and enabling certain optimizations may require other bits to be set. A clean capability report allows the hypervisor to choose safe defaults and to fall back when features are unavailable.

Another subtlety is the IA32_FEATURE_CONTROL MSR, which can lock VMX use or allow it only in certain modes. Even if CPUID reports VMX support, VMX may be disabled by firmware until the feature control MSR is configured. This is why the explorer must check both CPUID and feature control state and report actionable findings (e.g., “VMX supported but disabled in BIOS”).

Finally, capability discovery is a debugging tool. When VM entry fails, you want to compare your desired controls with the CPU’s allowed controls. A good explorer outputs a clear matrix that shows which features are supported and which are not. This makes VMX development practical rather than guesswork.

A practical VMX control set is often built by starting from the allowed-0/allowed-1 masks, then enabling only the minimum needed features. This reduces the surface area for bugs and makes exit behavior more predictable. As you add features (like EPT or unrestricted guest), you must revisit the dependencies and make sure each control bit is still legal. Many entry failures are simply caused by invalid combinations. A solid capability report helps you avoid trial-and-error debugging and lets you reason about why a feature is unavailable on a given CPU.

A practical VMX control set is often built by starting from the allowed-0/allowed-1 masks, then enabling only the minimum needed features. This reduces the surface area for bugs and makes exit behavior more predictable. As you add features (like EPT or unrestricted guest), you must revisit the dependencies and make sure each control bit is still legal. Many entry failures are simply caused by invalid combinations. A solid capability report helps you avoid trial-and-error debugging and lets you reason about why a feature is unavailable on a given CPU.How this fit on projects This concept is the entire purpose of the project: it produces the capability map you will use in later hypervisor builds.

Definitions & key terms

  • CPUID: CPU instruction that reports feature flags.
  • MSR: Model Specific Register, used for CPU control and capability reporting.
  • Allowed-0/Allowed-1: Bit masks defining legal control values.
  • VMX controls: Sets of bits that configure virtualization behavior.

Mental model diagram

CPUID -> VMX supported?
MSRs  -> allowed controls
Result -> valid control set

How it works (step-by-step, with invariants and failure modes)

  1. Read CPUID feature flags for VMX.
  2. Check feature control MSR for VMX enablement.
  3. Read VMX capability MSRs.
  4. Compute legal control masks.
  5. Generate a feature report.

Invariants: do not attempt VM entry without validating control legality. Failure modes include invalid control bits and BIOS-disabled VMX.

Minimal concrete example

READ: CPUID.VMX = 1
READ: FEATURE_CONTROL.LOCK = 1, VMXON allowed
READ: PINBASED_CTLS
OUTPUT: "NMI exiting supported, preemption timer unsupported"

Common misconceptions

  • If CPUID says VMX, VMX is always usable.
  • You can set any control bit you want.

Check-your-understanding questions

  1. Why do VMX controls have allowed-0 and allowed-1 masks?
  2. What happens if you set a disallowed control bit?
  3. Why must you check the feature control MSR?

Check-your-understanding answers

  1. They define the legal control settings for a given CPU.
  2. VM entry fails with an invalid control error.
  3. Firmware can disable VMX even if CPUID advertises it.

Real-world applications

  • Hypervisor feature negotiation
  • Debugging VM entry failures

Where you’ll apply it

  • Apply in §3.2 (functional requirements) and §4.1 (architecture)
  • Also used in: P01-toy-kvm-hypervisor, P10-mini-cloud-control-plane

References

  • Intel SDM Vol. 3C (VMX capability MSRs)
  • AMD64 APM (SVM feature discovery)

Key insights VMX is only usable when control bits are validated against CPU capability masks.

Summary You now understand how to discover and validate VMX capabilities before VM entry.

Homework/Exercises to practice the concept

  1. Explain the difference between “supported” and “allowed” controls.
  2. Draw a simple capability matrix for three VMX features.

Solutions to the homework/exercises

  1. Supported features may still be disallowed in specific control contexts.
  2. Matrix shows CPUID, allowed-0/allowed-1, and dependency notes.

3. Project Specification

3.1 What You Will Build

A capability explorer that outputs a readable VMX feature report for the current CPU.

3.2 Functional Requirements

  1. Detect VMX support via CPUID.
  2. Read feature control MSR and report enablement.
  3. Parse key VMX control MSRs.
  4. Output a feature matrix.

3.3 Non-Functional Requirements

  • Performance: Runs in under 1 second.
  • Reliability: Safe error handling without system instability.
  • Usability: Clear, actionable output.

3.4 Example Usage / Output

$ sudo ./vmx-explorer
[VMX] VMX supported: yes
[VMX] Feature control: locked, VMXON allowed
[VMX] EPT: supported
[VMX] Unrestricted guest: supported

3.5 Data Formats / Schemas / Protocols

  • Output should list each control category and its supported features.

3.6 Edge Cases

  • VMX supported but disabled in BIOS
  • Running on AMD host (report SVM instead)

3.7 Real World Outcome

A report that a hypervisor developer can use to construct valid control settings.

3.7.1 How to Run (Copy/Paste)

  • Run as root on a VMX-capable CPU
  • Ensure kernel allows MSR reads

3.7.2 Golden Path Demo (Deterministic)

  • Report VMX support and list key features

3.7.3 If CLI: exact terminal transcript

$ sudo ./vmx-explorer
[VMX] VMCS revision ID: 0x12
[VMX] Pin-based controls: NMI exiting allowed
[VMX] Secondary controls: EPT supported

4. Solution Architecture

4.1 High-Level Design

CPUID/MSR reader -> Control parser -> Human-readable report

4.2 Key Components

| Component | Responsibility | Key Decisions | |———–|—————-|—————| | CPUID reader | Detect VMX | Validate vendor | | MSR reader | Fetch controls | Safe access | | Reporter | Print matrix | Clear wording |

4.3 Data Structures (No Full Code)

  • Control category -> list of features -> allowed states

4.4 Algorithm Overview

Key Algorithm: control parsing

  1. Read MSR
  2. Split allowed-0/allowed-1 masks
  3. Classify each control bit

Complexity Analysis

  • Time: O(number of control bits)
  • Space: O(number of features)

5. Implementation Guide

5.1 Development Environment Setup

# Ensure MSR access and root privileges

5.2 Project Structure

project-root/
├── src/
│   ├── main.c
│   └── vmx_caps.c
├── tests/
│   └── sample_output.txt
└── README.md

5.3 The Core Question You’re Answering

“What virtualization features does this CPU actually support?”

5.4 Concepts You Must Understand First

  1. CPUID feature flags
  2. VMX capability MSRs
  3. Control dependencies

5.5 Questions to Guide Your Design

  1. Which controls are essential for your later VMM?
  2. How will you present dependencies in output?

5.6 Thinking Exercise

Sketch a report format that is readable and actionable by a VMM developer.

5.7 The Interview Questions They’ll Ask

  1. “Why do VMX controls have allowed-0/allowed-1 masks?”
  2. “What is IA32_FEATURE_CONTROL and why does it matter?”
  3. “How do you determine EPT support?”

5.8 Hints in Layers

Hint 1: Start with CPUID feature bits. Hint 2: Parse one control category before adding more. Hint 3: Pseudocode

READ MSR -> split masks -> print allowed features

Hint 4: If output is empty, verify MSR access permissions.

5.9 Books That Will Help

| Topic | Book | Chapter | |——-|——|———| | VMX | Intel SDM Vol. 3C | Ch. 23-33 | | Low-level CPU | “CSAPP” | Ch. 3 |

5.10 Implementation Phases

  • Phase 1: CPUID detection
  • Phase 2: MSR parsing
  • Phase 3: Output formatting

5.11 Key Implementation Decisions

| Decision | Options | Recommendation | Rationale | |———-|———|—————-|———–| | Output format | Text vs JSON | Text first | Easier to read |

6. Testing Strategy

6.1 Test Categories

| Category | Purpose | Examples | |———-|———|———-| | Unit Tests | Parser correctness | Mask interpretation | | Integration Tests | End-to-end | CPUID + MSR output |

6.2 Critical Test Cases

  1. VMX supported and enabled.
  2. VMX supported but disabled in BIOS.

6.3 Test Data

Sample MSR values and expected feature flags

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

| Pitfall | Symptom | Solution | |———|———|———-| | Missing privilege | Read fails | Run as root | | Misread masks | Wrong output | Verify bit order |

7.2 Debugging Strategies

  • Cross-check with known-good output from a VM or reference CPU.

7.3 Performance Traps

  • None significant; this is a lightweight tool.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add AMD SVM detection.

8.2 Intermediate Extensions

  • Emit JSON report for automation.

8.3 Advanced Extensions

  • Compare results across multiple hosts.

9. Real-World Connections

9.1 Industry Applications

  • Hypervisor feature detection and fallback
  • KVM, QEMU, libvirt

9.3 Interview Relevance

  • VMX controls, CPUID, MSRs

10. Resources

10.1 Essential Reading

  • Intel SDM Vol. 3C

10.2 Video Resources

  • Talks on VMX internals