Project 8: Mini Operating System Kernel

Build a bootable OS kernel that initializes paging, handles interrupts, schedules tasks, and exposes a tiny syscall interface.

Quick Reference

Attribute Value
Difficulty Level 5: Master
Time Estimate 2-3 months
Main Programming Language C + Assembly (Alternative: Rust)
Alternative Programming Languages Rust
Coolness Level Level 5: Pure Magic
Business Potential Level 4: Infrastructure / deep systems
Prerequisites C, assembly, paging, interrupts, toolchain basics
Key Topics boot process, paging, IDT, scheduler, syscalls

1. Learning Objectives

By completing this project, you will:

  1. Explain the boot sequence from reset to kernel entry.
  2. Initialize GDT/IDT and handle interrupts safely.
  3. Set up paging and manage basic physical memory.
  4. Implement a minimal scheduler and task switching.
  5. Provide a syscall mechanism and a tiny shell.
  6. Debug early-boot failures with QEMU and logging.

2. All Theory Needed (Per-Concept Breakdown)

2.1 Boot Process and CPU Modes

Fundamentals

At power-on, the CPU starts in real mode, executes firmware, loads a bootloader, and eventually jumps to your kernel. To run 32-bit or 64-bit code, you must set up protected/long mode with correct descriptors.

Deep Dive into the concept

The bootloader’s job is to load the kernel into memory and transfer control. Multiboot loaders like GRUB standardize this process. The kernel must set up a Global Descriptor Table (GDT) and enable protected mode (and long mode if 64-bit). The transition includes disabling interrupts, setting CR0/CR4, and loading segment selectors. A mistake here results in a triple fault and CPU reset, so careful step-by-step initialization is essential.

How this fits on projects

This is the required foundation for Section 3.1 and Section 5.10 Phase 1.

Definitions & key terms

  • real mode -> 16-bit CPU mode after reset
  • protected mode -> 32-bit mode with paging support
  • GDT -> descriptor table for segment selectors
  • bootloader -> code that loads the kernel

Mental model diagram (ASCII)

Reset -> BIOS/UEFI -> Bootloader -> Kernel entry -> Protected/Long mode

How it works (step-by-step)

  1. CPU starts in real mode at reset vector.
  2. Bootloader loads kernel into memory.
  3. Kernel sets up GDT and enables protected/long mode.
  4. Jump to higher-level C kernel entry.

Minimal concrete example

lgdt [gdt_ptr]
mov eax, cr0
or eax, 1
mov cr0, eax
jmp CODE_SEG:init_pm

Common misconceptions

  • Misconception: you can jump directly to C without mode setup. Correction: C assumes protected/long mode and proper stack.

Check-your-understanding questions

  1. Why is a GDT required in protected mode?
  2. What causes a triple fault?

Check-your-understanding answers

  1. It defines segment descriptors used by the CPU.
  2. An exception while handling an exception, often due to bad IDT/GDT.

Real-world applications

  • Bootloaders, hypervisors, embedded OSes

Where you’ll apply it

References

  • OSDev Wiki: Boot and Protected Mode

Key insights

Boot is a fragile, deterministic sequence–small mistakes crash instantly.

Summary

A kernel starts life in firmware; you must safely transition into your own execution environment.

Homework/Exercises to practice the concept

  1. Build a boot sector that prints text in VGA mode.

Solutions to the homework/exercises

  1. Use BIOS int 0x10 to print characters, then halt.

2.2 Paging and Virtual Memory Initialization

Fundamentals

Paging maps virtual addresses to physical memory and enforces isolation. Your kernel must create page tables and enable paging before accessing higher memory.

Deep Dive into the concept

On x86, paging uses multi-level page tables (e.g., PML4/PDPT/PD/PT in 64-bit). A minimal kernel can identity-map low memory and optionally map the kernel at a higher-half address. You must set CR3 to the page table base and enable paging in CR0. Common early errors include missing page table entries or incorrect permissions, causing immediate faults.

How this fits on projects

Paging is required for Section 3.1 and scheduler safety in Section 4.4.

Definitions & key terms

  • page table -> maps virtual pages to physical frames
  • CR3 -> register holding page table base
  • higher-half -> kernel mapped at high virtual addresses

Mental model diagram (ASCII)

VA 0xC0000000 -> PT -> PA 0x00100000 (kernel)

How it works (step-by-step)

  1. Allocate page tables.
  2. Identity-map low memory for early boot.
  3. Map kernel code/data region.
  4. Load CR3 and enable paging in CR0.

Minimal concrete example

PML4[0] -> PDPT -> PD -> PT -> 0x00000000

Common misconceptions

  • Misconception: paging is optional in protected mode. Correction: most modern kernels require paging for isolation.

Check-your-understanding questions

  1. Why identity-map early pages?
  2. What happens if a page is marked non-present?

Check-your-understanding answers

  1. So code can keep running after paging is enabled.
  2. Access triggers a page fault.

Real-world applications

  • Memory isolation in all modern OSes

Where you’ll apply it

References

  • CS:APP Ch. 9, OSDev Wiki paging sections

Key insights

Paging is the memory backbone of your OS.

Summary

Enable paging carefully or the kernel crashes immediately.

Homework/Exercises to practice the concept

  1. Build identity-mapped paging for the first 4MB.

Solutions to the homework/exercises

  1. Use 4KB pages and fill PT entries for 0x0-0x400000.

2.3 Interrupts, IDT, and Timer Ticks

Fundamentals

Interrupts deliver asynchronous events (timer, keyboard). The IDT maps interrupt vectors to handler addresses. A periodic timer interrupt drives scheduling.

Deep Dive into the concept

The IDT is a table of descriptors pointing to interrupt handlers. The CPU uses it to jump into handler code when an interrupt occurs. The timer interrupt (PIT or APIC) fires at a fixed rate and increments a tick counter. Without a valid IDT, enabling interrupts will cause a triple fault. You must also acknowledge interrupts to the PIC/APIC to avoid lockups.

How this fits on projects

You need timer interrupts for Section 3.1 and the scheduler in Section 4.4.

Definitions & key terms

  • IDT -> interrupt descriptor table
  • PIC/APIC -> interrupt controller hardware
  • IRQ0 -> timer interrupt line

Mental model diagram (ASCII)

IRQ0 -> IDT[32] -> timer_handler -> tick++ -> scheduler

How it works (step-by-step)

  1. Build IDT entries for exception and IRQ handlers.
  2. Remap PIC to avoid conflicts.
  3. Enable interrupts with sti.
  4. On each tick, increment a counter and schedule.

Minimal concrete example

lidt [idt_ptr]
sti

Common misconceptions

  • Misconception: interrupts are safe immediately after boot. Correction: IDT must be valid or you crash.

Check-your-understanding questions

  1. Why remap the PIC?
  2. What happens if you forget to send EOI?

Check-your-understanding answers

  1. To avoid conflicts with CPU exception vectors.
  2. The interrupt may not re-fire, causing stalls.

Real-world applications

  • Scheduling, device drivers, timers

Where you’ll apply it

References

  • OSDev Wiki: IDT and PIC

Key insights

Interrupts are the heartbeat of your OS.

Summary

Without a correct IDT and timer, your kernel cannot multitask.

Homework/Exercises to practice the concept

  1. Write a timer handler that prints a dot every 100 ticks.

Solutions to the homework/exercises

  1. Increment a counter and output via VGA when counter % 100 == 0.

2.4 Syscalls and Minimal Process Model

Fundamentals

A syscall is a controlled entry into the kernel. Your kernel can expose a tiny syscall interface for printing, yielding, or reading input. Processes require a minimal context (registers + stack).

Deep Dive into the concept

A syscall can be implemented via software interrupt (e.g., int 0x80) or a dedicated instruction. The kernel saves the user context, executes the syscall handler, and returns. A minimal process model can be just a kernel thread structure with a stack and instruction pointer. The scheduler selects the next runnable task and switches contexts. Even a two-task system (idle + shell) demonstrates core OS principles.

How this fits on projects

This is the core interaction in Section 3.2 and Section 5.10 Phase 3.

Definitions & key terms

  • syscall -> user-to-kernel call
  • context -> saved registers and stack
  • scheduler -> selects next runnable task

Mental model diagram (ASCII)

user -> int 0x80 -> kernel -> syscall -> return

How it works (step-by-step)

  1. User triggers syscall.
  2. Kernel saves context and dispatches.
  3. Handler executes and sets return value.
  4. Kernel restores context and returns.

Minimal concrete example

sys_write("hi") -> kernel prints to VGA -> return

Common misconceptions

  • Misconception: syscalls require full ELF + user mode immediately. Correction: you can start with kernel tasks and add user mode later.

Check-your-understanding questions

  1. Why are syscalls safer than arbitrary jumps into kernel?
  2. What is the minimum state needed to schedule a task?

Check-your-understanding answers

  1. The kernel validates and controls entry points.
  2. Instruction pointer, stack pointer, and registers.

Real-world applications

  • All OS user/kernel interactions

Where you’ll apply it

References

  • OSDev Wiki: syscalls

Key insights

Syscalls and scheduling are the minimum contract of an OS.

Summary

A kernel without syscalls is just a bootloader.

Homework/Exercises to practice the concept

  1. Implement a syscall to print a string to the screen.

Solutions to the homework/exercises

  1. Map syscall number 1 to a VGA print routine.

3. Project Specification

3.1 What You Will Build

A bootable kernel that runs in QEMU, prints a banner, sets up paging, handles interrupts, runs a basic scheduler, and exposes a tiny shell with a few commands.

3.2 Functional Requirements

  1. Boot via GRUB or multiboot loader.
  2. Initialize GDT and switch to protected/long mode.
  3. Set up paging with identity map and kernel map.
  4. Configure IDT and timer interrupts.
  5. Implement a basic scheduler with at least two tasks.
  6. Provide minimal syscall interface (print, yield).
  7. Implement a shell with commands: help, ps, mem, halt.

3.3 Non-Functional Requirements

  • Reliability: no triple fault on boot.
  • Usability: clear output in QEMU console.
  • Maintainability: clean separation of boot, mmu, irq, sched.

3.4 Example Usage / Output

$ qemu-system-i386 -kernel myos.bin
MyOS v0.1 booting...
> help
commands: ps, mem, echo, halt
> ps
PID 1 shell
PID 2 idle

3.5 Data Formats / Schemas / Protocols

  • Syscall numbers: 1=print, 2=yield, 3=exit
  • Task struct: {pid, state, esp, eip}

3.6 Edge Cases

  • Bad page tables -> immediate page fault.
  • Interrupts enabled before IDT -> triple fault.
  • Scheduler runs with empty run queue.

3.7 Real World Outcome

3.7.1 How to Run (Copy/Paste)

make
qemu-system-i386 -kernel build/myos.bin

3.7.2 Golden Path Demo (Deterministic)

  • Use fixed tick rate and deterministic shell commands.

3.7.3 CLI Transcript (Success + Failure)

$ qemu-system-i386 -kernel build/myos.bin
MyOS v0.1 booting...
> ps
PID 1 shell
PID 2 idle

$ qemu-system-i386 -kernel build/bad.bin
qemu: fatal: triple fault (see log)
exit code: 1

3.7.4 Exit Codes

  • 0 success
  • 1 boot failure (triple fault)

4. Solution Architecture

4.1 High-Level Design

Bootloader -> Kernel init -> MMU -> IDT/IRQ -> Scheduler -> Shell

4.2 Key Components

| Component | Responsibility | Key Decisions | |———–|—————-|—————| | boot | load kernel | multiboot/GRUB | | mmu | paging setup | identity + kernel map | | irq | IDT + PIC | PIT timer | | sched | task switch | round-robin | | shell | CLI | built-in commands |

4.3 Data Structures (No Full Code)

struct task {
    int pid;
    uint32_t esp, eip;
    enum { READY, RUNNING } state;
};

4.4 Algorithm Overview

  1. Boot and set up protected mode.
  2. Initialize paging and IDT.
  3. Start timer interrupt.
  4. Run scheduler loop and shell.

5. Implementation Guide

5.1 Development Environment Setup

sudo apt install gcc make qemu-system-x86

5.2 Project Structure

myos/
|-- boot/
|   |-- boot.S
|-- kernel/
|   |-- main.c
|   |-- mmu.c
|   |-- idt.c
|   |-- sched.c
|   `-- shell.c
`-- Makefile

5.3 The Core Question You’re Answering

“What is the minimum set of mechanisms required for a working OS?”

5.4 Concepts You Must Understand First

  1. Boot process and CPU modes.
  2. Paging initialization.
  3. Interrupts + timer.
  4. Syscalls and scheduling.

5.5 Questions to Guide Your Design

  1. Will you use multiboot or custom bootloader?
  2. Identity map only or higher-half?
  3. What is your minimal process model?

5.6 Thinking Exercise

If your kernel triple-faults after enabling paging, list the top three causes.

5.7 The Interview Questions They’ll Ask

  1. Explain the path from power-on to first user process.
  2. Why is an IDT required before enabling interrupts?

5.8 Hints in Layers

  • Hint 1: print early via VGA text mode.
  • Hint 2: identity-map low memory first.
  • Hint 3: add timer interrupt last.

5.9 Books That Will Help

| Topic | Book | Chapter | |——|——|———| | OS basics | OSTEP | Mechanisms chapters | | CPU/boot | CS:APP | Ch. 3 | | OSDev | OSDev Wiki | Boot + paging |

5.10 Implementation Phases

Phase 1: boot + VGA output. Phase 2: GDT + paging. Phase 3: IDT + timer. Phase 4: scheduler + shell.

6. Testing Strategy

6.1 Test Categories

| Category | Purpose | Examples | |———|———|———-| | Unit | paging | map/unmap correctness | | Integration | boot | QEMU boot test | | Edge | invalid IDT | verify safe failure |

6.2 Critical Test Cases

  1. Kernel prints banner in QEMU.
  2. Timer interrupt increments tick counter.
  3. Shell command ps shows tasks.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

| Pitfall | Symptom | Solution | |———|———|———-| | Bad IDT | triple fault | verify descriptor layout | | Wrong page table | immediate fault | identity map early |

7.2 Debugging Strategies

  • Use QEMU -d int,cpu_reset and serial logging.

8. Extensions & Challenges

  • Add ELF loader for user programs.
  • Add a basic filesystem (ramdisk).

9. Real-World Connections

  • Bootloaders, hypervisors, embedded kernels

10. Resources

  • OSDev Wiki
  • Intel SDM (interrupts + paging)

11. Self-Assessment Checklist

  • I can explain the boot sequence.
  • I can set up paging and IDT.

12. Submission / Completion Criteria

Minimum: boot + banner + halt. Full: paging + interrupts + shell. Excellence: user-mode tasks + syscalls.

13. Determinism Notes

  • Use fixed tick rate and scripted shell commands in demos.