Project 25: Build a Debugger Using ptrace
Build a tiny debugger from scratch: launch and control child processes, set software breakpoints, single-step through instructions, and inspect registers and memory. You will understand exactly how GDB works at the system call level.
Quick Reference
| Attribute | Value |
|---|---|
| Language | C (alt: C++, Rust) |
| Difficulty | Master |
| Time | 1 month+ |
| Chapters | 3, 7, 8 |
| Coolness | ***** Mind-Blowing |
| Portfolio Value | Resume Gold |
Learning Objectives
By completing this project, you will:
- Master the ptrace system call: Understand how one process can control another’s execution at the instruction level
- Implement software breakpoints: Learn how INT3 (0xCC) works and how debuggers pause execution at specific addresses
- Read and write CPU registers: Use PTRACE_GETREGS and PTRACE_SETREGS to inspect and modify the register file
- Inspect and modify process memory: Use PTRACE_PEEKDATA and PTRACE_POKEDATA to read/write arbitrary memory
- Understand process states: Master the stopped, running, and traced states and how they interact
- Handle signals in traced processes: Learn how SIGTRAP, SIGSTOP, and other signals work during debugging
- Implement single-stepping: Execute exactly one instruction at a time using PTRACE_SINGLESTEP
- Parse ELF symbol tables: Look up function addresses by name (e.g., “break main”)
- Build a debugger state machine: Manage breakpoints, continue/step logic, and user commands
- Connect machine code to debugging: Apply your CS:APP Chapter 3 knowledge to real debugging scenarios
Deep Theoretical Foundation
What is ptrace?
The ptrace() system call is the foundation of debugging on Unix-like systems. It allows one process (the tracer or debugger) to observe and control the execution of another process (the tracee or debuggee). Every feature of GDB, LLDB, and strace is built on top of ptrace.
+---------------------------------------------------------------------------+
| THE PTRACE ARCHITECTURE |
+---------------------------------------------------------------------------+
| |
| TRACER (debugger) TRACEE (program being debugged) |
| +-------------------+ +-------------------------+ |
| | | | | |
| | mydb (your | ptrace() | target program | |
| | debugger) | <-----------> | (hello, ls, etc.) | |
| | | syscall | | |
| +-------------------+ +-------------------------+ |
| | | |
| | | |
| v v |
| +-----------------------------------------------+ |
| | KERNEL | |
| | - Stops tracee on breakpoints | |
| | - Delivers signals to tracer | |
| | - Provides register/memory access | |
| | - Manages tracee execution state | |
| +-----------------------------------------------+ |
| |
| KEY INSIGHT: The kernel mediates ALL interaction between tracer |
| and tracee. The debugger never directly touches the target process. |
| |
+---------------------------------------------------------------------------+
ptrace System Call Overview
#include <sys/ptrace.h>
long ptrace(enum __ptrace_request request, pid_t pid,
void *addr, void *data);
Key ptrace requests you will use:
| Request | Purpose |
|---|---|
PTRACE_TRACEME |
Child requests to be traced by parent |
PTRACE_ATTACH |
Attach to an already-running process |
PTRACE_DETACH |
Detach from tracee, let it continue |
PTRACE_PEEKDATA |
Read a word from tracee memory |
PTRACE_POKEDATA |
Write a word to tracee memory |
PTRACE_GETREGS |
Read all general-purpose registers |
PTRACE_SETREGS |
Write all general-purpose registers |
PTRACE_CONT |
Continue tracee execution |
PTRACE_SINGLESTEP |
Execute exactly one instruction |
Process States During Debugging
Understanding process states is crucial for building a debugger:
+---------------------------------------------------------------------------+
| TRACEE PROCESS STATES |
+---------------------------------------------------------------------------+
| |
| +---------------+ |
| | RUNNING | |
| | (executing) | |
| +-------+-------+ |
| | |
| +-------------------+-------------------+ |
| | | | |
| v v v |
| +--------+------+ +-------+-------+ +-------+-------+ |
| | HIT BREAKPOINT| | SIGNAL | | SYSCALL | |
| | (SIGTRAP) | | (any signal) | | (if tracing) | |
| +--------+------+ +-------+-------+ +-------+-------+ |
| | | | |
| +-------------------+-------------------+ |
| | |
| v |
| +-------+-------+ |
| | STOPPED | |
| | (waiting) | |
| +-------+-------+ |
| | |
| +-------------------+-------------------+ |
| | | | |
| v v v |
| +--------+------+ +-------+-------+ +-------+-------+ |
| | PTRACE_CONT | | PTRACE_ | | PTRACE_DETACH | |
| | (continue) | | SINGLESTEP | | (release) | |
| +--------+------+ +-------+-------+ +-------+-------+ |
| | | | |
| v | v |
| +----+----+ | +------+------+ |
| | RUNNING | | | RUNNING | |
| | (free) |<-------------+ | (untraced) | |
| +---------+ +-------------+ |
| |
| STATE TRANSITIONS: |
| - Running -> Stopped: breakpoint, signal, or syscall |
| - Stopped -> Running: PTRACE_CONT or PTRACE_SINGLESTEP |
| - Stopped -> Running (untraced): PTRACE_DETACH |
| |
+---------------------------------------------------------------------------+
How Software Breakpoints Work
This is the heart of debugging. A software breakpoint works by:
- Saving the original byte at the breakpoint address
- Replacing it with
0xCC(the INT3 instruction) - Waiting for the tracee to hit the breakpoint (SIGTRAP)
- Restoring the original byte so execution can continue
+---------------------------------------------------------------------------+
| SOFTWARE BREAKPOINT MECHANISM |
+---------------------------------------------------------------------------+
| |
| ORIGINAL CODE at address 0x401000: |
| +--------+--------+--------+--------+--------+--------+ |
| | 0x48 | 0x89 | 0xe5 | 0x55 | 0x48 | 0x83 | |
| +--------+--------+--------+--------+--------+--------+ |
| mov rbp,rsp push rbp sub rsp,... |
| |
| STEP 1: Save original byte (0x48) and insert breakpoint |
| |
| CODE WITH BREAKPOINT at 0x401000: |
| +--------+--------+--------+--------+--------+--------+ |
| | 0xCC | 0x89 | 0xe5 | 0x55 | 0x48 | 0x83 | |
| +--------+--------+--------+--------+--------+--------+ |
| INT3 (rest of mov corrupted, but OK - we won't execute it) |
| |
| STEP 2: Tracee executes, hits INT3, kernel sends SIGTRAP |
| |
| CPU STATE WHEN STOPPED: |
| - RIP = 0x401001 (points PAST the INT3!) |
| - Tracee is in STOPPED state |
| - waitpid() returns in tracer |
| |
| STEP 3: Tracer handles breakpoint: |
| a) Restore original byte (0x48) |
| b) Decrement RIP by 1 (back to 0x401000) |
| c) Optionally single-step and re-insert breakpoint |
| |
| RESTORED CODE: |
| +--------+--------+--------+--------+--------+--------+ |
| | 0x48 | 0x89 | 0xe5 | 0x55 | 0x48 | 0x83 | |
| +--------+--------+--------+--------+--------+--------+ |
| ^ |
| RIP points here again |
| |
+---------------------------------------------------------------------------+
CRITICAL INSIGHT: INT3 is a ONE-BYTE instruction (0xCC). This means:
- It can replace ANY instruction's first byte
- We only corrupt one byte, which we save and restore
- RIP advances by 1 after INT3, so we must decrement it
The INT3 Instruction in Detail
+---------------------------------------------------------------------------+
| INT3 (0xCC) EXPLAINED |
+---------------------------------------------------------------------------+
| |
| WHAT IS INT3? |
| - A special single-byte interrupt instruction |
| - Raises interrupt vector 3 (breakpoint exception) |
| - Specifically designed for debuggers |
| |
| WHY INT3 AND NOT INT 0x03? |
| +-------------------+-------------------+ |
| | INT3 (0xCC) | INT 0x03 | |
| | 1 byte | 2 bytes (CD 03) | |
| +-------------------+-------------------+ |
| |
| One byte matters because: |
| - Can replace first byte of ANY x86 instruction |
| - Only need to save/restore one byte |
| - Works even if next instruction starts at current_addr + 1 |
| |
| WHAT HAPPENS WHEN INT3 EXECUTES: |
| 1. CPU pushes flags, CS, and RIP (return address) to stack |
| 2. CPU looks up interrupt vector 3 in IDT |
| 3. Kernel's breakpoint handler runs |
| 4. Kernel sees process is traced -> sends SIGTRAP to tracee |
| 5. Tracee stops, tracer's waitpid() returns |
| |
| IMPORTANT: RIP after INT3 points to the NEXT instruction |
| (the byte after 0xCC), so debugger must adjust RIP -= 1 |
| |
+---------------------------------------------------------------------------+
Hardware Breakpoints vs Software Breakpoints
+---------------------------------------------------------------------------+
| SOFTWARE vs HARDWARE BREAKPOINTS |
+---------------------------------------------------------------------------+
| |
| SOFTWARE BREAKPOINTS (INT3): |
| +------------------------------------------------------------------+ |
| | Pros: | |
| | - Unlimited quantity (just patch memory) | |
| | - Works on any executable address | |
| | | |
| | Cons: | |
| | - Modifies process memory (can cause issues) | |
| | - Doesn't work on read-only memory without remapping | |
| | - Can't watch data accesses (only instruction fetch) | |
| +------------------------------------------------------------------+ |
| |
| HARDWARE BREAKPOINTS (Debug Registers DR0-DR3): |
| +------------------------------------------------------------------+ |
| | Pros: | |
| | - No memory modification | |
| | - Can break on data READ/WRITE (watchpoints!) | |
| | - Works on ROM/read-only memory | |
| | | |
| | Cons: | |
| | - Limited to 4 breakpoints (DR0, DR1, DR2, DR3) | |
| | - Platform-specific (x86 only in this form) | |
| +------------------------------------------------------------------+ |
| |
| x86 DEBUG REGISTERS: |
| DR0-DR3: Breakpoint addresses |
| DR6: Debug status (which breakpoint hit) |
| DR7: Debug control (enable/disable, conditions) |
| |
| DR7 Control Bits per Breakpoint: |
| - L0-L3: Local enable (this task only) |
| - G0-G3: Global enable (all tasks) |
| - R/W0-R/W3: Break condition (exec/write/read-write) |
| - LEN0-LEN3: Data size (1/2/4/8 bytes) |
| |
+---------------------------------------------------------------------------+
Register Access with ptrace
+---------------------------------------------------------------------------+
| x86-64 REGISTER FILE LAYOUT |
+---------------------------------------------------------------------------+
| |
| struct user_regs_struct { // From <sys/user.h> |
| |
| // General-purpose registers |
| unsigned long r15, r14, r13, r12; // Callee-saved |
| unsigned long rbp; // Base pointer (callee-saved)|
| unsigned long rbx; // Callee-saved |
| unsigned long r11, r10; // Caller-saved |
| unsigned long r9, r8; // Args 5, 6 |
| unsigned long rax; // Return value |
| unsigned long rcx, rdx; // Args 4, 3 |
| unsigned long rsi, rdi; // Args 2, 1 |
| |
| // Special registers |
| unsigned long orig_rax; // Syscall number |
| unsigned long rip; // Instruction pointer |
| unsigned long cs; // Code segment |
| unsigned long eflags; // Flags register |
| unsigned long rsp; // Stack pointer |
| unsigned long ss; // Stack segment |
| unsigned long fs_base, gs_base; // Segment bases |
| unsigned long ds, es, fs, gs; // Segment selectors |
| }; |
| |
| USAGE: |
| struct user_regs_struct regs; |
| ptrace(PTRACE_GETREGS, pid, NULL, ®s); |
| printf("RIP = 0x%lx\n", regs.rip); |
| printf("RSP = 0x%lx\n", regs.rsp); |
| |
| MODIFICATION: |
| regs.rip -= 1; // Back up over INT3 |
| ptrace(PTRACE_SETREGS, pid, NULL, ®s); |
| |
+---------------------------------------------------------------------------+
+---------------------------------------------------------------------------+
| EFLAGS REGISTER BREAKDOWN |
+---------------------------------------------------------------------------+
| |
| Bit Name Meaning |
| --- ---- ------- |
| 0 CF Carry flag |
| 2 PF Parity flag |
| 4 AF Auxiliary carry flag |
| 6 ZF Zero flag |
| 7 SF Sign flag |
| 8 TF Trap flag (single-step mode!) |
| 9 IF Interrupt enable flag |
| 10 DF Direction flag |
| 11 OF Overflow flag |
| |
| THE TRAP FLAG (TF): |
| - When TF=1, CPU generates debug exception after EVERY instruction |
| - This is how PTRACE_SINGLESTEP works internally |
| - Kernel sets TF=1, resumes tracee, next instruction triggers trap |
| |
+---------------------------------------------------------------------------+
Memory Inspection with ptrace
+---------------------------------------------------------------------------+
| READING/WRITING TRACEE MEMORY |
+---------------------------------------------------------------------------+
| |
| PTRACE_PEEKDATA: |
| long data = ptrace(PTRACE_PEEKDATA, pid, addr, NULL); |
| // Returns 8 bytes (word) at 'addr' in tracee's address space |
| // Returns -1 and sets errno on error (but -1 could be valid data!) |
| |
| PTRACE_POKEDATA: |
| ptrace(PTRACE_POKEDATA, pid, addr, data); |
| // Writes 8 bytes (word) to 'addr' in tracee's address space |
| |
| CHALLENGE: Reading arbitrary-length data |
| |
| void read_memory(pid_t pid, void *addr, void *buf, size_t len) { |
| size_t bytes_read = 0; |
| long *ptr = (long *)buf; |
| unsigned long target = (unsigned long)addr; |
| |
| while (bytes_read < len) { |
| long word = ptrace(PTRACE_PEEKDATA, pid, target, NULL); |
| if (word == -1 && errno != 0) { |
| perror("PEEKDATA failed"); |
| break; |
| } |
| *ptr++ = word; |
| bytes_read += sizeof(long); |
| target += sizeof(long); |
| } |
| } |
| |
| ALTERNATIVE: /proc/[pid]/mem |
| - Open /proc/[pid]/mem as file |
| - lseek() to target address |
| - read()/write() arbitrary lengths |
| - More efficient for large reads |
| |
+---------------------------------------------------------------------------+
Single-Stepping Execution
+---------------------------------------------------------------------------+
| SINGLE-STEP MECHANISM |
+---------------------------------------------------------------------------+
| |
| PTRACE_SINGLESTEP: |
| 1. Kernel sets Trap Flag (TF) in tracee's EFLAGS |
| 2. Tracee resumes execution |
| 3. CPU executes ONE instruction |
| 4. CPU generates debug exception (because TF=1) |
| 5. Kernel clears TF, delivers SIGTRAP to tracee |
| 6. Tracee stops, tracer's waitpid() returns |
| |
| TIMELINE: |
| |
| TRACER TRACEE |
| ------ ------ |
| ptrace(SINGLESTEP, pid) |
| | |
| +-----------------------> (kernel sets TF=1, resumes) |
| | |
| waitpid(pid, ...) | executes one instruction |
| | | |
| | <----------------------- (debug exception, SIGTRAP) |
| | | |
| (returns with (stopped) |
| WIFSTOPPED=true, |
| WSTOPSIG=SIGTRAP) |
| | |
| Read new RIP |
| with PTRACE_GETREGS |
| |
| DISTINGUISHING SINGLE-STEP FROM BREAKPOINT: |
| Both deliver SIGTRAP! How do you tell them apart? |
| |
| Option 1: Track state - know if you just issued SINGLESTEP |
| Option 2: Check if RIP matches a breakpoint address |
| Option 3: Use PTRACE_GETSIGINFO to check si_code |
| - SI_KERNEL: breakpoint |
| - TRAP_TRACE: single-step |
| |
+---------------------------------------------------------------------------+
The Debugger-Tracee Interaction Flow
+---------------------------------------------------------------------------+
| DEBUGGER MAIN LOOP |
+---------------------------------------------------------------------------+
| |
| STARTUP: |
| |
| debugger |
| | |
| | fork() |
| | |
| +------------+ |
| | | |
| v v |
| parent child |
| (tracer) (tracee) |
| | | |
| | | ptrace(TRACEME) |
| | | execve(program) |
| | | | |
| | v v |
| | [stopped on exec] |
| | | |
| | waitpid() | |
| | <----------+ |
| | |
| v |
| MAIN LOOP: |
| |
| +---> Read command from user |
| | | |
| | v |
| | +-----------+ |
| | | "break" | --> enable_breakpoint(addr) |
| | +-----------+ PTRACE_PEEKDATA (save byte) |
| | | PTRACE_POKEDATA (write 0xCC) |
| | v |
| | +-----------+ |
| | | "run" | --> ptrace(PTRACE_CONT, pid) |
| | +-----------+ waitpid(pid, &status) |
| | | if SIGTRAP: handle_breakpoint() |
| | v |
| | +-----------+ |
| | | "step" | --> ptrace(PTRACE_SINGLESTEP, pid) |
| | +-----------+ waitpid(pid, &status) |
| | | print current RIP |
| | v |
| | +-----------+ |
| | | "regs" | --> ptrace(PTRACE_GETREGS, pid, ..., ®s) |
| | +-----------+ print all registers |
| | | |
| | v |
| | +-----------+ |
| | | "x" | --> ptrace(PTRACE_PEEKDATA, pid, addr) |
| | +-----------+ print memory contents |
| | | |
| +----<----+ |
| |
+---------------------------------------------------------------------------+
ELF Symbol Table Basics
To support break main instead of break 0x401000, you need to parse ELF symbols:
+---------------------------------------------------------------------------+
| ELF SYMBOL TABLE STRUCTURE |
+---------------------------------------------------------------------------+
| |
| ELF FILE LAYOUT: |
| +-------------------+ |
| | ELF Header | <-- Contains entry point, section header offset |
| +-------------------+ |
| | Program Headers | <-- Describe segments for loading |
| +-------------------+ |
| | .text section | <-- Executable code |
| +-------------------+ |
| | .data section | <-- Initialized data |
| +-------------------+ |
| | .symtab section | <-- Symbol table (for -g builds) |
| +-------------------+ |
| | .strtab section | <-- String table (symbol names) |
| +-------------------+ |
| | Section Headers | <-- Describe all sections |
| +-------------------+ |
| |
| SYMBOL TABLE ENTRY (Elf64_Sym): |
| typedef struct { |
| Elf64_Word st_name; // Offset into string table |
| unsigned char st_info; // Type (FUNC, OBJECT, etc.) and binding |
| unsigned char st_other; // Visibility |
| Elf64_Half st_shndx; // Section index |
| Elf64_Addr st_value; // Symbol value (address for functions) |
| Elf64_Xword st_size; // Symbol size |
| } Elf64_Sym; |
| |
| FINDING A FUNCTION ADDRESS: |
| 1. Open executable file |
| 2. Parse ELF header to find section headers |
| 3. Find .symtab and .strtab sections |
| 4. Iterate through symbols |
| 5. For each symbol, compare name (from strtab) with target |
| 6. If match and type is FUNC, return st_value |
| |
| ALTERNATIVE: Use libelf or libbfd for easier parsing |
| |
+---------------------------------------------------------------------------+
DWARF Debug Information (Advanced)
For source-level debugging (line numbers, variable names), you need DWARF:
+---------------------------------------------------------------------------+
| DWARF DEBUG INFORMATION |
+---------------------------------------------------------------------------+
| |
| WHAT DWARF PROVIDES: |
| +------------------------------------------------------------------+ |
| | Information | Use Case | |
| +------------------------+------------------------------------------+ |
| | Line number table | "break main.c:42" | |
| | Variable locations | "print x" shows local variable | |
| | Type information | "print *ptr" knows struct layout | |
| | Call frame info | Stack unwinding for backtraces | |
| +------------------------------------------------------------------+ |
| |
| KEY DWARF SECTIONS: |
| .debug_info - DIE (Debug Info Entry) tree |
| .debug_line - Line number to address mapping |
| .debug_frame - Call frame information |
| .debug_abbrev - Abbreviation tables |
| .debug_str - String table for debug info |
| |
| FOR THIS PROJECT: |
| DWARF is optional. You can implement basic debugging without it. |
| For line-number debugging, you can shell out to addr2line: |
| |
| $ addr2line -e ./program 0x401000 |
| /path/to/main.c:10 |
| |
| Or use libdwarf/libelf for full DWARF parsing. |
| |
+---------------------------------------------------------------------------+
Project Specification
What You Will Build
A command-line debugger called mydb that can:
- Launch a program under debugging control
- Set and manage breakpoints
- Single-step through instructions
- Continue execution until next breakpoint
- Inspect CPU registers
- Examine memory contents
Command-Line Interface
$ ./mydb ./target_program [args...]
mydb> break 0x401000
Breakpoint 1 at 0x401000
mydb> break main
Breakpoint 2 at 0x401126 (main)
mydb> run
Starting program: ./target_program
Breakpoint 1 hit at 0x401000
mydb> regs
RAX: 0x0000000000000000 RBX: 0x0000000000000000
RCX: 0x0000000000000000 RDX: 0x00007fffffffde88
RSI: 0x00007fffffffde78 RDI: 0x0000000000000001
RBP: 0x0000000000000000 RSP: 0x00007fffffffdd80
R8: 0x0000000000000000 R9: 0x0000000000000000
R10: 0x0000000000000000 R11: 0x0000000000000000
R12: 0x0000000000000000 R13: 0x0000000000000000
R14: 0x0000000000000000 R15: 0x0000000000000000
RIP: 0x0000000000401000 EFLAGS: 0x0000000000000246
mydb> step
0x401002: mov rbp, rsp
mydb> x/4x 0x7fffffffdd80
0x7fffffffdd80: 0x00000001 0x00000000 0x00007fff 0xffffde78
mydb> continue
Continuing...
Breakpoint 2 hit at 0x401126
mydb> quit
Required Commands
| Command | Description |
|---|---|
break <addr|symbol> |
Set breakpoint at address or function name |
delete <n> |
Delete breakpoint number n |
run |
Start/restart program execution |
continue (or c) |
Continue execution until next breakpoint |
step (or s) |
Execute one instruction (single-step) |
regs |
Display all registers |
x/<n><f> <addr> |
Examine n units of memory at addr (f=x,d,s) |
quit (or q) |
Exit debugger |
Functional Requirements
+---------------------------------------------------------------------------+
| FUNCTIONAL REQUIREMENTS |
+---------------------------------------------------------------------------+
| |
| PROCESS CONTROL: |
| - Launch target program as child process |
| - Use ptrace(TRACEME) in child before exec |
| - Parent waits for initial stop after exec |
| - Support clean detach on quit |
| |
| BREAKPOINTS: |
| - Set breakpoints before or during execution |
| - Save original byte when setting breakpoint |
| - Restore byte when breakpoint hit |
| - Handle multiple breakpoints |
| - Support breakpoint by address (0x...) or symbol name |
| |
| EXECUTION CONTROL: |
| - Continue: PTRACE_CONT until breakpoint or exit |
| - Step: PTRACE_SINGLESTEP, wait, report new RIP |
| - Correctly handle SIGTRAP from breakpoints vs single-step |
| |
| INSPECTION: |
| - Read all general-purpose registers with PTRACE_GETREGS |
| - Read memory with PTRACE_PEEKDATA |
| - Format output clearly (hex addresses, register names) |
| |
| ERROR HANDLING: |
| - Handle ptrace failures gracefully |
| - Detect when tracee exits |
| - Handle signals other than SIGTRAP |
| |
+---------------------------------------------------------------------------+
Non-Functional Requirements
- Correctness: Breakpoints fire exactly at the right address
- Reliability: No crashes when tracee exits or receives signals
- Usability: Clear error messages, intuitive commands
- Code Quality: Well-structured, documented code
Real World Outcome
When you complete this project, here’s exactly what you’ll see:
Launching and Basic Operation
$ ./mydb ./hello
mydb - A tiny debugger
Loaded executable: ./hello
Child process: 12345
Tracee stopped (initial stop after exec)
mydb>
Setting Breakpoints
mydb> break main
Breakpoint 1 at 0x401126 (main)
mydb> break 0x40112a
Breakpoint 2 at 0x40112a
mydb> info break
Num Address Symbol Enabled
1 0x0000000000401126 main yes
2 0x000000000040112a <unknown> yes
mydb>
Running and Hitting Breakpoints
mydb> run
Starting program...
Breakpoint 1 hit at 0x401126
Stopped at main (0x401126)
mydb>
Inspecting Registers
mydb> regs
General Purpose Registers:
RAX: 0x0000000000401126 RBX: 0x0000000000000000
RCX: 0x00007ffff7fa4718 RDX: 0x00007fffffffdf08
RSI: 0x00007fffffffdef8 RDI: 0x0000000000000001
RBP: 0x0000000000000000 RSP: 0x00007fffffffdde8
R8: 0x0000000000000000 R9: 0x00007ffff7fcfb50
R10: 0x00007ffff7fcb878 R11: 0x00000000000000c0
R12: 0x0000000000401040 R13: 0x00007fffffffdef8
R14: 0x0000000000000000 R15: 0x0000000000000000
Instruction Pointer:
RIP: 0x0000000000401126
Flags:
EFLAGS: 0x0000000000000246 [PF ZF IF]
mydb>
Single-Stepping
mydb> step
0x401127: push rbp
mydb> step
0x401128: mov rbp, rsp
mydb> step
0x40112b: lea rdi, [rip+0xed2] ; "Hello, World!"
mydb>
Examining Memory
mydb> x/4x $rsp
0x7fffffffdde8: 0xf7de50d0 0x00007fff 0xfffddef8 0x00007fff
mydb> x/8x 0x401126
0x401126: 0xe5894855 0x00000aed 0x050febed 0x00002000
mydb> x/s 0x402004
0x402004: "Hello, World!"
mydb>
Continuing to Next Breakpoint
mydb> continue
Continuing...
Breakpoint 2 hit at 0x40112a
Stopped at 0x40112a
mydb>
Program Exit
mydb> continue
Continuing...
Hello, World!
Program exited normally (status: 0)
mydb> quit
Goodbye.
$
Solution Architecture
High-Level Design
+---------------------------------------------------------------------------+
| DEBUGGER ARCHITECTURE |
+---------------------------------------------------------------------------+
| |
| +----------------+ |
| | main.c | |
| | (entry point) | |
| +-------+--------+ |
| | |
| v |
| +-------+--------+ |
| | debugger.c | |
| | (main loop) | |
| +-------+--------+ |
| | |
| +---------------------------+---------------------------+ |
| | | | |
| v v v |
| +-------+--------+ +--------+-------+ +--------+------+ |
| | breakpoint.c | | registers.c | | memory.c | |
| | (breakpoint | | (register | | (memory | |
| | management) | | inspection) | | inspection) | |
| +----------------+ +----------------+ +---------------+ |
| | | | |
| +---------------------------+---------------------------+ |
| | |
| v |
| +-------+--------+ |
| | ptrace_util.c | |
| | (ptrace | |
| | wrappers) | |
| +-------+--------+ |
| | |
| v |
| +-------+--------+ |
| | KERNEL | |
| | (ptrace syscall)| |
| +----------------+ |
| |
+---------------------------------------------------------------------------+
Data Structures
/* ==================== BREAKPOINT STRUCTURE ==================== */
typedef struct breakpoint {
unsigned long addr; // Address where breakpoint is set
unsigned char saved_byte; // Original byte at that address
int enabled; // Is breakpoint active?
int id; // Breakpoint number for user
} breakpoint_t;
#define MAX_BREAKPOINTS 256
/* ==================== DEBUGGER STATE ==================== */
typedef struct debugger {
pid_t pid; // Tracee process ID
char *program; // Path to executable
int running; // Is tracee currently running?
breakpoint_t breakpoints[MAX_BREAKPOINTS];
int num_breakpoints;
// For distinguishing breakpoint vs single-step SIGTRAP
int waiting_for_step;
// ELF information for symbol lookup
void *elf_data; // Mapped ELF file or parsed data
} debugger_t;
/* ==================== COMMAND STRUCTURE ==================== */
typedef struct command {
const char *name;
void (*handler)(debugger_t *dbg, char *args);
const char *help;
} command_t;
Debugger State Machine
+---------------------------------------------------------------------------+
| DEBUGGER STATE MACHINE |
+---------------------------------------------------------------------------+
| |
| States: |
| [NOT_STARTED] - Debugger launched, tracee not yet started |
| [STOPPED] - Tracee is stopped (breakpoint, step, signal) |
| [RUNNING] - Tracee is running (after continue) |
| [EXITED] - Tracee has exited |
| |
| |
| +---------------+ |
| | NOT_STARTED | |
| +-------+-------+ |
| | |
| | "run" command |
| | fork() + ptrace(TRACEME) + execve() |
| | |
| v |
| +-------+-------+ |
| +-------->| STOPPED |<-----------+ |
| | +-------+-------+ | |
| | | | |
| | +-------------+-------------+ | |
| | | | | | |
| | v v v | |
| | "step" "continue" "regs" | |
| | "break" "x" "quit" | |
| | | | | | |
| | | | | | |
| | v v | | |
| | SINGLESTEP CONT | | |
| | | | | | |
| | +------+------+ | | |
| | | | | |
| | v | | |
| | +-------+-------+ | | |
| | | RUNNING | | | |
| | +-------+-------+ | | |
| | | | | |
| | | waitpid() | | |
| | | | | |
| | +-------+-------+ | | |
| | | SIGTRAP |------------+ | |
| | | SIGSTOP |-------------------+ |
| +--| (other signal)| |
| +-------+-------+ |
| | |
| | exit status |
| v |
| +-------+-------+ |
| | EXITED | |
| +---------------+ |
| |
+---------------------------------------------------------------------------+
Breakpoint Management Flow
+---------------------------------------------------------------------------+
| BREAKPOINT ENABLE/DISABLE FLOW |
+---------------------------------------------------------------------------+
| |
| ENABLE BREAKPOINT (user: "break 0x401000"): |
| |
| 1. Find or create breakpoint entry |
| bp = &dbg->breakpoints[n]; |
| bp->addr = 0x401000; |
| |
| 2. Read and save original byte |
| long word = ptrace(PTRACE_PEEKDATA, pid, addr, NULL); |
| bp->saved_byte = word & 0xFF; |
| |
| 3. Write INT3 (0xCC) in place of original |
| long new_word = (word & ~0xFF) | 0xCC; |
| ptrace(PTRACE_POKEDATA, pid, addr, new_word); |
| |
| 4. Mark breakpoint enabled |
| bp->enabled = 1; |
| |
| |
| HANDLE BREAKPOINT HIT: |
| |
| 1. waitpid() returns with SIGTRAP |
| |
| 2. Get RIP |
| ptrace(PTRACE_GETREGS, pid, NULL, ®s); |
| hit_addr = regs.rip - 1; // RIP is PAST the INT3! |
| |
| 3. Find matching breakpoint |
| bp = find_breakpoint_at(dbg, hit_addr); |
| |
| 4. Restore original byte |
| long word = ptrace(PTRACE_PEEKDATA, pid, bp->addr, NULL); |
| word = (word & ~0xFF) | bp->saved_byte; |
| ptrace(PTRACE_POKEDATA, pid, bp->addr, word); |
| |
| 5. Back up RIP to point at original instruction |
| regs.rip -= 1; |
| ptrace(PTRACE_SETREGS, pid, NULL, ®s); |
| |
| 6. (To continue past breakpoint) |
| - Single-step one instruction |
| - Re-enable breakpoint (write INT3 again) |
| - Then PTRACE_CONT |
| |
+---------------------------------------------------------------------------+
Implementation Guide
The Core Question You’re Answering
“When I type
break mainandrunin GDB, what system calls happen, how does the kernel stop execution at exactly that instruction, and how does GDB read my variables?”
This project demystifies the magic of debugging. You will see that debuggers are built on a simple but powerful kernel interface. There is no magic - just careful use of ptrace to control another process’s execution.
Concepts You Must Understand First
Before starting this project, ensure you understand these concepts:
| Concept | Why It Matters | Where to Learn |
|---|---|---|
| x86-64 registers and calling convention | You’ll inspect and modify them | CS:APP 3.4-3.7 |
| How functions are called (stack frames) | Essential for backtraces | CS:APP 3.7 |
| Process creation (fork/exec) | You’ll create the tracee process | CS:APP 8.2-8.4 |
| Signals (especially SIGTRAP) | Kernel notifies debugger via signals | CS:APP 8.5 |
| ELF file format basics | Need to find function addresses | CS:APP 7.4 |
| Machine code encoding | Understanding INT3 and instruction lengths | CS:APP 3.1-3.3 |
Questions to Guide Your Design
Work through these questions BEFORE writing code:
-
Process Launch: How do you launch a program under ptrace control? What’s the sequence of fork, ptrace(TRACEME), and execve?
-
Initial Stop: After execve, where does the tracee stop? How do you know it’s ready?
-
Breakpoint Address: When a breakpoint is hit, where does RIP point - at the INT3, or after it? How do you find the original instruction address?
-
Continuing Past Breakpoints: After hitting a breakpoint, if the user says “continue”, how do you execute the original instruction AND keep the breakpoint for next time?
-
Single-Step vs Breakpoint: Both generate SIGTRAP. How do you tell them apart?
-
Memory Access Alignment: ptrace reads/writes words (8 bytes). What if you want to read 1 byte? What if the address isn’t word-aligned?
-
Symbol Lookup: How do you find what address “main” is at? What sections of the ELF file contain this information?
Thinking Exercise
Before writing any code, trace through this scenario by hand:
You have a simple program:
// test.c - compile with: gcc -g -O0 -o test test.c
#include <stdio.h>
int add(int a, int b) { // Assume at address 0x401106
return a + b;
}
int main() { // Assume at address 0x401126
int x = add(5, 3);
printf("%d\n", x);
return 0;
}
Debugger session:
mydb> break main
mydb> run
mydb> step
mydb> step
mydb> step
mydb> continue
Exercise: On paper, answer:
-
After
break main: What byte was saved? What byte is now at 0x401126? -
After
run: Where is RIP when the debugger regains control? What signal was delivered? -
After each
step: What address does RIP point to? What instruction just executed? -
When
add(5,3)is called: What’s on the stack? What registers hold the arguments? -
After
continue: What happens when execution reaches theaddfunction and then returns?
Verify your answers by implementing and adding debug printing.
Interview Questions They’ll Ask
After completing this project, you’ll be ready for these common interview questions:
- “How does a debugger set a breakpoint?”
- Expected: Replace instruction byte with INT3 (0xCC), save original, restore when hit
- Bonus: Explain RIP adjustment, single-step-and-restore for continuing
- “How does ptrace work?”
- Expected: Kernel mediates interaction between tracer and tracee, provides register/memory access
- Bonus: Discuss ptrace requests, wait status values, signal delivery
- “What happens when you hit a breakpoint?”
- Expected: CPU executes INT3, raises exception, kernel sends SIGTRAP, tracee stops
- Bonus: Explain that RIP points past INT3, debugger must adjust
- “How does single-stepping work?”
- Expected: Trap flag (TF) in EFLAGS causes debug exception after each instruction
- Bonus: Explain how kernel sets TF, clears it after trap
- “How does GDB know where ‘main’ is?”
- Expected: Parses ELF symbol table (.symtab section)
- Bonus: Discuss DWARF for line numbers, debug_info section
- “What’s the difference between software and hardware breakpoints?”
- Expected: Software modifies code (INT3), hardware uses debug registers (DR0-DR3)
- Bonus: Discuss limits (4 hw breakpoints), watchpoints (data access)
Hints in Layers
If you’re stuck, reveal hints one at a time:
Hint 1: Starting the Tracee Process
The basic pattern for launching a program under ptrace control:
pid_t pid = fork();
if (pid == 0) {
// Child process (will become tracee)
// Request to be traced by parent
if (ptrace(PTRACE_TRACEME, 0, NULL, NULL) < 0) {
perror("ptrace TRACEME");
exit(1);
}
// Stop ourselves so parent can set up
// (Alternative: execve will stop automatically after TRACEME)
execve(program, argv, envp);
// If we get here, execve failed
perror("execve");
exit(1);
}
// Parent process (tracer)
// Wait for child to stop on execve
int status;
waitpid(pid, &status, 0);
if (WIFSTOPPED(status)) {
printf("Child stopped, signal %d\n", WSTOPSIG(status));
// Child is now stopped and ready for commands
}
After TRACEME + execve, the child stops with SIGTRAP. This is your initial stop.
Hint 2: Setting a Breakpoint
To set a breakpoint:
int set_breakpoint(pid_t pid, unsigned long addr, unsigned char *saved) {
// 1. Read the current word at addr
errno = 0;
long word = ptrace(PTRACE_PEEKDATA, pid, (void *)addr, NULL);
if (word == -1 && errno != 0) {
perror("PEEKDATA failed");
return -1;
}
// 2. Save the original byte (lowest byte on little-endian)
*saved = (unsigned char)(word & 0xFF);
// 3. Replace lowest byte with INT3 (0xCC)
long new_word = (word & ~0xFF) | 0xCC;
// 4. Write the modified word back
if (ptrace(PTRACE_POKEDATA, pid, (void *)addr, (void *)new_word) < 0) {
perror("POKEDATA failed");
return -1;
}
return 0;
}
Remember: x86 is little-endian, so the first byte of an instruction is in the lowest byte of the word.
Hint 3: Handling a Breakpoint Hit
When waitpid returns with SIGTRAP:
void handle_breakpoint_hit(debugger_t *dbg) {
struct user_regs_struct regs;
// 1. Get current registers
ptrace(PTRACE_GETREGS, dbg->pid, NULL, ®s);
// 2. RIP points PAST the INT3 (it's 1 byte), so back up
unsigned long bp_addr = regs.rip - 1;
// 3. Find which breakpoint was hit
breakpoint_t *bp = find_breakpoint(dbg, bp_addr);
if (!bp) {
// Not one of our breakpoints - could be single-step SIGTRAP
return;
}
printf("Breakpoint %d hit at 0x%lx\n", bp->id, bp_addr);
// 4. Restore the original byte
long word = ptrace(PTRACE_PEEKDATA, dbg->pid, (void *)bp_addr, NULL);
word = (word & ~0xFF) | bp->saved_byte;
ptrace(PTRACE_POKEDATA, dbg->pid, (void *)bp_addr, (void *)word);
// 5. Back up RIP to point at the original instruction
regs.rip = bp_addr;
ptrace(PTRACE_SETREGS, dbg->pid, NULL, ®s);
// Now the tracee is positioned to execute the original instruction
}
Hint 4: Continuing Past a Breakpoint
The tricky part: after hitting a breakpoint, you restored the original byte. But if you just CONT, the breakpoint is gone! To keep it:
void continue_from_breakpoint(debugger_t *dbg, breakpoint_t *bp) {
// 1. Original byte is already restored (from handling the hit)
// 2. RIP already points at the original instruction
// 3. Single-step to execute the original instruction
ptrace(PTRACE_SINGLESTEP, dbg->pid, NULL, NULL);
waitpid(dbg->pid, &status, 0);
// 4. Re-enable the breakpoint (write INT3 again)
if (bp->enabled) {
long word = ptrace(PTRACE_PEEKDATA, dbg->pid, (void *)bp->addr, NULL);
word = (word & ~0xFF) | 0xCC;
ptrace(PTRACE_POKEDATA, dbg->pid, (void *)bp->addr, (void *)word);
}
// 5. Now continue normally
ptrace(PTRACE_CONT, dbg->pid, NULL, NULL);
}
Hint 5: Looking Up Symbols in ELF
To find function addresses by name, you need to parse the ELF file:
#include <elf.h>
#include <fcntl.h>
#include <sys/mman.h>
unsigned long find_symbol(const char *filename, const char *symname) {
int fd = open(filename, O_RDONLY);
struct stat st;
fstat(fd, &st);
// Map the file
void *map = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
Elf64_Ehdr *ehdr = (Elf64_Ehdr *)map;
// Find section header table
Elf64_Shdr *shdr = (Elf64_Shdr *)((char *)map + ehdr->e_shoff);
// Find .symtab and .strtab
Elf64_Shdr *symtab = NULL;
Elf64_Shdr *strtab = NULL;
char *shstrtab = (char *)map + shdr[ehdr->e_shstrndx].sh_offset;
for (int i = 0; i < ehdr->e_shnum; i++) {
char *name = shstrtab + shdr[i].sh_name;
if (strcmp(name, ".symtab") == 0) symtab = &shdr[i];
if (strcmp(name, ".strtab") == 0) strtab = &shdr[i];
}
if (!symtab || !strtab) {
munmap(map, st.st_size);
close(fd);
return 0; // No symbols
}
// Search symbols
Elf64_Sym *syms = (Elf64_Sym *)((char *)map + symtab->sh_offset);
char *strings = (char *)map + strtab->sh_offset;
int nsyms = symtab->sh_size / sizeof(Elf64_Sym);
for (int i = 0; i < nsyms; i++) {
if (strcmp(strings + syms[i].st_name, symname) == 0) {
unsigned long addr = syms[i].st_value;
munmap(map, st.st_size);
close(fd);
return addr;
}
}
munmap(map, st.st_size);
close(fd);
return 0; // Not found
}
For stripped binaries, .symtab may not exist. Use .dynsym for dynamic symbols.
Testing Strategy
Test Programs
Create simple test programs compiled with -g -O0 -fno-omit-frame-pointer:
test1.c - Basic function call:
#include <stdio.h>
void greet() {
printf("Hello from greet!\n");
}
int main() {
printf("Before greet\n");
greet();
printf("After greet\n");
return 0;
}
test2.c - Loop for multiple breakpoint hits:
int main() {
int sum = 0;
for (int i = 0; i < 5; i++) {
sum += i; // Set breakpoint here
}
return sum;
}
test3.c - Function with arguments:
int add(int a, int b) {
return a + b;
}
int main() {
int x = add(10, 20);
return x;
}
Breakpoint Verification
# Test 1: Basic breakpoint
mydb> break main
mydb> run
# Verify: should stop at main
# Test 2: Multiple breakpoints
mydb> break main
mydb> break greet
mydb> run
# Verify: hits main first
mydb> continue
# Verify: hits greet next
# Test 3: Breakpoint hit multiple times (loop)
mydb> break 0x401xxx # Address inside loop
mydb> run
# Verify: can hit same breakpoint 5 times with continue
Register Verification
# After hitting breakpoint in add(10, 20):
mydb> regs
# Verify: RDI should be 10 (first arg)
# Verify: RSI should be 20 (second arg)
Memory Verification
# Examine stack:
mydb> x/8x $rsp
# Verify: shows stack contents
# Examine code:
mydb> x/4x $rip
# Verify: shows instructions (after breakpoint restore)
Common Pitfalls
Breakpoint Pitfalls
+---------------------------------------------------------------------------+
| BREAKPOINT BUGS |
+---------------------------------------------------------------------------+
| |
| BUG: Not decrementing RIP after breakpoint hit |
| ----------------------------------------------------- |
| |
| Wrong: // RIP points past INT3, but we use it as-is |
| printf("Stopped at 0x%lx\n", regs.rip); |
| |
| Right: unsigned long bp_addr = regs.rip - 1; |
| printf("Stopped at 0x%lx\n", bp_addr); |
| |
| |
| BUG: Forgetting to re-enable breakpoint after continue |
| -------------------------------------------------------- |
| |
| Symptom: Breakpoint only fires once |
| |
| Fix: After single-stepping past original instruction, |
| write 0xCC back before PTRACE_CONT |
| |
| |
| BUG: Byte order confusion on big-endian systems |
| ------------------------------------------------- |
| |
| On little-endian (x86), first byte is in low bits of word |
| On big-endian, first byte is in high bits |
| |
| // Little-endian (x86): |
| saved = word & 0xFF; |
| word = (word & ~0xFF) | 0xCC; |
| |
+---------------------------------------------------------------------------+
Signal Handling Pitfalls
+---------------------------------------------------------------------------+
| SIGNAL HANDLING BUGS |
+---------------------------------------------------------------------------+
| |
| BUG: Confusing breakpoint SIGTRAP with single-step SIGTRAP |
| ----------------------------------------------------------- |
| |
| Both generate SIGTRAP! Distinguish by: |
| - Tracking state: did we just issue SINGLESTEP? |
| - Checking if RIP-1 matches a breakpoint |
| - Using ptrace(PTRACE_GETSIGINFO) to check si_code |
| |
| BUG: Not handling other signals |
| ---------------------------------- |
| |
| Tracee might receive SIGSEGV, SIGFPE, etc. |
| Your debugger should report these, not crash! |
| |
| if (WIFSTOPPED(status)) { |
| int sig = WSTOPSIG(status); |
| if (sig == SIGTRAP) { |
| handle_trap(dbg); |
| } else { |
| printf("Received signal %d (%s)\n", sig, strsignal(sig)); |
| } |
| } |
| |
| BUG: Not detecting tracee exit |
| ---------------------------------- |
| |
| if (WIFEXITED(status)) { |
| printf("Program exited with status %d\n", WEXITSTATUS(status)); |
| dbg->running = 0; |
| } |
| if (WIFSIGNALED(status)) { |
| printf("Program killed by signal %d\n", WTERMSIG(status)); |
| dbg->running = 0; |
| } |
| |
+---------------------------------------------------------------------------+
Wait Status Pitfalls
+---------------------------------------------------------------------------+
| WAIT STATUS BUGS |
+---------------------------------------------------------------------------+
| |
| UNDERSTANDING WAIT STATUS: |
| |
| waitpid(pid, &status, 0); |
| |
| WIFEXITED(status) - true if child exited normally |
| WEXITSTATUS(status) - exit code (0-255) |
| |
| WIFSIGNALED(status) - true if child killed by signal |
| WTERMSIG(status) - signal number that killed it |
| |
| WIFSTOPPED(status) - true if child is stopped (traced) |
| WSTOPSIG(status) - signal that caused stop |
| |
| COMMON BUG: Checking these in wrong order |
| |
| Wrong: |
| if (WSTOPSIG(status) == SIGTRAP) { ... } // Without WIFSTOPPED! |
| |
| Right: |
| if (WIFSTOPPED(status) && WSTOPSIG(status) == SIGTRAP) { ... } |
| |
+---------------------------------------------------------------------------+
Memory Access Pitfalls
+---------------------------------------------------------------------------+
| MEMORY ACCESS BUGS |
+---------------------------------------------------------------------------+
| |
| BUG: Checking PEEKDATA return value incorrectly |
| ------------------------------------------------- |
| |
| ptrace(PEEKDATA) returns the word value on success. |
| It returns -1 on error, BUT -1 could also be valid data! |
| |
| Wrong: |
| long data = ptrace(PTRACE_PEEKDATA, pid, addr, NULL); |
| if (data == -1) { error!; } // What if memory contains 0xFFFFFFFF? |
| |
| Right: |
| errno = 0; |
| long data = ptrace(PTRACE_PEEKDATA, pid, addr, NULL); |
| if (data == -1 && errno != 0) { error!; } |
| |
| |
| BUG: Unaligned access assumptions |
| ---------------------------------- |
| |
| ptrace reads/writes in word-size (8 byte) chunks. |
| Address doesn't need to be aligned, but you get the whole word. |
| |
| To read a single byte at unaligned address: |
| unsigned long aligned = addr & ~7; // Align down |
| unsigned long offset = addr & 7; // Byte offset within word |
| long word = ptrace(PTRACE_PEEKDATA, pid, aligned, NULL); |
| unsigned char byte = (word >> (offset * 8)) & 0xFF; |
| |
+---------------------------------------------------------------------------+
Extensions
Beginner Extensions
- List breakpoints:
info breakcommand to show all breakpoints - Delete breakpoints:
delete <n>to remove a breakpoint - Disable/enable: Temporarily disable breakpoints without deleting
- Print current instruction: Disassemble instruction at RIP
Intermediate Extensions
- Watchpoints: Hardware breakpoints for memory access (use debug registers DR0-DR3)
- Backtrace: Walk the stack and show return addresses
- Source line mapping: Use addr2line or DWARF to show source lines
- Print variables: Parse DWARF to find local variable locations
Advanced Extensions
- Conditional breakpoints: Only stop if expression is true
- Attach to running process: Use PTRACE_ATTACH instead of fork+TRACEME
- Multi-threaded debugging: Handle multiple threads with ptrace
- Remote debugging stub: Implement GDB remote protocol
Watchpoint Implementation Sketch
// Using hardware debug registers for watchpoints
void set_watchpoint(pid_t pid, unsigned long addr, int len, int type) {
// type: 0=exec, 1=write, 3=read/write
// Set address in DR0
ptrace(PTRACE_POKEUSER, pid,
offsetof(struct user, u_debugreg[0]), addr);
// Enable in DR7
unsigned long dr7 = ptrace(PTRACE_PEEKUSER, pid,
offsetof(struct user, u_debugreg[7]), NULL);
// Set condition and length for DR0
dr7 |= (1 << 0); // Local enable for DR0
dr7 |= (type << 16); // R/W condition
dr7 |= ((len-1) << 18); // Length
ptrace(PTRACE_POKEUSER, pid,
offsetof(struct user, u_debugreg[7]), dr7);
}
Books That Will Help
| Topic | Book | Chapter/Section |
|---|---|---|
| x86-64 registers and instructions | CS:APP 3rd Ed | Chapter 3 “Machine-Level Representation” |
| Process control (fork, exec, wait) | CS:APP 3rd Ed | Chapter 8.2-8.4 “Process Control” |
| Signals and signal handling | CS:APP 3rd Ed | Chapter 8.5 “Signals” |
| ELF object files | CS:APP 3rd Ed | Chapter 7 “Linking” |
| ptrace details | The Linux Programming Interface | Chapter 26 “Monitoring Child Processes” |
| ptrace in depth | Linux Kernel Development (Love) | Chapter 5 “System Calls” |
| DWARF debugging format | DWARF Debugging Information Format | (specification document) |
| GDB internals | GDB Internals Manual | (online at sourceware.org) |
Real-World Connections
Production Debuggers
GDB (GNU Debugger):
- Uses ptrace on Linux, Mach ports on macOS
- Supports remote debugging via RSP (Remote Serial Protocol)
- Extensive DWARF support for source-level debugging
- Python scripting for automation
LLDB (LLVM Debugger):
- Modern architecture with plugin system
- Native macOS debugger using Mach APIs
- Uses ptrace on Linux
- Reuses Clang’s type system for expression evaluation
strace:
- Also built on ptrace
- Uses PTRACE_SYSCALL to stop at every system call
- Great for understanding program behavior
Kernel Debuggers
kdb/kgdb:
- Linux kernel debugger
- Works through serial port or network
- Can’t use ptrace (kernel can’t trace itself!)
WinDbg:
- Windows kernel/user debugger
- Different architecture (debug API, not ptrace)
- Similar concepts: breakpoints, stepping, memory/register access
Security Tools
Frida:
- Dynamic instrumentation toolkit
- Uses ptrace to inject JavaScript engine
- Popular for mobile app reverse engineering
GEF/PEDA (GDB plugins):
- Enhanced GDB for exploit development
- Heap visualization, ROP gadget search
- Built on GDB’s Python API
Self-Assessment Checklist
Understanding
- I can explain how ptrace enables one process to control another
- I understand why INT3 is used for breakpoints (1 byte, specific exception)
- I can describe what happens when a breakpoint is hit (RIP adjustment, byte restore)
- I know how single-stepping works (Trap Flag in EFLAGS)
- I understand the difference between software and hardware breakpoints
- I can explain how to look up function addresses in an ELF file
Implementation
- My debugger can launch a program under ptrace control
- Breakpoints work correctly (program stops at right address)
- RIP is correctly adjusted after breakpoint hit
- Breakpoints persist after continuing (single-step and re-enable)
- Single-step executes exactly one instruction
- Register display shows all general-purpose registers
- Memory examination works for arbitrary addresses
- Symbol lookup finds function addresses
Robustness
- Handles tracee exit gracefully
- Reports signals other than SIGTRAP
- Doesn’t crash on invalid commands
- Handles multiple breakpoints correctly
- Distinguishes breakpoint SIGTRAP from single-step SIGTRAP
Growth
- I debugged at least one tricky issue (wrong RIP, byte order, etc.)
- I can explain debugger internals in an interview
- I understand what additional work would be needed for source-level debugging
- I know how this project relates to security tools and reverse engineering
Submission / Completion Criteria
Minimum Viable Completion:
- Launch program under ptrace control
- Set breakpoints by address
- Continue and step commands work
- Register display implemented
- Handles basic test programs
Full Completion:
- All of the above plus symbol lookup (break by function name)
- Memory examination command
- Multiple breakpoints work correctly
- Breakpoints persist after continue
- Clean exit handling
Excellence (Going Above & Beyond):
- Watchpoints using debug registers
- Backtrace command
- Source line display (via addr2line or DWARF)
- Conditional breakpoints
- Attach to running process
This guide was expanded from CSAPP_3E_DEEP_LEARNING_PROJECTS.md. For the complete learning path, see the project index.