Project 7: Memory Visualizer & Debugger

Build a tool that attaches to a running process and visualizes its memory layout in real-time, revealing the stack, heap, code sections, and how they change during execution.

Quick Reference

Attribute	Value
Language	C (alt: Rust, C++, Python)
Difficulty	Advanced
Time	2-3 weeks
Knowledge Area	Memory Layout / Debugging
Coolness	★★★★★ Genuinely Powerful
Portfolio Value	Resume Gold / Interview Ace

Learning Objectives

By completing this project, you will:

Master virtual memory concepts: Explain how processes see memory vs how it maps to physical RAM
Understand process memory layout: Draw and explain text, data, bss, heap, stack, and mmap regions
Use system debugging APIs: Write code using ptrace (Linux) or mach APIs (macOS) to inspect another process
Parse memory mapping information: Read and interpret /proc/[pid]/maps or vmmap output
Visualize stack frames: Show function call chains with local variables and return addresses
Track memory allocations: Observe heap growth, fragmentation, and allocation patterns over time
Bridge CPU and OS concepts: Connect how the CPU’s MMU translates addresses to what the OS presents to processes

The Core Question You’re Answering

“Where exactly does my program’s data live in memory, and how can I observe the CPU’s view of memory change as my program executes?”

This question forces you to confront the reality that memory is not a simple array of bytes. It is a virtualized, segmented, protected abstraction provided by the OS with the CPU’s MMU hardware. Understanding this is essential for debugging memory corruption, optimizing cache usage, analyzing security vulnerabilities, and reasoning about program behavior at the lowest level.

Concepts You Must Understand First

Before starting this project, ensure you understand these concepts:

Concept	Why It Matters	Where to Learn
Pointers and pointer arithmetic	You’ll read and manipulate memory addresses	Any C book, CS:APP 3.8
Function calling conventions	You’ll decode stack frames and return addresses	CS:APP 3.7
Virtual vs physical memory	Core concept of this project	CS:APP 9.1-9.3
Process memory layout (text, data, heap, stack)	You’ll visualize these regions	CS:APP 7.9, 9.7
System calls (especially fork, exec)	You’ll attach to processes	TLPI Chapters 24-26
Hexadecimal notation	Memory addresses are in hex	CS:APP 2.1
ELF binary format basics	Understanding where code/data come from	CS:APP 7.3-7.4

Key Concepts Deep Dive

Virtual Memory and Address Spaces
- Why does every process think it owns all of memory from 0x0 to 0x7fff…?
- How does the MMU translate virtual addresses to physical addresses?
- What happens when a process accesses unmapped memory (SIGSEGV)?
- CS:APP Ch. 9.1-9.4
Process Memory Layout
- What are the text, data, bss, heap, and stack segments?
- Why does the heap grow up and the stack grow down?
- What lives in the “gap” between heap and stack?
- CS:APP Ch. 7.9, Ch. 9.7
The Stack Frame
- What is stored in a stack frame (return address, saved registers, locals, arguments)?
- How do rbp and rsp define the current frame?
- What is the red zone (on x86-64 System V ABI)?
- CS:APP Ch. 3.7
Heap Management
- How does malloc/free work at a high level?
- What is heap fragmentation and why does it matter?
- How can you detect memory leaks and corruption?
- CS:APP Ch. 9.9
ptrace and Process Debugging
- How does a debugger attach to and control another process?
- What operations can ptrace perform (read memory, single-step, etc.)?
- What are the security implications of ptrace?
- The Linux Programming Interface Ch. 26

Theoretical Foundation

Virtual Memory: The Great Illusion

When your program runs, it doesn’t see physical RAM. It sees a carefully crafted illusion called virtual memory:

Physical Reality                           Virtual Illusion (Per Process)

┌───────────────────────────┐             ┌─────────────────────────────────┐
│      Physical RAM         │             │  Process A's View               │
│      (e.g., 16 GB)        │             │                                 │
│                           │             │  0x7fff...  ┌──────────────┐    │
│  ┌─────┐ ┌─────┐ ┌─────┐  │             │             │    Stack     │    │
│  │     │ │     │ │     │  │             │             └──────────────┘    │
│  │ A's │ │ B's │ │ A's │  │             │                                 │
│  │stack│ │heap │ │text │  │             │             ┌──────────────┐    │
│  │     │ │     │ │     │  │             │             │    Heap      │    │
│  └─────┘ └─────┘ └─────┘  │    MMU      │             └──────────────┘    │
│                           │ ◄──────────►│                                 │
│  ┌─────┐ ┌─────┐ ┌─────┐  │ Translation │  0x555...   ┌──────────────┐    │
│  │ B's │ │OS   │ │Shared│  │             │             │    Text      │    │
│  │text │ │kern │ │libs  │  │             │             └──────────────┘    │
│  └─────┘ └─────┘ └─────┘  │             │  0x0                            │
│                           │             │  (Unmapped - null pointer trap) │
└───────────────────────────┘             └─────────────────────────────────┘

Every process thinks it has its own private, contiguous address space.
The MMU hardware and OS kernel collaborate to maintain this illusion.

Why Virtual Memory Exists

Isolation: Process A cannot access Process B’s memory (security)
Simplification: Every program can assume it starts at the same address
Efficiency: Physical memory can be shared (shared libraries) or overcommitted
Protection: Different regions have different permissions (read, write, execute)

Process Memory Layout

Every Unix/Linux process has a standardized memory layout:

High Addresses (0x7fffffff...)
┌─────────────────────────────────────────────────────────────┐
│                     KERNEL SPACE                            │
│            (Not accessible to user code)                    │
│           Addresses >= 0x8000000000000000 (64-bit)          │
├─────────────────────────────────────────────────────────────┤
│                      STACK                                  │
│                        │                                    │
│                        ▼ grows down                         │
│     - Function stack frames                                 │
│     - Local variables                                       │
│     - Return addresses                                      │
│     - Saved registers                                       │
│                                                             │
│     [RSP points here - current stack top]                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│                    UNMAPPED REGION                          │
│              (Guard page, causes SIGSEGV)                   │
│                                                             │
│            ~140 TB of unused address space!                 │
│                                                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│              MEMORY-MAPPED REGIONS                          │
│     - Shared libraries (libc.so, libpthread.so, etc.)       │
│     - Memory-mapped files                                   │
│     - Anonymous mappings (large mallocs)                    │
│                                                             │
├─────────────────────────────────────────────────────────────┤
│                        ▲ grows up                           │
│                        │                                    │
│                      HEAP                                   │
│     - malloc()'d memory                                     │
│     - Dynamic allocations                                   │
│                                                             │
│     [brk() / sbrk() controls end of heap]                   │
├─────────────────────────────────────────────────────────────┤
│                     .bss                                    │
│     - Uninitialized global/static variables                 │
│     - Zeroed by OS at load time                             │
│     - Doesn't take space in executable file!                │
├─────────────────────────────────────────────────────────────┤
│                     .data                                   │
│     - Initialized global/static variables                   │
│     - int global_var = 42;                                  │
├─────────────────────────────────────────────────────────────┤
│                    .rodata                                  │
│     - Read-only data (string literals, const)               │
│     - "Hello, World!\n"                                     │
├─────────────────────────────────────────────────────────────┤
│                     .text                                   │
│     - Executable code                                       │
│     - main(), printf(), etc.                                │
│     - Read + Execute (no write!)                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│              UNMAPPED LOW ADDRESSES                         │
│           (NULL pointer detection)                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘
Low Addresses (0x0...)

Stack Frame Structure

Each function call creates a stack frame. Understanding the frame layout is crucial for your visualizer:

High Addresses (earlier in stack)
┌──────────────────────────────────────────────────────┐
│                 CALLER'S FRAME                       │
│                                                      │
├──────────────────────────────────────────────────────┤
│  Arguments passed on stack (if > 6 args)             │  ← Arg 7+
│  [Only if needed - first 6 args in registers]       │
├──────────────────────────────────────────────────────┤
│  Return Address (8 bytes)                            │  ← pushed by CALL
│  [Address in caller to return to]                    │     instruction
├──────────────────────────────────────────────────────┤ ◄── Previous RSP
│  Saved RBP (8 bytes)                                 │  ← pushed by callee
│  [Caller's frame pointer, enables stack walking]     │     (frame pointer)
├──────────────────────────────────────────────────────┤ ◄── Current RBP
│                                                      │
│  Local Variables                                     │
│    - int x;           (4 bytes)                      │
│    - char buf[32];    (32 bytes)                     │
│    - double y;        (8 bytes)                      │
│                                                      │
├──────────────────────────────────────────────────────┤
│  Saved Callee-Save Registers                         │
│    - RBX, R12, R13, R14, R15 (if used)              │
├──────────────────────────────────────────────────────┤
│                                                      │
│  Space for outgoing arguments (if calling others)    │
│                                                      │
├──────────────────────────────────────────────────────┤
│  Alignment padding (16-byte alignment required)      │
└──────────────────────────────────────────────────────┘ ◄── Current RSP
Low Addresses (stack grows down)

Key relationships:
- RBP points to saved RBP (start of current frame)
- [RBP + 8] is the return address
- [RBP + 0] is the previous RBP (for unwinding)
- [RBP - N] are local variables
- RSP points to current stack top

ptrace: The Debugger’s Superpower

The ptrace system call is how debuggers like GDB control other processes:

┌──────────────────────────────────────────────────────────────────────┐
│                        ptrace INTERACTION MODEL                       │
└──────────────────────────────────────────────────────────────────────┘

Debugger (Parent)                          Target (Child)
┌──────────────┐                          ┌──────────────┐
│              │                          │              │
│   memviz     │   PTRACE_ATTACH          │   target     │
│              │ ───────────────────────► │   program    │
│              │                          │              │
│              │   Target stops (SIGSTOP) │   STOPPED    │
│              │ ◄─────────────────────── │              │
│              │                          │              │
│   Read       │   PTRACE_PEEKDATA        │              │
│   memory     │ ───────────────────────► │   memory     │
│              │         word             │              │
│              │ ◄─────────────────────── │              │
│              │                          │              │
│   Read       │   PTRACE_GETREGS         │              │
│   registers  │ ───────────────────────► │   RSP, RBP   │
│              │      all regs            │   RIP, etc   │
│              │ ◄─────────────────────── │              │
│              │                          │              │
│   Single     │   PTRACE_SINGLESTEP      │              │
│   step       │ ───────────────────────► │  execute 1   │
│              │                          │  instruction │
│              │        SIGTRAP           │              │
│              │ ◄─────────────────────── │   STOPPED    │
│              │                          │              │
│   Continue   │   PTRACE_CONT            │              │
│              │ ───────────────────────► │   RUNNING    │
│              │                          │              │
│   Detach     │   PTRACE_DETACH          │              │
│              │ ───────────────────────► │   RUNNING    │
│              │                          │  (free)      │
└──────────────┘                          └──────────────┘

Key ptrace operations:
PTRACE_ATTACH    - Attach to running process (becomes tracer)
PTRACE_TRACEME   - Child requests to be traced
PTRACE_PEEKDATA  - Read one word from target's memory
PTRACE_POKEDATA  - Write one word to target's memory
PTRACE_GETREGS   - Get all CPU registers
PTRACE_SETREGS   - Set all CPU registers
PTRACE_SINGLESTEP - Execute one instruction, then stop
PTRACE_CONT      - Continue execution until next signal
PTRACE_DETACH    - Stop tracing, let process run free

Virtual Memory Mapping (/proc/[pid]/maps)

The kernel exposes each process’s memory mappings through the proc filesystem:

$ cat /proc/12345/maps

Address Range              Perms  Offset  Dev   Inode    Path
─────────────────────────────────────────────────────────────────────────
00400000-00401000          r--p   00000000 08:01 1234567  /home/user/myprogram
│        │                 │ │    │        │     │        │
│        │                 │ │    │        │     │        └── Mapped file
│        │                 │ │    │        │     └── Inode number
│        │                 │ │    │        └── Device (major:minor)
│        │                 │ │    └── Offset in file
│        │                 │ └── p=private, s=shared
│        │                 └── rwxp (read, write, execute, private/shared)
│        └── End address (exclusive)
└── Start address (inclusive)

00401000-00402000          r-xp   00001000 08:01 1234567  /home/user/myprogram
                              │
                              └── executable (.text section)

00402000-00403000          r--p   00002000 08:01 1234567  /home/user/myprogram
                              │
                              └── read-only (.rodata)

00403000-00404000          rw-p   00003000 08:01 1234567  /home/user/myprogram
                               │
                               └── read-write (.data, .bss)

00404000-00425000          rw-p   00000000 00:00 0        [heap]
                                                          │
                                                          └── anonymous mapping

7f1234560000-7f1234700000  r-xp   00000000 08:01 7890123  /lib/x86_64.../libc.so.6
                                                          │
                                                          └── shared library

7ffff7ffd000-7ffff8000000  rw-p   00000000 00:00 0        [stack]
                                                          │
                                                          └── the stack

The MMU and Page Tables

The CPU’s Memory Management Unit (MMU) translates every memory access:

┌─────────────────────────────────────────────────────────────────────────┐
│                    MMU ADDRESS TRANSLATION                               │
└─────────────────────────────────────────────────────────────────────────┘

Virtual Address (48 bits on x86-64):
┌─────────┬─────────┬─────────┬─────────┬───────────────┐
│ PML4    │ PDPT    │  PD     │  PT     │   Offset      │
│ Index   │ Index   │ Index   │ Index   │   (12 bits)   │
│ (9 bits)│ (9 bits)│ (9 bits)│ (9 bits)│               │
└────┬────┴────┬────┴────┬────┴────┬────┴───────────────┘
     │         │         │         │
     ▼         ▼         ▼         ▼
  ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐
  │PML4  │──│PDPT  │──│ PD   │──│ PT   │──► Physical Page ──► Byte
  │Table │  │Table │  │Table │  │Table │    Frame Number      in RAM
  └──────┘  └──────┘  └──────┘  └──────┘
     ▲
     │
  CR3 Register (points to current process's page tables)


What the MMU checks at each level:
┌─────────────────────────────────────────────────┐
│  Page Table Entry (PTE)                          │
├─────────────────────────────────────────────────┤
│  Bits 51:12  │ Physical page frame number       │
│  Bit 0       │ Present (P) - is page in RAM?    │
│  Bit 1       │ Read/Write (R/W)                 │
│  Bit 2       │ User/Supervisor (U/S)            │
│  Bit 5       │ Accessed (A) - has been read?    │
│  Bit 6       │ Dirty (D) - has been written?    │
│  Bit 63      │ No Execute (NX)                  │
└─────────────────────────────────────────────────┘

If Present=0: Page Fault → OS loads page from disk or SIGSEGV
If U/S wrong: Protection Fault → SIGSEGV
If writing to R/W=0: Protection Fault → SIGSEGV
If executing with NX=1: Protection Fault → SIGSEGV

Why This Matters: Real-World Implications

Understanding memory layout enables you to:

Debug crashes: A crash at 0x7ffff7xxx means shared library; 0x7fffffff… means stack overflow; 0x0 means null pointer
Understand security: Buffer overflows corrupt return addresses on the stack; heap overflows corrupt adjacent allocations
Optimize performance: Data locality in memory affects cache performance; understanding layout helps optimize
Analyze malware: Understanding memory layout is essential for reverse engineering
Write debuggers/profilers: Every debugger needs to read and interpret memory

Project Specification

What You Will Build

A command-line memory visualization tool (memviz) that:

Attaches to a running process (or launches a new one under its control)
Displays the complete memory map with segment identification
Shows detailed stack frames with local variables (when debug info available)
Tracks heap allocations and fragmentation over time
Provides an interactive interface for exploration

Functional Requirements

Process Attachment (memviz --attach <pid> or memviz <program> [args]):
- Attach to an existing running process
- Or launch a new process under control (using PTRACE_TRACEME)
- Handle permission denied errors gracefully
Memory Map Display (memviz -m):
- Parse /proc/[pid]/maps (Linux) or use vmmap (macOS)
- Categorize regions (stack, heap, text, data, libraries, etc.)
- Show permissions, sizes, and mapped files
- Calculate fragmentation and total usage
Stack Visualization (memviz -s):
- Walk the stack using saved frame pointers
- Show each frame with function name (if symbols available)
- Display local variables with their values
- Show return addresses and calling chain
Heap Analysis (memviz -h):
- Identify heap boundaries
- Track allocations over time (basic: show current size)
- Optionally intercept malloc/free (advanced)
Register Display (memviz -r):
- Show all CPU registers
- Highlight RSP, RBP, RIP
- Show flags register decoded
Interactive Mode (memviz -i):
- Step through execution
- Examine arbitrary memory
- Set breakpoints (stretch goal)
- Watch memory locations change

Non-Functional Requirements

Safety: Never crash the target process (except by user request)
Portability: Linux primary, macOS stretch goal (different APIs)
Performance: Handle large memory maps efficiently
Education: Output explains what each region is and why

Real World Outcome

When complete, you will have a CLI tool that produces output like this:

$ ./memviz ./my_program
Attaching to PID 12345...

================================================================================
                           MEMORY VISUALIZER - PID 12345
================================================================================

┌─────────────────────────────────────────────────────────────────────────────┐
│                            MEMORY MAP OVERVIEW                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  0x7fffffffdf00 ────────────────────────────────────── STACK (grows down)   │
│  │ main() frame: 64 bytes                                                   │
│  │ ├─ int argc = 1                           [RBP-0x04] = 0x00000001        │
│  │ ├─ char **argv                            [RBP-0x10] = 0x7fffffffe0a8    │
│  │ ├─ int x = 42                             [RBP-0x14] = 0x0000002a        │
│  │ └─ char buf[32]                           [RBP-0x34] = "Hello\0..."      │
│  │                                                                          │
│  │ foo() frame: 32 bytes                                                    │
│  │ ├─ return addr → main+0x1a                [RBP+0x08] = 0x00401234        │
│  │ ├─ int local_var = 17                     [RBP-0x04] = 0x00000011        │
│  │ └─ double result = 3.14                   [RBP-0x10] = 0x40091eb851...   │
│  ▼                                                                          │
│                                                                              │
│  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [~3.8 GB unused] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~│
│                                                                              │
│  ▲                                                                          │
│  │                                                                          │
│  0x555555559000 ────────────────────────────────────── HEAP (grows up)      │
│  │ [0x555555559000 - 0x55555557a000] 132 KB allocated                       │
│  │ ├─ malloc(1024) at 0x555555559010 - active                               │
│  │ ├─ malloc(4096) at 0x555555559420 - active                               │
│  │ └─ [free block] at 0x55555555a430 - 2048 bytes                           │
│                                                                              │
├─────────────────────────────────────────────────────────────────────────────┤
│  Segment          Start              End              Size      Perm        │
├─────────────────────────────────────────────────────────────────────────────┤
│  [heap]           0x555555559000     0x55555557a000   132 KB    rw-p        │
│  .bss             0x555555558000     0x555555559000   4 KB      rw-p        │
│  .data            0x555555557000     0x555555558000   4 KB      rw-p        │
│  .rodata          0x555555402000     0x555555403000   4 KB      r--p        │
│  .text            0x555555401000     0x555555402000   4 KB      r-xp        │
├─────────────────────────────────────────────────────────────────────────────┤
│  Shared Libraries:                                                          │
│  libc.so.6        0x7ffff7c00000     0x7ffff7e00000   2 MB      r-xp        │
│  ld-linux.so.2    0x7ffff7fc3000     0x7ffff7ff0000   180 KB    r-xp        │
└─────────────────────────────────────────────────────────────────────────────┘

REGISTERS:
  RIP: 0x555555401234 (foo+0x15)    RSP: 0x7fffffffde70    RBP: 0x7fffffffde90
  RAX: 0x000000000042               RBX: 0x0000000000000   RCX: 0x00007ffff7e01...
  FLAGS: CF=0 ZF=1 SF=0 OF=0

[Press 's' to step, 'c' to continue, 'm' to show map, 'q' to quit]
> s
Single stepping... stopped at foo+0x17 (0x555555401236)

  0x555555401234:  mov    eax, DWORD PTR [rbp-0x4]   ; load local_var
  0x555555401237:  add    eax, 0x1                    ; local_var + 1
> 0x55555540123a:  mov    DWORD PTR [rbp-0x4], eax   ; store back  <-- HERE

> x/16x $rsp
0x7fffffffde70: 0x00000011 0x00000000 0x40091eb8 0x51eb851f
0x7fffffffde80: 0x7fffffffde90 0x00000000 0x00401234 0x00000000

Questions to Guide Your Design

Work through these questions BEFORE writing code:

How will you attach to a process? What happens if the target is running? What if you don’t have permission?
How will you read memory from another process? ptrace(PTRACE_PEEKDATA, ...) reads one word at a time. Is that efficient enough? What about /proc/[pid]/mem?
How will you identify memory regions? Parsing /proc/[pid]/maps gives you ranges, but how do you identify which is stack, heap, etc.?
How will you walk the stack? You need to read RBP, then read [RBP] to get the previous RBP, etc. What if frame pointers are omitted (-fomit-frame-pointer)?
How will you get function names? Without debug symbols, you only have addresses. With symbols, you can use nm or read the symbol table.
How will you handle threads? Each thread has its own stack. How do you enumerate threads?
How will you make the output useful? Just dumping hex is overwhelming. What visualization makes the data understandable?
How will you handle macOS differences? macOS doesn’t have /proc. It uses mach_vm_read and task_for_pid instead.

Thinking Exercise

Before writing any code, work through this exercise on paper:

Given this C program:

// target.c
#include <stdio.h>
#include <stdlib.h>

int global_init = 100;
int global_uninit;

void inner(int n) {
    int local = n * 2;
    printf("inner: local = %d\n", local);
}

void outer(int x) {
    int arr[10] = {0};
    arr[0] = x;
    inner(arr[0]);
}

int main(int argc, char **argv) {
    char *heap_ptr = malloc(1024);
    outer(global_init);
    free(heap_ptr);
    return 0;
}

Answer these questions:

Memory Map: When this program is stopped inside inner(), list all the memory segments you expect to see in /proc/[pid]/maps. For each, state:
- Address range (use placeholder like 0x555… for text)
- Permissions (r/w/x/p)
- What lives there
Stack Layout: Draw the stack at the moment execution is inside inner(). Show:
- All three frames (main, outer, inner)
- Return addresses
- Local variables with their values
- The direction the stack grows
Register State: What will these registers contain when stopped at the first line of inner()?
- RIP: ?
- RSP: ?
- RBP: ?
- RDI: ? (first argument)
Memory Reading: If you want to read the value of arr[0] from outer()’s frame while stopped in inner(), what address do you need to read? How do you calculate it?
Heap Identification: How can you tell from the memory map that heap_ptr points into the heap? What if the heap has been allocated but appears empty?
Stack Walking: Starting from the current RBP in inner(), describe the algorithm to walk back and print all return addresses up to main().

Hints in Layers

If you’re stuck, reveal hints one at a time:

Hint 1: Getting Started with ptrace

Start with the absolute minimum - attach to a process and read one word:

#include <sys/ptrace.h>
#include <sys/wait.h>

pid_t pid = /* target pid */;

// Attach - this sends SIGSTOP to target
ptrace(PTRACE_ATTACH, pid, NULL, NULL);

// Wait for it to stop
int status;
waitpid(pid, &status, 0);

// Now you can read memory
long value = ptrace(PTRACE_PEEKDATA, pid, (void*)address, NULL);

// Read registers
struct user_regs_struct regs;
ptrace(PTRACE_GETREGS, pid, NULL, &regs);
printf("RIP: %llx, RSP: %llx\n", regs.rip, regs.rsp);

// Detach when done
ptrace(PTRACE_DETACH, pid, NULL, NULL);

Hint 2: Parsing /proc/[pid]/maps

The maps file format is well-defined. Parse it line by line:

// Open /proc/[pid]/maps
char path[64];
snprintf(path, sizeof(path), "/proc/%d/maps", pid);
FILE *f = fopen(path, "r");

char line[512];
while (fgets(line, sizeof(line), f)) {
    unsigned long start, end;
    char perms[5];
    char pathname[256] = "";

    // Format: start-end perms offset dev inode pathname
    sscanf(line, "%lx-%lx %4s %*x %*s %*d %255s",
           &start, &end, perms, pathname);

    printf("Region: %lx-%lx (%s) %s\n", start, end, perms, pathname);

    // Identify special regions
    if (strcmp(pathname, "[stack]") == 0) { /* it's the stack */ }
    if (strcmp(pathname, "[heap]") == 0) { /* it's the heap */ }
}

Hint 3: Walking the Stack

Stack walking using frame pointers:

struct user_regs_struct regs;
ptrace(PTRACE_GETREGS, pid, NULL, &regs);

unsigned long rbp = regs.rbp;
unsigned long rip = regs.rip;

printf("Current function: 0x%lx\n", rip);

// Walk the chain
while (rbp != 0) {
    // [RBP + 8] = return address
    // [RBP + 0] = saved RBP (previous frame's RBP)

    long ret_addr = ptrace(PTRACE_PEEKDATA, pid, (void*)(rbp + 8), NULL);
    long prev_rbp = ptrace(PTRACE_PEEKDATA, pid, (void*)rbp, NULL);

    printf("  Return to: 0x%lx (frame at 0x%lx)\n", ret_addr, rbp);

    rbp = prev_rbp;
}

Note: This only works if frame pointers are preserved (-fno-omit-frame-pointer).

Hint 4: Reading More Than One Word

PTRACE_PEEKDATA reads 8 bytes at a time. For efficiency with larger regions, use /proc/[pid]/mem:

char mem_path[64];
snprintf(mem_path, sizeof(mem_path), "/proc/%d/mem", pid);
int mem_fd = open(mem_path, O_RDONLY);

// Seek to the address you want to read
lseek(mem_fd, address, SEEK_SET);

// Read arbitrary amounts
char buffer[4096];
read(mem_fd, buffer, sizeof(buffer));

This is much faster than calling ptrace for each word.

Hint 5: Symbol Resolution

To get function names, you need to read the symbol table. Quick approach using external tools:

// Use addr2line for quick symbol resolution
char cmd[256];
snprintf(cmd, sizeof(cmd), "addr2line -f -e %s 0x%lx", executable_path, address);
FILE *p = popen(cmd, "r");
char func_name[128], file_line[256];
fgets(func_name, sizeof(func_name), p);
fgets(file_line, sizeof(file_line), p);
pclose(p);

For a proper implementation, parse the ELF symbol table (.symtab / .dynsym) yourself.

Hint 6: Interactive Mode with ncurses

For a nice TUI, consider ncurses:

#include <ncurses.h>

initscr();
noecho();
cbreak();

WINDOW *mem_win = newwin(20, 60, 0, 0);
WINDOW *regs_win = newwin(10, 60, 20, 0);
WINDOW *cmd_win = newwin(4, 60, 30, 0);

// Draw memory map in mem_win
// Draw registers in regs_win
// Get commands in cmd_win

int ch;
while ((ch = getch()) != 'q') {
    switch(ch) {
        case 's': single_step(); break;
        case 'c': continue_exec(); break;
        // ...
    }
    update_displays();
}

endwin();

Solution Architecture

High-Level Design

┌─────────────────────────────────────────────────────────────────────────────┐
│                              memviz                                          │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌────────────────┐     ┌────────────────┐     ┌────────────────┐           │
│  │   Process      │     │   Memory       │     │   Stack        │           │
│  │   Controller   │────▶│   Reader       │────▶│   Walker       │           │
│  │   (ptrace)     │     │ (/proc/[pid]/) │     │   (frames)     │           │
│  └────────────────┘     └────────────────┘     └────────────────┘           │
│         │                      │                      │                      │
│         ▼                      ▼                      ▼                      │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │                        Data Structures                                  │ │
│  │  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐   │ │
│  │  │MemoryRegion[]│ │StackFrame[] │ │Register      │ │SymbolTable   │   │ │
│  │  └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘   │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
│                                    │                                         │
│                                    ▼                                         │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │                         UI / Output                                     │ │
│  │  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐                    │ │
│  │  │ Text Output  │ │ ncurses TUI  │ │ JSON Export  │                    │ │
│  │  └──────────────┘ └──────────────┘ └──────────────┘                    │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Key Components

Component	Responsibility	Key Decisions
Process Controller	Attach/detach, single-step, continue	Uses ptrace; handles signals properly
Memory Reader	Read target’s memory efficiently	Uses /proc/[pid]/mem for bulk reads
Maps Parser	Parse memory layout	Parses /proc/[pid]/maps line by line
Stack Walker	Walk frames, extract locals	Uses RBP chain; handles missing frame pointers
Symbol Resolver	Address → function name	Uses addr2line or parses ELF directly
UI Layer	Display information clearly	Text mode default, ncurses optional

Data Structures

// Memory region from /proc/[pid]/maps
typedef struct {
    unsigned long start;
    unsigned long end;
    char perms[5];      // "rwxp" or "r-xp" etc
    unsigned long offset;
    char pathname[256];

    // Derived
    enum {
        REGION_TEXT,
        REGION_DATA,
        REGION_BSS,
        REGION_HEAP,
        REGION_STACK,
        REGION_SHARED_LIB,
        REGION_MMAP,
        REGION_UNKNOWN
    } type;
} MemoryRegion;

// Stack frame
typedef struct {
    unsigned long rbp;        // Frame pointer
    unsigned long rsp;        // Stack pointer (approximate end of frame)
    unsigned long ret_addr;   // Return address
    char func_name[128];      // Resolved function name
    char source_loc[256];     // file:line if available

    // Local variables (if debug info available)
    struct {
        char name[64];
        unsigned long offset_from_rbp;
        size_t size;
        unsigned char value[64];  // Raw bytes
    } locals[32];
    int num_locals;
} StackFrame;

// CPU registers
typedef struct {
    unsigned long rax, rbx, rcx, rdx;
    unsigned long rsi, rdi, rbp, rsp;
    unsigned long r8, r9, r10, r11;
    unsigned long r12, r13, r14, r15;
    unsigned long rip;
    unsigned long rflags;
    unsigned long cs, ss, ds, es, fs, gs;
} Registers;

// Main state
typedef struct {
    pid_t pid;
    char executable_path[PATH_MAX];

    MemoryRegion *regions;
    int num_regions;

    StackFrame *frames;
    int num_frames;

    Registers regs;

    int is_attached;
    int is_running;
} DebuggerState;

Algorithm Overview

Main Execution Flow:

Initialization
- Parse command line (attach vs launch)
- If launching: fork, child calls PTRACE_TRACEME, then exec
- If attaching: call PTRACE_ATTACH, wait for SIGSTOP
Memory Map Collection
- Open /proc/[pid]/maps
- Parse each line into MemoryRegion structure
- Classify regions by type (stack, heap, etc.)
Register Collection
- Call PTRACE_GETREGS
- Store in Registers structure
Stack Walking
- Start from current RBP
- For each frame:
  - Read return address at [RBP+8]
  - Read previous RBP at [RBP]
  - Resolve function name from return address
  - Calculate frame size
  - (Optional) Extract local variables from debug info
- Stop when RBP is 0 or invalid
Display
- Format memory map with aligned columns
- Show stack with nested indentation
- Highlight current instruction
- Show register values
Interactive Loop
- Accept user commands (step, continue, examine, quit)
- Update state after each command
- Refresh display

Implementation Guide

Development Environment Setup

# Required tools (Linux)
sudo apt-get install gcc gdb build-essential libncurses5-dev

# Optional but helpful
sudo apt-get install binutils-dev  # for libbfd (symbol parsing)

# Verify ptrace is allowed
cat /proc/sys/kernel/yama/ptrace_scope
# Should be 0 or 1 (0 = any process, 1 = parent only)

# If you get permission denied, temporarily allow:
# echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

# Create project structure
mkdir -p memviz/{src,include,tests}
cd memviz

Project Structure

memviz/
├── src/
│   ├── main.c              # Entry point, CLI parsing
│   ├── ptrace.c            # ptrace operations wrapper
│   ├── maps.c              # Parse /proc/[pid]/maps
│   ├── memory.c            # Read target memory
│   ├── stack.c             # Stack walking
│   ├── symbols.c           # Symbol resolution
│   ├── display.c           # Output formatting
│   └── interactive.c       # Command loop
├── include/
│   ├── memviz.h            # Main structures
│   ├── ptrace.h            # ptrace wrappers
│   └── display.h           # Output functions
├── tests/
│   ├── target_simple.c     # Simple test target
│   ├── target_recursive.c  # Recursive calls
│   └── test_maps_parser.c  # Unit test for maps parsing
├── Makefile
└── README.md

Implementation Phases

Phase 1: Basic Attachment (Days 1-3)

Goals:

Fork/exec a child process under ptrace control
Attach to an existing process
Read and display registers
Single-step one instruction

Tasks:

Implement ptrace_attach() and ptrace_detach() wrappers
Implement ptrace_launch() that forks, sets PTRACE_TRACEME, and execs
Read and print all registers
Implement single-step and continue

Checkpoint: Can launch ./target_simple, stop at first instruction, show registers, step once.

Phase 2: Memory Map Parsing (Days 4-6)

Goals:

Parse /proc/[pid]/maps completely
Classify all region types
Display formatted memory map

Tasks:

Open and read /proc/[pid]/maps
Parse each line into MemoryRegion structure
Identify [stack], [heap], [vdso], executables, libraries
Calculate sizes and percentages
Format and display the map

Checkpoint: Running on any process shows complete memory map with all regions identified.

Phase 3: Stack Walking (Days 7-10)

Goals:

Walk stack frames using RBP chain
Show return addresses
Resolve function names

Tasks:

Read current RBP from registers
Implement frame walking loop
Validate memory accesses (check if in stack region)
Use addr2line for symbol resolution
Display stack with proper formatting

Checkpoint: Shows correct call chain for recursive test program.

Phase 4: Memory Reading & Display (Days 11-13)

Goals:

Read arbitrary memory from target
Hexdump display
Show local variables (basic)

Tasks:

Implement bulk memory reading via /proc/[pid]/mem
Create hexdump function with ASCII display
Read local variables from stack (positions known from frame size)
Format combined display

Checkpoint: Can examine any memory region, show hex + ASCII.

Phase 5: Interactive Mode (Days 14-17)

Goals:

Command-driven interface
Step, continue, examine, quit
Watch memory changes

Tasks:

Implement command parser
Handle step, continue, examine commands
Refresh display after state changes
Add breakpoint support (stretch)
Polish output formatting

Checkpoint: Full interactive session works smoothly.

Phase 6: Polish & Extensions (Days 18-21)

Goals:

Error handling
Edge cases
Documentation
Optional: ncurses TUI

Tasks:

Handle all error cases (permission denied, process exited, etc.)
Test with various target programs
Write documentation
(Optional) Add ncurses interface

Checkpoint: Tool is robust and well-documented.

Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Memory reading	ptrace word-by-word vs /proc/[pid]/mem	/proc/[pid]/mem	Much faster for bulk reads
Symbol resolution	External tools vs parse ELF	External (addr2line) first	Simpler; can add ELF parsing later
UI	Plain text vs ncurses	Plain text first	Gets functionality working; add TUI later
Threads	Single-thread vs multi-thread aware	Start with single	Add thread enumeration as extension
Frame pointers	Require vs handle missing	Require first	Much simpler; note limitation

Testing Strategy

Test Categories

Category	Purpose	Examples
Unit Tests	Test parsing/formatting in isolation	Maps parser handles all line formats
Integration Tests	Test against real processes	Correctly walks 10-deep recursion
Regression Tests	Verify fixes don’t break	Compare output to baselines
Edge Cases	Handle unusual situations	Process exits mid-analysis, no debug info

Test Target Programs

Minimal:
```
int main() { return 0; }
```
Expected: Small memory map, one stack frame

Recursive:

void recurse(int n) { if (n > 0) recurse(n-1); }
int main() { recurse(10); return 0; }

Expected: 11 stack frames

With globals:

int init = 42;
int uninit;
const char *str = "hello";
int main() { return init + uninit; }

Expected: .data, .bss, .rodata all populated

With heap:

int main() {
    void *p1 = malloc(1024);
    void *p2 = malloc(4096);
    free(p1);
    // Stop here and examine
    pause();
}

Expected: Heap region shows allocations

Threaded (advanced):

void *thread_func(void *arg) { pause(); return NULL; }
int main() {
    pthread_t t;
    pthread_create(&t, NULL, thread_func, NULL);
    pause();
}

Expected: Multiple stacks visible

Critical Validation

After each phase, verify:

Attachment works: Process stops, doesn’t crash
Registers are correct: Compare with GDB output
Memory map matches: Compare with cat /proc/[pid]/maps
Stack walk is correct: Compare with GDB backtrace
Memory reads are accurate: Compare with GDB x command

Common Pitfalls & Debugging

Frequent Mistakes

Pitfall	Symptom	Solution
Not waiting after attach	Operations fail with ESRCH	Always `waitpid()` after PTRACE_ATTACH
Reading from wrong address	Garbage values	Verify address is in valid region first
Ignoring signals	Target crashes or hangs	Properly handle SIGTRAP, SIGSEGV
Frame pointer optimized away	Stack walk fails	Compile targets with `-fno-omit-frame-pointer`
Endianness confusion	Values look wrong	x86 is little-endian; least significant byte first
Not handling threads	Miss stacks	Each thread has its own stack

Debugging Strategies

Compare with GDB: Run GDB on the same process and compare outputs
Print syscall returns: Every ptrace call should be checked for errors
Use strace: strace ./memviz target shows what syscalls are made
Start simple: Get basic attachment working before adding features
Test on yourself: Attach to a known process like cat for predictable behavior

Platform-Specific Issues

Linux:

/proc/sys/kernel/yama/ptrace_scope may restrict attachment
Docker containers may have ptrace disabled
AppArmor/SELinux may block

macOS (if implementing):

No /proc filesystem
Use task_for_pid(), mach_vm_read(), mach_vm_region()
Requires entitlements or SIP disabled
Very different API, essentially a separate implementation

Extensions & Challenges

Beginner Extensions

Color output: Use ANSI colors for different region types
JSON export: Output memory map as JSON for other tools
Watch mode: Continuously update display as process runs
Memory diff: Show what changed between steps

Intermediate Extensions

DWARF parsing: Read debug info to get real variable names and types
Breakpoints: Set software breakpoints (INT3 injection)
Thread awareness: List all threads, show all stacks
Heap analysis: Intercept malloc/free to track allocations

Advanced Extensions

ncurses TUI: Full-screen interactive interface
macOS support: Implement using mach APIs
Expression evaluation: Parse expressions like $rsp + 0x10
Scripting: Allow Lua/Python scripts to automate analysis
Time-travel debugging: Record memory states for replay

Self-Assessment Checklist

Before considering this project complete, verify:

Understanding

I can explain virtual vs physical memory without notes
I can draw a process memory layout and explain each region
I understand how ptrace works and its security implications
I can explain how the stack grows and how frames are organized
I understand why the heap and stack grow toward each other

Implementation

Can attach to and detach from processes cleanly
Memory map parsing correctly identifies all regions
Stack walking shows correct call chain
Handles target process exit gracefully
Works on various test programs

Growth

I can now debug programs at the memory level
I understand what GDB does internally
I’m comfortable reading /proc filesystem entries
I can explain a segfault by examining memory layout

The Interview Questions They’ll Ask

After completing this project, you’ll be ready for these common interview questions:

“Explain the memory layout of a running process”
- They want: Stack, heap, text, data, bss, mmap regions; explain growth directions and purposes
- Bonus: Discuss virtual memory, page tables, and the role of the MMU
“How does a debugger work?”
- They want: ptrace for attachment, reading memory/registers, breakpoints via INT3, single-stepping
- Discuss: The parent-child relationship, signals (SIGTRAP), and register manipulation
“What happens when you dereference a NULL pointer?”
- They want: MMU translation fails, page fault, kernel sends SIGSEGV
- Discuss: Why low addresses are unmapped, guard pages, crash handling
“How does the stack work? Walk me through a function call.”
- They want: CALL pushes return address, RBP chain, local variables, calling convention
- Bonus: Explain buffer overflow attacks using this knowledge
“What’s the difference between stack and heap?”
- They want: Stack is automatic/fast/limited/LIFO; heap is dynamic/slower/flexible/fragmentation
- Discuss: When to use each, common bugs with each (overflow, leak)
“How would you debug a crash at address X?”
- They want: Check if X is in a valid region, examine memory map, check for NULL deref, use debugger
- Discuss: Core dumps, address sanitizer, valgrind

Real-World Connections

Industry Applications

Debuggers (GDB, LLDB): This is exactly how they work internally
Profilers (perf, valgrind): Use similar techniques for memory analysis
Security tools: Exploit development, forensics, malware analysis
Embedded systems: Memory-constrained debugging
Containers/VMs: Understanding isolation mechanisms

GDB: The standard debugger; study its architecture
strace/ltrace: System call and library call tracing
/proc interface: Linux kernel documentation on /proc
ptrace man page: Complete API documentation
Radare2/Ghidra: Advanced binary analysis

DWARF debug format: How debuggers know variable names
ELF format deep dive: Symbol tables, relocations
Memory allocators (jemalloc, tcmalloc): Heap internals
Operating system virtual memory: Page tables, TLB, swap

Resources

Essential Reading

CS:APP Chapter 9: “Virtual Memory” - The foundation for this project
CS:APP Chapter 7.9: “Loading Executable Object Files”
The Linux Programming Interface Chapter 26: “Monitoring Child Processes” (ptrace)
ptrace(2) man page: Complete API reference

Books That Will Help

Topic	Book	Chapter
Virtual Memory fundamentals	Computer Systems: A Programmer’s Perspective	Ch. 9
Process memory layout	Computer Systems: A Programmer’s Perspective	Ch. 7.9
The stack and calling conventions	Computer Systems: A Programmer’s Perspective	Ch. 3.7
ptrace and process tracing	The Linux Programming Interface	Ch. 26
Debugging techniques	Debug Hacks (O’Reilly Japan)	Ch. 1-3
ELF format details	Practical Binary Analysis	Ch. 2
Memory allocators	The Art of Exploitation	Ch. 3

Video Resources

LiveOverflow YouTube series on binary exploitation
pwn.college lectures on memory layout
MIT 6.828 OS course lectures on virtual memory

Tools & Documentation

proc(5): man proc - Documentation for /proc filesystem
ptrace(2): man ptrace - ptrace system call
gdb documentation: Understanding GDB internals
/proc/[pid]/maps: Format documentation in kernel docs

Submission / Completion Criteria

Minimum Viable Completion:

Can attach to a running process
Displays memory map with region types
Shows current registers
Basic stack walk with return addresses

Full Completion:

All basic functionality works
Stack frames show function names
Interactive mode with step/continue/examine
Clean error handling for all edge cases
Tested on multiple target programs

Excellence (Going Above & Beyond):

DWARF parsing for real variable names
ncurses TUI with real-time updates
Thread-aware multi-stack display
Breakpoint support
macOS port using mach APIs

This guide was expanded from CPU_ISA_ARCHITECTURE_PROJECTS.md. For the complete learning path, see the project index.

Project 7: Memory Visualizer & Debugger

Quick Reference

Learning Objectives

The Core Question You’re Answering

Concepts You Must Understand First

Key Concepts Deep Dive

Theoretical Foundation

Virtual Memory: The Great Illusion

Why Virtual Memory Exists

Process Memory Layout

Stack Frame Structure

ptrace: The Debugger’s Superpower

Virtual Memory Mapping (/proc/[pid]/maps)

The MMU and Page Tables

Why This Matters: Real-World Implications

Project Specification

What You Will Build

Functional Requirements

Non-Functional Requirements

Real World Outcome

Questions to Guide Your Design

Thinking Exercise

Hints in Layers

Solution Architecture

High-Level Design

Key Components

Data Structures

Algorithm Overview

Implementation Guide

Development Environment Setup

Project Structure

Implementation Phases

Phase 1: Basic Attachment (Days 1-3)

Phase 2: Memory Map Parsing (Days 4-6)

Phase 3: Stack Walking (Days 7-10)

Phase 4: Memory Reading & Display (Days 11-13)

Phase 5: Interactive Mode (Days 14-17)

Phase 6: Polish & Extensions (Days 18-21)

Key Implementation Decisions

Testing Strategy

Test Categories

Test Target Programs

Critical Validation

Common Pitfalls & Debugging

Frequent Mistakes

Debugging Strategies

Platform-Specific Issues

Extensions & Challenges

Beginner Extensions

Intermediate Extensions

Advanced Extensions

Self-Assessment Checklist

Understanding

Implementation

Growth

The Interview Questions They’ll Ask

Real-World Connections

Industry Applications

Related Open Source Projects

Related Topics to Explore

Resources

Essential Reading

Books That Will Help

Video Resources

Tools & Documentation

Submission / Completion Criteria