Sprint 1: Memory & Control - Real World Projects

Goal: Build a mental model of how memory actually works in a running C program and how control flow can be bent, broken, or defended. By the end of this sprint you will be able to reason about stack vs heap, read raw addresses, design safe APIs around unsafe primitives, and debug memory failures with confidence. You will also understand why these skills directly map to performance, security, and reliability in production systems.

Introduction

Memory & control in C means understanding how data is laid out, how code reaches that data, and what can go wrong when the program reads or writes the wrong bytes. In practice, it is the difference between software that is fast and predictable and software that leaks, crashes, or is exploited.

What you will build (by the end of this guide):

A memory inspector CLI that prints real addresses and process layout segments
A safe string library with explicit lengths and overflow protection
A memory leak detector that hooks allocation and produces leak reports
A custom arena allocator built on mmap() and alignment rules
A buffer overflow lab to understand exploitation and mitigations
A terminal text editor capstone that forces you to manage memory at scale

Scope (what is included):

Virtual memory layout, stack frames, heap allocation, pointers, C strings
Memory bugs, ownership models, and undefined behavior
Debugging with GDB/LLDB, AddressSanitizer, and Valgrind
Hardening and exploit mitigation basics (ASLR, NX, stack canaries)

Out of scope (for this guide):

Full compiler internals
OS kernel memory management
Advanced exploitation (heap feng shui, JOP, kernel ROP)

The Big Picture (Mental Model)

Source Code
   |
   v
Compiler + Linker
   |
   v
Executable (ELF/Mach-O)
   |
   v
Loader maps segments into process memory
   |
   v
Process runs in virtual address space
   |
   v
Stack + Heap + Mapped Regions + Code
   |
   v
Bugs and tools (ASan, Valgrind, GDB) observe behavior

Key Terms You Will See Everywhere

Virtual address space: The memory map the OS gives a process.
Stack frame: The layout of local variables and return state for a function.
Heap: Dynamically allocated memory returned by malloc().
Undefined behavior (UB): C behavior that the standard does not define, often leading to security bugs.
ASLR: Address Space Layout Randomization, which makes addresses unpredictable.

How to Use This Guide

Read the Theory Primer first. It is your mini-textbook. The projects assume those concepts.
Implement each project in order. The tools and mental models stack on each other.
Keep a lab notebook. Record addresses, outputs, and what surprised you.
Always run the same code through three lenses:
- Normal run (./program)
- Sanitized run (-fsanitize=address)
- Valgrind run (valgrind --leak-check=full)
Treat mistakes as the curriculum. Every crash is a lesson in how memory truly works.

Prerequisites & Background

Essential Prerequisites (Must Have)

Programming Skills:

Write and compile C programs with gcc or clang
Comfortable with functions, arrays, structs, and pointers
Basic command line usage (compile, run, read files)

Systems Fundamentals:

What a process is
What a stack and heap are at a high level
Basic familiarity with Linux/Unix shells

Recommended Reading:

“The C Programming Language” by Kernighan & Ritchie - Ch. 5 (Pointers and Arrays)
“Computer Systems: A Programmer’s Perspective” - Ch. 9 (Virtual Memory)

Helpful But Not Required

Assembly basics (helps with stack frames)
Linux /proc filesystem (helps with memory maps)
Security basics (helps with exploitation project)

Self-Assessment Questions

Can you explain what *p does if p is an int *?
Can you explain the difference between stack and heap lifetime?
Can you compile a program with -g and run it under GDB/LLDB?
Have you ever seen or fixed a segmentation fault?
Do you understand why strcpy is unsafe?

If you answered “no” to 1-3, spend a week reviewing K&R Chapter 5 and CSAPP Chapter 3.

Development Environment Setup

Required Tools:

Linux or macOS system (Linux preferred)
gcc or clang
gdb or lldb
make

Recommended Tools:

valgrind
readelf, objdump, nm
checksec (optional, for exploit project)

Testing Your Setup:

$ clang --version
$ gdb --version || lldb --version
$ valgrind --version

Time Investment

Small projects (1, 2): 6-10 hours each
Medium projects (3, 4): 10-20 hours each
Advanced project (5): 15-25 hours
Capstone (6): 20-40 hours

Important Reality Check

Memory mastery is slow because it rewires how you think. Expect to re-read concepts and re-run experiments. The point is not just to finish projects, but to build intuition for what each byte is doing.

Big Picture / Mental Model

           Your Code
              |
              v
         Compiler/Linker
              |
              v
        ELF/Mach-O Binary
              |
              v
 OS Loader maps segments
   (text, data, bss)
              |
              v
     Virtual Address Space
  +-------------------------+
  |   Stack (downwards)     |
  |-------------------------|
  |   Heap (upwards)        |
  |-------------------------|
  |   Mapped Libraries      |
  |-------------------------|
  |   Text (code)           |
  +-------------------------+
              |
              v
      Bugs (UB, overflow)
              |
              v
  Tools (GDB, ASan, Valgrind)

Theory Primer

Chapter 1: Virtual Memory and Process Layout

Fundamentals

Virtual memory is the illusion that every process owns a large, contiguous address space. The OS and the CPU’s memory management unit translate those virtual addresses into physical memory behind the scenes. This allows each process to run as if it is the only program on the machine, even though it is not. In C, every pointer you use is a virtual address, not a raw physical address. Understanding this abstraction is the foundation for everything in this sprint because the stack, heap, code, and mapped libraries are all just regions inside this virtual address space. The layout is predictable in structure (code near low addresses, stack near high addresses), but the exact addresses can change due to ASLR and loader choices.

Another important piece is that virtual memory is page-based. That means every address you see is really an offset within a fixed-size page (usually 4KB). The OS can map, unmap, and protect pages independently, which is why a single invalid access can crash a program even if nearby addresses are valid. This page granularity is also why memory debugging tools talk about pages, not just bytes.

Deep Dive into the Concept

Virtual memory is implemented using page tables. The CPU translates a virtual address into a physical one by walking these tables or consulting a cache called the TLB (translation lookaside buffer). Each virtual page maps to a physical page or triggers a page fault if it is not mapped. When a process starts, the OS loader maps the executable’s code and data segments into memory. It also maps shared libraries, allocates a stack, and prepares space for heap growth. The result is a process memory layout with distinct regions: text (code), data (initialized globals), bss (zero-initialized globals), heap (dynamic allocations), stack (function frames), and mapped regions (shared libraries, file mappings, anonymous mappings).

On Linux, you can see this layout in /proc/<pid>/maps. That file shows ranges of virtual addresses and the permissions associated with them (read, write, execute). Permissions matter because they enforce data vs code separation. The OS can prevent execution in writable regions (NX), and it can randomize region positions (ASLR). The allocator uses two different mechanisms to grow memory: brk()/sbrk() for the traditional heap region, and mmap() for large allocations or separate arenas. That means a pointer you get from malloc() might come from either region.

When your program accesses an address, the OS does not immediately allocate physical memory. Many mappings are created lazily. The first time you touch a page, you get a page fault, and the kernel allocates physical memory or loads the page from disk. This is why memory can appear “allocated” but still not consume physical RAM. It also explains why your program can crash with a SIGSEGV: the address was not mapped or the access violated permissions. Page faults are part of normal operation; they are how lazy allocation and copy-on-write work.

Copy-on-write (COW) is a critical optimization: when a process forks, the parent and child initially share the same physical pages marked read-only. When either writes, the OS copies that page, giving each process its own copy. This is why fork() is cheap until you modify memory. Understanding COW explains why large allocations or writes can suddenly spike memory usage. It also explains why memory leaks matter: they cause more pages to remain resident and increase the working set, reducing cache performance and increasing paging.

In practice, this means that every memory operation in your C program is mediated by the OS’s virtual memory subsystem. Debugging memory issues often starts by looking at mappings, permissions, and boundaries. If you free memory and then access it, the pointer still holds a virtual address, but that address may now point to an unmapped or re-used page. If ASLR is enabled, addresses will change from run to run, but the relative layout stays similar. Your memory inspector tool will rely on this concept to show you how the OS sees your process.

Another subtle but critical detail is permission enforcement. The kernel tracks permissions at the page level: read, write, execute, and sometimes shared/private flags. The loader sets these based on ELF segment headers. That is why code pages are typically r-x and why data pages are rw-. This separation prevents accidental self-modifying code and is a prerequisite for NX. Memory-mapped files add another layer: you can map a file as read-only and let the OS page data in on demand. Databases, search engines, and language runtimes use this to avoid copying large files into memory. When you map a file and then access beyond its length, you may see SIGBUS rather than SIGSEGV because the mapped file does not have backing storage for that page. Understanding this helps you debug mysterious crashes in file-backed memory.

How this fits on projects

You will use this to interpret /proc/self/maps, explain why stack addresses change between runs, and understand why arena allocators often use mmap() to carve dedicated regions.

Definitions & key terms

Virtual address: Address seen by your program.
Physical address: Actual RAM location.
Page: Fixed-size block of memory (often 4KB).
Page fault: Trap when accessing an unmapped or protected page.
VMA: Virtual Memory Area in Linux (range with permissions).

Mental model diagram

Virtual Address Space
+-----------------------------+  High
| Stack (rw-)                 |
|     |                       |
|     v  grows down           |
|-----------------------------|
| mmap regions (r-x, rw-)     |
|-----------------------------|
| Heap (rw-)                  |
|     ^ grows up              |
|-----------------------------|
| BSS (rw-)                   |
| Data (rw-)                  |
| Text (r-x)                  |
+-----------------------------+  Low

How it works (step-by-step, with invariants and failure modes)

Loader maps ELF segments into memory with permissions.
Kernel allocates a stack region and initializes the heap base.
Pages are mapped lazily; first access triggers page fault.
Address translation via page tables + TLB.
Access violations produce SIGSEGV or SIGBUS.

Invariant: Code pages are typically non-writable; data pages are typically non-executable.

Minimal concrete example

#include <stdio.h>
#include <stdlib.h>

int global = 42;        // Data segment
static int zero;        // BSS

int main() {
    int local = 7;      // Stack
    int *heap = malloc(4);
    printf("&global=%p &zero=%p &local=%p heap=%p\n",
           (void*)&global, (void*)&zero, (void*)&local, (void*)heap);
    free(heap);
}

Common misconceptions

“A pointer is a physical address.” -> It is virtual.
“The heap is one contiguous block.” -> It is often multiple mappings.
“Page faults mean failure.” -> Many are normal and expected.

Check-your-understanding questions

Why do stack addresses change between runs on modern Linux?
What is the difference between SIGSEGV and SIGBUS?
Why does malloc() sometimes return memory from mmap()?

Check-your-understanding answers

ASLR randomizes the stack base address.
SIGSEGV is invalid access; SIGBUS often indicates invalid alignment or access beyond mapped file length.
Large allocations or fragmentation may cause the allocator to use mmap().

Real-world applications

Memory-mapped databases
High-performance caching systems
Crash triage and core dump analysis

Where you’ll apply it

Project 1 (Memory Inspector Tool)
Project 4 (Arena Allocator)
Project 5 (Exploit Lab)

References

https://man7.org/linux/man-pages/man2/mmap.2.html
https://man7.org/linux/man-pages/man5/proc_sys_kernel.5.html
“Computer Systems: A Programmer’s Perspective” - Ch. 9

Key insight

Virtual memory is the OS-controlled map that makes every pointer meaningful or dangerous.

Summary

You now understand the basic layout of a process address space, how pages are mapped, and why permissions and randomization exist. This will let you reason about addresses you print in later projects.

Homework/Exercises to practice the concept

Print addresses of globals, locals, and heap allocations across three runs.
Read /proc/self/maps and annotate each region.

Solutions to the homework/exercises

You should see stack and heap addresses change on each run (ASLR).
The maps file should show text segments (r-x), data (rw-), stack (rw-), and mapped libraries.

Chapter 2: Stack Frames and Calling Conventions

Fundamentals

The stack is a per-thread memory region that stores function call state. Every time a function is called, the CPU pushes a return address and the function allocates a stack frame for local variables. When the function returns, the frame is discarded. This makes the stack fast and predictable, but also limited in size. Understanding the stack is critical because buffer overflows and return address corruption happen here. It is also where recursion depth, alignment, and calling conventions matter. In C, stack lifetime is automatic: if you return a pointer to a local variable, you are returning a pointer to memory that will soon be overwritten.

The stack is also per-thread. Each thread gets its own stack region and guard page. This matters for concurrency because stack overflows in one thread do not affect another, but deep recursion can still crash the process. Stack size is a resource you must respect just like heap memory.

Deep Dive into the Concept

Calling conventions define how functions receive arguments and return values. On x86-64 System V, the first few arguments go in registers (RDI, RSI, RDX, RCX, R8, R9), while additional arguments spill to the stack. The stack pointer (RSP) moves downward as the stack grows, and the stack must remain 16-byte aligned at call boundaries. The typical function prologue saves the old base pointer (RBP), sets up a new frame, and reserves space for locals. The epilogue restores RBP and returns by popping the return address into RIP.

Stack frames also contain saved registers, spill slots, and sometimes padding for alignment. If your function allocates a local array or uses alloca, that memory lives directly in the frame. If the array is too large, you risk stack overflow. If you write past the end of the array, you overwrite neighboring locals, saved registers, or the return address. This is the classic stack buffer overflow. Modern compilers insert a stack canary (via -fstack-protector), which places a guard value before the return address and checks it on function exit. If it changes, the program aborts. This is a software mitigation, not a guarantee.

The stack also interacts with interrupts, signals, and thread creation. Each thread has its own stack, which means stack size and guard pages matter. A guard page is an unmapped page placed below the stack to catch overflows. A large stack allocation can “jump” over the guard page, which is why stack-clash protection exists. When debugging, the backtrace you see is literally the chain of saved return addresses in stack frames. Corrupt one, and the backtrace becomes unreliable.

Optimizations can change stack layout. With -O2, the compiler may omit the frame pointer (-fomit-frame-pointer), inline functions, or keep variables in registers instead of the stack. This is why debugging optimized builds is harder. For learning, compile with -O0 -fno-omit-frame-pointer so the layout is stable. You will see how local variables appear at specific offsets from RBP, and how arguments are passed. This becomes essential in the exploit lab where you intentionally overwrite the return address.

Finally, stack frames encode lifetime: local variables are valid only inside the function. This is not just a semantic rule. It is enforced by the fact that the stack pointer moves when the function returns. Returning pointers to locals is undefined behavior because the memory will be reused for later calls. Understanding the stack teaches you why some bugs only appear under certain call patterns or recursion depths. The stack is also your first performance win: allocation is a single pointer move, and deallocation is free.

Calling conventions also define how variadic functions work. Functions like printf do not know how many arguments they will receive, so the caller must follow ABI rules precisely. The callee uses the format string to interpret the stack and registers. If you pass the wrong type, you create undefined behavior that often manifests as stack corruption or incorrect output. The ABI also defines which registers are caller-saved and callee-saved. If you violate that, you corrupt state across calls. These details are not trivia; they are the reason why memory corruption bugs can appear when you mix inline assembly, variadic calls, or nonstandard calling conventions. Finally, stack unwinding during exceptions or signals depends on consistent frame information, which is why debugging optimized code without frame pointers can be painful.

How this fits on projects

You will inspect stack frames in Project 1, rely on stack discipline to avoid buffer overflow in Project 2, and exploit stack corruption in Project 5.

Definitions & key terms

Stack frame: Memory region for one function call.
Calling convention: ABI rules for arguments/returns.
Prologue/Epilogue: Compiler-generated frame setup/teardown.
Stack canary: Guard value that detects overwrites.

Mental model diagram

Stack (grows down)
+--------------------+  High
| Return Address     |
| Saved RBP          |
| Local vars         |
| Padding/Alignment  |
+--------------------+  Low

How it works (step-by-step, with invariants and failure modes)

Caller places args (registers/stack).
CPU pushes return address.
Callee prologue sets up frame.
Locals live at fixed offsets from RBP.
Epilogue restores RBP and returns.

Invariant: Stack pointer alignment is preserved at call boundaries.

Minimal concrete example

#include <stdio.h>

void foo(int a, int b) {
    int local = a + b;
    printf("local=%d\n", local);
}

int main() {
    foo(1, 2);
}

Common misconceptions

“Stack memory is always zeroed.” -> It is not.
“Returning a pointer to a local is fine if I don’t use it later.” -> Still UB.
“The stack is infinite.” -> It is limited and guarded.

Check-your-understanding questions

Why do debuggers show cleaner backtraces with frame pointers?
What is the purpose of a stack canary?
Why can alloca() be dangerous?

Check-your-understanding answers

The frame pointer forms a stable chain of stack frames.
It detects stack buffer overwrites by checking a guard value.
It allocates on the stack and can bypass guard pages or exhaust stack space.

Real-world applications

Crash dump analysis
Security hardening
Performance tuning (recursion vs iteration)

Where you’ll apply it

Project 1 (Memory Inspector Tool)
Project 5 (Exploit Lab)
Project 6 (Mini Text Editor)

References

https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
“Computer Systems: A Programmer’s Perspective” - Ch. 3

Key insight

Stack frames are the physical record of control flow, and corruption here means control flow loss.

Summary

You now understand how function calls allocate memory, why stack memory is fast but fragile, and how calling conventions shape the layout you will inspect and exploit.

Homework/Exercises to practice the concept

Compile a simple program with -fno-omit-frame-pointer and examine the stack in GDB.
Create a local array and deliberately write past its end under a debugger.

Solutions to the homework/exercises

Use info frame and x/32gx $rsp in GDB to see the frame contents.
You should see nearby values (like saved registers or return address) change, often leading to a crash.

Chapter 3: Heap Allocation and Allocator Metadata

Fundamentals

The heap is the region of memory used for dynamic allocation via malloc, calloc, realloc, and free. Unlike stack memory, heap allocations persist until you free them. This gives you flexibility, but also responsibility. Allocators must track which blocks are free and which are in use. They do this by storing metadata alongside each allocation. If your program corrupts that metadata, the allocator itself becomes unreliable. Heap management is therefore both a performance and security issue.

Heap allocation is always a negotiation between speed, space, and safety. The allocator must keep data structures describing free blocks and in-use blocks, and your program must never corrupt those structures. This is why even a one-byte overflow can cause huge downstream failures in later allocations.

Deep Dive into the Concept

Most modern C runtimes implement a malloc-like allocator. On Linux, glibc uses ptmalloc, which divides memory into chunks. Each chunk has a header containing size fields and flags. When you call malloc(size), the allocator finds a free chunk of sufficient size, splits it if needed, and returns a pointer to the user area (just after the header). When you call free(ptr), the allocator marks the chunk as free and attempts to coalesce adjacent free chunks to reduce fragmentation.

Allocators maintain free lists or bins. Small allocations are often stored in size-segregated bins for fast lookup. Large allocations may come directly from mmap(). Recent glibc versions also use per-thread caches (tcache) to speed up malloc/free by avoiding global locks. This means the same allocation pattern can behave differently under multithreading and can influence memory reuse patterns that lead to use-after-free bugs.

Fragmentation is a major issue. Internal fragmentation happens when you receive a chunk larger than you requested due to alignment or bin sizes. External fragmentation happens when free space is split into many small pieces that cannot satisfy a large request. Allocator design is a trade-off between speed and fragmentation. For performance-critical systems, this is why custom allocators exist: they reduce fragmentation by allocating fixed-size blocks or using arena-style bulk allocation.

Heap allocations have alignment requirements. Most allocators align to 8 or 16 bytes. If you break alignment assumptions by casting to a stricter type, you can trigger undefined behavior on some architectures. This is why an allocator must return properly aligned pointers for any type.

Heap errors are common: double-free, invalid free, off-by-one writes, and use-after-free. These bugs can corrupt free lists, causing allocator crashes or, worse, controlled exploitation. AddressSanitizer detects many of these by using red zones and a shadow memory map. But sanitizers are not perfect; they introduce overhead and should not be used in production. Valgrind’s Memcheck uses dynamic binary instrumentation to detect invalid accesses and leaks, but it is much slower. Both tools are essential for finding allocator-related issues in this sprint.

Understanding heap allocation also means understanding the difference between malloc and mmap. Small allocations typically come from a heap region grown by brk() (the program break). Large allocations often use mmap() because they can be individually returned to the OS with munmap(). This is why you might see malloc return a pointer far away from other heap allocations.

Finally, the heap is where ownership decisions become real. When you return a pointer from a function, you transfer the obligation to free it. When you store heap pointers in data structures, you must decide who owns the memory. Leaks and double frees are both ownership failures.

The behavior of realloc is also important: it may return the same pointer (growing in place), or it may allocate a new block, copy data, and free the old one. This means any pointer aliases become invalid after realloc. Many bugs come from forgetting this. calloc is another special case: it zeroes memory, which can mask uninitialized-read bugs in debug builds and make them reappear in release builds. Some allocators provide trimming (returning unused heap pages to the OS). This can reduce memory footprint but may hurt performance if your program allocates again soon after. These are real trade-offs that systems programmers must weigh.

How this fits on projects

You will build a leak detector that tracks heap allocations, and an arena allocator that implements a simplified heap strategy. Understanding heap metadata is crucial for the exploit lab.

Definitions & key terms

Chunk: Allocator-managed block of memory.
Coalescing: Merging adjacent free blocks.
Fragmentation: Wasted memory due to allocation patterns.
Alignment: Pointer address must be multiple of specific value.

Mental model diagram

Heap
+------------------------+
| Header | User Data     |
+------------------------+
| Header | Free Block    |
+------------------------+
| Header | User Data     |
+------------------------+

How it works (step-by-step, with invariants and failure modes)

malloc(size) searches bins for a fit.
Chunk is split and marked in-use.
Pointer returned points after header.
free(ptr) marks chunk free and coalesces neighbors.

Invariant: Free list metadata must remain consistent or allocator crashes.

Minimal concrete example

#include <stdlib.h>
#include <string.h>

int main() {
    char *buf = malloc(16);
    strcpy(buf, "hello");
    free(buf);
    // buf is now dangling
}

Common misconceptions

“Freeing twice just does nothing.” -> It corrupts allocator metadata.
“Small allocations always come from one region.” -> Large ones use mmap().
“The heap is a single contiguous block.” -> It is not guaranteed.

Check-your-understanding questions

Why do allocators store metadata before user data?
What is the difference between internal and external fragmentation?
Why are large allocations often backed by mmap()?

Check-your-understanding answers

They need size and state to manage free lists and coalescing.
Internal is wasted inside a block; external is unusable space between blocks.
To allow independent mapping/unmapping and reduce heap fragmentation.

Real-world applications

High-performance servers
Game engines with custom allocators
Embedded systems with fixed memory pools

Where you’ll apply it

Project 3 (Memory Leak Detector)
Project 4 (Arena Allocator)
Project 5 (Exploit Lab)

References

https://man7.org/linux/man-pages/man2/mmap.2.html
“C Interfaces and Implementations” - Ch. 5-6
“Computer Systems: A Programmer’s Perspective” - Ch. 9

Key insight

The heap is flexible because it is managed, and that management is a major source of bugs and performance trade-offs.

Summary

You now understand allocator structure, metadata, and the root causes of heap-related bugs.

Homework/Exercises to practice the concept

Write a loop that allocates and frees blocks of varying sizes and record addresses.
Use valgrind --leak-check=full on a program with a deliberate leak.

Solutions to the homework/exercises

You should see reuse of freed addresses for same-size allocations.
Valgrind should report leaked blocks with stack traces.

Chapter 4: Pointers, Types, and Pointer Arithmetic

Fundamentals

Pointers are just numbers that refer to memory locations, but in C they are typed numbers. The type tells the compiler how many bytes to move when you do pointer arithmetic and how to interpret the bits at that address. This means pointer arithmetic is scaled by the size of the pointed-to type. Misunderstanding this is one of the most common sources of bugs in C. It is also why void * is special: it has no size, so you must cast it to a specific type before arithmetic.

Pointers can also point to pointers, which enables dynamic data structures but increases the chance of confusion about ownership and lifetime. Understanding how arrays decay to pointers when passed to functions is also essential: the array size information is lost, which is why bounds checking must be explicit.

Deep Dive into the Concept

In C, the difference between int * and char * is not just semantics. If p is an int *, then p + 1 means “advance by sizeof(int) bytes.” For char *, it means “advance by 1 byte.” This is the foundation of array indexing: arr[i] is defined as *(arr + i).

Pointers also encode aliasing relationships. The compiler assumes that pointers of different types do not alias the same memory (strict aliasing rule). Violating this rule can produce undefined behavior and surprising optimizations. For example, writing to memory through an int * and reading through a float * may be optimized incorrectly. The solution is to use memcpy or union carefully, or compile with -fno-strict-aliasing if needed.

Pointer alignment is another subtlety. Many architectures require that a pointer to a type be aligned to the type’s size. For example, a uint64_t * should be aligned to an 8-byte boundary. Misaligned access can be slower or even crash on some systems. Allocators therefore return pointers aligned for the most restrictive type. When you cast a char * to a uint64_t *, you are asserting that the address is aligned. If it is not, you have undefined behavior.

Pointer provenance is an emerging area: the C standard does not fully define how far you can move a pointer beyond the object it points to. The common rule is that you can compute one past the end, but you cannot dereference it. This matters in custom allocators and when using pointer arithmetic on raw memory buffers. It also matters for exploit lab work: attackers exploit the fact that pointer arithmetic and bounds checks are often absent.

Null pointers are also special. The null pointer is guaranteed not to point to any valid object. Dereferencing it is undefined behavior. But note that null is not guaranteed to be zero bits at the machine level (although in practice it usually is). This matters for low-level systems and cross-platform code. Also, malloc(0) can return either NULL or a unique pointer that must not be dereferenced but can be passed to free.

Finally, pointers to functions exist. They are used in callbacks, dispatch tables, and dynamic loading. Function pointers follow different rules for arithmetic and are a major ingredient in control flow. If you corrupt a function pointer in memory, you can redirect control flow even without touching a return address. This is a key concept in exploitation and in defensive coding.

Pointer comparison is also subtle. You can legally compare pointers that point into the same array object, but comparing pointers from different allocations is undefined behavior according to the C standard. In practice, most platforms implement a total ordering of addresses, but relying on it is non-portable. The restrict keyword is another pointer-related tool: it tells the compiler that two pointers do not alias, enabling aggressive optimizations. If you lie to the compiler about restrict, your program becomes undefined. This matters in performance-critical code like memcpy or image processing, where aliasing assumptions can change output correctness. Finally, pointer-to-array types (int (*)[N]) preserve size information, which is useful for safe APIs, but they are often misunderstood and avoided.

How this fits on projects

You will use pointer arithmetic in the memory inspector, in safe string operations, and in the arena allocator. You will also interpret pointers and their types in the exploit lab.

Definitions & key terms

Pointer arithmetic: Adding offsets scaled by type size.
Strict aliasing: Compiler assumption about non-overlapping types.
Alignment: Address must be multiple of type size.
Provenance: Which object a pointer is derived from.

Mental model diagram

int arr[3] = {10,20,30}
Address: 0x1000 0x1004 0x1008
Value:     10      20      30
Pointer p: 0x1000
p+1 => 0x1004
p+2 => 0x1008

How it works (step-by-step, with invariants and failure modes)

Pointer stores an address.
Type determines scaling for arithmetic.
Dereference reads/writes that memory.
Misaligned or out-of-bounds access causes UB.

Invariant: Pointer arithmetic must remain within the allocated object.

Minimal concrete example

int arr[3] = {10, 20, 30};
int *p = arr;
printf("%d %d %d\n", *p, *(p+1), *(p+2));

Common misconceptions

“Pointer arithmetic is byte-based.” -> It is type-based.
“Casting makes it safe.” -> Casting can hide UB.
“Out-of-bounds pointer arithmetic is harmless if you don’t dereference.” -> It can still be UB in some cases.

Check-your-understanding questions

Why does p+1 move by 4 bytes for int * on most systems?
What does strict aliasing allow the compiler to assume?
Why can misaligned access be dangerous?

Check-your-understanding answers

Because sizeof(int) is 4 bytes.
That different types do not point to the same memory.
Some CPUs fault or perform slower on misaligned access.

Real-world applications

Binary parsing
Memory-mapped file formats
Custom allocators and serializers

Where you’ll apply it

Project 1 (Memory Inspector Tool)
Project 2 (Safe String Library)
Project 4 (Arena Allocator)

References

“The C Programming Language” - Ch. 5
“Understanding and Using C Pointers” - Ch. 1-4

Key insight

Pointers are typed addresses, and the type is what makes them safe or unsafe.

Summary

You now understand how pointer arithmetic works, why alignment matters, and how aliasing can break assumptions.

Homework/Exercises to practice the concept

Write a program that walks a byte buffer using uint8_t * and uint32_t *.
Force a misaligned access and observe behavior on your system.

Solutions to the homework/exercises

You should see different increments and different interpretations of bytes.
Some systems will crash, others will work but be slower.

Chapter 5: C Strings and Buffer Safety

Fundamentals

A C string is an array of bytes terminated by a \0 (null terminator). There is no length stored alongside the string; instead, functions like strlen scan memory until they find the terminator. This is simple but dangerous: if the terminator is missing, the function will read past the buffer. Many memory vulnerabilities come from unsafe string handling. To be safe, you must track both length and capacity, or use bounded functions like snprintf and strnlen.

Because C strings do not store their length, every API call that manipulates them depends on an implied contract about buffer size. Once you see strings as just byte arrays with a sentinel, you realize why explicit length tracking is the only safe long-term strategy.

Deep Dive into the Concept

The C standard library provides classic string functions: strcpy, strncpy, strcat, strcmp, strlen, and friends. The problem is that many of these assume the destination buffer is large enough. strcpy does not check bounds. strncpy is safer but tricky: if the source is longer than the destination, it does not null-terminate. This means blindly using strncpy can still lead to non-terminated strings and later overreads. Similarly, strncat requires you to manage remaining capacity yourself.

A safe string library usually includes explicit length and capacity. For example, you can define a struct { char *data; size_t len; size_t cap; } and ensure every operation checks len < cap. This is the same pattern as the C++ std::string or Rust’s String. It forces you to separate allocated size from used length and provides a place to store that metadata. Another option is to use a length-prefixed string (first 4 or 8 bytes store length), which is common in databases and network protocols. The trade-off is compatibility with existing C APIs that expect null-terminated strings.

Buffer overflows often occur during input. Functions like gets are unsafe and removed from modern standards. Safer alternatives include fgets, getline, and read with explicit bounds. But you still need to manage the buffer size, check return values, and handle truncation. A robust string library provides operations like sbuf_append, sbuf_insert, sbuf_trim, sbuf_split, and sbuf_format. Each of these must handle growth, typically by doubling capacity. The growth strategy is also important for performance: too small grows too often, too large wastes memory.

Another source of bugs is mixing binary data with C strings. If a buffer contains zero bytes (like a binary file), string functions will stop early. This is why safe libraries separate binary buffers from text strings. Your safe string library should make this explicit, perhaps by providing buf_* for raw bytes and str_* for null-terminated strings.

Finally, encoding matters. UTF-8 strings are variable-length. If you assume 1 byte per character, your length calculations will break for non-ASCII. If your library is ASCII-only, document it. If you support UTF-8, implement functions that count code points and validate sequences. This is advanced, but even acknowledging it prevents hidden bugs.

The safer APIs in C have nuances. snprintf returns the number of bytes that would have been written, which lets you detect truncation. If you ignore that return value, you silently drop data. strlcpy and strlcat are widely used but non-standard; they are not available on all platforms. getline dynamically allocates and resizes buffers, which reduces overflow risk but shifts ownership responsibilities to the caller. These subtleties are why many security guidelines recommend writing thin wrappers with consistent semantics. For your library, choose one rule (e.g., always return error on truncation) and enforce it everywhere. That consistency is what prevents half-safe code.

Another common pitfall is confusing memcpy and memmove. memcpy assumes non-overlapping regions; if the source and destination overlap, the behavior is undefined and can corrupt data. memmove is safe for overlaps but slightly slower. Many string operations (like insertions or deletions) require memmove because you are shifting bytes within the same buffer. This detail is easy to miss and leads to subtle corruption bugs in editors and parsers.

How this fits on projects

Project 2 is about building a safe string library. You will use these ideas directly to design the API and to build safe append/insert operations.

Definitions & key terms

Null terminator: The \0 byte that marks end of string.
Capacity: Allocated size of the buffer.
Length: Number of bytes used (excluding terminator).
Truncation: Input exceeds capacity and is shortened.

Mental model diagram

struct s { char *data; size_t len; size_t cap; }

[H][e][l][l][o][\0][?][?]
 len=5 cap=8

How it works (step-by-step, with invariants and failure modes)

Allocate buffer with capacity.
Track current length.
Every write checks capacity.
Always maintain null terminator for text strings.

Invariant: len < cap and data[len] == '\0'.

Minimal concrete example

typedef struct {
    char *data;
    size_t len;
    size_t cap;
} sstr;

int sstr_append(sstr *s, const char *src) {
    size_t add = strlen(src);
    if (s->len + add + 1 > s->cap) return -1;
    memcpy(s->data + s->len, src, add + 1);
    s->len += add;
    return 0;
}

Common misconceptions

“strncpy always makes strings safe.” -> It can leave strings unterminated.
“strlen is O(1).” -> It scans until \0.
“Binary data can be handled with string functions.” -> It cannot.

Check-your-understanding questions

Why is strcpy unsafe?
Why can strncpy still be dangerous?
What invariant should a safe string maintain?

Check-your-understanding answers

It does not check the destination size.
It may not null-terminate if source is too long.
Length must be tracked, and capacity must not be exceeded.

Real-world applications

Secure network protocol implementations
Safer CLI tools
Libraries used in embedded systems

Where you’ll apply it

Project 2 (Safe String Library)
Project 6 (Mini Text Editor)

References

“Effective C” - Ch. 4 (Strings and arrays)
“The C Programming Language” - Ch. 5

Key insight

C strings are simple but dangerous; safe strings require explicit length tracking and strict invariants.

Summary

You now understand why C string APIs are risky and how to build a safer alternative.

Homework/Exercises to practice the concept

Write a function sstr_concat that refuses to overflow.
Implement sstr_trim to remove trailing whitespace safely.

Solutions to the homework/exercises

It should return an error if len + add + 1 > cap.
It should decrement len while last byte is whitespace and update the terminator.

Chapter 6: Memory Bugs, Ownership, and Undefined Behavior

Fundamentals

Most memory bugs are really ownership or lifetime bugs. If you free memory too early, you get use-after-free. If you free twice, you corrupt allocator state. If you never free, you leak. Undefined behavior is the C standard’s way of saying “anything can happen” when you break rules such as out-of-bounds access, uninitialized reads, or misaligned pointer use. Understanding these failure modes is crucial because they explain why a program can appear to work for months and then crash or become exploitable in production.

Ownership is a human contract. A good C codebase makes that contract explicit with naming conventions, comments, and API design. If you cannot clearly answer “who frees this?” for every pointer, the program is already broken, even if it still runs.

Deep Dive into the Concept

Ownership means answering “who is responsible for freeing this memory?” In C, this is a convention, not enforced by the compiler. Patterns like “caller frees” or “callee frees” must be documented. If you violate the contract, you leak or double free. Complex data structures (trees, graphs, lists) make this harder because ownership can be shared. One solution is reference counting, but that introduces its own bugs. Another is arena allocation, which frees everything at once. Each has trade-offs.

Undefined behavior is broader than memory safety. It includes integer overflow of signed values, shifting by out-of-range amounts, dereferencing invalid pointers, or reading uninitialized variables. Compilers exploit UB to optimize. That means UB can produce different results under different optimization levels or compilers. This is why a program might appear to work at -O0 but fail at -O2. Memory bugs are a subset of UB, and they can manifest as silent corruption rather than immediate crashes.

Use-after-free is especially dangerous because the pointer still looks valid. The memory may be reused by a later allocation, so your program will silently corrupt new data. Double free corrupts allocator metadata, and in some allocators it can be exploited to write arbitrary memory. Buffer overflows (both stack and heap) allow writes past the end of arrays, which can corrupt neighboring state. Off-by-one errors are common: if you forget to reserve space for the null terminator, you write one byte past the buffer and corrupt the next object.

Another class of bug is uninitialized memory. Reading uninitialized values can leak sensitive data or cause inconsistent behavior. This is why tools like Valgrind track definedness of memory. Similarly, integer overflow can cause buffer size miscalculations. For example, if you compute size = count * sizeof(T) without checking for overflow, the result may wrap around, causing malloc to allocate less than intended and leading to overflow on writes.

Ownership rules can be enforced with patterns: naming conventions (*_new returns owned pointer), helper functions (*_free), and explicit documentation. Unit tests and sanitizers help catch violations. But the real skill is mental modeling: always know the lifetime and owner of every pointer. This is what makes C both powerful and dangerous.

A practical way to manage ownership is the “create/destroy” pattern: every constructor-like function (foo_new) has a matching destructor (foo_free). You can also centralize cleanup with a single goto cleanup exit path, which makes it easier to ensure all resources are released on error. In large codebases, bugs often appear at boundaries: a function frees a buffer it did not allocate, or a caller assumes a callee will free something it will not. This is why good APIs document ownership explicitly. Testing helps, but design is the first line of defense.

Static analyzers can help enforce ownership by flagging paths where pointers are freed or left dangling. Tools like clang-tidy and static analyzers are not perfect, but they can catch mistakes that slip past unit tests. You should view them as another layer in the defense-in-depth stack alongside sanitizers and careful code review.

How this fits on projects

Project 3 (Leak Detector) directly exposes ownership rules. Project 5 (Exploit Lab) demonstrates how UB becomes exploitation. Project 6 forces you to manage ownership across multiple data structures.

Definitions & key terms

Use-after-free: Accessing memory after it is freed.
Double free: Freeing the same pointer twice.
Dangling pointer: Pointer to memory that is no longer valid.
Undefined behavior: Behavior not defined by the C standard.

Mental model diagram

Allocation -> Use -> Free
          \-> Use after free (BUG)
          \-> Free twice (BUG)
          \-> Never free (LEAK)

How it works (step-by-step, with invariants and failure modes)

Allocate memory.
Use it within bounds.
Free it exactly once.
Never touch it again.

Invariant: Every allocation has exactly one responsible owner.

Minimal concrete example

char *p = malloc(8);
strcpy(p, "hello");
free(p);
// BUG: use-after-free
printf("%s\n", p);

Common misconceptions

“Undefined behavior just means a crash.” -> It can be silent corruption.
“Leaks don’t matter if the OS cleans up on exit.” -> Long-running services suffer.
“If I set pointer to NULL after free, I’m safe.” -> Only safe if all aliases are cleared too.

Check-your-understanding questions

Why is use-after-free often worse than a crash?
How can integer overflow cause a buffer overflow?
Why do compilers optimize around undefined behavior?

Check-your-understanding answers

It can silently corrupt data and be exploited.
The size calculation wraps, allocating too little memory.
UB allows compilers to assume impossible cases for optimization.

Real-world applications

Security exploit analysis
Long-running servers and daemons
Debugging intermittent crashes

Where you’ll apply it

Project 3 (Memory Leak Detector)
Project 5 (Exploit Lab)
Project 6 (Mini Text Editor)

References

“Expert C Programming” - Ch. 2-3
“Effective C” - Ch. 7 (Undefined behavior)

Key insight

Most memory bugs are not about syntax; they are about ownership contracts and lifetimes.

Summary

You now understand the main classes of memory bugs, why they matter, and how undefined behavior amplifies risk.

Homework/Exercises to practice the concept

Write a program that intentionally leaks, double-frees, and uses after free, then run ASan.
Write a function that overflows due to integer multiplication and fix it.

Solutions to the homework/exercises

ASan should report each bug with stack traces and abort at the first.
Use checked multiplication or compare with SIZE_MAX / sizeof(T).

Chapter 7: Debugging and Instrumentation

Fundamentals

Debugging memory bugs requires visibility. GDB/LLDB let you inspect registers, memory, and stack frames. Sanitizers instrument your code to detect invalid accesses, and Valgrind instruments the binary at runtime. These tools are the difference between guessing and knowing. They also teach you what the program is actually doing rather than what you think it is doing.

Think of instrumentation as adding sensors to a machine. Without sensors, you only know it failed; with sensors, you know where and why. The earlier you add these sensors, the cheaper the fixes become.

Debugging is also about controlling the environment. Disabling optimizations, fixing random seeds, and reducing input size makes bugs reproducible. Reproducibility is what turns a mystery into a fixable problem.

Deep Dive into the Concept

Start with GDB/LLDB. Compile with -g and minimal optimization. You can break on malloc, free, or specific lines, then inspect memory with x/32bx or x/8gx. Watchpoints allow you to pause when a specific address is written. This is invaluable for tracking corruption. When a segmentation fault occurs, the backtrace (bt) shows you the call chain. If you enable frame pointers, the backtrace is more reliable.

AddressSanitizer (ASan) is a compiler-based tool that adds red zones around allocations and maintains a shadow memory map. It detects out-of-bounds accesses, use-after-free, and some stack errors. It is fast enough for everyday use but typically adds ~2x overhead. ASan terminates the program at the first detected error to prevent further corruption, which makes it great for early debugging but not ideal for long-running tests. The ASan runtime also supports options like detect_leaks=1 and detect_stack_use_after_return to catch more subtle bugs.

Valgrind’s Memcheck uses dynamic binary instrumentation. It does not require recompilation, but it is much slower (10x to 50x). Memcheck tracks definedness of memory and can detect uninitialized reads that ASan might miss. It also has a detailed leak checker. The downside is performance, so you use it on smaller test cases, not full workloads.

Another key technique is differential testing: run the same input under normal execution and under ASan or Valgrind, compare outputs. If outputs differ, you almost certainly have a memory bug. Also use ulimit -c unlimited to generate core dumps, then inspect them with GDB for post-mortem analysis.

Instrumentation is not only for bug detection. It shapes how you write code. When you know you will run ASan and Valgrind, you write more deterministic, testable code. You create smaller test cases. You isolate components. These are habits that scale far beyond memory debugging.

Finally, use compiler warnings as a form of instrumentation. Compile with -Wall -Wextra -Wshadow -Wconversion -Werror for stricter checking. Many memory bugs are caught at compile time if you enable these warnings. The combination of compiler warnings, sanitizer runs, and Valgrind is the best practical safety net for C programming.

Beyond ASan and Valgrind, modern toolchains include UBSan (Undefined Behavior Sanitizer) and LSan (LeakSanitizer). UBSan catches issues like integer overflow, misaligned access, and invalid shifts. LSan focuses specifically on leaks and can run alongside ASan in many configurations. Integrating these tools into your build (even for a subset of tests) gives you a continuous safety net. Another powerful debugging technique is to generate a core dump and use addr2line to map instruction addresses back to source. This is the workflow used in production incident response when no debugger was attached.

For recurring bugs, create a minimal repro case and commit it as a regression test. This practice prevents the same memory bug from reappearing later. It also teaches you how to shrink complex failures into simple, understandable inputs, which is one of the most valuable debugging skills in systems programming.

One more practical tip: always log the exact build flags and tool versions you used when reporting a memory bug. Small differences in compiler version or sanitizer runtime can change behavior, and a reproducible environment is often the fastest path to a fix.

How this fits on projects

You will use these tools in every project. They are mandatory for verifying correctness in the leak detector and text editor.

Definitions & key terms

ASan: AddressSanitizer, compiler-based memory error detector.
Valgrind Memcheck: Binary instrumentation for memory checking.
Watchpoint: Debugger trap on memory write.
Core dump: Snapshot of process memory after crash.

Mental model diagram

Program -> Compiler w/ ASan -> Instrumented binary
         -> Run -> ASan runtime -> Crash report

Program -> Run under Valgrind -> Emulated CPU -> Report

How it works (step-by-step, with invariants and failure modes)

Compile with -g -O1 -fsanitize=address.
Run program; ASan intercepts memory accesses.
On invalid access, ASan prints stack trace and aborts.
Valgrind runs program under instrumentation and logs errors.

Invariant: Reports point to the first invalid access, not necessarily the root cause.

Minimal concrete example

clang -O1 -g -fsanitize=address -fno-omit-frame-pointer main.c -o main
./main
valgrind --leak-check=full ./main

Common misconceptions

“Valgrind is always right.” -> It can have false positives.
“ASan finds everything.” -> It does not catch all bugs (e.g., uninitialized reads).
“Debuggers slow you down too much.” -> They save time overall.

Check-your-understanding questions

Why does ASan abort immediately on error?
What kinds of bugs can Valgrind catch that ASan might miss?
Why should you compile with frame pointers during debugging?

Check-your-understanding answers

Because memory corruption makes subsequent behavior unreliable.
Uninitialized reads and some undefined value flows.
Frame pointers produce stable backtraces.

Real-world applications

Debugging production crashes
Security audits of legacy code
Testing C libraries in CI

Where you’ll apply it

Projects 1-6 (all)

References

https://clang.llvm.org/docs/AddressSanitizer.html
https://valgrind.org/docs/manual/quick-start.html

Key insight

Visibility is the real power tool; debugging is a skill you can systematize.

Summary

You now know how to use debuggers and instrumentation to turn mysterious crashes into actionable fixes.

Homework/Exercises to practice the concept

Run a small program under ASan and Valgrind and compare outputs.
Use a watchpoint to detect which line overwrites a variable.

Solutions to the homework/exercises

ASan will report the first invalid access; Valgrind may report multiple issues.
In GDB: watch *ptr and run until it triggers.

Chapter 8: Exploit Mitigations and Hardening

Fundamentals

Exploit mitigations are defenses that make memory bugs harder to turn into control-flow hijacks. They do not prevent bugs; they reduce the chance that a bug becomes an exploit. The most common mitigations are ASLR (randomizing addresses), NX (making data non-executable), stack canaries (detecting overwrites), PIE (position-independent executables), RELRO (read-only relocation data), and FORTIFY (bounds-checked libc calls). Understanding these is critical for the exploit lab.

Hardening is about defense in depth. No single mitigation is perfect, but the combination of multiple mitigations can make exploitation so expensive that attackers move on to softer targets.

Mitigations are most effective when they are combined with secure coding and testing. Think of them as airbags, not brakes: they help, but you still must steer carefully.

Deep Dive into the Concept

A classic stack buffer overflow works by overwriting the return address. Without mitigations, an attacker can write the address of injected shellcode and hijack execution. NX (no-execute) breaks this by marking data pages as non-executable. The CPU will fault if you try to execute code from the stack or heap. This forces attackers to reuse existing code (return-to-libc or ROP).

ASLR makes those code addresses unpredictable by randomizing the location of stack, heap, libraries, and sometimes the executable itself. On Linux, the randomize_va_space sysctl controls this policy. When ASLR is enabled, the same binary has different addresses on every run, which breaks static exploit offsets. Attackers often need info leaks to bypass ASLR. PIE (position independent executables) ensures the main binary itself is also relocatable, so its base address is randomized too.

Stack canaries are a compiler feature. A random guard value is placed between local buffers and control data. Before returning, the function checks if the guard changed. If it did, the program aborts. This doesn’t prevent overflows, but it detects them before control flow is hijacked. GCC provides -fstack-protector and stronger variants. However, canaries can be bypassed if attackers can leak their value or exploit heap overflows instead of stack overflows.

RELRO (Relocation Read-Only) makes sections like the Global Offset Table (GOT) read-only after relocation, preventing certain overwrites. Full RELRO is stronger but slower at startup. FORTIFY_SOURCE adds checks to certain libc functions when the compiler can determine buffer sizes. It provides extra protection against trivial overflows, but it is not a substitute for correct code.

The security landscape is a layered defense. Modern exploitation often uses a chain: find a bug, bypass ASLR with an info leak, use ROP to bypass NX, and target a non-hardened binary. This is why your exploit lab should explicitly toggle mitigations on and off and observe how they change exploit difficulty. Understanding mitigations also helps you write safer code: you know which failures are catastrophic and which are likely to be detected.

Modern CPUs add additional defenses like CET (Control-flow Enforcement Technology), which introduces shadow stacks and indirect branch tracking. These make ROP and JOP attacks more difficult, but they are not universally deployed. Another defense is stack-clash protection, which ensures large stack allocations cannot skip guard pages. These mitigations illustrate an important point: hardening is a moving target. Attackers adapt, so defenders layer mitigations, use safe languages where possible, and instrument code heavily. Your exploit lab is about understanding both the old vulnerabilities and the modern defenses that evolved to contain them.

In modern build pipelines, hardening flags are often standardized: -fstack-protector-strong, -D_FORTIFY_SOURCE=2, -Wl,-z,relro,-z,now, and PIE flags are enabled by default in many distros. Knowing these defaults helps you interpret why an exploit that works in a lab fails in production. It also teaches you to read binary security metadata before making assumptions about exploitability.

Even with mitigations, memory safety bugs remain costly because they can be chained with logic flaws or configuration mistakes. Understanding mitigations therefore improves both your defensive mindset and your ability to assess real risk in production systems.

How this fits on projects

Project 5 will ask you to bypass or observe mitigations. Projects 2 and 3 benefit from understanding how buffer overflows and double frees lead to exploitation.

Definitions & key terms

ASLR: Address Space Layout Randomization.
NX: Non-executable memory protection.
Stack canary: Guard value to detect stack smashing.
PIE: Position Independent Executable.
RELRO: Read-only relocation sections.

Mental model diagram

Without defenses:
[Buffer][Return Addr] -> overwrite -> jump to shellcode

With defenses:
[Buffer][Canary][Ret] -> canary check fails -> abort
NX -> injected code cannot execute
ASLR -> address unpredictable

How it works (step-by-step, with invariants and failure modes)

Compiler inserts canary and checks on return.
Loader randomizes addresses (ASLR/PIE).
OS enforces NX on data pages.
Relocations locked down with RELRO.

Invariant: Mitigations reduce exploit reliability, not bug existence.

Minimal concrete example

# Compile with stack protection and PIE
clang -fstack-protector-strong -fPIE -pie vuln.c -o vuln

# Inspect protections (if checksec available)
checksec --file=./vuln

Common misconceptions

“ASLR stops all exploits.” -> Info leaks can bypass it.
“NX means no exploits.” -> ROP can still work.
“Stack canaries make code safe.” -> Heap overflows remain.

Check-your-understanding questions

Why does PIE matter for ASLR?
What does NX actually prevent?
Why is RELRO useful?

Check-your-understanding answers

It allows the main binary to be relocated like shared libraries.
Execution of injected code in data pages.
It prevents overwriting GOT entries for function hijacking.

Real-world applications

Secure build pipelines
Vulnerability analysis
Hardening legacy C services

Where you’ll apply it

Project 5 (Exploit Lab)
Project 2 (Safe String Library)

References

https://man7.org/linux/man-pages/man5/proc_sys_kernel.5.html
https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
https://en.wikipedia.org/wiki/NX_bit

Key insight

Mitigations change the economics of exploitation, not the existence of bugs.

Summary

You now understand the most common exploit defenses and how they raise the bar for attackers.

Homework/Exercises to practice the concept

Compile a simple overflow program with and without -fstack-protector and observe behavior.
Toggle ASLR with sysctl (on a non-production system) and observe address changes.

Solutions to the homework/exercises

With stack protector, the program should abort with “stack smashing detected”.
With ASLR off, addresses should remain stable between runs.

Chapter 9: Custom Allocators and Arenas

Fundamentals

A custom allocator is a memory manager tailored to a specific workload. An arena allocator (also called a bump allocator) is the simplest form: it allocates a big block up front and hands out memory by moving a pointer forward. Freeing individual objects is not supported; instead, you reset or destroy the entire arena at once. This is extremely fast and avoids fragmentation, which makes it a powerful tool for systems programming.

Allocators also influence cache behavior. By allocating related objects contiguously, you improve spatial locality, reduce cache misses, and speed up traversal-heavy workloads. That performance win is often as important as the reduced fragmentation.

Deep Dive into the Concept

General-purpose allocators like malloc must handle many workloads. They are optimized for average cases, which can be inefficient for specific patterns. If your program allocates many small objects with similar lifetimes (e.g., parsing a file, building a request), an arena allocator is perfect. You allocate one big chunk (often with mmap) and then bump a pointer for each allocation. Deallocation is a single pointer reset. This is fast and predictable. The trade-off is that you cannot free individual objects early; you must accept bulk lifetime.

Arena allocators must handle alignment. Suppose you have a current pointer p and you need to allocate an object with alignment a. You compute aligned = (p + a-1) & ~(a-1) to round up. You then advance the pointer to aligned + size. If the arena runs out of space, you either allocate a new arena or return NULL. Many implementations support chained arenas: when one fills, allocate another and link it. A reset operation can free all arenas at once.

Beyond bump allocators, there are pool allocators (fixed-size blocks) and slab allocators (caches of objects by type). These reduce fragmentation and improve cache locality. The design decision is based on object size variability and lifetime. Pools are great for fixed-size objects; slabs for type-specific caches; arenas for bulk lifetime. The best allocator is the one that matches your allocation pattern.

When you write a custom allocator, you must also expose diagnostics: current capacity, bytes used, allocations count, and high-water mark. These metrics let you verify that your allocator behaves as expected. You also need a strategy for error handling: if an allocation fails, do you return NULL, abort, or allocate a new arena? This should be explicit in your API.

One subtlety is that mmap returns page-aligned memory and allows you to reserve large address ranges without touching physical memory. That means arenas can be large without immediately consuming RAM. But the first write to each page still incurs a page fault. Measuring this cost is part of the arena project: you should see the trade-off between speed (fast allocations) and initialization cost (touching pages).

Custom allocators are not only for performance. They are also security tools. By controlling allocation patterns, you can reduce fragmentation and make exploitation harder. You can also use arenas to simplify ownership: if all objects in a parse belong to a request, you can free them all at once without tracking each pointer.

If you make your allocator thread-safe, you must decide how to synchronize access. A global lock is simple but can become a bottleneck. Thread-local arenas reduce contention but can increase memory usage if many threads allocate infrequently. Some systems use a hybrid approach with per-thread caches and a shared global pool. These are advanced designs, but understanding the trade-offs will make you better at interpreting real-world allocator behavior when you debug memory issues.

Debug features are also valuable. A debug build of an allocator can fill new allocations with a known pattern (like 0xAA) and freed allocations with another (like 0xDD). This makes use-after-free and uninitialized reads much easier to spot during development.

How this fits on projects

Project 4 is an arena allocator. You will implement alignment, tracking, and reset semantics. The text editor capstone can optionally use arenas for line storage.

Definitions & key terms

Arena/Bump allocator: Allocation by pointer bump.
Pool allocator: Fixed-size blocks.
Slab allocator: Type-based cache of objects.
High-water mark: Maximum memory used.

Mental model diagram

Arena
+-------------------------------+
| used | used | used | free ... |
+-------------------------------+
          ^ bump pointer

How it works (step-by-step, with invariants and failure modes)

Allocate large block with mmap or malloc.
Set cursor = base.
For each allocation, align cursor and advance.
If out of space, fail or chain a new arena.
Reset by setting cursor = base.

Invariant: Allocations are monotonically increasing.

Minimal concrete example

typedef struct {
    char *base;
    size_t size;
    size_t offset;
} arena;

void *arena_alloc(arena *a, size_t sz, size_t align) {
    size_t p = (size_t)(a->base + a->offset);
    size_t aligned = (p + align - 1) & ~(align - 1);
    size_t new_offset = (aligned - (size_t)a->base) + sz;
    if (new_offset > a->size) return NULL;
    a->offset = new_offset;
    return (void *)aligned;
}

Common misconceptions

“Arenas are always better than malloc.” -> Only when lifetimes are aligned.
“Alignment is optional.” -> Misalignment can cause crashes on some CPUs.
“Arenas leak memory.” -> They trade fine-grained free for bulk reset.

Check-your-understanding questions

Why do arenas reduce fragmentation?
Why is alignment necessary in a custom allocator?
When is an arena a bad choice?

Check-your-understanding answers

They allocate linearly and never free individual blocks.
Many types require aligned addresses.
When objects have independent lifetimes or need precise freeing.

Real-world applications

Game engines
Compilers and parsers
High-throughput servers

Where you’ll apply it

Project 4 (Arena Allocator)
Project 6 (Mini Text Editor) (optional optimization)

References

https://man7.org/linux/man-pages/man2/mmap.2.html
“C Interfaces and Implementations” - Ch. 5-6

Key insight

Custom allocators turn memory management into a design choice rather than a hidden cost.

Summary

You now understand how to design and implement a simple arena allocator and why it can outperform general-purpose malloc.

Homework/Exercises to practice the concept

Implement an arena that can be reset and reused across multiple tasks.
Add a high-water mark counter and print it after program exit.

Solutions to the homework/exercises

Reset is simply offset = 0 if the arena is a single block.
Track max offset reached and print it in a summary report.

Glossary

ASLR: Randomizes memory layout to make exploits harder.
ABI: Application Binary Interface, defines calling conventions.
BSS: Memory region for zero-initialized globals.
Canary: Guard value to detect stack smashing.
Heap: Dynamically allocated memory region.
Stack: Memory region for call frames and locals.
UB: Undefined Behavior, results not defined by the C standard.

Why Memory & Control Matters

The Modern Problem It Solves

In modern systems, performance and security both depend on correct memory handling. A single out-of-bounds write can lead to data corruption or remote code execution. This is why memory safety is one of the highest priority issues in modern infrastructure.

Real-world impact (recent data):

Memory safety dominates vulnerability counts: The NSA noted in 2022 that Microsoft and Google report about 70% of their vulnerabilities are caused by memory safety issues. (NSA press release, 2022)
Vulnerability rates are dropping with memory-safe code: Google reported that Android memory safety vulnerabilities dropped from 76% in 2019 to 35% in 2022, and later reported a drop in raw counts from over 220 in 2019 to a projected 36 in 2024, alongside a shift to memory-safe languages. (Google Security Blog, 2022, 2024)
Linux remains the dominant server OS: W3Techs reported that Linux powers about 59.8% of websites with known OS as of January 1, 2026. (W3Techs, 2026)

OLD APPROACH                      MODERN REALITY
+----------------------+          +---------------------------+
| "It works"           |          | "It must be safe"         |
| Minimal testing      |          | Sanitizers + fuzzing       |
| Debug after crash    |          | Debug before release       |
+----------------------+          +---------------------------+

Context & Evolution

C was designed to give developers direct control of memory for efficiency. That power enabled the operating systems and high-performance software of the last 50 years. Today, that same power makes memory safety the hardest class of bugs to eradicate. This sprint teaches you how to work safely in that world.

Concept Summary Table

Concept	What You Need to Internalize
Virtual Memory & Process Layout	How OS maps code/data/stack/heap and how permissions affect access
Stack Frames & Calling Conventions	How function calls create stack frames and why overflows matter
Heap Allocation & Metadata	How allocators manage memory and why metadata corruption is dangerous
Pointers & Arithmetic	How type controls pointer math, alignment, and aliasing
C Strings & Buffer Safety	Why null-terminated strings are risky and how to enforce invariants
Memory Bugs & Ownership	How leaks, double frees, UAF, and UB happen in practice
Debugging & Instrumentation	How to use GDB/LLDB, ASan, Valgrind to find bugs
Exploit Mitigations & Hardening	How ASLR, NX, canaries, PIE, RELRO reduce exploitability
Custom Allocators & Arenas	How to build a fast allocator tuned to a workload

Project-to-Concept Map

Project	What It Builds	Primer Chapters It Uses
Project 1: Memory Inspector Tool	A process memory map explorer	1, 2, 4
Project 2: Safe String Library	Safe bounded string API	4, 5, 6
Project 3: Memory Leak Detector	Allocation tracker + leak report	3, 6, 7
Project 4: Arena Allocator	Custom allocator with alignment	3, 9
Project 5: Exploit Lab	Controlled overflow + mitigations	2, 6, 8
Project 6: Mini Text Editor	Real system with complex memory use	4, 5, 6, 7, 9

Deep Dive Reading by Concept

Fundamentals & Architecture

Concept	Book & Chapter	Why This Matters
Virtual Memory	Computer Systems: A Programmer’s Perspective - Ch. 9	Canonical explanation of VM and paging
Process Memory Layout	Operating Systems: Three Easy Pieces - Ch. 14-16	OS-level view of address spaces
Stack Frames	Computer Systems: A Programmer’s Perspective - Ch. 3	Machine-level view of calls

C Language & Memory

Concept	Book & Chapter	Why This Matters
Pointers	The C Programming Language - Ch. 5	Foundational pointer semantics
Strings & Arrays	Effective C - Ch. 4	Safe handling patterns
Undefined Behavior	Expert C Programming - Ch. 2-3	Common pitfalls and UB

Debugging & Security

Concept	Book & Chapter	Why This Matters
Debugging	The Art of Debugging with GDB - Ch. 1-4	Practical debugging workflows
Exploit Mitigations	Practical Binary Analysis - Ch. 5-7	Modern binary security features
Allocators	C Interfaces and Implementations - Ch. 5-6	Custom allocation strategies

Quick Start

Your First 48 Hours

Day 1 (4 hours):

Read Chapters 1, 2, and 4 in the Theory Primer.
Compile a tiny C program and print stack/heap addresses.
Install and run AddressSanitizer on a trivial buffer overflow.
Start Project 1 and print /proc/self/maps.

Day 2 (4 hours):

Finish Project 1 with readable output and annotations.
Start Project 2: implement sstr_init and sstr_append.
Run valgrind --leak-check=full on Project 2 tests.

End of 48 hours: You should be able to explain where your data lives in memory and why unsafe string handling is dangerous. The rest of the sprint is deepening that intuition.

Recommended Learning Paths

Path 1: The Systems Programmer (Recommended)

Project 1 -> Project 2 -> Project 3 -> Project 4 -> Project 6
Add Project 5 after Project 2 if you are security curious.

Path 2: The Security Track

Project 1 -> Project 2 -> Project 5 -> Project 3
Finish with Project 6 to test your skills in a real system.

Path 3: The Performance Track

Project 1 -> Project 4 -> Project 3
Add Project 2 only as needed, then build Project 6 with arenas.

Path 4: The Completionist

Projects 1-4 in order
Project 5 for exploit intuition
Project 6 as capstone

Success Metrics

You can draw the process memory layout without looking it up.
You can explain why a given memory bug is stack vs heap.
You can fix a segmentation fault using GDB in under 30 minutes.
Your projects run clean under ASan and Valgrind.
You can explain how ASLR and NX change exploitability.

Appendix: Tooling & Debugging Cheat Sheet

# Compile with debug + ASan
clang -O1 -g -fsanitize=address -fno-omit-frame-pointer file.c -o file

# Valgrind memory check
valgrind --leak-check=full --show-leak-kinds=all ./file

# GDB basics
(gdb) break main
(gdb) run
(gdb) bt
(gdb) x/32bx $rsp
(gdb) watch *ptr

Project Overview Table

#	Project	Core Topics Covered	Difficulty
1	Memory Inspector Tool	Process layout, pointers, stack/heap, `/proc/self/maps`	Intermediate
2	Safe String Library	C strings, bounds checking, safe APIs	Intermediate
3	Memory Leak Detector	malloc/free tracking, leak reports, ownership	Intermediate
4	Arena Allocator	custom allocators, alignment, mmap	Intermediate
5	Exploit Lab (Buffer Overflow)	stack frames, mitigation bypass, ASLR/NX/canaries	Advanced
6	Mini Text Editor (Capstone)	complex memory management, data structures, terminal I/O	Advanced

Project List

Project 1: Memory Inspector Tool

Main Programming Language: C
Difficulty: Intermediate
Knowledge Area: Process memory layout, pointers, /proc
Main Book: “Computer Systems: A Programmer’s Perspective”

What you’ll build: A CLI that prints a live memory map of your process, highlights stack/heap/code regions, and shows example addresses for variables, functions, and heap allocations.

Why it teaches memory & control: You will map abstract concepts into real addresses and learn how the OS actually arranges your program in memory.

Core challenges you’ll face:

Parsing /proc/self/maps reliably
Printing addresses with correct formatting
Explaining why addresses move across runs (ASLR)

Real World Outcome

When you run your tool, you should see a readable memory map plus a snapshot of actual addresses:

$ ./mem_inspect
== Memory Inspector ==
PID: 32145

[Segments]
0x00400000-0x00452000 r-xp  /home/user/mem_inspect
0x00652000-0x00654000 r--p  /home/user/mem_inspect
0x00654000-0x00656000 rw-p  /home/user/mem_inspect
0x7f3a8c000000-0x7f3a8c021000 rw-p  [heap]
0x7ffd9d9f9000-0x7ffd9da1a000 rw-p  [stack]

[Addresses]
&global_var  = 0x00655124  (data)
&static_var  = 0x00655128  (bss)
&local_var   = 0x7ffd9da18c4c (stack)
heap_alloc   = 0x7f3a8c001260 (heap)
func main()  = 0x004012a0 (text)

The Core Question You’re Answering

“Where does my program actually live in memory, and how can I prove it?”

This project turns every abstract memory region into a real address you can see and reason about.

Concepts You Must Understand First

Virtual Memory Layout
- What does /proc/self/maps show?
- Why does ASLR change addresses across runs?
- Book Reference: CSAPP Ch. 9
Pointers and Formatting
- Why does %p require (void*)?
- What is the difference between &var and var?
- Book Reference: K&R Ch. 5
Stack vs Heap
- How can you identify stack and heap addresses?
- Book Reference: OSTEP Ch. 14-16

Questions to Guide Your Design

Input and Parsing
- How will you read /proc/self/maps safely?
- How will you parse permissions and labels?
Output Formatting
- How will you display ranges in a human-readable way?
- How will you color or annotate regions?
Address Collection
- Which variables should you print to demonstrate each region?
- How will you prove a function address is in the text segment?

Thinking Exercise

Trace the address of a local variable:

void foo() {
    int x = 42;
    printf("%p\n", (void*)&x);
}

Questions:

Why is x close to the top of the stack?
How does calling foo() multiple times change the address?

The Interview Questions They’ll Ask

“What is the difference between virtual and physical addresses?”
“How do you inspect a process’s memory layout in Linux?”
“Why do stack addresses change between runs?”
“What is ASLR and how does it affect debugging?”
“How can you tell if an address belongs to the heap?”

Hints in Layers

Hint 1: Print Addresses First

printf("&local=%p\n", (void*)&local);

Hint 2: Parse /proc/self/maps

FILE *f = fopen("/proc/self/maps", "r");
char line[256];
while (fgets(line, sizeof(line), f)) {
    // print or parse
}

Hint 3: Label Regions Look for “[heap]” and “[stack]” labels in each line to classify segments.

Hint 4: Show Permissions Print permission bits and highlight any region with r-x to show code sections.

Books That Will Help

Topic	Book	Chapter
Virtual memory	“Computer Systems: A Programmer’s Perspective”	Ch. 9
Process memory layout	“Operating Systems: Three Easy Pieces”	Ch. 14-16
Pointers	“The C Programming Language”	Ch. 5

Common Pitfalls & Debugging

Problem 1: “Addresses change every run”

Why: ASLR is enabled.
Fix: Accept it; display relative offsets instead of absolute addresses.
Quick test: Run twice and compare output.

Problem 2: “Permission parsing is wrong”

Why: You did not split fields correctly.
Fix: Use sscanf with proper format or tokenize by spaces.
Quick test: Print raw line and parsed tokens.

Definition of Done

Prints stack, heap, text, data, and bss addresses
Reads /proc/self/maps and displays permissions
Explains ASLR in README
Runs clean under ASan and Valgrind

Project 2: Safe String Library

Main Programming Language: C
Difficulty: Intermediate
Knowledge Area: Strings, bounds checking, API design
Main Book: “Effective C”

What you’ll build: A small library that provides safe string operations with explicit length and capacity tracking.

Why it teaches memory & control: You will replace unsafe APIs with safe invariants and enforce bounds checks everywhere.

Core challenges you’ll face:

Designing a clean API around length/capacity
Avoiding off-by-one errors
Maintaining invariants across operations

Real World Outcome

$ ./sstr_demo
Input: "hello"
Append: " world"
Result: "hello world" (len=11 cap=32)

Trying to append 100 bytes...
Error: sstr_append overflow (requested 100, remaining 20)

The Core Question You’re Answering

“How can I design a C string API that makes unsafe operations impossible by default?”

Concepts You Must Understand First

C String Internals
- What does the null terminator mean?
- Book Reference: K&R Ch. 5
Buffer Invariants
- How do length and capacity differ?
- Book Reference: Effective C Ch. 4
Pointer Arithmetic
- How does data + len work?
- Book Reference: Understanding and Using C Pointers Ch. 1-2

Questions to Guide Your Design

How will you represent strings (struct, length, capacity)?
How will you enforce null termination after every operation?
How will errors be reported (return code vs errno)?

Thinking Exercise

Given a buffer of size 8, what happens when you append 8 bytes?

cap=8, len=0
append "12345678" (8 bytes)

Question: Why must you reject this, even though it “fits”?

The Interview Questions They’ll Ask

“Why is strcpy unsafe?”
“What invariant should a safe string maintain?”
“How would you design a string API in C?”
“What is the cost of length tracking?”

Hints in Layers

Hint 1: Define the struct

typedef struct { char *data; size_t len; size_t cap; } sstr;

Hint 2: Initialize with capacity

int sstr_init(sstr *s, size_t cap) {
    s->data = malloc(cap);
    s->len = 0;
    s->cap = cap;
    s->data[0] = '\0';
}

Hint 3: Append safely Check len + add + 1 <= cap before copying.

Hint 4: Test overflow cases Write unit tests that attempt to overflow and expect errors.

Books That Will Help

Topic	Book	Chapter
Strings	“Effective C”	Ch. 4
Pointers	“The C Programming Language”	Ch. 5
Memory bugs	“Expert C Programming”	Ch. 2-3

Common Pitfalls & Debugging

Problem 1: “String loses null terminator”

Why: You forgot to append \0 after an operation.
Fix: Always set data[len] = '\0'.
Quick test: Print with printf("%s").

Problem 2: “Overflow not detected”

Why: You forgot the +1 for the terminator.
Fix: Check len + add + 1 <= cap.
Quick test: Append a string of length cap.

Definition of Done

All operations enforce bounds checks
All strings remain null-terminated
Library has tests for overflow and truncation
Runs clean under ASan and Valgrind

Project 3: Memory Leak Detector

Main Programming Language: C
Difficulty: Intermediate
Knowledge Area: malloc/free tracking, debugging
Main Book: “The Linux Programming Interface”

What you’ll build: A library or tool that wraps malloc/free and reports leaked blocks and double frees at program exit.

Why it teaches memory & control: You will implement ownership tracking and learn how allocator misuse shows up in practice.

Core challenges you’ll face:

Tracking allocations with metadata
Handling double frees safely
Reporting leaks with stack traces (optional)

Real World Outcome

$ ./leak_demo
[leakdet] malloc(32) -> 0x55555556a2a0 (main.c:12)
[leakdet] malloc(64) -> 0x55555556a2d0 (main.c:13)
[leakdet] free(0x55555556a2a0) (main.c:17)

== Leak Report ==
Leaked blocks: 1
Total leaked bytes: 64
Leaked at:
  0x55555556a2d0 size=64  (main.c:13)

The Core Question You’re Answering

“How can I make memory ownership visible and enforceable in a C program?”

Concepts You Must Understand First

Heap Allocation
- How does malloc return memory?
- Book Reference: CSAPP Ch. 9
Ownership & Lifetimes
- Who frees allocations?
- Book Reference: Expert C Programming Ch. 2-3
Debugging Tools
- How do ASan and Valgrind report leaks?
- Book Reference: Art of Debugging with GDB Ch. 1-2

Questions to Guide Your Design

How will you store metadata (hash table, linked list)?
How will you detect double free or invalid free?
Do you want to support realloc?

Thinking Exercise

Consider a program that allocates 10 blocks, frees 5, and exits. How will you report only the 5 leaked ones?

The Interview Questions They’ll Ask

“How do leak detectors work?”
“What is a double free and why is it dangerous?”
“How do you track allocation site information in C?”

Hints in Layers

Hint 1: Wrap malloc/free with macros

#define malloc(sz) leak_malloc(sz, __FILE__, __LINE__)
#define free(p) leak_free(p, __FILE__, __LINE__)

Hint 2: Store metadata in a linked list

typedef struct alloc { void *ptr; size_t size; const char *file; int line; struct alloc *next; } alloc;

Hint 3: Remove on free Search the list; if not found, report invalid free.

Hint 4: Report at exit Register a atexit() handler to print leaks.

Books That Will Help

Topic	Book	Chapter
Heap allocation	“Computer Systems: A Programmer’s Perspective”	Ch. 9
Debugging	“The Art of Debugging with GDB”	Ch. 1-4
C memory pitfalls	“Expert C Programming”	Ch. 2-3

Common Pitfalls & Debugging

Problem 1: “Double free not detected”

Why: You remove nodes but don’t mark freed pointers.
Fix: Keep a hash set of freed addresses or detect missing entries.
Quick test: Free the same pointer twice.

Problem 2: “Leak detector itself leaks”

Why: Your metadata list is never freed.
Fix: Use a static pool or ignore metadata leaks by design.
Quick test: Run under Valgrind and ensure only user leaks appear.

Definition of Done

Reports leaked blocks with size and allocation site
Detects double free
Supports realloc or documents limitations
Runs clean under ASan

Project 4: Arena Allocator

Main Programming Language: C
Difficulty: Intermediate
Knowledge Area: Custom allocators, alignment, mmap
Main Book: “C Interfaces and Implementations”

What you’ll build: A custom allocator that allocates a large region and serves fast bump allocations with alignment support.

Why it teaches memory & control: You will learn how allocators work by building one from scratch.

Core challenges you’ll face:

Correct alignment
Handling out-of-memory
Reset or destroy semantics

Real World Outcome

$ ./arena_demo
Arena size: 1MB
alloc(24) -> 0x7f3a90000000
alloc(64) -> 0x7f3a90000020
alloc(128) -> 0x7f3a90000060

Arena used: 216 bytes
Arena high-water: 216 bytes
Resetting arena...
Arena used: 0 bytes

The Core Question You’re Answering

“How can I trade flexibility for speed by controlling memory allocation patterns?”

Concepts You Must Understand First

Heap Allocation Basics
- What does malloc do internally?
- Book Reference: C Interfaces and Implementations Ch. 5-6
Alignment Rules
- Why must allocations be aligned?
- Book Reference: CSAPP Ch. 3 (data alignment)
mmap
- How do you request a large contiguous region?
- Book Reference: TLPI Ch. 49 (Memory Mapping)

Questions to Guide Your Design

How will you ensure alignment for all allocations?
What happens when the arena fills?
How will you expose usage statistics?

Thinking Exercise

If your arena is 4096 bytes and you allocate 1024 bytes at alignment 64, how much space is wasted due to alignment padding?

The Interview Questions They’ll Ask

“What is an arena allocator and why is it fast?”
“How does alignment work in allocators?”
“When would you not use an arena?”

Hints in Layers

Hint 1: Use mmap for large blocks

void *mem = mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);

Hint 2: Align your pointer

size_t aligned = (p + align - 1) & ~(align - 1);

Hint 3: Track high-water mark Update max_used whenever offset increases.

Hint 4: Implement reset Reset by setting offset = 0.

Books That Will Help

Topic	Book	Chapter
Allocators	“C Interfaces and Implementations”	Ch. 5-6
Memory mapping	“The Linux Programming Interface”	Ch. 49
VM and alignment	“Computer Systems: A Programmer’s Perspective”	Ch. 9

Common Pitfalls & Debugging

Problem 1: “Misaligned pointers”

Why: Alignment rounding logic is wrong.
Fix: Use power-of-two alignment and proper mask.
Quick test: assert(((uintptr_t)ptr % align)==0).

Problem 2: “Out of memory not handled”

Why: You return NULL but caller doesn’t check.
Fix: Document and assert on OOM in tests.
Quick test: Allocate more than arena size.

Definition of Done

Allocations are aligned correctly
Arena supports reset and reuse
Usage metrics are reported
Works under ASan

Project 5: Exploit Lab (Buffer Overflow Playground)

Main Programming Language: C
Difficulty: Advanced
Knowledge Area: Stack frames, mitigations, exploitation
Main Book: “Practical Binary Analysis”

What you’ll build: A controlled lab with a vulnerable program, step-by-step overflow exploit, and toggled mitigations to observe defenses.

Why it teaches memory & control: You will see exactly how control flow is hijacked and how mitigations stop it.

Core challenges you’ll face:

Understanding stack layout
Crafting precise payloads
Observing mitigation impact

Real World Outcome

$ ./vuln
Enter your name: AAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault (core dumped)

$ gdb ./vuln core
(gdb) bt
#0  0x41414141 in ?? ()

$ ./vuln_protected
*** stack smashing detected ***: terminated

The Core Question You’re Answering

“How does a single out-of-bounds write become control over execution?”

Concepts You Must Understand First

Stack Frames
- Where is the return address stored?
- Book Reference: CSAPP Ch. 3
Buffer Overflows
- How does a buffer overwrite control data?
- Book Reference: Practical Binary Analysis Ch. 5-6
Mitigations
- What do ASLR, NX, and canaries do?
- Book Reference: Practical Binary Analysis Ch. 7

Questions to Guide Your Design

Will you build a single vulnerable binary or multiple variants?
How will you demonstrate ASLR on/off?
Will you use GDB or a scripted exploit to show offsets?

Thinking Exercise

Given a 16-byte buffer and a saved return address 24 bytes above it, how many bytes do you need to overwrite the return address?

The Interview Questions They’ll Ask

“What is a stack buffer overflow?”
“How does ASLR work?”
“What is the purpose of NX?”
“How does a stack canary detect overflows?”

Hints in Layers

Hint 1: Build a vulnerable function

void vuln() {
    char buf[16];
    gets(buf); // intentionally unsafe
}

Hint 2: Use pattern generation Use a cyclic pattern to find exact overwrite offset.

Hint 3: Toggle mitigations Compile with and without -fstack-protector-strong -D_FORTIFY_SOURCE=2.

Hint 4: Observe ASLR Compare addresses across runs in GDB.

Books That Will Help

Topic	Book	Chapter
Stack frames	“Computer Systems: A Programmer’s Perspective”	Ch. 3
Exploits	“Practical Binary Analysis”	Ch. 5-7
Security basics	“Foundations of Information Security”	Ch. 2-3

Common Pitfalls & Debugging

Problem 1: “Exploit not reproducible”

Why: ASLR changes addresses.
Fix: Disable ASLR in a lab VM or use info leaks.
Quick test: Check /proc/sys/kernel/randomize_va_space.

Problem 2: “Stack canary aborts”

Why: Compiler inserted canary.
Fix: Compile without stack protector for the baseline exercise.
Quick test: checksec output.

Definition of Done

Demonstrates overflow with and without mitigations
Shows control-flow hijack in GDB
Explains why mitigations changed the outcome
Runs only in a safe lab environment

Project 6: Mini Text Editor (Capstone)

Main Programming Language: C
Difficulty: Advanced
Knowledge Area: Complex memory management, terminal I/O
Main Book: “Advanced Programming in the UNIX Environment”

What you’ll build: A terminal-based text editor with file loading, editing, saving, and undo/redo.

Why it teaches memory & control: You must manage multiple dynamic data structures, handle real-world input, and prevent memory bugs under continuous use.

Core challenges you’ll face:

Efficient text buffer design
Correct cursor movement and bounds
Undo/redo memory ownership

Real World Outcome

$ ./mini_editor notes.txt
[STATUS] notes.txt  |  12 lines  |  UTF-8  |  CTRL-S to save  CTRL-Q to quit

# You see your file in the terminal
# Cursor moves, typing edits live
# Saving writes back to disk

The Core Question You’re Answering

“Can I build a real interactive system in C without leaking or corrupting memory?”

Concepts You Must Understand First

Strings and Buffers
- How do you store lines safely?
- Book Reference: Effective C Ch. 4
Memory Ownership
- Who owns line buffers and undo stack?
- Book Reference: Expert C Programming Ch. 2-3
Debugging Tools
- How do you detect leaks in long-running programs?
- Book Reference: Art of Debugging with GDB Ch. 1-4

Questions to Guide Your Design

Will you use a gap buffer, rope, or array of lines?
How will you manage undo/redo memory?
How will you ensure no leaks on exit?

Thinking Exercise

If you implement undo as a stack of operations, what memory do you need to free when the user types after undo?

The Interview Questions They’ll Ask

“What data structure would you use for a text editor buffer?”
“How would you implement undo/redo safely?”
“What causes memory leaks in long-running apps?”

Hints in Layers

Hint 1: Start with raw mode

raw.c_lflag &= ~(ECHO | ICANON | ISIG);

Hint 2: Use a simple line array first Store each line as a dynamically allocated string in an array.

Hint 3: Add undo/redo with ownership rules Each undo node should own its own copy of text.

Hint 4: Run ASan after every milestone Compile and test frequently to catch leaks early.

Books That Will Help

Topic	Book	Chapter
Terminal programming	“Advanced Programming in the UNIX Environment”	Ch. 18
Strings	“Effective C”	Ch. 4
Debugging	“The Art of Debugging with GDB”	Ch. 1-4

Common Pitfalls & Debugging

Problem 1: “Terminal stuck in raw mode”

Why: You didn’t restore terminal settings on exit.
Fix: Use atexit() and signal handlers.
Quick test: Run and press Ctrl-C, verify terminal recovers.

Problem 2: “Undo leaks memory”

Why: Undo nodes are never freed.
Fix: Free nodes when clearing redo stack.
Quick test: Run under Valgrind.

Problem 3: “Cursor jumps out of bounds”

Why: You didn’t clamp cursor_x to line length.
Fix: Clamp after every movement.
Quick test: Move between short and long lines.

Definition of Done

Can open, edit, and save files
Undo/redo works with no leaks
Cursor movement is correct on variable-length lines
Runs clean under ASan and Valgrind