Project 8: Heap Overflow Detector

A custom allocator wrapper that adds “red zones” (guard bytes) around allocations to detect heap buffer overflows at runtime.

Quick Reference

Attribute Value
Primary Language C
Alternative Languages Python (for analysis)
Difficulty Level 3: Advanced
Time Estimate 2 weeks
Knowledge Area Exploit Defense / Heap Security
Tooling glibc malloc, GDB, custom allocator
Prerequisites Project 6 (safe allocator), heap internals

What You Will Build

A custom allocator wrapper that adds “red zones” (guard bytes) around allocations to detect heap buffer overflows at runtime.

Why It Matters

This project builds core skills that appear repeatedly in real-world systems and tooling.

Core Challenges

  • Adding metadata to allocations → maps to header/footer design
  • Checking red zones → maps to when to verify
  • Handling realloc → maps to preserving guards during resize
  • Performance impact → maps to memory overhead

Key Concepts

  • Heap Chunk Layout: “The Shellcoder’s Handbook” Ch. 6
  • Memory Debugging: Electric Fence, DUMA documentation
  • Guard Pages/Bytes: AddressSanitizer design

Real-World Outcome

$ ./test_heap_overflow

Testing red zone detection...
Allocating 64 bytes with guard zones...
Memory layout:
  [REDZONE 16 bytes][USER DATA 64 bytes][REDZONE 16 bytes]
  0xDEADBEEF x 4    |  your data here  |  0xDEADBEEF x 4

Writing one byte past buffer...
Calling checked_free()...
*** HEAP OVERFLOW DETECTED ***
Corruption in trailing red zone at offset 0
Expected: 0xDEADBEEF, Found: 0xDEADBE41
Allocation backtrace:
  #0 checked_malloc() at heap_guard.c:42
  #1 main() at test.c:15

Testing underflow...
Writing before buffer...
*** HEAP UNDERFLOW DETECTED ***
Corruption in leading red zone at offset 12

Implementation Guide

  1. Reproduce the simplest happy-path scenario.
  2. Build the smallest working version of the core feature.
  3. Add input validation and error handling.
  4. Add instrumentation/logging to confirm behavior.
  5. Refactor into clean modules with tests.

Milestones

  • Milestone 1: Minimal working program that runs end-to-end.
  • Milestone 2: Correct outputs for typical inputs.
  • Milestone 3: Robust handling of edge cases.
  • Milestone 4: Clean structure and documented usage.

Validation Checklist

  • Output matches the real-world outcome example
  • Handles invalid inputs safely
  • Provides clear errors and exit codes
  • Repeatable results across runs

References

  • Main guide: LEARN_SECURE_C_AND_EXPLOIT_AWARENESS.md
  • “The Shellcoder’s Handbook”