Project 7: Buffer Overflow Exploitation

Expanded deep-dive guide for Project 7 from the Binary Analysis sprint.

Quick Reference

Attribute Value
Difficulty Level 3: Advanced
Time Estimate 3-4 weeks
Main Programming Language C (targets), Python (exploits)
Alternative Programming Languages Assembly for shellcode
Coolness Level Level 5: Pure Magic (Super Cool)
Business Potential 1. The “Resume Gold”
Knowledge Area Binary Exploitation / Memory Corruption
Software or Tool GDB, pwntools, checksec
Main Book “Hacking: The Art of Exploitation” by Jon Erickson

1. Learning Objectives

  1. Build a working implementation with reproducible outputs.
  2. Justify key design choices with binary-analysis principles.
  3. Produce an evidence-backed report of findings and limitations.
  4. Document hardening or next-step improvements.

2. All Theory Needed (Per-Concept Breakdown)

This project depends on concepts from the main sprint primer: loader semantics, control/data-flow recovery, runtime observation, and mitigation-aware vulnerability reasoning. Before implementation, restate the project’s core assumptions in your own words and define how you will validate them.

3. Project Specification

3.1 What You Will Build

Working exploits for buffer overflow vulnerabilities, progressing from simple stack smashing to bypass ASLR and stack canaries.

3.2 Functional Requirements

  1. Accept the target binary/input and validate format assumptions.
  2. Produce analyzable outputs (console report and/or artifacts).
  3. Handle malformed inputs safely with explicit errors.

3.3 Non-Functional Requirements

  • Reproducibility: same input should produce equivalent findings.
  • Safety: unknown samples run only in isolated lab contexts.
  • Clarity: separate facts, hypotheses, and inferred conclusions.

3.4 Expanded Project Brief

  • File: P07-buffer-overflow-exploitation.md

  • Main Programming Language: C (targets), Python (exploits)
  • Alternative Programming Languages: Assembly for shellcode
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Binary Exploitation / Memory Corruption
  • Software or Tool: GDB, pwntools, checksec
  • Main Book: “Hacking: The Art of Exploitation” by Jon Erickson

What you’ll build: Working exploits for buffer overflow vulnerabilities, progressing from simple stack smashing to bypass ASLR and stack canaries.

Why it teaches binary analysis: Understanding exploitation gives you insight into why security mitigations exist and how low-level memory works.

Core challenges you’ll face:

  • Finding the offset → maps to pattern generation, EIP/RIP control
  • Controlling execution → maps to return address overwrite
  • Bypassing NX → maps to return-to-libc, ROP
  • Bypassing ASLR → maps to info leaks, partial overwrite

Resources for key challenges:

Key Concepts:

  • Stack Layout: “Hacking: Art of Exploitation” Ch. 2
  • Shellcode: “Hacking: Art of Exploitation” Ch. 5
  • Return-Oriented Programming: “Practical Binary Analysis” Ch. 10

Difficulty: Advanced Time estimate: 3-4 weeks Prerequisites: Projects 1-6, solid C and assembly

Real World Outcome

Deliverables:

  • Analysis output or tooling scripts
  • Report with control/data flow notes

Validation checklist:

  • Parses sample binaries correctly
  • Findings are reproducible in debugger
  • No unsafe execution outside lab ```python from pwn import *

Connect to target

p = process(‘./vulnerable’)

Find offset with pattern

offset = 72

Build payload

payload = b’A’ * offset # Fill buffer payload += p64(0x401337) # Overwrite return address with win()

Send payload

p.sendline(payload)

Get shell!

p.interactive()

Output:

[*] Switching to interactive mode

$ whoami

root

$ cat flag.txt

FLAG{buffer_overflow_mastered}


#### Hints in Layers
Progression:
1. **ret2win**: Overwrite return address to call `win()` function
2. **ret2shellcode**: Jump to shellcode on stack (no NX)
3. **ret2libc**: Return to `system("/bin/sh")` (bypass NX)
4. **ROP chain**: Chain gadgets for complex operations
5. **GOT overwrite**: Hijack function pointers
6. **Format string**: Arbitrary read/write

Finding offset:
```python
from pwn import *

# Generate cyclic pattern
pattern = cyclic(200)
# Feed to program, get crash address
# Use cyclic_find to get offset
offset = cyclic_find(0x61616168)  # 'haaa' in little-endian

Key questions:

  • How do you find the offset to the return address?
  • What’s the difference between 32-bit and 64-bit exploitation?
  • How do you find useful libc functions when ASLR is enabled?

Learning milestones:

  1. Control EIP/RIP → Overwrite return address
  2. Execute shellcode → Spawn a shell (no NX)
  3. ROP chains → Bypass NX with gadgets
  4. Leak addresses → Bypass ASLR

The Core Question You Are Answering

How do you exploit unsafe memory operations to hijack program control flow, execute arbitrary code, and bypass modern security mitigations—all by understanding the precise layout of memory at runtime?

This project bridges theory and practice: you’ll see how textbook stack diagrams become real exploitable conditions, and how security features (NX, ASLR, stack canaries) force increasingly sophisticated attack techniques.

Concepts You Must Understand First

1. The Stack Memory Layout

The stack grows downward (high to low addresses) and stores local variables, saved registers, and return addresses. Understanding this layout is essential for exploitation.

High addresses
+------------------+
| Command-line args|
| (argv, envp)     |
+------------------+
| Stack            |
| (grows down)     |
|                  |
|  +-----------+   |  <-- Current function's stack frame
|  | Local vars|   |
|  | (buffer)  |   |
|  +-----------+   |
|  | Saved RBP |   |  <-- Frame pointer (base of previous frame)
|  +-----------+   |
|  | Ret addr  |   |  <-- Return address (TARGET FOR OVERWRITE)
|  +-----------+   |
|  | Arguments |   |
|  +-----------+   |
|      ...         |
|                  |
+------------------+
| Heap             |
| (grows up)       |
+------------------+
| .bss (uninit)    |
+------------------+
| .data (init)     |
+------------------+
| .text (code)     |
+------------------+
Low addresses

Guiding Questions:

  • Why does the stack grow downward while arrays grow upward (creating overflow)?
  • What’s stored between a buffer and the return address?
  • How does the saved frame pointer (RBP) help identify stack frame boundaries?

Book Reference: “Computer Systems: A Programmer’s Perspective” Ch. 3.7 - Procedures

2. Buffer Overflow Mechanics

When strcpy(buffer, user_input) copies more data than the buffer can hold, it overwrites adjacent memory—including saved RBP and return address.

Guiding Questions:

  • Why are functions like gets(), strcpy(), and sprintf() dangerous?
  • What’s the difference between stack overflow (too much data) and stack smashing (deliberate overwrite)?
  • How do you calculate the exact offset from buffer start to return address?

Book Reference: “Hacking: The Art of Exploitation” Ch. 2.5 - Buffer Overflows

3. Return Address Hijacking

The return address (pushed by call, popped by ret) determines where execution goes after a function. Overwriting it redirects control flow.

Guiding Questions:

  • What does the ret instruction do at the assembly level?
  • Why must your payload preserve stack alignment (especially on x64)?
  • What happens if you overwrite the return address with an invalid address?

Book Reference: “Computer Systems: A Programmer’s Perspective” Ch. 3.7.3 - Data Transfer

4. Shellcode Development

Shellcode is position-independent assembly code that spawns a shell or executes commands. It must avoid null bytes (which terminate string copies).

Guiding Questions:

  • Why does shellcode use execve("/bin/sh", NULL, NULL) instead of system("/bin/sh")?
  • How do you write position-independent code (no hardcoded addresses)?
  • What techniques eliminate null bytes (e.g., xor eax, eax instead of mov eax, 0)?

Book Reference: “Hacking: The Art of Exploitation” Ch. 5 - Shellcode

5. NX (No-Execute) Protection

Modern systems mark the stack as non-executable, preventing shellcode execution. This forces attackers to use existing code (return-to-libc, ROP).

Guiding Questions:

  • How does the NX bit work at the hardware level (page table permissions)?
  • Why can’t you just mark the stack executable from your exploit?
  • What’s the difference between DEP (Windows) and NX (Linux)?

Book Reference: “Practical Binary Analysis” Ch. 10.2 - Code-Reuse Attacks

6. ASLR (Address Space Layout Randomization)

ASLR randomizes the base addresses of stack, heap, and libraries, making hardcoded addresses unreliable. Defeating it requires information leaks.

Guiding Questions:

  • What parts of memory are randomized (stack, heap, libraries)?
  • Why is the code section (.text) often NOT randomized in binaries without PIE?
  • How do format string bugs or read overflows leak addresses?

Book Reference: “Practical Binary Analysis” Ch. 10.3 - Randomization-Based Defenses

7. Stack Canaries

Canaries are random values placed between the buffer and return address. Before returning, the program checks if the canary is intact; if not, it aborts.

Guiding Questions:

  • Where exactly is the canary placed in the stack frame?
  • How are canaries generated (random, constant, TLS-based)?
  • Can you bypass canaries by leaking their value or using partial overwrites?

Book Reference: “Computer Systems: A Programmer’s Perspective” Ch. 3.10.3 - Stack Corruption Detection

8. Pwntools and Exploit Development

Pwntools is a Python library for writing exploits. It handles process interaction, payload generation, and address packing.

Guiding Questions:

  • What’s the difference between p32() and p64() for packing addresses?
  • How does cyclic() help find the exact overflow offset?
  • When should you use process() vs remote() for local vs remote targets?

Book Reference: Official pwntools documentation (docs.pwntools.com)

Questions to Guide Your Design

  1. How do you determine the exact offset from the buffer start to the return address without source code? Consider pattern generation, crash analysis, and GDB inspection.

  2. When NX is enabled, what existing code can you reuse to achieve your goals? Think about libc functions, PLT entries, and gadgets.

  3. How would you leak a libc address to defeat ASLR in a two-stage exploit? Consider using puts() to print GOT entries.

  4. What strategies work when you can only overflow a small number of bytes (not enough for shellcode)? Think about partial overwrites, ROP, or pointer manipulation.

  5. How do you write shellcode that works regardless of where it’s placed in memory? Consider relative addressing, stack pivoting, and position-independent techniques.

  6. When would you choose ret2libc over ROP, or vice versa? Think about complexity, reliability, and available gadgets.

  7. How do you test your exploits reliably when ASLR is enabled locally? Consider disabling ASLR (echo 0 > /proc/sys/kernel/randomize_va_space) or handling it properly.

  8. What debugging workflow helps when your exploit crashes the program in unexpected ways? Think about core dumps, GDB breakpoints, and payload inspection.

Thinking Exercise

Before writing any exploits, complete these exercises:

  1. Manual Stack Diagram: Draw the complete stack layout for this function call: 
    void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No bounds checking!
}

int main(int argc, char **argv) {
    if (argc > 1) {
        vulnerable(argv[1]);
    }
    return 0;
}
    • Compile with gcc -fno-stack-protector -z execstack -o vuln vuln.c
    • Run in GDB with breakpoint in vulnerable() after the strcpy()
    • Print the stack: x/40wx $rsp-0x50
    • Identify: buffer location, saved RBP, return address
    • Calculate the offset: how many bytes from buffer[0] to return address?
  2. Shellcode Analysis: Examine this x64 shellcode: 
    xor rsi, rsi         ; NULL (argv)
mul rsi              ; RAX = RDX = 0
mov rbx, 0x68732f2f6e69622f  ; "/bin//sh" reversed
push rbx
push rsp
pop rdi              ; RDI points to "/bin//sh"
mov al, 0x3b         ; syscall number for execve
syscall
    • Why use xor rsi, rsi instead of mov rsi, 0?
    • Why is the string “/bin//sh” instead of “/bin/sh”?
    • What’s the syscall number for execve on x64 Linux?
    • Assemble it and verify it has no null bytes
  3. Pattern Offset Calculation: 
    from pwn import *

# Generate a cyclic pattern
pattern = cyclic(200)
print(pattern)

# Feed it to the vulnerable program
# Say it crashes with RIP = 0x6161616c ('laaa')

# Find the offset
offset = cyclic_find(0x6161616c)
print(f"Offset to RIP: {offset}")
    • Run this against a vulnerable binary
    • Verify the offset by sending b'A' * offset + b'BBBBBBBB'
    • Confirm RIP becomes 0x4242424242424242 (BBBBBBBB)
  4. NX Bypass Conceptual: Given a binary with NX enabled:
    • List all functions in the PLT (objdump -d vuln | grep @plt)
    • Find system@plt and puts@plt addresses
    • Locate the string “/bin/sh” in libc using strings -a -t x /lib/x86_64-linux-gnu/libc.so.6 | grep /bin/sh
    • Conceptually design a ret2libc attack: 
      payload = 'A' * offset
payload += p64(pop_rdi_ret)    # Gadget to set RDI
payload += p64(binsh_addr)     # Argument: "/bin/sh"
payload += p64(system_addr)    # Call system

The Interview Questions They’ll Ask

  1. “Walk me through the exact steps of a buffer overflow from overwrite to code execution.” Expected: Unsafe function → overflow buffer → overwrite saved RBP → overwrite return address → ret instruction loads attacker’s address → control flow hijacked.

  2. “Why does the stack grow downward but arrays grow upward? How does this enable overflows?” Expected: Historical architecture decision. Arrays grow toward higher addresses, so overflow overwrites later stack data (saved pointers, return addresses).

  3. “Explain the difference between controlling RIP and actually executing your payload.” Expected: Controlling RIP just redirects execution. Without executable stack (NX), you must point to existing code or use ROP. With exec stack, you can point to your shellcode.

  4. “How does ASLR prevent exploitation, and how do you defeat it?” Expected: ASLR randomizes addresses, breaking hardcoded values. Defeat with info leaks (format strings, read overflows) or partial overwrites (only modify least significant bytes).

  5. “What’s a stack canary and how would you bypass it?” Expected: Random value between buffer and return address. Bypass by: leaking canary value, overwriting without corrupting it, or using other vulnerabilities (format string).

  6. “Explain ret2libc. Why is it used when NX is enabled?” Expected: Return to existing library functions (like system()) instead of shellcode. Works because libc is executable and always loaded.

  7. “You have a 12-byte overflow but need 100+ bytes for shellcode. What are your options?” Expected: (1) ROP chain, (2) stack pivot to larger buffer elsewhere, (3) two-stage exploit (small stub to read larger payload), (4) ret2libc (no shellcode needed).

  8. “How do you calculate the exact offset to the return address?” Expected: Methods: (1) cyclic pattern + crash analysis, (2) GDB to examine stack, (3) source code analysis, (4) trial and error with increasing payloads.

  9. “What’s the purpose of NOP sled in shellcode exploits?” Expected: Provides margin of error. If you’re not sure of exact shellcode address, point anywhere in the NOPs (0x90) and execution slides to the shellcode.

  10. “Describe a real-world scenario where buffer overflow exploitation is still relevant today.” Expected: IoT devices (often no ASLR/NX), legacy systems, kernel exploits, CTF competitions, security research/testing.

Books That Will Help

Topic Book Chapter/Section
Stack Layout & Function Calls “Computer Systems: A Programmer’s Perspective” Ch. 3.7 (Procedures, Stack Frames)
Buffer Overflow Fundamentals “Hacking: The Art of Exploitation” Ch. 2.5 (Buffer Overflows)
Shellcode Writing “Hacking: The Art of Exploitation” Ch. 5 (Shellcode)
Return Address Hijacking “Computer Systems: A Programmer’s Perspective” Ch. 3.7.3 (Data Transfer)
Security Mitigations (NX, ASLR, Canaries) “Computer Systems: A Programmer’s Perspective” Ch. 3.10.3 (Stack Corruption Detection)
Code-Reuse Attacks (ret2libc) “Practical Binary Analysis” Ch. 10.2 (Code-Reuse Attacks)
ASLR and Randomization “Practical Binary Analysis” Ch. 10.3 (Randomization Defenses)
Low-Level Memory Layout “Low-Level Programming” Ch. 8 (Memory Management)
Exploitation Techniques “The Shellcoder’s Handbook” Ch. 4-5 (Stack Overflows)
Assembly for Exploitation “Low-Level Programming” Ch. 3-4 (Assembly Language)
Debugging with GDB “Hacking: The Art of Exploitation” Ch. 2 (Programming, Debugging)
Format String Exploits “Hacking: The Art of Exploitation” Ch. 3 (Exploitation)
Heap Exploitation Intro “The Shellcoder’s Handbook” Ch. 7 (Heap Overflows)
Pwntools Usage Official Pwntools Docs docs.pwntools.com
Modern Exploitation “A Guide to Kernel Exploitation” Ch. 1-2 (Background, Stack Overflows)

Common Pitfalls and Debugging

Problem 1: “Your interpretation does not match runtime behavior”

  • Why: Static analysis can hide runtime-resolved addresses, lazy binding, and input-dependent branches.
  • Fix: Reproduce the path with debugger or tracer, then compare static assumptions against live register/memory state.
  • Quick test: Run the same sample through both your static workflow and a debugger transcript, and confirm control-flow decisions align.

Problem 2: “Tool output is inconsistent across machines”

  • Why: ASLR, tool version drift, and different binary build flags (PIE, RELRO, symbols stripped) change observed addresses and metadata.
  • Fix: Pin tool versions, capture checksec/metadata, and document environment assumptions in your report.
  • Quick test: Re-run analysis in a container or VM with pinned tools and compare hashes of generated outputs.

Problem 3: “Analysis accidentally executes unsafe code”

  • Why: Dynamic workflows run binaries in host context without sufficient isolation.
  • Fix: Use disposable snapshots, no-network execution, and non-privileged users for all unknown samples.
  • Quick test: Validate isolation controls first (network disabled, snapshot active, unprivileged user), then execute sample.

Definition of Done

  • Core functionality works on reference inputs
  • Edge cases are tested and documented
  • Results are reproducible (same binary, same tools, same report output)
  • Analysis notes clearly separate observations, assumptions, and conclusions
  • Lab safety controls were applied for any dynamic execution

4. Solution Architecture

Input Artifact -> Parse/Decode -> Analysis Engine -> Validation Layer -> Report

Design each stage so intermediate artifacts are inspectable (JSON/text/notes), which makes debugging and peer review much easier.

5. Implementation Phases

Phase 1: Foundation

  • Define input assumptions and format checks.
  • Produce a minimal golden output on one known sample.

Phase 2: Core Functionality

  • Implement full analysis pass for normal cases.
  • Add validation against an external ground-truth tool.

Phase 3: Hard Cases and Reporting

  • Add malformed/edge-case handling.
  • Finalize report template and reproducibility notes.

6. Testing Strategy

  • Unit-level checks for parser/decoder helpers.
  • Integration checks against known binaries/challenges.
  • Regression tests for previously failing cases.

7. Extensions & Challenges

  • Add automation for batch analysis and comparative reports.
  • Add confidence scoring for each major finding.
  • Add export formats suitable for CI/security pipelines.

8. Production Reflection

Map your project output to a production analogue: what reliability, observability, and security controls would be required to run this continuously in an engineering organization?