← Back to all projects

LEARN GDB DEEP DIVE

Learn GDB: From Zero to Debugging Master

Goal: Deeply understand the GNU Debugger (GDB)—from basic commands and crash analysis to advanced scripting, reverse debugging, and the underlying mechanisms that make it all work.

---

Why Learn GDB?

GDB is the quintessential tool for inspecting a program’s soul. While IDE debuggers are convenient, mastering GDB gives you a superpower: the ability to debug anything, anywhere—on a remote server, in a minimal Docker container, or on an embedded device—with maximum control and insight. It’s the difference between driving a car and being the mechanic who can build the engine.

After completing these projects, you will:

• Confidently navigate the state of any C/C++ program: stack, heap, registers.
• Diagnose and fix segmentation faults and other crashes in minutes.
• Use advanced features like conditional breakpoints and watchpoints to find complex bugs.
• Automate your debugging workflows with Python scripting.
• Understand how a debugger actually controls a process at the OS level.

---

Core Concept Analysis

1. The Debugger-Debuggee Relationship (via ptrace)

On Linux, GDB works its magic using the ptrace (process trace) system call. This is the kernel-level API that allows one process (GDB) to observe and control another (the “debuggee”).

┌──────────────────┐                              ┌──────────────────┐
│       GDB        │                              │  Target Program  │
│  (The Tracer)    │                              │  (The "Tracee")  │
└──────────────────┘                              └──────────────────┘
         │                                                  ▲
         │ PTRACE_ATTACH                                    │ (Process stops)
         ├──────────────────────────────────────────────────►
         │                                                  │
         │ PTRACE_PEEKDATA (Read memory/registers)          │
         ├──────────────────────────────────────────────────►
         │                                                  │
         │ PTRACE_POKEDATA (Write memory/registers)         │
         ├──────────────────────────────────────────────────►
         │                                                  │
         │ PTRACE_SINGLESTEP (Execute one instruction)      │
         ├──────────────────────────────────────────────────►
         │                                                  ▲
         │ PTRACE_CONT (Continue execution)                 │ (Process runs until
         ├──────────────────────────────────────────────────►    next signal)
         │                                                  │

2. Anatomy of a Breakpoint

A software breakpoint is not magic. GDB simply replaces an instruction in the target’s code with a special trap instruction (int 3 on x86).

Before GDB:
   Address      Instruction
   0x401000     mov eax, ebx
   0x401002     add ecx, edx   <-- You want to break here
   0x401004     sub eax, 1

After `break *0x401002`:
   Address      Instruction
   0x401000     mov eax, ebx
   0x401002     int 3          <-- Trap instruction (1 byte)
   0x401004     sub eax, 1

When the CPU hits `int 3`:
1. The CPU generates a "trap" signal.
2. The kernel catches the signal and notifies the tracer (GDB).
3. GDB inspects the program state.
4. To continue, GDB replaces `int 3` with the original instruction (`add ecx, edx`), single-steps the CPU, and then puts the `int 3` back.

---

Project List

The best way to learn GDB is to solve a series of increasingly difficult debugging challenges. Each “project” is a C program with a specific bug, designed to teach you a new GDB skill.

---

Project 1: The Basics - First Steps

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 2: Practical but Forgettable
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 1: Beginner
• Knowledge Area: Debugging Fundamentals
• Software or Tool: GDB, GCC
• Main Book: “The Art of Debugging with GDB” by Matloff & Salzman

What you’ll build: A simple C program with a for loop and a function call. You will use GDB to step through it, inspect variables, and understand the flow of execution.

Why it teaches GDB: This project builds the foundational muscle memory for the most common debugging loop: run, stop, inspect, continue.

Core challenges you’ll face:

• Compiling for debugging → maps to using the -g flag in GCC
• Starting a GDB session → maps to gdb ./a.out
• Controlling execution → maps to run, break, continue, next, step
• Inspecting state → maps to print for variables, backtrace for the call stack

Key Concepts:

• Symbols and the -g flag: “The Art of Debugging” Chapter 1
• Basic GDB Commands: GDB help command
• Stack Frames: “The Art of Debugging” Chapter 3

Difficulty: Beginner Time estimate: 1-2 hours Prerequisites: Basic C knowledge.

Real world outcome: You will compile this C code with gcc -g -o target target.c and debug it.

// target.c
#include <stdio.h>

void greet(int count) {
    printf("Hello for the %dth time!\n", count);
}

int main() {
    int i;
    for (i = 0; i < 5; ++i) {
        greet(i);
    }
    return 0;
}

Debugging Session:

$ gdb ./target
(gdb) break main       # Set a breakpoint at the start of main
(gdb) run              # Start the program
(gdb) next             # Execute 'int i;'
(gdb) next             # Execute 'for (i = 0; ...)'
(gdb) print i          # See the value of i
$1 = 0
(gdb) step             # Step *into* the greet() function
greet (count=0) at target.c:4
(gdb) backtrace        # See the call stack (main -> greet)
(gdb) continue         # Run until the program finishes

Learning milestones:

1. Set a breakpoint and run to it → You can control where execution starts.
2. Distinguish next from step → You understand stepping over vs. into functions.
3. Print a variable’s value → You can inspect program state.
4. View the call stack → You understand program structure.

---

Project 2: The Crash - Core Dump Analysis

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 1: Beginner
• Knowledge Area: Crash Analysis / Debugging
• Software or Tool: GDB, ulimit
• Main Book: “The Art of Debugging with GDB” by Matloff & Salzman

What you’ll build: A C program that reliably segfaults (e.g., by dereferencing a NULL pointer). You will learn to use GDB to perform a “post-mortem” analysis on the resulting core dump file.

Why it teaches GDB: Debugging a live process isn’t always possible. Understanding how to analyze a core dump is a critical skill for debugging crashes in production or non-interactive environments.

Core challenges you’ll face:

• Triggering a segfault → maps to writing buggy code
• Enabling core dumps → maps to ulimit -c unlimited shell command
• Loading a core dump → maps to gdb <executable> <corefile>
• Inspecting the crashed state → maps to finding the exact line and reason for the crash

Key Concepts:

• Pointers and Memory: “The C Programming Language” Chapter 5
• Core Dump Analysis: “The Linux Programming Interface” Chapter 23

Difficulty: Beginner Time estimate: 1-2 hours Prerequisites: Project 1.

Real world outcome: You will compile gcc -g -o crash crash.c, run ./crash, and then analyze the crash.

// crash.c
#include <stdio.h>

void crash_me() {
    char *p = NULL;
    *p = 'A'; // Segfault!
}

int main() {
    crash_me();
    return 0;
}

Debugging Session:

# First, enable core dumps in your shell
$ ulimit -c unlimited

# Run the program to make it crash
$ ./crash
Segmentation fault (core dumped)

# Analyze the core dump with GDB
$ gdb ./crash core
(gdb) backtrace
#0  0x000055555555513d in crash_me () at crash.c:5
#1  0x0000555555555152 in main () at crash.c:9

(gdb) frame 0
#0  0x000055555555513d in crash_me () at crash.c:5
5           *p = 'A'; // Segfault!

(gdb) print p
$1 = 0x0 <_start-0x100>  # The pointer is NULL!

Learning milestones:

1. Generate a core dump → You know how to configure the shell for debugging.
2. Load the core dump into GDB → You can start a post-mortem session.
3. Use backtrace to find the faulting function → You can pinpoint the location of a crash.
4. Inspect variables to find the root cause → You can determine why it crashed.

---

Project 3: The Hang - Attaching to a Running Process

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 2: Intermediate
• Knowledge Area: Dynamic Analysis / Process Control
• Software or Tool: GDB, ps
• Main Book: “The Linux Programming Interface” by Michael Kerrisk

What you’ll build: A program with an infinite loop. You’ll then attach GDB to the already-running process, pause it, inspect its state, and detach without killing it.

Why it teaches GDB: This simulates debugging a production server or long-running service that has become unresponsive. You can’t just restart it; you need to attach to it live, figure out what it’s doing, and get out cleanly.

Core challenges you’ll face:

• Finding the Process ID (PID) → maps to using ps aux | grep <name>
• Attaching GDB → maps to gdb -p <PID>
• Interrupting execution → maps to using Ctrl-C inside GDB
• Detaching cleanly → maps to the detach command

Key Concepts:

• Process IDs: “How Linux Works” Chapter 4
• Attaching and Detaching: help attach, help detach in GDB.

Difficulty: Intermediate Time estimate: 1 hour Prerequisites: Basic Linux command-line knowledge (ps).

Real world outcome: Compile gcc -g -o hang hang.c, run it in one terminal, and debug it from another.

// hang.c
#include <stdio.h>
#include <unistd.h> 

int main() {
    int counter = 0;
    while (1) {
        printf("Looping... counter = %d\n", counter++);
        sleep(1);
    }
    return 0;
}

Debugging Session:

# Terminal 1: Run the program
$ ./hang
Looping... counter = 0
Looping... counter = 1
...

# Terminal 2: Find the PID and attach GDB
$ ps aux | grep ./hang
user      12345   ... ./hang  # The PID is 12345
$ gdb -p 12345

(gdb) # Program is now paused. Let's see where it is.
(gdb) backtrace
#0  __GI___nanosleep (remaining=0x0, requested_time=0x7ffc...) at ...
#1  __sleep (seconds=0) at ../sysdeps/posix/sleep.c:113
#2  0x000055555555518d in main () at hang.c:9

(gdb) # It's in the sleep() call. Let's step out of it.
(gdb) finish
Run till exit from #0  __GI___nanosleep ...
main () at hang.c:10
10      }

(gdb) print counter
$1 = 2

(gdb) # Okay, we're done. Let the program continue.
(gdb) detach
Detaching from program: ./hang, process 12345
[Inferior 1 (process 12345) detached]

# Back in Terminal 1, the program resumes running
Looping... counter = 2
Looping... counter = 3
...

Learning milestones:

1. Successfully attach to a running process → You can debug live systems.
2. Interrupt and inspect the program → You can find out what a “stuck” program is doing.
3. Modify a variable in a live process → set var counter = 100.
4. Detach without killing the process → You can perform non-destructive inspection.

---

Project 4: The Corruption - Using Watchpoints

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 2: Intermediate
• Knowledge Area: Advanced Debugging / Memory Analysis
• Software or Tool: GDB
• Main Book: “The Art of Debugging with GDB” by Matloff & Salzman

What you’ll build: A C program where a variable is mysteriously overwritten by a buggy function. Instead of stepping through the whole program, you’ll use a “watchpoint” to make GDB stop at the exact moment the variable’s value changes.

Why it teaches GDB: This is one of GDB’s most powerful features. For “what the heck changed my variable?” bugs, watchpoints are infinitely faster than manual searching. It feels like magic.

Core challenges you’ll face:

• Identifying the symptom → maps to noticing a variable has the wrong value
• Setting a watchpoint → maps to the watch command
• Running to the trigger point → maps to continue and letting the watchpoint stop execution
• Analyzing the context → maps to using backtrace to see the culprit

Key Concepts:

• Watchpoints: help watch in GDB.
• Hardware vs. Software Watchpoints: GDB documentation. Hardware watchpoints are much faster but limited in number.

Difficulty: Intermediate Time estimate: 2 hours Prerequisites: Project 1.

Real world outcome: Compile and debug gcc -g -o corrupt corrupt.c.

// corrupt.c
#include <stdio.h>

int global_value = 100;

void buggy_function() {
    int *p = &global_value;
    // Oops, off-by-one pointer arithmetic
    *(p + 1) = 0;
}

int main() {
    int local_value = 200;
    printf("Before: global=%d, local=%d\n", global_value, local_value);
    buggy_function();
    printf("After:  global=%d, local=%d\n", global_value, local_value);
    // Why is local_value 0?
    return 0;
}

Debugging Session:

$ gdb ./corrupt
(gdb) break 17     # Break before the second printf
(gdb) run
(gdb) # We are stopped. Let's check the values.
(gdb) print local_value
$1 = 0  # That's wrong! It should be 200.

(gdb) # Let's restart and watch that variable.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

(gdb) # Set a watchpoint *before* the corruption happens.
(gdb) watch local_value
Hardware watchpoint 1: local_value

(gdb) continue
Continuing.
Hardware watchpoint 1: local_value

Old value = 200
New value = 0
buggy_function () at corrupt.c:8
8           *(p + 1) = 0;

(gdb) # GDB stopped on the exact line that changed the value!
(gdb) backtrace
#0  buggy_function () at corrupt.c:8
#1  0x00005555555551b8 in main () at corrupt.c:15

Learning milestones:

1. Set a watchpoint on a variable → You can monitor memory for changes.
2. Let GDB find the exact line of corruption → You’ve automated the search for memory bugs.
3. Use rwatch and awatch → You can break on read-access and read/write-access, respectively.

---

Project 5: The Assembly Level - disassemble and stepi

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands / x86 Assembly
• Alternative Programming Languages: N/A
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 3: Advanced
• Knowledge Area: Low-level Debugging / Reverse Engineering
• Software or Tool: GDB
• Main Book: “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron

What you’ll build: A simple C program which you will compile with optimizations (-O2) and debug at the assembly-instruction level.

Why it teaches GDB: Sometimes the source code lies. Optimizations can reorder operations, eliminate variables, and make line-by-line debugging misleading. Dropping down to the assembly level shows you the ground truth of what your program is actually doing.

Core challenges you’ll face:

• Viewing disassembly → maps to the disassemble command
• Correlating source to assembly → maps to GDB’s TUI or layout asm
• Stepping instruction-by-instruction → maps to stepi and nexti
• Inspecting CPU registers → maps to info registers or layout regs

Key Concepts:

• x86 Assembly: “Computer Systems: A Programmer’s Perspective” Chapter 3
• GDB TUI (Text User Interface): Press Ctrl-X A in GDB.
• CPU Registers: rip (instruction pointer), rsp (stack pointer), rax (return value).

Difficulty: Advanced Time estimate: 3-4 hours Prerequisites: Basic understanding of what assembly and registers are.

Real world outcome: Compile gcc -g -O2 -o optimized optimized.c and debug it.

// optimized.c
int calculate(int a, int b) {
    int result = (a + b) * 2;
    return result;
}

int main() {
    calculate(10, 20);
    return 0;
}

Debugging Session:

$ gdb ./optimized
(gdb) break main
(gdb) run
(gdb) layout asm  # Show disassembly
(gdb) layout regs # Show registers

(gdb) # Step through assembly instructions
(gdb) nexti
... (GDB output showing assembly instructions)
(gdb) nexti
   0x55555555513d <main+4>    mov    $0x14,%esi   # Move 20 into esi (arg 2)
(gdb) nexti
   0x555555555142 <main+9>    mov    $0xa,%edi    # Move 10 into edi (arg 1)
(gdb) nexti
   0x555555555147 <main+14>   call   0x555555555129 <calculate>
(gdb) # Let's look at the calculate function
(gdb) disassemble calculate
Dump of assembler code for function calculate:
   0x0000555555555129 <+0>:     lea    (%rdi,%rsi,1),%eax  # eax = rdi + rsi (a + b)
   0x000055555555512c <+3>:     add    %eax,%eax           # eax = eax + eax (result * 2)
   0x000055555555512e <+5>:     ret
End of dump.
# The optimizer got rid of the 'result' variable entirely!

(gdb) # After the call, the return value is in RAX
(gdb) nexti
(gdb) info registers rax
rax            0x3c     60   # The correct result, 30 * 2

Learning milestones:

1. View the disassembly of a function → You can see the machine code.
2. Use stepi to execute a single instruction → You have fine-grained control.
3. Inspect registers → You can see data outside of C variables.
4. Understand how optimizations change your code → You trust the assembly, not the source.

---

Project 6: GDB Python Scripting

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: Python (in GDB)
• Alternative Programming Languages: GDB’s own command language
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 3: Advanced
• Knowledge Area: Debugging Automation
• Software or Tool: GDB with Python support
• Main Book: GDB Documentation (Chapter “Python API”)

What you’ll build: A Python script that runs inside GDB to automate a debugging task. For example, a script that automatically prints all string arguments every time printf is called.

Why it teaches GDB: Manual debugging is repetitive. Scripting is the path to mastery. It allows you to create your own powerful debugging commands and automate complex analysis that would be impossible by hand.

Core challenges you’ll face:

• Running a Python script → maps to the source command in GDB
• Accessing GDB from Python → maps to the gdb Python module
• Creating a new GDB command → maps to subclassing gdb.Command
• Scripting breakpoints → maps to subclassing gdb.Breakpoint

Key Concepts:

• GDB Python API: Official GDB documentation is the best source.
• Extending GDB: gdb.Command, gdb.Function, gdb.Breakpoint.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Python knowledge, Project 1.

Real world outcome: A Python script (myscripts.py) that you load into GDB to add a new command.

# myscripts.py
import gdb

class PrintfTracer(gdb.Breakpoint):
    def __init__(self):
        super(PrintfTracer, self).__init__("printf", gdb.BP_BREAKPOINT, internal=True)

    def stop(self):
        # x86-64 calling convention: first arg is in RDI
        # Let's get the string at the address stored in RDI
        string_addr = gdb.parse_and_eval("$rdi")
        first_arg = gdb.selected_inferior().read_memory(string_addr, 64).tobytes().split(b'\x00')[0]
        print(f"[printf called with: {first_arg.decode('utf-8', 'ignore')}]")
        return False # Don't actually stop, just print and continue

class HelloCommand(gdb.Command):
    def __init__(self):
        super(HelloCommand, self).__init__("hello_gdb", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        print("Hello from your custom GDB command!")

HelloCommand()
PrintfTracer()

Debugging Session:

$ gdb ./my_program
(gdb) source myscripts.py  # Load the script
(gdb) hello_gdb
Hello from your custom GDB command!

(gdb) run
[printf called with: Hello for the 0th time!]
Hello for the 0th time!
[printf called with: Hello for the 1st time!]
Hello for the 1st time!
...

Learning milestones:

1. Create a custom GDB command → You can extend GDB’s vocabulary.
2. Create a scripted breakpoint → You can run code automatically when a breakpoint is hit.
3. Inspect memory and registers from Python → You can perform complex analysis.
4. Build a useful tool → e.g., a “pretty printer” for one of your custom data structures.

---

Project Comparison Table

Project	Difficulty	Time	Depth of Understanding	Fun Factor
The Basics	Level 1: Beginner	1-2 hours	Low	★★☆☆☆
The Crash	Level 1: Beginner	1-2 hours	Medium	★★★☆☆
The Hang	Level 2: Intermediate	1 hour	Medium	★★★☆☆
The Corruption	Level 2: Intermediate	2 hours	High	★★★★☆
The Assembly Level	Level 3: Advanced	3-4 hours	High	★★★★☆
Python Scripting	Level 3: Advanced	1-2 weeks	Very High	★★★★★

Recommendation

Follow the projects in order. Projects 1-4 are essential for any developer. They cover the most common use cases and provide the biggest return on time investment. If you can analyze a crash, attach to a running process, and use watchpoints, you are already a more effective debugger than most.

Once you are comfortable, move to Project 5 (The Assembly Level). This will forever change how you think about your code.

Finally, invest time in Project 6 (Python Scripting). This is the gateway to true GDB mastery and will allow you to build your own toolkit tailored to your specific needs.

---

Final Overall Project: Build a Mini-Debugger

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: C
• Alternative Programming Languages: Rust, Go
• Coolness Level: Level 5: Pure Magic (Super Cool)
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 5: Master
• Knowledge Area: Systems Programming / OS Internals
• Software or Tool: A ptrace-based debugger
• Main Book: “The Linux Programming Interface” by Michael Kerrisk

What you’ll build: A command-line program that uses the ptrace system call to start, stop, and inspect another program. It will be a very simplified GDB.

Why it’s the final goal: This project demystifies debugging entirely. You will no longer see GDB as a black box, but as a C program that is masterfully using the ptrace API. You will understand the fundamental OS mechanics that enable all debuggers.

Core challenges you’ll face:

• Controlling a child process → maps to PTRACE_TRACEME and the fork/exec pattern
• Setting a breakpoint → maps to reading memory (PTRACE_PEEKTEXT), writing an int 3 instruction (PTRACE_POKETEXT), and then restoring it
• Reading registers → maps to PTRACE_GETREGS
• Single-stepping → maps to PTRACE_SINGLESTEP

Real world outcome: A simple REPL that lets you control a target program.

$ ./mini_gdb ./my_program
mini_gdb> break 0x40100a
Breakpoint set at 0x40100a
mini_gdb> continue
Stopped at breakpoint 1: 0x40100a
mini_gdb> regs
rax: 0x5
rbx: 0x0
rip: 0x40100a
mini_gdb> step
Stopped at 0x40100b
mini_gdb> quit

---

Summary

Project	Main Programming Language
The Basics	GDB Commands (on a C target)
The Crash	GDB Commands (on a C target)
The Hang	GDB Commands (on a C target)
The Corruption	GDB Commands (on a C target)
The Assembly Level	GDB Commands / x86 Assembly
Python Scripting	Python (in GDB)
Build a Mini-Debugger	C