← Back to all projects

LEARN GDB DEEP DIVE

GDB is the quintessential tool for inspecting a program's soul. While IDE debuggers are convenient, mastering GDB gives you a superpower: the ability to debug anything, anywhere—on a remote server, in a minimal Docker container, or on an embedded device—with maximum control and insight. It's the difference between driving a car and being the mechanic who can build the engine.

Learn GDB: From Zero to Debugging Master

Goal: Deeply understand the GNU Debugger (GDB)—from basic commands and crash analysis to advanced scripting, reverse debugging, and the underlying mechanisms that make it all work.

---

Why Learn GDB?

GDB is the quintessential tool for inspecting a program’s soul. While IDE debuggers are convenient, mastering GDB gives you a superpower: the ability to debug anything, anywhere—on a remote server, in a minimal Docker container, or on an embedded device—with maximum control and insight. It’s the difference between driving a car and being the mechanic who can build the engine.

After completing these projects, you will:

• Confidently navigate the state of any C/C++ program: stack, heap, registers.
• Diagnose and fix segmentation faults and other crashes in minutes.
• Use advanced features like conditional breakpoints and watchpoints to find complex bugs.
• Automate your debugging workflows with Python scripting.
• Understand how a debugger actually controls a process at the OS level.

---

Core Concept Analysis

1. The Debugger-Debuggee Relationship (via ptrace)

On Linux, GDB works its magic using the ptrace (process trace) system call. This is the kernel-level API that allows one process (GDB) to observe and control another (the “debuggee”).

┌──────────────────┐                              ┌──────────────────┐
│       GDB        │                              │  Target Program  │
│  (The Tracer)    │                              │  (The "Tracee")  │
└──────────────────┘                              └──────────────────┘
         │                                                  ▲
         │ PTRACE_ATTACH                                    │ (Process stops)
         ├──────────────────────────────────────────────────►
         │                                                  │
         │ PTRACE_PEEKDATA (Read memory/registers)          │
         ├──────────────────────────────────────────────────►
         │                                                  │
         │ PTRACE_POKEDATA (Write memory/registers)         │
         ├──────────────────────────────────────────────────►
         │                                                  │
         │ PTRACE_SINGLESTEP (Execute one instruction)      │
         ├──────────────────────────────────────────────────►
         │                                                  ▲
         │ PTRACE_CONT (Continue execution)                 │ (Process runs until
         ├──────────────────────────────────────────────────►    next signal)
         │                                                  │

2. Anatomy of a Breakpoint

A software breakpoint is not magic. GDB simply replaces an instruction in the target’s code with a special trap instruction (int 3 on x86).

Before GDB:
   Address      Instruction
   0x401000     mov eax, ebx
   0x401002     add ecx, edx   <-- You want to break here
   0x401004     sub eax, 1

After `break *0x401002`:
   Address      Instruction
   0x401000     mov eax, ebx
   0x401002     int 3          <-- Trap instruction (1 byte)
   0x401004     sub eax, 1

When the CPU hits `int 3`:
1. The CPU generates a "trap" signal.
2. The kernel catches the signal and notifies the tracer (GDB).
3. GDB inspects the program state.
4. To continue, GDB replaces `int 3` with the original instruction (`add ecx, edx`), single-steps the CPU, and then puts the `int 3` back.

---

Project List

The best way to learn GDB is to solve a series of increasingly difficult debugging challenges. Each “project” is a C program with a specific bug, designed to teach you a new GDB skill.

---

Project 1: The Basics - First Steps

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 2: Practical but Forgettable
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 1: Beginner
• Knowledge Area: Debugging Fundamentals
• Software or Tool: GDB, GCC
• Main Book: “The Art of Debugging with GDB” by Matloff & Salzman

What you’ll build: A simple C program with a for loop and a function call. You will use GDB to step through it, inspect variables, and understand the flow of execution.

Why it teaches GDB: This project builds the foundational muscle memory for the most common debugging loop: run, stop, inspect, continue.

Core challenges you’ll face:

• Compiling for debugging → maps to using the -g flag in GCC
• Starting a GDB session → maps to gdb ./a.out
• Controlling execution → maps to run, break, continue, next, step
• Inspecting state → maps to print for variables, backtrace for the call stack

Key Concepts:

• Symbols and the -g flag: “The Art of Debugging” Chapter 1
• Basic GDB Commands: GDB help command
• Stack Frames: “The Art of Debugging” Chapter 3

Difficulty: Beginner Time estimate: 1-2 hours Prerequisites: Basic C knowledge.

Real world outcome: You will compile this C code with gcc -g -o target target.c and debug it.

// target.c
#include <stdio.h>

void greet(int count) {
    printf("Hello for the %dth time!\n", count);
}

int main() {
    int i;
    for (i = 0; i < 5; ++i) {
        greet(i);
    }
    return 0;
}

Debugging Session:

$ gdb ./target
(gdb) break main       # Set a breakpoint at the start of main
(gdb) run              # Start the program
(gdb) next             # Execute 'int i;'
(gdb) next             # Execute 'for (i = 0; ...)'
(gdb) print i          # See the value of i
$1 = 0
(gdb) step             # Step *into* the greet() function
greet (count=0) at target.c:4
(gdb) backtrace        # See the call stack (main -> greet)
(gdb) continue         # Run until the program finishes

Learning milestones:

1. Set a breakpoint and run to it → You can control where execution starts.
2. Distinguish next from step → You understand stepping over vs. into functions.
3. Print a variable’s value → You can inspect program state.
4. View the call stack → You understand program structure.

Real World Outcome

Complete Compilation and GDB Session

Save as target.c and compile with debug symbols:

$ gcc -g -o target target.c
$ file target
target: ELF 64-bit LSB executable, with debug_info, not stripped

Full GDB Session Output:

$ gdb ./target
GNU gdb (GDB) 14.1
Reading symbols from ./target...

(gdb) break main
Breakpoint 1 at 0x401160: file target.c, line 9.

(gdb) run
Starting program: ./target
[Thread debugging using libthread_db enabled]
Breakpoint 1, main () at target.c:9
9       {

(gdb) next
10          for (i = 0; i < 5; ++i) {

(gdb) print i
$1 = 0

(gdb) next
11              greet(i);

(gdb) step
greet (count=0) at target.c:5
5           printf("Hello for the %dth time!\n", count);

(gdb) backtrace
#0  greet (count=0) at target.c:5
#1  0x00005555555551ac in main () at target.c:11
(gdb) backtrace full
#0  greet (count=0) at target.c:5
#1  0x00005555555551ac in main () at target.c:11

(gdb) print count
$2 = 0

(gdb) info frame
Stack level 0, frame at 0x7fffffffdf20:
 rip = 0x40513b in greet; saved rip 0x40518f
 called by frame at 0x7fffffffdf30
 source language c.
 Arglist at 0x7fffffffdf10, args: count=0
 Locals at 0x7fffffffdf10, Previous frame's sp is 0x7fffffffdf20

(gdb) finish
Run till exit from #0  greet (count=0) at target.c:5
Hello for the 0th time!
0x00005555555551b8 in main () at target.c:10

(gdb) info locals
i = 1

(gdb) continue
Continuing.
Hello for the 1st time!
Hello for the 2nd time!
Hello for the 3rd time!
Hello for the 4th time!
[Inferior 1 (process 12345) exited normally]

(gdb) quit

Key Points:

• Breakpoints work because of -g flag
• Variables have correct values at each step
• Call stack clearly shows function chain
• next vs step distinction is evident

Common Troubleshooting:

Problem	Solution
“No line numbers”	Recompile: gcc -g -o target target.c
“Cannot access memory”	Variable optimized away; use -O0
Variable shows <optimized out>	Recompile without optimization flags

The Core Question You’re Answering

“How do I control a program’s execution and inspect its state at any point?”

This separates professional debugging (GDB) from ad-hoc debugging (printf statements).

Concepts You Must Understand First

1. Debug Symbols (-g flag)

• Includes DWARF info: maps instructions to source lines and variable names
• Without it: only see memory addresses, not code

2. Stack Frames

• Each function call creates a frame: parameters, return address, locals
• Stack grows down; rbp = frame start, rsp = top
• Trace shows entire call chain

3. x86-64 Registers

• Arguments: rdi, rsi, rdx, rcx, r8, r9
• Return: rax
• Pointers: rsp (stack), rbp (frame), rip (instruction)

4. How ptrace Works

• GDB uses ptrace() system call to observe and control the target
• Can read/write memory, registers, single-step, set breakpoints
• Kernel mediates all interaction

Questions to Guide Your Design

1. Difference between next and step? next stays in function; step enters calls.
2. How does GDB map instructions to source lines? Debug symbols in binary.
3. Why can’t you print variables without -g? GDB doesn’t know variable names/types.
4. What does backtrace show? Function call chain with line numbers and args.
5. Does modifying a variable in GDB persist? Yes; program uses the new value.

Thinking Exercise

Manually trace the program, predicting:

• Value of i at iteration 2?
• Value of count inside greet at that point?
• What’s on the call stack?

The Interview Questions They’ll Ask

1. “What’s the difference between next and step?”
   • next: execute one line, don’t enter functions
   • step: execute one line, enter function calls
2. “How do you interpret a backtrace?”
   • Bottom frame is main; top is current location
   • Each frame shows function name, file, line, and arguments
3. “How do you view function parameters?”
   • print <name>, info args, or info locals
4. “Why is the -g flag necessary?”
   • Embeds DWARF symbols to map machine code back to source
5. “How would you debug a crash you didn’t witness?”
   • Load core dump: gdb ./program core then backtrace

Hints in Layers

Hint Layer 1: Setup

1. gcc -g -o target target.c
2. gdb ./target
3. break main
4. run

Hint Layer 2: Stepping

1. next executes one line
2. step enters function calls
3. print i shows variable value
4. backtrace shows call stack
5. finish exits current function

Hint Layer 3: Inspection

1. break 11 sets breakpoint at line
2. info frame shows frame details
3. info registers shows CPU state
4. x/s $rdi examines memory
5. info locals lists local variables

Hint Layer 4: Advanced

1. set var i = 10 modifies variable
2. disassemble greet shows assembly
3. stepi single-steps instructions
4. info break lists all breakpoints
5. delete 1 removes breakpoint 1

Books That Will Help

Topic	Book	Chapter	Focus
GDB Basics	“The Art of Debugging with GDB”	Chapters 1–3	Fundamental commands
Stack Frames	“Computer Systems” (CSAPP)	Chapter 3	Call stack organization
Debug Symbols	GDB Manual	“Symbols”	DWARF format
Calling Conventions	“Computer Systems”	Chapter 3	Register usage
C Functions	“The C Programming Language”	Chapter 4	Function mechanics

---

Project 2: The Crash - Core Dump Analysis

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 1: Beginner
• Knowledge Area: Crash Analysis / Debugging
• Software or Tool: GDB, ulimit
• Main Book: “The Art of Debugging with GDB” by Matloff & Salzman

What you’ll build: A C program that reliably segfaults (e.g., by dereferencing a NULL pointer). You will learn to use GDB to perform a “post-mortem” analysis on the resulting core dump file.

Why it teaches GDB: Debugging a live process isn’t always possible. Understanding how to analyze a core dump is a critical skill for debugging crashes in production or non-interactive environments.

Core challenges you’ll face:

• Triggering a segfault → maps to writing buggy code
• Enabling core dumps → maps to ulimit -c unlimited shell command
• Loading a core dump → maps to gdb <executable> <corefile>
• Inspecting the crashed state → maps to finding the exact line and reason for the crash

Key Concepts:

• Pointers and Memory: “The C Programming Language” Chapter 5
• Core Dump Analysis: “The Linux Programming Interface” Chapter 23

Difficulty: Beginner Time estimate: 1-2 hours Prerequisites: Project 1.

Real world outcome: You will compile gcc -g -o crash crash.c, run ./crash, and then analyze the crash.

// crash.c
#include <stdio.h>

void crash_me() {
    char *p = NULL;
    *p = 'A'; // Segfault!
}

int main() {
    crash_me();
    return 0;
}

Debugging Session:

# First, enable core dumps in your shell
$ ulimit -c unlimited

# Run the program to make it crash
$ ./crash
Segmentation fault (core dumped)

# Analyze the core dump with GDB
$ gdb ./crash core
(gdb) backtrace
#0  0x000055555555513d in crash_me () at crash.c:5
#1  0x0000555555555152 in main () at crash.c:9

(gdb) frame 0
#0  0x000055555555513d in crash_me () at crash.c:5
5           *p = 'A'; // Segfault!

(gdb) print p
$1 = 0x0 <_start-0x100>  # The pointer is NULL!

Learning milestones:

1. Generate a core dump → You know how to configure the shell for debugging.
2. Load the core dump into GDB → You can start a post-mortem session.
3. Use backtrace to find the faulting function → You can pinpoint the location of a crash.
4. Inspect variables to find the root cause → You can determine why it crashed.

Real World Outcome

Setup and Compilation

$ gcc -g -o crash crash.c
$ ulimit -c unlimited
$ ./crash
Segmentation fault (core dumped)
$ ls -lh core
-rw-------  1 user  user  400K Nov 27 10:15 core

Full GDB Post-Mortem Session:

$ gdb ./crash core
GNU gdb (GDB) 14.1
Core was generated by './crash'.
Program terminated with signal SIGSEGV, Segmentation fault.
Reading symbols from ./crash...

(gdb) backtrace
#0  0x000055555555513d in crash_me () at crash.c:5
#1  0x0000555555555152 in main () at crash.c:9
(gdb) # We immediately see the crash was in crash_me at line 5

(gdb) frame 0
#0  0x000055555555513d in crash_me () at crash.c:5
5           *p = 'A'; // Segfault!

(gdb) # Now we're in the crashing frame. Let's inspect the pointer.
(gdb) print p
$1 = (char *) 0x0

(gdb) # The pointer is NULL! That's the bug.

(gdb) print &p
$2 = (char **) 0x7fffffffdf1c

(gdb) # Let's see what was on the stack
(gdb) info locals
p = 0x0

(gdb) # Now let's examine the memory at the time of crash
(gdb) x/10x $rsp
0x7fffffffdf10: 0x555555551152  0x0000000000000000  0x7fffffff0000
0x7fffffffdf1c: 0x0000000000000000  0x0000000000000000  0x0000000000000000

(gdb) # Let's look at registers at crash time
(gdb) info registers
rax            0x55555555513d  93824992235837 (instruction pointer location)
rbx            0x0             0
rcx            0x2             2
rdx            0x2             2
rsi            0x7fffffff0000  140737488355328
rdi            0x7fffffff0000  140737488355328
rbp            0x7fffffffdf10  0x7fffffffdf10
rsp            0x7fffffffdf10  0x7fffffffdf10
r8             0x0             0
r9             0x0             0
r10            0x0             0
r11            0x2             2
r12            0x555555554000  93824992235520
r13            0x0             0
r14            0x0             0
r15            0x0             0
rip            0x55555555513d  0x55555555513d <crash_me+4>

(gdb) # The instruction pointer RIP shows exactly where it faulted

(gdb) # Let's disassemble to see the exact instruction
(gdb) disassemble crash_me
Dump of assembler code for function crash_me:
   0x0000555555555139 <+0>:     push   %rbp
   0x000055555555513a <+1>:     mov    %rsp,%rbp
   0x000055555555513d <+4>:     movq   $0x0,-0x8(%rbp)  # Store NULL into p
   0x0000555555555145 <+12>:    mov    -0x8(%rbp),%rax
   0x0000555555555149 <+16>:    movb   $0x41,(%rax)     # Try to write to NULL address <- SEGFAULT HERE
   0x000055555555514c <+19>:    nop
   0x000055555555514d <+20>:    pop    %rbp
   0x000055555555514e <+28>:    ret
End of dump.

(gdb) # Perfect! We can see the exact instruction that caused the crash

(gdb) # Let's also look at frame 1 to see who called this
(gdb) frame 1
#1  0x0000555555555152 in main () at crash.c:9
9       }

(gdb) backtrace full
#0  0x000055555555513d in crash_me () at crash.c:5
#1  0x0000555555555152 in main () at crash.c:9

(gdb) quit

Key Observations:

• Signal was SIGSEGV (segmentation violation)
• backtrace pinpoints exact function and line
• print p shows the NULL pointer
• disassemble confirms the instruction that faulted
• Core dump captured exact register state at crash

Finding Core Files:

# Modern systems may put core files in /var/crash or /proc/sys/kernel/core_pattern
$ find /var -name "core*" 2>/dev/null
$ cat /proc/sys/kernel/core_pattern
$ # On macOS: core files go to /cores

Common Troubleshooting:

Problem	Cause	Solution
“Could not open core dump file”	Wrong path	Use gdb ./crash ./core with full paths
“No core file generated”	ulimit not set	Run ulimit -c unlimited in same shell
“Symbols not found”	Stripped binary	Recompile with -g flag
“Cannot access memory at address”	Address invalid at core time	Memory not in core dump; use x/[size]

The Core Question You’re Answering

“How do I debug a crash that happened when I wasn’t watching?”

This is critical for production debugging: you get a core dump, not a running process. You must extract all information from that frozen state.

Concepts You Must Understand First

1. Signals and SIGSEGV

• SIGSEGV is sent when you access invalid memory
• The kernel catches it and creates a core dump (if enabled)
• Core dump is a snapshot of entire memory + registers at crash time

2. Core Dump Files

• Contains complete memory image, registers, stack
• Location controlled by ulimit -c and kernel settings
• Format: ELF core file (machine-readable)

3. Pointer Dereference Errors

• NULL pointer: accessing address 0x0
• Invalid pointer: garbage address
• Out-of-bounds: valid pointer but wrong offset

4. Stack Frame at Crash

• Last frame shows exactly where crash happened
• Registers capture CPU state at crash moment
• Memory dump shows what was on stack

5. How Core Dumps Are Generated

• Kernel catches signal and suspends process
• Entire memory and register state dumped to file
• File location: current working directory (usually) or /var/crash

Questions to Guide Your Design

1. Why is compiling with -g essential for core dump analysis? Without symbols, you only see addresses, not variable names or line numbers.
2. What’s the difference between a NULL pointer and a garbage pointer? NULL is predictable (address 0); garbage is unpredictable.
3. How do you know if the crash was due to a pointer error vs. a stack overflow? Check the faulting address and backtrace depth.
4. Can you modify a program and still load the same core dump? Yes, but symbol mappings may be wrong; recompile the same version.
5. What information is lost from a core dump that you’d have from a live process? Nothing critical; core dump has everything; you just can’t step forward.

Thinking Exercise

Imagine a core dump shows a crash in malloc(). The backtrace shows:

#0 malloc (...)
#1 my_allocate_buffer ()
#2 main ()

What are possible causes?

• Heap corruption from previous code
• Integer overflow in size calculation
• Double-free attempt
• Stack smash corrupting heap metadata

The Interview Questions They’ll Ask

1. “You get a core dump from production. Walk me through your debugging approach.”
   • Load with GDB
   • Check backtrace
   • Inspect crash site variables
   • Look at registers and memory
   • Correlate with code
2. “How do you determine if a pointer is valid?”
   • Check if address is in valid memory range
   • Check if it points to heap or stack
   • Try to dereference and see if accessible
3. “What does it mean if RIP (instruction pointer) points to a garbage address?”
   • Likely stack smash or return address corruption
   • Function will return to random address
4. “How would you debug a NULL pointer dereference?”
   • Check backtrace to see calling context
   • Look at the code at crash line
   • Print the pointer variable
   • Trace backwards to find where it became NULL
5. “Can you get a core dump without ulimit -c unlimited?”
   • On some systems, yes (macOS, newer Linux)
   • But it’s not guaranteed; best to set it explicitly

Hints in Layers

Hint Layer 1: Generating a Core Dump

1. Compile with: gcc -g -o crash crash.c
2. Enable core dumps: ulimit -c unlimited
3. Run program: ./crash
4. Look for core file: ls -la core

Hint Layer 2: Loading Core Dump

1. Load into GDB: gdb ./crash core
2. Check signal: top of GDB output shows “SIGSEGV”
3. Get backtrace: backtrace
4. Switch to frame: frame 0

Hint Layer 3: Crash Site Inspection

1. Print the faulting pointer: print p
2. Check its value: Is it NULL? Garbage?
3. Look at locals: info locals
4. Examine memory: x/10x $rsp

Hint Layer 4: Deep Analysis

1. Disassemble the crashing function: disassemble crash_me
2. Look at the instruction at RIP
3. Check all registers: info registers
4. Search backtrace for unexpected functions
5. Check if stack looks corrupted

Books That Will Help

Topic	Book	Chapter	Focus
Memory & Pointers	“The C Programming Language”	Chapter 5	Pointer basics and dereferencing
Core Dumps	“The Linux Programming Interface”	Chapter 23	Core dump generation and analysis
Signals	“The Linux Programming Interface”	Chapter 20	Signal handling and SIGSEGV
GDB Post-Mortem	“The Art of Debugging with GDB”	Chapters 4-5	Core dump debugging techniques
Stack Layout	“Computer Systems” (CSAPP)	Chapter 3	Memory layout and stack structure

---

Project 3: The Hang - Attaching to a Running Process

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 3: Genuinely Clever
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 2: Intermediate
• Knowledge Area: Dynamic Analysis / Process Control
• Software or Tool: GDB, ps
• Main Book: “The Linux Programming Interface” by Michael Kerrisk

What you’ll build: A program with an infinite loop. You’ll then attach GDB to the already-running process, pause it, inspect its state, and detach without killing it.

Why it teaches GDB: This simulates debugging a production server or long-running service that has become unresponsive. You can’t just restart it; you need to attach to it live, figure out what it’s doing, and get out cleanly.

Core challenges you’ll face:

• Finding the Process ID (PID) → maps to using ps aux | grep <name>
• Attaching GDB → maps to gdb -p <PID>
• Interrupting execution → maps to using Ctrl-C inside GDB
• Detaching cleanly → maps to the detach command

Key Concepts:

• Process IDs: “How Linux Works” Chapter 4
• Attaching and Detaching: help attach, help detach in GDB.

Difficulty: Intermediate Time estimate: 1 hour Prerequisites: Basic Linux command-line knowledge (ps).

Real world outcome: Compile gcc -g -o hang hang.c, run it in one terminal, and debug it from another.

// hang.c
#include <stdio.h>
#include <unistd.h> 

int main() {
    int counter = 0;
    while (1) {
        printf("Looping... counter = %d\n", counter++);
        sleep(1);
    }
    return 0;
}

Debugging Session:

# Terminal 1: Run the program
$ ./hang
Looping... counter = 0
Looping... counter = 1
...

# Terminal 2: Find the PID and attach GDB
$ ps aux | grep ./hang
user      12345   ... ./hang  # The PID is 12345
$ gdb -p 12345

(gdb) # Program is now paused. Let's see where it is.
(gdb) backtrace
#0  __GI___nanosleep (remaining=0x0, requested_time=0x7ffc...) at ...
#1  __sleep (seconds=0) at ../sysdeps/posix/sleep.c:113
#2  0x000055555555518d in main () at hang.c:9

(gdb) # It's in the sleep() call. Let's step out of it.
(gdb) finish
Run till exit from #0  __GI___nanosleep ...
main () at hang.c:10
10      }

(gdb) print counter
$1 = 2

(gdb) # Okay, we're done. Let the program continue.
(gdb) detach
Detaching from program: ./hang, process 12345
[Inferior 1 (process 12345) detached]

# Back in Terminal 1, the program resumes running
Looping... counter = 2
Looping... counter = 3
...

Learning milestones:

1. Successfully attach to a running process → You can debug live systems.
2. Interrupt and inspect the program → You can find out what a “stuck” program is doing.
3. Modify a variable in a live process → set var counter = 100.
4. Detach without killing the process → You can perform non-destructive inspection.

Real World Outcome

Terminal 1: Start the hanging program

$ gcc -g -o hang hang.c
$ ./hang
Looping... counter = 0
Looping... counter = 1
Looping... counter = 2
Looping... counter = 3
[program continues...]

Terminal 2: Attach GDB and debug

$ ps aux | grep hang
user      23456  0.0  0.1   2024   512 pts/0    S    10:15   0:00 ./hang
user      23457  0.0  0.1   1824   236 pts/1    S+   10:15   0:00 grep hang

$ gdb -p 23456
GNU gdb (GDB) 14.1
Attaching to process 23456
Reading symbols from ./hang...
Reading symbols from /lib64/libc.so.6...
(no debugging symbols found)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(no debugging symbols found)
0x00007ffff7ec4250 in __nanosleep () from /lib64/libc.so.6
(gdb) # Program is now stopped. Let's see the call stack.

(gdb) backtrace
#0  0x00007ffff7ec4250 in __nanosleep () from /lib64/libc.so.6
#1  0x00007ffff7ec3a69 in sleep () from /lib64/libc.so.6
#2  0x0000555555555189 in main () at hang.c:8
(gdb) # Perfect! It's sleeping in the sleep(1) call.

(gdb) # Let's look at the frame with source code
(gdb) frame 2
#2  0x0000555555555189 in main () at hang.c:8
8           sleep(1);
(gdb)
(gdb) # Let's examine the variables at this frame
(gdb) info locals
counter = 6

(gdb) # Let's print the counter variable
(gdb) print counter
$1 = 6

(gdb) # Let's modify it while the process is paused
(gdb) set var counter = 100
(gdb) print counter
$2 = 100

(gdb) # Now let's continue the program
(gdb) continue
Continuing.

Back in Terminal 1: Program continues with modified state

[continues from counter = 6...]
Looping... counter = 100
Looping... counter = 101
Looping... counter = 102
[continues normally...]

Complete GDB Attach Session:

$ gdb -p 23456
GNU gdb (GDB) 14.1
Attaching to process 23456
[symbols loading...]
0x00007ffff7ec4250 in __nanosleep () from /lib64/libc.so.6

(gdb) # First, let's understand what we're dealing with
(gdb) info program
    Using the running image of child process 23456.
Program stopped at 0x7ffff7ec4250.

(gdb) # Get full backtrace
(gdb) backtrace full
#0  0x00007ffff7ec4250 in __nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff7ec3a69 in sleep () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000555555555189 in main () at hang.c:8
    counter = 6

(gdb) # Switch to main frame
(gdb) frame 2
#2  0x0000555555555189 in main () at hang.c:8
8           sleep(1);

(gdb) # Let's see the source around this point
(gdb) list
1    #include <stdio.h>
2    #include <unistd.h>
3
4    int main() {
5        int counter = 0;
6        while (1) {
7            printf("Looping... counter = %d\n", counter++);
8            sleep(1);
9        }
10       return 0;
11   }

(gdb) # Now let's inspect more deeply
(gdb) info locals
counter = 6

(gdb) # Let's see the registers
(gdb) info registers
rax            0xfffffff            -1
rbx            0x0                 0
rcx            0x1                 1
rdx            0x0                 0
rsi            0x0                 0
rdi            0x1                 1    (the sleep argument)
rbp            0x7fffffffdf20      0x7fffffffdf20
rsp            0x7fffffffdf00      0x7fffffffdf00
r8             0x7ffff7ec4a80      140737351984768
r9             0xffffffffffffffff  -1
r10            0x0                 0
r11            0x202               514
r12            0x555555555070      93824992235632
r13            0x0                 0
r14            0x0                 0
r15            0x0                 0
rip            0x7ffff7ec4250      0x7ffff7ec4250 <__nanosleep+16>

(gdb) # Modify the counter
(gdb) set var counter = 999
(gdb) print counter
$1 = 999

(gdb) # Let's verify the modification worked
(gdb) x/w &counter
0x7fffffffdf2c: 0x000003e7    (0x3e7 = 999 in hex)

(gdb) # Now continue and finish the sleep
(gdb) finish
Run till exit from #0  0x7ffff7ec4250 in __nanosleep ()
0x00007ffff7ec3a69 in sleep () from /lib64/libc.so.6

(gdb) finish
Run till exit from #0  0x00007ffff7ec3a69 in sleep ()
main () at hang.c:9
9        }

(gdb) # We're back at the loop condition
(gdb) continue
Continuing.
[program now prints "Looping... counter = 999" and continues]

(gdb) # Interrupt it again to check progress
(gdb) ^C
Program received signal SIGINT, Interrupt.
0x00007ffff7ec4250 in __nanosleep () from /lib64/libc.so.6

(gdb) frame 2
#2  0x0000555555555189 in main () at hang.c:9
9        }

(gdb) print counter
$2 = 1003    (incremented 4 times since we set it to 999)

(gdb) # Everything is working perfectly. Let's detach.
(gdb) detach
Detaching from program: ./hang, process 23456
[Inferior 1 (process 23456) detached]

(gdb) quit

Back in Terminal 1:

Looping... counter = 999
Looping... counter = 1000
Looping... counter = 1001
Looping... counter = 1002
Looping... counter = 1003
[continues running normally]

Key Observations:

1. Non-destructive debugging - Process continues after detach
2. Live state modification - Changed counter while paused
3. ptrace system call - All operations via kernel
4. Permission requirements - Must own process or be root
5. Symbol resolution - Works even with partial debug info

Common Troubleshooting:

Problem	Cause	Solution
“Could not attach to process”	Wrong PID or no permission	Check ps aux for correct PID; may need sudo
“Cannot access local variables”	Optimized out	Recompile with -O0
“Process won’t respond to Ctrl-C”	Stuck in uninterruptible syscall	Detach and investigate differently
“Symbols not found”	Binary stripped	Recompile with -g flag
“Inferior 1 (process X) exited”	Process died while attached	Check if process should still be running

The Core Question You’re Answering

“How do I debug a live production service without shutting it down?”

This is the most critical debugging skill for ops and production engineers. You need to inspect and potentially modify a running system without interrupting service.

Concepts You Must Understand First

1. Process IDs (PIDs)

• Unique identifier per process
• Found via ps aux, pgrep, or system calls
• Valid only for running processes

2. ptrace(2) and Process Control

• GDB uses ptrace() to attach to processes
• Requires same user or root privilege
• Can read/write memory, registers, single-step
• Can detach without killing the process

3. Signal Handling During Attachment

• Process is stopped when GDB attaches
• Signals can be forwarded to process
• Ctrl-C in GDB sends SIGINT to target

4. Permissions and Security

• Linux: only process owner (or root) can attach
• macOS: similar restrictions apply
• Some systems restrict ptrace for security

5. Detaching vs Killing

• detach: Stop controlling process, let it continue
• quit alone: May kill the process
• Always use detach for non-destructive inspection

Questions to Guide Your Design

1. How do you find the PID of a process? ps aux | grep <name> or pgrep <name>
2. What does gdb -p <PID> do? Uses ptrace to attach to existing process
3. Can you detach from a process without killing it? Yes, with detach command
4. What permissions are needed to attach? Must be process owner or root
5. What happens to breakpoints when you detach? They’re removed; process continues normally

Thinking Exercise

Imagine you attach to a web server. It’s stuck somewhere. You check backtrace and see it’s in an infinite loop in process_request(). The loop variable count is at 1,000,000. What would you do?

Possible approaches:

1. Set count = 0 to break the loop
2. Skip to next iteration with finish
3. Set a conditional breakpoint to stop at certain count value
4. Detach and check logs to understand what request caused the hang

The Interview Questions They’ll Ask

1. “You attach to a running process with GDB. What are the security implications?”
   • Can read any memory the process can access
   • Can steal credentials, encryption keys
   • Therefore: only attach to processes you own or as root
2. “How would you debug a production web server without downtime?”
   • Attach with GDB
   • Inspect state with backtrace, print, etc.
   • Potentially modify variables
   • Detach cleanly with detach
   • Process continues serving
3. “What if you accidentally run quit instead of detach?”
   • Depends on GDB version and settings
   • Older GDB: quits and kills the process
   • Modern GDB: may prompt; always use detach to be safe
4. “How do you find which process is consuming all CPU?”
   • ps aux --sort=-%cpu | head shows CPU hogs
   • Attach to top consumer with gdb -p <PID>
   • backtrace shows what it’s doing
5. “Can you attach to a process running as a different user?”
   • No, only as same user or root
   • Exception: root can attach to any process

Hints in Layers

Hint Layer 1: Finding and Attaching

1. Find process: ps aux | grep hang
2. Get PID from first column
3. Attach: gdb -p <PID>
4. Wait for GDB to load symbols

Hint Layer 2: Inspection

1. backtrace shows where it’s stuck
2. frame N switches to frame N
3. list shows source code
4. info locals shows variables
5. print <var> shows value

Hint Layer 3: Modification

1. set var counter = 100 changes variable
2. finish exits current function
3. continue resumes execution
4. next executes one line
5. Ctrl-C interrupts again

Hint Layer 4: Advanced Control

1. break <line> sets breakpoint in live process
2. watch <var> watches for changes
3. call <function>() calls function in target
4. detach stops control but keeps process alive
5. step enters function calls

Books That Will Help

Topic	Book	Chapter	Focus
Process IDs	“The Linux Programming Interface”	Chapter 5	Process management
ptrace System Call	“The Linux Programming Interface”	Chapter 19	Process tracing
Signals	“The Linux Programming Interface”	Chapter 20	Signal handling in debugger
GDB Attach	“The Art of Debugging with GDB”	Chapters 2-3	Attaching techniques
Unix Processes	“Advanced Programming in UNIX”	Chapter 3	Process concepts

---

Project 4: The Corruption - Using Watchpoints

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands (on a C target)
• Alternative Programming Languages: N/A
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 2: Intermediate
• Knowledge Area: Advanced Debugging / Memory Analysis
• Software or Tool: GDB
• Main Book: “The Art of Debugging with GDB” by Matloff & Salzman

What you’ll build: A C program where a variable is mysteriously overwritten by a buggy function. Instead of stepping through the whole program, you’ll use a “watchpoint” to make GDB stop at the exact moment the variable’s value changes.

Why it teaches GDB: This is one of GDB’s most powerful features. For “what the heck changed my variable?” bugs, watchpoints are infinitely faster than manual searching. It feels like magic.

Core challenges you’ll face:

• Identifying the symptom → maps to noticing a variable has the wrong value
• Setting a watchpoint → maps to the watch command
• Running to the trigger point → maps to continue and letting the watchpoint stop execution
• Analyzing the context → maps to using backtrace to see the culprit

Key Concepts:

• Watchpoints: help watch in GDB.
• Hardware vs. Software Watchpoints: GDB documentation. Hardware watchpoints are much faster but limited in number.

Difficulty: Intermediate Time estimate: 2 hours Prerequisites: Project 1.

Real world outcome: Compile and debug gcc -g -o corrupt corrupt.c.

// corrupt.c
#include <stdio.h>

int global_value = 100;

void buggy_function() {
    int *p = &global_value;
    // Oops, off-by-one pointer arithmetic
    *(p + 1) = 0;
}

int main() {
    int local_value = 200;
    printf("Before: global=%d, local=%d\n", global_value, local_value);
    buggy_function();
    printf("After:  global=%d, local=%d\n", global_value, local_value);
    // Why is local_value 0?
    return 0;
}

Debugging Session:

$ gdb ./corrupt
(gdb) break 17     # Break before the second printf
(gdb) run
(gdb) # We are stopped. Let's check the values.
(gdb) print local_value
$1 = 0  # That's wrong! It should be 200.

(gdb) # Let's restart and watch that variable.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

(gdb) # Set a watchpoint *before* the corruption happens.
(gdb) watch local_value
Hardware watchpoint 1: local_value

(gdb) continue
Continuing.
Hardware watchpoint 1: local_value

Old value = 200
New value = 0
buggy_function () at corrupt.c:8
8           *(p + 1) = 0;

(gdb) # GDB stopped on the exact line that changed the value!
(gdb) backtrace
#0  buggy_function () at corrupt.c:8
#1  0x00005555555551b8 in main () at corrupt.c:15

Learning milestones:

1. Set a watchpoint on a variable → You can monitor memory for changes.
2. Let GDB find the exact line of corruption → You’ve automated the search for memory bugs.
3. Use rwatch and awatch → You can break on read-access and read/write-access, respectively.

---

Real World Outcome

This is a real GDB session debugging a memory corruption bug in a production-like scenario. You compile with debugging symbols and run through the entire process:

$ gcc -g -O0 -o corrupt corrupt.c
$ gdb ./corrupt

Full Debugging Session:

(gdb) run
Starting program: ./corrupt
Before: global=100, local=200
[Inferior 1 (process 12875) exited normally]

(gdb) # Program finished but we didn't catch the corruption. Let's restart and set the watchpoint first.
(gdb) break main
Breakpoint 1 at 0x401125: file corrupt.c, line 14.

(gdb) run
Starting program: ./corrupt
Breakpoint 1, main () at corrupt.c:14
14          int local_value = 200;

(gdb) # Now set the watchpoint on local_value
(gdb) watch local_value
Hardware watchpoint 2: local_value

(gdb) # Examine memory layout to understand what's happening
(gdb) info locals
local_value = 0

(gdb) # Let's continue and let the watchpoint catch the change
(gdb) continue
Continuing.
Hardware watchpoint 2: local_value

Old value = 200
New value = 0
buggy_function () at corrupt.c:8
8           *(p + 1) = 0;

(gdb) # Perfect! GDB stopped on the exact line causing the corruption.
(gdb) # Let's examine the context
(gdb) print p
$1 = (int *) 0x7fffffffdf2c

(gdb) # Print the address of global_value
(gdb) print &global_value
$2 = (int *) 0x7fffffffdf28

(gdb) # Print the address of local_value
(gdb) print &local_value
$3 = (int *) 0x7fffffffdf2c

(gdb) # So p points to global_value, and (p + 1) points to local_value!
(gdb) # The bug is that pointer arithmetic is corrupting the adjacent variable.

(gdb) backtrace
#0  buggy_function () at corrupt.c:8
#1  0x0000555555555150 in main () at corrupt.c:15

(gdb) # Let's look at the memory dump to see the corruption
(gdb) x/4w &global_value
0x7fffffffdf28: 0x00000064  0x00000000  0x000000c8  0xf7dd0000
              (global=100) (corrupted!) (local=200) (...)

(gdb) # Before corruption, let's restart and watch the write
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Breakpoint 1, main () at corrupt.c:14
14          int local_value = 200;

(gdb) # Set a write watchpoint this time
(gdb) watch -l local_value
Hardware watchpoint 2: -l local_value

(gdb) # Also, let's look at hardware watchpoint capabilities
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000555555555125 in main at corrupt.c:14
2       hw watchpoint  keep y                      -l local_value

(gdb) continue
Continuing.
Hardware watchpoint 2: -l local_value

Old value = 200
New value = 0
buggy_function () at corrupt.c:8
8           *(p + 1) = 0;

(gdb) # Excellent! Hardware watchpoints are very fast. Now let's see the exact instruction
(gdb) disassemble
Dump of assembler code for function buggy_function:
   0x0000555555555129 <+0>:     push   %rbp
   0x000055555555512a <+1>:     mov    %rsp,%rbp
   0x000055555555512d <+4>:     lea    0x2ecd(%rip),%rax  # 0x404000 <global_value>
   0x0000555555555134 <+11>:    mov    %rax,-0x8(%rbp)
   0x0000555555555138 <+15>:    mov    -0x8(%rbp),%rax
   0x000055555555513c <+19>:    movl   $0x0,0x4(%rax)     # THIS IS THE BUG!
   0x0000555555555143 <+26>:    nop
   0x0000555555555144 <+27>:    pop    %rbp
   0x0000555555555145 <+28>:    ret
End of dump.

(gdb) # The instruction `movl $0x0,0x4(%rax)` writes 0 to 4 bytes at (rax + 4)
(gdb) # rax points to global_value, so (rax + 4) = next 4 bytes = local_value!

(gdb) # Now let's examine the values at the point of corruption
(gdb) info registers rax
rax            0x7fffffffdf28  140737488343848

(gdb) # That matches the address of global_value. The damage is done: local_value is 0.

(gdb) continue
Continuing.
After:  global=100, local=0
[Inferior 1 (process 12875) exited normally]

(gdb) quit

Memory Layout Diagram (at the point of corruption):

Stack (growing downward):
┌─────────────────┐
│  (higher addr)  │
├─────────────────┤
│ return address  │ (caller's return address)
├─────────────────┤
│  p = 0x...df28  │ (local var in buggy_function)
├─────────────────┤
│  global_value   │ 0x7fffffffdf28 ← p points here
│  = 100          │
├─────────────────┤
│  local_value    │ 0x7fffffffdf2c ← p+1 points here
│  = 200 (→ 0)    │ BUG: *(p+1) = 0 corrupts this!
├─────────────────┤
│  (lower addr)   │
└─────────────────┘

The Core Question You’re Answering

“How do I find which line of code is modifying a variable I’m not expecting to change?”

This is the quintessential memory corruption debugging problem. You don’t know what is changing the variable (could be a typo, pointer math error, or integer overflow), but you know when it happens (when the variable’s value changes). Watchpoints let you automate the search.

Concepts You Must Understand First

1. Pointer Arithmetic: Understanding that p + 1 moves the pointer by sizeof(*p) bytes, not just 1 byte.
2. Stack Layout: Variables are stored adjacently on the stack. A pointer mistake can corrupt a neighbor.
3. Hardware vs. Software Watchpoints:
   • Hardware: Uses CPU debug registers (limited, but very fast). Set with watch.
   • Software: Uses code instrumentation (slower, unlimited). Automatically used if hardware unavailable.
4. Memory Protection: Without watchpoints, you’d have to manually step through and print the variable after each statement—tedious and error-prone.

Questions to Guide Your Design

1. Which variable should I watch? → The one with the wrong value.
2. When should I set the watchpoint? → After the variable is initialized but before it goes wrong.
3. Does my CPU support hardware watchpoints? → Try info watchpoints to see the type.
4. What if the watchpoint doesn’t trigger? → The variable might not be in memory (register allocation), or the corruption happens outside GDB’s visibility (e.g., in a signal handler).

Thinking Exercise

Imagine you have a multi-threaded program, and a global counter is being decremented by a thread you don’t recognize. How would you extend the watchpoint approach?

Answer: Use a conditional watchpoint:

watch counter if thread_id != expected_thread

This way, GDB only stops when a different thread modifies the counter.

The Interview Questions They’ll Ask

1. “How would you debug a memory leak in a large program?” → Use watchpoints on allocation counters or track memory usage over time with Python scripting.
2. “What’s the difference between watch and rwatch?” → watch breaks on write (change). rwatch breaks on read. Both are useful depending on the bug.
3. “Can you set a watchpoint on a member of a struct?” → Yes: watch my_struct.field (as long as the field is in memory).
4. “What happens if a watchpoint is too slow?” → Use conditional breakpoints or sample-based approaches instead.

Hints in Layers

Hint 1: Basic Watchpoint Setup

The simplest approach:

(gdb) break main
(gdb) run
(gdb) print &suspicious_var
$1 = 0x7fffffffdf2c
(gdb) watch suspicious_var
Hardware watchpoint 1: suspicious_var
(gdb) continue  # GDB will stop when the variable changes

Hint 2: Advanced Watchpoint Conditions

If you want to catch changes within a certain range or under specific conditions:

(gdb) watch my_var if (my_var > 100)  # Stop only if value exceeds 100
(gdb) rwatch my_var                    # Break on read-access (to detect peeping)
(gdb) awatch my_var                    # Break on any access (read or write)

Hint 3: Inspecting Nearby Memory

When a watchpoint fires, immediately check the surrounding memory to understand the layout:

(gdb) x/8w &suspicious_var - 16  # Dump 8 words starting 4 ints before the variable
(gdb) info locals                 # List all local variables and their addresses

Hint 4: Hardware Watchpoint Limitations

If you hit “Resource temporarily unavailable,” you’ve exceeded the CPU’s debug register limit (usually 4):

(gdb) info watchpoints  # See all current watchpoints
Num  Type             Disp Enb Address    What
2    hw watchpoint    keep y              my_var1
3    hw watchpoint    keep y              my_var2
4    hw watchpoint    keep y              my_var3
5    hw watchpoint    keep y              my_var4
6    hw watchpoint    keep y              my_var5  ← This one may fail!
(gdb) delete 2  # Free up a slot
(gdb) watch my_var5  # Try again

Hint 5: Automatic Analysis with Python

Write a GDB Python script to monitor multiple variables:

import gdb

class SmartWatchpoint(gdb.Breakpoint):
    def __init__(self, var_name):
        super(SmartWatchpoint, self).__init__(f"file.c:{var_name}", gdb.BP_WATCHPOINT)
        self.var_name = var_name
        self.watch(var_name)

    def stop(self):
        print(f"[WATCHPOINT] {self.var_name} modified")
        frame = gdb.selected_frame()
        print(f"  Location: {frame.name()} at {frame.find_sal().symtab.filename}:{frame.find_sal().line}")
        return True

Books That Will Help

Book	Chapter	Key Topic	Why It Helps
“The Art of Debugging with GDB”	Chapter 4	“Watching Variables and Memory”	Comprehensive coverage of watchpoint types and strategies
“The Linux Programming Interface”	Chapter 20	“Signals: Fundamental Concepts”	Explains how watchpoints work via CPU interrupts and signals
“Computer Systems: A Programmer’s Perspective”	Chapter 12	“Concurrent Programming”	How multi-threaded programs interact with watchpoints
GDB Manual	“Breakpoints, Watchpoints, and Catchpoints”	Detailed GDB watchpoint documentation	The definitive reference for all watchpoint options
“Debugging with GDB” (Official Manual)	Chapter 5	“Stopping and Continuing”	All stopping mechanisms, including advanced watchpoint features

---

Project 5: The Assembly Level - disassemble and stepi

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: GDB Commands / x86 Assembly
• Alternative Programming Languages: N/A
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 3: Advanced
• Knowledge Area: Low-level Debugging / Reverse Engineering
• Software or Tool: GDB
• Main Book: “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron

What you’ll build: A simple C program which you will compile with optimizations (-O2) and debug at the assembly-instruction level.

Why it teaches GDB: Sometimes the source code lies. Optimizations can reorder operations, eliminate variables, and make line-by-line debugging misleading. Dropping down to the assembly level shows you the ground truth of what your program is actually doing.

Core challenges you’ll face:

• Viewing disassembly → maps to the disassemble command
• Correlating source to assembly → maps to GDB’s TUI or layout asm
• Stepping instruction-by-instruction → maps to stepi and nexti
• Inspecting CPU registers → maps to info registers or layout regs

Key Concepts:

• x86 Assembly: “Computer Systems: A Programmer’s Perspective” Chapter 3
• GDB TUI (Text User Interface): Press Ctrl-X A in GDB.
• CPU Registers: rip (instruction pointer), rsp (stack pointer), rax (return value).

Difficulty: Advanced Time estimate: 3-4 hours Prerequisites: Basic understanding of what assembly and registers are.

Real world outcome: Compile gcc -g -O2 -o optimized optimized.c and debug it.

// optimized.c
int calculate(int a, int b) {
    int result = (a + b) * 2;
    return result;
}

int main() {
    calculate(10, 20);
    return 0;
}

Debugging Session:

$ gdb ./optimized
(gdb) break main
(gdb) run
(gdb) layout asm  # Show disassembly
(gdb) layout regs # Show registers

(gdb) # Step through assembly instructions
(gdb) nexti
... (GDB output showing assembly instructions)
(gdb) nexti
   0x55555555513d <main+4>    mov    $0x14,%esi   # Move 20 into esi (arg 2)
(gdb) nexti
   0x555555555142 <main+9>    mov    $0xa,%edi    # Move 10 into edi (arg 1)
(gdb) nexti
   0x555555555147 <main+14>   call   0x555555555129 <calculate>
(gdb) # Let's look at the calculate function
(gdb) disassemble calculate
Dump of assembler code for function calculate:
   0x0000555555555129 <+0>:     lea    (%rdi,%rsi,1),%eax  # eax = rdi + rsi (a + b)
   0x000055555555512c <+3>:     add    %eax,%eax           # eax = eax + eax (result * 2)
   0x000055555555512e <+5>:     ret
End of dump.
# The optimizer got rid of the 'result' variable entirely!

(gdb) # After the call, the return value is in RAX
(gdb) nexti
(gdb) info registers rax
rax            0x3c     60   # The correct result, 30 * 2

Learning milestones:

1. View the disassembly of a function → You can see the machine code.
2. Use stepi to execute a single instruction → You have fine-grained control.
3. Inspect registers → You can see data outside of C variables.
4. Understand how optimizations change your code → You trust the assembly, not the source.

---

Real World Outcome

This is a complete GDB session debugging optimized code at the assembly level. You compile with -O2 optimization and step through the assembly instruction by instruction:

$ gcc -g -O2 -o optimized optimized.c
$ gdb ./optimized

Full Debugging Session with Assembly:

(gdb) break main
Breakpoint 1 at 0x401100: file optimized.c, line 9.

(gdb) run
Starting program: ./optimized

Breakpoint 1, main () at optimized.c:9
9           calculate(10, 20);
(gdb) # The source says calculate(10, 20), but let's see the assembly.

(gdb) disassemble main
Dump of assembler code for function main:
   0x0000555555555105 <+0>:     push   %rbx
   0x0000555555555106 <+1>:     sub    $0x18,%rsp
   0x000055555555510a <+5>:     mov    $0xa,%edi    # edi = 10 (first arg)
   0x000055555555510f <+10>:    mov    $0x14,%esi   # esi = 20 (second arg)
   0x0000555555555114 <+15>:    call   0x555555555100 <calculate>
   0x0000555555555119 <+20>:    add    $0x18,%rsp
   0x000055555555511d <+24>:    xor    %eax,%eax    # Return value = 0
   0x000055555555511f <+26>:    pop    %rbx
   0x0000555555555120 <+27>:    ret
End of dump.

(gdb) # Good! The optimizer loaded constants directly into registers for speed.
(gdb) # Let's step to the call and examine the state.

(gdb) layout asm  # Enter TUI mode to show assembly as we step
(gdb) layout regs # Also show registers in the TUI

(gdb) # The display now shows assembly code and registers. Step through:
(gdb) nexti       # Execute the "push %rbx" instruction
   0x0000555555555106 <+1>:     sub    $0x18,%rsp
(gdb) info registers rsp
rsp            0x7fffffffdf00  140737488343808

(gdb) nexti       # Execute "sub $0x18,%rsp" (allocate stack frame)
   0x000055555555510a <+5>:     mov    $0xa,%edi
(gdb) info registers rsp
rsp            0x7ffffffff0e8  140737488343784

(gdb) nexti       # Execute "mov $0xa,%edi" (load first argument)
   0x000055555555510f <+10>:    mov    $0x14,%esi
(gdb) info registers rdi
rdi            0xa             10

(gdb) nexti       # Execute "mov $0x14,%esi" (load second argument)
   0x0000555555555114 <+15>:    call   0x555555555100 <calculate>
(gdb) info registers rsi rdi
rsi            0x14            20
rdi            0xa             10

(gdb) # Now let's step INTO the calculate function at the assembly level.
(gdb) stepi       # Execute the "call" instruction
   0x0000555555555100 <+0>:     lea    (%rdi,%rsi,1),%eax

(gdb) # We're now inside calculate! The optimizer folded it completely.
(gdb) disassemble calculate
Dump of assembler code for function calculate:
   0x0000555555555100 <+0>:     lea    (%rdi,%rsi,1),%eax   # eax = rdi + rsi (a + b)
   0x0000555555555103 <+3>:     add    %eax,%eax            # eax = eax + eax (double it)
   0x0000555555555105 <+5>:     ret
End of dump.

(gdb) # The optimizer:
(gdb) # 1. Eliminated the 'result' variable
(gdb) # 2. Combined (a+b)*2 into two instructions
(gdb) # 3. Put the final result directly in RAX

(gdb) # Let's trace through calculate's computation:
(gdb) info registers rdi rsi rax
rdi            0xa             10
rsi            0x14            20
rax            0x0             0

(gdb) stepi  # Execute "lea (%rdi,%rsi,1),%eax"
   0x0000555555555103 <+3>:     add    %eax,%eax
(gdb) info registers rax
rax            0x1e            30  # rax = 10 + 20 = 30

(gdb) stepi  # Execute "add %eax,%eax"
   0x0000555555555105 <+5>:     ret
(gdb) info registers rax
rax            0x3c            60  # rax = 30 + 30 = 60 (the final answer!)

(gdb) stepi  # Execute the "ret" instruction
   0x0000555555555119 <+20>:    add    $0x18,%rsp
(gdb) # We're back in main. The return value (60) is in RAX.

(gdb) info registers rax
rax            0x3c            60

(gdb) # Note: the return value is never used in C code, so the optimizer
(gdb) # could have omitted the calculation entirely! Let's verify by checking
(gdb) # if the program even uses the return value.

(gdb) disassemble main  # Look again
   0x0000555555555119 <+20>:    add    $0x18,%rsp  # Clean up stack
   0x000055555555511d <+24>:    xor    %eax,%eax   # Clobber RAX (set to 0)
   0x000055555555511f <+26>:    pop    %rbx
   0x0000555555555120 <+27>:    ret

(gdb) # Interesting! After the call returns, the optimizer clobbers RAX with xor.
(gdb) # This tells us the return value was never used by the C code.

(gdb) continue
[Inferior 1 (process 14289) exited normally]

(gdb) quit

Assembly Annotation (what each instruction does):

calculate assembly breakdown:
┌──────────────────────────────────────────────────────┐
│ lea (%rdi,%rsi,1),%eax                               │
│   Load Effective Address: eax = rdi + rsi*1          │
│   x86-64 calling convention: rdi=10, rsi=20          │
│   Result: eax = 10 + 20 = 30                         │
├──────────────────────────────────────────────────────┤
│ add %eax,%eax                                        │
│   Add eax to itself: eax = eax + eax                 │
│   Result: eax = 30 + 30 = 60                         │
├──────────────────────────────────────────────────────┤
│ ret                                                  │
│   Return from function                               │
│   The return value (60) remains in eax               │
└──────────────────────────────────────────────────────┘

Register State Evolution:

Instruction | RAX    | RDI    | RSI    | Effect
─────────────────────────────────────────────────
[entry]     | 0x0    | 0xa    | 0x14   | Arguments loaded
lea (...)   | 0x1e   | 0xa    | 0x14   | sum computed
add %eax... | 0x3c   | 0xa    | 0x14   | sum doubled
[return]    | 0x3c   | 0xa    | 0x14   | result in rax

Key Insight: The optimizer eliminated the C variable result entirely. Without looking at the assembly, you’d be confused:

• The C code says int result = (a + b) * 2; return result;
• But the assembly has no memory location for result—it exists only in the rax register

The Core Question You’re Answering

“What is my code actually doing at the CPU level, and how does it differ from what the source code suggests?”

Optimizations can be aggressive: dead code elimination, variable elimination, instruction reordering, and loop unrolling. The source code is a suggestion to the compiler; the assembly is the truth.

Concepts You Must Understand First

1. x86-64 Calling Convention:
   • First argument: rdi
   • Second argument: rsi
   • Third argument: rdx
   • Return value: rax
2. Common x86 Instructions:
   • lea: Load Effective Address (address calculation without memory access)
   • mov: Move data
   • add: Addition
   • call/ret: Function calls
3. Optimization Levels: -O0 (no optimization), -O2 (balanced), -O3 (aggressive)
4. TUI Mode: layout asm, layout regs, Ctrl-X A in GDB

Questions to Guide Your Design

1. What instruction is executing right now? → Use info registers rip to find the instruction pointer.
2. Are my variables in registers or memory? → Use info locals and then x to dump memory.
3. How does the assembly differ from my source? → Compare line-by-line using disassemble.
4. Why did the optimizer eliminate that variable? → Check if the variable’s value is ever read.

Thinking Exercise

You have a loop that the compiler optimized away. How would you force the compiler to keep it?

Answer: Mark the variable as volatile or force a side-effect (e.g., write to a pointer). Example:

int total = 0;
for (int i = 0; i < 1000000; ++i) {
    total += i;
}
// The loop might be optimized to: total = 999999 * 1000000 / 2
// To prevent this: volatile int total = 0;
// Or: asm("" : "+r"(total));  (inline assembly barrier)

The Interview Questions They’ll Ask

1. “How would you debug a race condition in multi-threaded code?” → Look at the assembly to see if operations are truly atomic, or if the compiler reordered them.
2. “Can you explain why this optimization is unsafe?” → Show the assembly and point out memory access patterns.
3. “What’s the difference between push and mov for saving registers?” → push is faster on some CPUs; mov has fewer dependencies.
4. “Why does -O3 sometimes make code slower?” → Show assembly that has worse cache behavior due to aggressive loop unrolling.

Hints in Layers

Hint 1: Setting Up TUI Mode

The best way to debug assembly is with TUI (Text User Interface):

(gdb) layout asm      # Show assembly code
(gdb) layout regs     # Add register window
(gdb) layout split    # Show source + assembly side-by-side
(gdb) Ctrl-X A        # Toggle TUI on/off
(gdb) Ctrl-X 2        # Split windows

Hint 2: Key Assembly Commands

(gdb) disassemble function_name           # Show all assembly for a function
(gdb) disassemble /m main                 # Show mixed source + assembly
(gdb) stepi                               # Step one instruction
(gdb) nexti                               # Step over (not into) one instruction
(gdb) info registers rax rbx rcx          # Show specific registers
(gdb) info registers                      # Show all registers
(gdb) x/10i $rip                          # Dump 10 instructions at RIP

Hint 3: Correlating Source to Assembly

Use the “mixed” disassembly to see source and assembly together:

(gdb) disassemble /m calculate
Dump of assembler code for function calculate:
5       int result = (a + b) * 2;
   0x0000555555555100 <+0>:     lea    (%rdi,%rsi,1),%eax
   0x0000555555555103 <+3>:     add    %eax,%eax

6       return result;
   0x0000555555555105 <+5>:     ret
End of dump.

This shows exactly which assembly instructions implement each C statement.

Hint 4: Understanding Register Allocation

After stepping through code, print registers to understand allocation:

(gdb) info registers  # Shows all CPU registers
rax            0x0             0
rbx            0x0             0
rcx            0x0             0
rdx            0x0             0
rsi            0x14            20
rdi            0xa             10
...

Each register holds a value. The optimizer chose these to minimize memory accesses.

Hint 5: Detecting Instruction Reordering

Compare the source order to assembly order:

// Source
a = b + c;  // Line 10
d = a * 2;  // Line 11
e = f + g;  // Line 12

If assembly shows instruction for Line 12 before Line 11, the compiler reordered for parallelism.

(gdb) disassemble /m my_function
   10   a = b + c;
   0x...   mov    (%rax),%r8d   # Load b+c into r8d

   12   e = f + g;             # REORDERED UP (no data dependency on line 11!)
   0x...   mov    (%rdx),%r9d   # Load f+g into r9d

   11   d = a * 2;
   0x...   shl    %r8d          # Double a

The compiler moved the independent operation up to hide memory latency.

Hint 6: Assembly Debugging Script

Automate assembly inspection with a GDB Python script:

import gdb

class AssemblyBreakdown(gdb.Command):
    def __init__(self):
        super(AssemblyBreakdown, self).__init__("asm_state", gdb.COMMAND_USER)

    def invoke(self, args, from_tty):
        # Print current instruction and registers
        frame = gdb.selected_frame()
        pc = frame.pc()
        disasm = gdb.execute(f"disassemble /m {frame.name()}", to_string=True)
        print(disasm)

        regs = gdb.execute("info registers rax rdi rsi rdx", to_string=True)
        print("\n[Current Registers]")
        print(regs)

AssemblyBreakdown()

Load with source script.py and call with asm_state.

Books That Will Help

Book	Chapter	Key Topic	Why It Helps
“Computer Systems: A Programmer’s Perspective”	Chapter 3	“Machine-Level Representation of Programs”	Fundamental x86-64 assembly and calling conventions
“Computer Systems: A Programmer’s Perspective”	Chapter 5	“Optimizing Program Performance”	How compilers optimize and reorder code
“The Art of Debugging with GDB”	Chapter 5	“More Sophisticated Debugging Techniques”	GDB’s assembly debugging features
“Intel 64 and IA-32 Architectures Software Developer’s Manual”	Volume 1	x86-64 instruction set	The authoritative reference for assembly instructions
GDB Manual	Chapter 10	“Examining Source Files”	Mixed source/assembly display and disassembly commands

---

Project 6: GDB Python Scripting

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: Python (in GDB)
• Alternative Programming Languages: GDB’s own command language
• Coolness Level: Level 4: Hardcore Tech Flex
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 3: Advanced
• Knowledge Area: Debugging Automation
• Software or Tool: GDB with Python support
• Main Book: GDB Documentation (Chapter “Python API”)

What you’ll build: A Python script that runs inside GDB to automate a debugging task. For example, a script that automatically prints all string arguments every time printf is called.

Why it teaches GDB: Manual debugging is repetitive. Scripting is the path to mastery. It allows you to create your own powerful debugging commands and automate complex analysis that would be impossible by hand.

Core challenges you’ll face:

• Running a Python script → maps to the source command in GDB
• Accessing GDB from Python → maps to the gdb Python module
• Creating a new GDB command → maps to subclassing gdb.Command
• Scripting breakpoints → maps to subclassing gdb.Breakpoint

Key Concepts:

• GDB Python API: Official GDB documentation is the best source.
• Extending GDB: gdb.Command, gdb.Function, gdb.Breakpoint.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Python knowledge, Project 1.

Real world outcome: A Python script (myscripts.py) that you load into GDB to add a new command.

# myscripts.py
import gdb

class PrintfTracer(gdb.Breakpoint):
    def __init__(self):
        super(PrintfTracer, self).__init__("printf", gdb.BP_BREAKPOINT, internal=True)

    def stop(self):
        # x86-64 calling convention: first arg is in RDI
        # Let's get the string at the address stored in RDI
        string_addr = gdb.parse_and_eval("$rdi")
        first_arg = gdb.selected_inferior().read_memory(string_addr, 64).tobytes().split(b'\x00')[0]
        print(f"[printf called with: {first_arg.decode('utf-8', 'ignore')}]")
        return False # Don't actually stop, just print and continue

class HelloCommand(gdb.Command):
    def __init__(self):
        super(HelloCommand, self).__init__("hello_gdb", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        print("Hello from your custom GDB command!")

HelloCommand()
PrintfTracer()

Debugging Session:

$ gdb ./my_program
(gdb) source myscripts.py  # Load the script
(gdb) hello_gdb
Hello from your custom GDB command!

(gdb) run
[printf called with: Hello for the 0th time!]
Hello for the 0th time!
[printf called with: Hello for the 1st time!]
Hello for the 1st time!
...

Learning milestones:

1. Create a custom GDB command → You can extend GDB’s vocabulary.
2. Create a scripted breakpoint → You can run code automatically when a breakpoint is hit.
3. Inspect memory and registers from Python → You can perform complex analysis.
4. Build a useful tool → e.g., a “pretty printer” for one of your custom data structures.

---

Real World Outcome

This is a complete Python scripting session that extends GDB with custom commands and automated debugging. You create a Python script to trace printf calls and analyze their arguments:

Script: debug_printf_tracer.py

import gdb
import struct

class PrintfTracer(gdb.Breakpoint):
    """Automatically intercepts and logs printf calls with their arguments"""

    def __init__(self):
        super(PrintfTracer, self).__init__("printf", internal=True)
        self.call_count = 0

    def stop(self):
        """Called every time the breakpoint is hit"""
        self.call_count += 1

        # Get the current frame to inspect the call
        frame = gdb.selected_frame()

        # x86-64 calling convention: first arg (format string) is in RDI
        format_addr = int(gdb.parse_and_eval("$rdi"))

        # Read the format string from memory
        try:
            inferior = gdb.selected_inferior()
            format_bytes = inferior.read_memory(format_addr, 256)
            format_str = format_bytes.tobytes().split(b'\x00')[0].decode('utf-8', 'ignore')
        except:
            format_str = "[unable to read]"

        # Extract additional arguments (rsi, rdx, rcx, r8, r9 for subsequent args)
        args_regs = ["$rsi", "$rdx", "$rcx", "$r8", "$r9"]
        args = []

        for arg_reg in args_regs:
            try:
                arg_val = int(gdb.parse_and_eval(arg_reg))
                args.append(f"0x{arg_val:x}")
            except:
                break

        # Print the trace
        print(f"[printf #{self.call_count}] Format: \"{format_str}\"")
        if args:
            print(f"            Args: {', '.join(args)}")
        print(f"            Called from: {frame.name()} at {frame.find_sal().symtab.filename}:{frame.find_sal().line}")

        # Don't actually stop execution—just log and continue
        return False

class ListBreakpoints(gdb.Command):
    """Custom command to list all breakpoints with their status"""

    def __init__(self):
        super(ListBreakpoints, self).__init__("bp_list", gdb.COMMAND_USER)

    def invoke(self, args, from_tty):
        bps = gdb.breakpoints()
        print(f"Total breakpoints: {len(bps)}")
        for bp in bps:
            print(f"  #{bp.number}: {bp.location} (enabled: {bp.enabled})")

class DumpMemoryRegion(gdb.Command):
    """Dump a region of memory in hex format"""

    def __init__(self):
        super(DumpMemoryRegion, self).__init__("memdump", gdb.COMMAND_USER)

    def invoke(self, args, from_tty):
        """
        Usage: memdump ADDRESS SIZE
        Example: memdump 0x7fffffffdf00 64
        """
        parts = args.split()
        if len(parts) < 2:
            print("Usage: memdump ADDRESS SIZE")
            return

        address = int(parts[0], 16)
        size = int(parts[1])

        try:
            inferior = gdb.selected_inferior()
            memory = inferior.read_memory(address, size)

            # Print in hex + ASCII format (like 'xxd')
            for i in range(0, len(memory), 16):
                chunk = memory[i:i+16]
                hex_str = ' '.join(f'{b:02x}' for b in chunk)
                ascii_str = ''.join(chr(b) if 32 <= b < 127 else '.' for b in chunk)
                print(f"0x{address + i:08x}: {hex_str:<48} {ascii_str}")
        except Exception as e:
            print(f"Error reading memory: {e}")

class CallTracer(gdb.Breakpoint):
    """Trace function calls with arguments"""

    def __init__(self, function_name):
        super(CallTracer, self).__init__(function_name, internal=True)
        self.function_name = function_name
        self.call_stack = []

    def stop(self):
        frame = gdb.selected_frame()

        # x86-64: first 6 arguments are in rdi, rsi, rdx, rcx, r8, r9
        arg_regs = ["$rdi", "$rsi", "$rdx", "$rcx", "$r8", "$r9"]
        arg_vals = []

        for reg in arg_regs:
            try:
                val = gdb.parse_and_eval(reg)
                arg_vals.append(str(val))
            except:
                break

        indent = "  " * len(self.call_stack)
        print(f"{indent}→ {self.function_name}({', '.join(arg_vals)})")
        self.call_stack.append(self.function_name)

        return False  # Don't stop, just log

# Register all commands and breakpoints
ListBreakpoints()
DumpMemoryRegion()
PrintfTracer()  # Automatically trace all printf calls

Session with the Script:

$ gcc -g -O0 -o test_program test_program.c
$ gdb ./test_program
(gdb) source debug_printf_tracer.py
(gdb) # The script is now loaded. printf calls will be automatically traced.

(gdb) run
[printf #1] Format: "Hello for the %dth time!"
            Args: 0x0
            Called from: greet at test_program.c:5

[printf #2] Format: "Hello for the %dth time!"
            Args: 0x1
            Called from: greet at test_program.c:5

[Inferior 1 (process 14567) exited normally]

(gdb) # Let's try the custom memdump command:
(gdb) break main
(gdb) run
(gdb) # Allocate some data
(gdb) print &global_value
$1 = 0x404000 <global_value>

(gdb) memdump 0x404000 32
0x404000: 64 00 00 00 c8 00 00 00 00 00 00 00 00 00 00 00  d...........
0x404010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

More Advanced Example: Tracking struct modifications

import gdb

class StructFieldWatcher(gdb.Breakpoint):
    """Watch for changes to a struct field across all functions"""

    def __init__(self, struct_name, field_name):
        super(StructFieldWatcher, self).__init__(f"my_function", internal=True)
        self.struct_name = struct_name
        self.field_name = field_name
        self.last_value = None

    def stop(self):
        try:
            frame = gdb.selected_frame()
            # Evaluate the struct.field expression in the current frame
            expr = f"(({self.struct_name}*)$rdi)->{self.field_name}"
            current_value = gdb.parse_and_eval(expr)
            current_val_int = int(current_value)

            if self.last_value is None or self.last_value != current_val_int:
                print(f"[STRUCT] {self.struct_name}.{self.field_name} changed: {self.last_value} → {current_val_int}")
                self.last_value = current_val_int
        except Exception as e:
            print(f"[ERROR] {e}")

        return False

Pretty Printer Example: Make custom types display nicely

import gdb.printing

class LinkedListPrinter:
    """Pretty printer for a custom linked list struct"""

    def __init__(self, val):
        self.val = val

    def to_string(self):
        node = self.val
        items = []

        while node != 0:  # Loop until NULL
            # Read the data field
            data = int(node['data'])
            items.append(str(data))

            # Move to next node
            try:
                node = node['next']
            except:
                break

        return f"LinkedList({' -> '.join(items)})"

    def display_hint(self):
        return 'string'

def build_pretty_printers():
    pp = gdb.printing.RegexObjectPrinter("MyProgram")
    pp.add_printer('LinkedList', '^LinkedList$', LinkedListPrinter)
    return pp

gdb.printing.register_pretty_printer(gdb.current_objfile(), build_pretty_printers())

The Core Question You’re Answering

“How do I automate repetitive debugging tasks and extend GDB with custom tools tailored to my codebase?”

Manual debugging is tedious. With Python scripting, you can:

• Automatically log specific function calls
• Create custom commands for your domain
• Build “pretty printers” for complex data structures
• Analyze memory patterns automatically
• Create breakpoints with complex conditions

Concepts You Must Understand First

1. GDB’s Python API:
   • gdb.Breakpoint: Define breakpoints in Python
   • gdb.Command: Create custom GDB commands
   • gdb.selected_frame(): Get the current stack frame
   • gdb.parse_and_eval(): Evaluate expressions
   • gdb.selected_inferior(): Access the target process

2. x86-64 Calling Convention: Arguments are in rdi, rsi, rdx, rcx, r8, r9.

3. Memory Access: Use inferior.read_memory(addr, size) to read raw memory.

4. Frame Inspection: Access function names, file names, line numbers from the frame.

Questions to Guide Your Design

1. What information do I need to log? → Arguments, return values, memory state.
2. How often will the breakpoint be hit? → Affects performance; use conditional breakpoints.
3. Should I stop execution or just log? → Usually log without stopping (return False).
4. How do I handle errors gracefully? → Wrap memory reads in try/except.

Thinking Exercise

You have a program that crashes randomly. You want to log all memory allocations and frees to find a use-after-free bug. How would you script this?

Answer: Create breakpoints at malloc and free, log the addresses and sizes, and maintain a set of allocated blocks. When a block is freed twice or used after free, alert the user.

import gdb

allocated_blocks = {}

class MallocBreakpoint(gdb.Breakpoint):
    def stop(self):
        addr = int(gdb.parse_and_eval("$rax"))  # malloc returns in rax
        size = int(gdb.parse_and_eval("$rdi"))  # size is first arg
        allocated_blocks[addr] = size
        print(f"[MALLOC] 0x{addr:x} (size: {size})")
        return False

class FreeBreakpoint(gdb.Breakpoint):
    def stop(self):
        addr = int(gdb.parse_and_eval("$rdi"))  # free arg is in rdi
        if addr in allocated_blocks:
            del allocated_blocks[addr]
            print(f"[FREE] 0x{addr:x}")
        else:
            print(f"[FREE] WARNING: Freeing unallocated block 0x{addr:x}")
        return False

# Register the breakpoints
MallocBreakpoint("malloc")
FreeBreakpoint("free")

The Interview Questions They’ll Ask

1. “How would you create a GDB command that finds all calls to a specific function?” → Use gdb.Breakpoint with logging and frame inspection.
2. “Can you script a breakpoint that only stops under specific conditions?” → Yes, use conditional logic in the stop() method.
3. “How would you extract and pretty-print a complex data structure?” → Create a pretty printer that walks the structure recursively.
4. “What’s the performance impact of extensive Python scripting?” → Each stop() call adds overhead. Use internal=True and conditional logic to minimize impact.

Hints in Layers

Hint 1: Basic Custom Command

Here’s the minimum for a custom command:

import gdb

class HelloCommand(gdb.Command):
    def __init__(self):
        super(HelloCommand, self).__init__("hello", gdb.COMMAND_USER)

    def invoke(self, args, from_tty):
        print(f"Hello, {args}!")

HelloCommand()

Load with source script.py and call with hello World.

Hint 2: Accessing Registers and Memory

Common patterns for reading data:

import gdb

# Read a register value
rax = int(gdb.parse_and_eval("$rax"))
rdi = int(gdb.parse_and_eval("$rdi"))

# Read memory at an address
inferior = gdb.selected_inferior()
memory = inferior.read_memory(0x7fffffffdf00, 64)  # Read 64 bytes

# Parse the bytes as integers
value = int.from_bytes(memory[:4], byteorder='little')  # First 4 bytes as little-endian int

Hint 3: Conditional Breakpoints with State

Store state across breakpoint hits:

import gdb

class StatefulBreakpoint(gdb.Breakpoint):
    def __init__(self, location):
        super(StatefulBreakpoint, self).__init__(location, internal=True)
        self.hit_count = 0
        self.history = []

    def stop(self):
        self.hit_count += 1

        # Store data from this hit
        frame = gdb.selected_frame()
        self.history.append({
            'count': self.hit_count,
            'function': frame.name(),
            'line': frame.find_sal().line
        })

        # Only stop every 10th hit
        return self.hit_count % 10 == 0

StatefulBreakpoint("printf")

Hint 4: Iterating Frames in the Call Stack

Walk the entire call stack programmatically:

import gdb

def print_stack_trace():
    frame = gdb.selected_frame()
    depth = 0

    while frame:
        print(f"#{depth} {frame.name()} at {frame.find_sal().symtab.filename}:{frame.find_sal().line}")
        try:
            frame = frame.older()
        except:
            break
        depth += 1

# Call this from a breakpoint's stop() method
class StackPrinter(gdb.Breakpoint):
    def stop(self):
        print_stack_trace()
        return False

StackPrinter("malloc")

Hint 5: Building a Memory Analyzer

Automatically analyze memory patterns:

import gdb

class MemoryAnalyzer(gdb.Command):
    def __init__(self):
        super(MemoryAnalyzer, self).__init__("analyze_mem", gdb.COMMAND_USER)

    def invoke(self, args, from_tty):
        # Find all global variables and their addresses
        symtab_and_line = gdb.decode_line("*")
        print("[Global Variables]")

        for var in gdb.current_objfile().global_symbols():
            if var.is_variable:
                addr = var.value()
                print(f"  {var.name} @ {addr}")

MemoryAnalyzer()

Hint 6: Creating Custom Function Signatures

Define helper functions that wrap GDB operations:

import gdb

def read_string(addr):
    """Read a null-terminated string from memory"""
    inferior = gdb.selected_inferior()
    chars = []

    for i in range(1024):  # Limit to 1024 chars
        try:
            byte = inferior.read_memory(addr + i, 1)[0]
            if byte == 0:
                break
            chars.append(chr(byte))
        except:
            break

    return ''.join(chars)

def read_int(addr):
    """Read a 4-byte integer"""
    inferior = gdb.selected_inferior()
    data = inferior.read_memory(addr, 4)
    return int.from_bytes(data, byteorder='little')

# Use these in breakpoints:
class PrintfTracer(gdb.Breakpoint):
    def stop(self):
        fmt_addr = int(gdb.parse_and_eval("$rdi"))
        fmt_str = read_string(fmt_addr)
        print(f"printf: {fmt_str}")
        return False

PrintfTracer("printf")

Books That Will Help

Book	Chapter	Key Topic	Why It Helps
GDB Manual	“Python API”	Python scripting in GDB	The definitive reference for GDB’s Python API
“The Art of Debugging with GDB”	Chapter 7	“Debugging with Python”	Practical Python debugging patterns and examples
“Debugging with GDB” (Official Manual)	“Extending GDB”	GDB extensibility overview	Overview of Python API capabilities
Python Official Docs	“ctypes”	C data structures in Python	How to structure data when reading from memory
“The Linux Programming Interface”	Chapter 28	“Process Execution”	Understanding the process being debugged

---

Project Comparison Table

Project	Difficulty	Time	Depth of Understanding	Fun Factor
The Basics	Level 1: Beginner	1-2 hours	Low	★★☆☆☆
The Crash	Level 1: Beginner	1-2 hours	Medium	★★★☆☆
The Hang	Level 2: Intermediate	1 hour	Medium	★★★☆☆
The Corruption	Level 2: Intermediate	2 hours	High	★★★★☆
The Assembly Level	Level 3: Advanced	3-4 hours	High	★★★★☆
Python Scripting	Level 3: Advanced	1-2 weeks	Very High	★★★★★

Recommendation

Follow the projects in order. Projects 1-4 are essential for any developer. They cover the most common use cases and provide the biggest return on time investment. If you can analyze a crash, attach to a running process, and use watchpoints, you are already a more effective debugger than most.

Once you are comfortable, move to Project 5 (The Assembly Level). This will forever change how you think about your code.

Finally, invest time in Project 6 (Python Scripting). This is the gateway to true GDB mastery and will allow you to build your own toolkit tailored to your specific needs.

---

Final Overall Project: Build a Mini-Debugger

• File: LEARN_GDB_DEEP_DIVE.md
• Main Programming Language: C
• Alternative Programming Languages: Rust, Go
• Coolness Level: Level 5: Pure Magic (Super Cool)
• Business Potential: 1. The “Resume Gold”
• Difficulty: Level 5: Master
• Knowledge Area: Systems Programming / OS Internals
• Software or Tool: A ptrace-based debugger
• Main Book: “The Linux Programming Interface” by Michael Kerrisk

What you’ll build: A command-line program that uses the ptrace system call to start, stop, and inspect another program. It will be a very simplified GDB.

Why it’s the final goal: This project demystifies debugging entirely. You will no longer see GDB as a black box, but as a C program that is masterfully using the ptrace API. You will understand the fundamental OS mechanics that enable all debuggers.

Core challenges you’ll face:

• Controlling a child process → maps to PTRACE_TRACEME and the fork/exec pattern
• Setting a breakpoint → maps to reading memory (PTRACE_PEEKTEXT), writing an int 3 instruction (PTRACE_POKETEXT), and then restoring it
• Reading registers → maps to PTRACE_GETREGS
• Single-stepping → maps to PTRACE_SINGLESTEP

Real world outcome: A simple REPL that lets you control a target program.

$ ./mini_gdb ./my_program
mini_gdb> break 0x40100a
Breakpoint set at 0x40100a
mini_gdb> continue
Stopped at breakpoint 1: 0x40100a
mini_gdb> regs
rax: 0x5
rbx: 0x0
rip: 0x40100a
mini_gdb> step
Stopped at 0x40100b
mini_gdb> quit

---

Summary

Project	Main Programming Language
The Basics	GDB Commands (on a C target)
The Crash	GDB Commands (on a C target)
The Hang	GDB Commands (on a C target)
The Corruption	GDB Commands (on a C target)
The Assembly Level	GDB Commands / x86 Assembly
Python Scripting	Python (in GDB)
Build a Mini-Debugger	C