Project 2: The GDB Backtrace

Master the fundamental skill of post-mortem debugging: using GDB to extract a stack backtrace from a core dump and pinpoint the exact line of code that caused a crash.

Quick Reference

Attribute Value
Difficulty Beginner
Time Estimate Weekend
Language C
Prerequisites Project 1 (The First Crash), Basic C programming
Key Topics Debug symbols, Stack frames, GDB backtrace command, Symbol tables

1. Learning Objectives

By completing this project, you will:

  1. Compile C programs with debug symbols: Understand the -g flag and what information it embeds in your executable
  2. Load core dumps into GDB: Master the basic GDB command-line syntax for post-mortem analysis
  3. Execute and interpret the backtrace command: Learn to read the call stack that led to a crash
  4. Connect stack frames to source code: Trace from memory addresses back to exact file names and line numbers
  5. Understand the value of debug information: Appreciate the difference between debugging with and without symbols
  6. Navigate the GDB environment: Become comfortable with the debugger’s interactive prompt

2. Theoretical Foundation

2.1 Core Concepts

What is a Stack Backtrace?

A stack backtrace (also called a “stack trace” or “call stack”) is a snapshot of all active function calls at a specific moment in time. When your program crashes, the backtrace shows you the chain of function calls that led to the crash point.

The Call Stack at Crash Time
============================

┌─────────────────────────────────────────────────────────┐
│ Frame #0: crash_function()   ← Crash happened HERE      │
│   - Local variables: ptr = 0x0                          │
│   - Called from: helper_function() at line 15           │
├─────────────────────────────────────────────────────────┤
│ Frame #1: helper_function()                             │
│   - Local variables: input = "test"                     │
│   - Called from: process_data() at line 42              │
├─────────────────────────────────────────────────────────┤
│ Frame #2: process_data()                                │
│   - Local variables: filename = "config.txt"            │
│   - Called from: main() at line 78                      │
├─────────────────────────────────────────────────────────┤
│ Frame #3: main()                                        │
│   - argc = 1, argv[0] = "./my_program"                  │
│   - Called from: _start (C runtime)                     │
└─────────────────────────────────────────────────────────┘

Each “frame” represents one function call. Frame #0 is always where the crash occurred. Higher frame numbers show the chain of callers leading to that point.

What are Debug Symbols?

Debug symbols are metadata embedded in your executable that map machine code addresses back to source code. Without debug symbols, GDB can only show you raw memory addresses. With debug symbols, it shows you:

  • Function names
  • File names
  • Line numbers
  • Variable names and types
  • Type definitions
Without Debug Symbols                With Debug Symbols (-g)
=====================                =======================

(gdb) bt                             (gdb) bt
#0  0x000055555555513d in ?? ()      #0  0x000055555555513d in main ()
#1  0x00007ffff7de8b25 in ?? ()          at crashing_program.c:4
                                      #1  0x00007ffff7de8b25 in __libc_start_main ()
                                          from /lib/x86_64-linux-gnu/libc.so.6

The difference is dramatic. With symbols, you immediately know the crash was in main() at line 4 of crashing_program.c. Without symbols, you only have a hexadecimal address.

How Debug Information is Stored

When you compile with -g, the compiler adds extra sections to your executable:

ELF Executable Structure (with -g)
==================================

┌─────────────────────────────────────────────────────────┐
│ ELF Header                                              │
├─────────────────────────────────────────────────────────┤
│ .text section      ← Your executable code               │
├─────────────────────────────────────────────────────────┤
│ .data section      ← Initialized global variables       │
├─────────────────────────────────────────────────────────┤
│ .rodata section    ← String literals, constants         │
├─────────────────────────────────────────────────────────┤
│ .debug_info        ← Type information, variable info    │
├─────────────────────────────────────────────────────────┤
│ .debug_line        ← Line number → address mapping      │
├─────────────────────────────────────────────────────────┤
│ .debug_abbrev      ← Abbreviation tables for DWARF      │
├─────────────────────────────────────────────────────────┤
│ .debug_str         ← String table for debug info        │
├─────────────────────────────────────────────────────────┤
│ .symtab            ← Symbol table (functions, globals)  │
├─────────────────────────────────────────────────────────┤
│ .strtab            ← String table for symbols           │
└─────────────────────────────────────────────────────────┘

This debug information is stored in the DWARF format (Debugging With Attributed Record Formats), the standard debugging format on Linux.

The Stack Frame in Memory

Understanding what a stack frame looks like in memory helps you interpret backtraces:

Stack Memory Layout (x86-64)
============================

High Addresses
┌─────────────────────────────────────────────────────────┐
│              Command-line arguments (argv)              │
├─────────────────────────────────────────────────────────┤
│              Environment variables (envp)               │
├─────────────────────────────────────────────────────────┤
│                                                         │
│              main()'s Stack Frame                       │
│              ┌─────────────────────────────────────┐   │
│              │ Return address (to _start)          │   │
│              │ Saved base pointer (RBP)            │   │
│              │ Local variables                     │   │
│              └─────────────────────────────────────┘   │
│                          │                              │
│                          ▼ Stack grows down             │
│              ┌─────────────────────────────────────┐   │
│              │ Return address (to main)            │   │
│              │ Saved base pointer (RBP)            │   │
│              │ Local variables                     │   │
│              │ Function arguments (if >6)          │   │
│              └─────────────────────────────────────┘   │
│              helper_function()'s Stack Frame            │
│                          │                              │
│                          ▼                              │
│              ┌─────────────────────────────────────┐   │
│              │ Return address (to helper_function) │   │
│              │ Saved base pointer (RBP)            │   │
│              │ Local variables  ← Crash happens    │   │
│              └─────────────────────────────────────┘   │
│              crash_function()'s Stack Frame             │
│                                                         │
└─────────────────────────────────────────────────────────┘
Low Addresses (RSP points here)

When GDB generates a backtrace, it “walks” up this chain of saved base pointers and return addresses.

2.2 Why This Matters

The backtrace is the single most important piece of information when debugging a crash:

  1. Immediate Location: Tells you exactly where the crash occurred (file and line)
  2. Context: Shows the path your program took to get there
  3. State Information: Lets you inspect variables at each step
  4. Root Cause Identification: Often the bug is not at the crash site, but in a caller

Real-world impact:

  • Production crash reports always include backtraces
  • Bug reports in open-source projects expect backtraces
  • Support teams use backtraces to triage issues
  • Crash reporting systems (Sentry, Crashlytics) are built around backtraces

2.3 Historical Context

The concept of a stack trace dates back to the earliest debugging tools in the 1960s. GDB itself was first released in 1986 by Richard Stallman as part of the GNU Project. The backtrace command (and its shorthand bt) has been a core feature since GDB’s inception.

The DWARF debugging format, now the standard on Unix-like systems, was developed to improve upon earlier formats like STABS. DWARF version 5 (2017) is the current standard, supporting modern language features and optimized code debugging.

2.4 Common Misconceptions

Misconception 1: “I don’t need debug symbols in development” Reality: Always compile with -g during development. The size increase is negligible compared to the debugging time saved. You can strip symbols for release builds.

Misconception 2: “The crash always happens at frame #0” Reality: The instruction at frame #0 triggered the fault, but the bug might be in a caller. For example, a NULL pointer might be passed from main() and dereferenced in helper_function().

Misconception 3: “GDB needs the exact same executable” Reality: GDB needs an executable that matches the core dump. If you recompile between crash and analysis, addresses won’t match and the backtrace will be meaningless.

Misconception 4: “Debug builds are much slower” Reality: The -g flag alone doesn’t affect code generation or optimization. You can combine -g with -O2 for optimized code with debug symbols (though debugging optimized code can be confusing).

3. Project Specification

3.1 What You Will Build

Using the crashing program and core dump from Project 1, you will:

  1. Recompile the program with debug symbols
  2. Generate a new core dump
  3. Load the executable and core dump into GDB
  4. Execute the backtrace command
  5. Interpret the output to identify the crash location
  6. Compare the output with a version compiled without debug symbols

This is primarily an exploration and learning project rather than a code-building project. Your deliverable is understanding, demonstrated through documented analysis.

3.2 Functional Requirements

  1. Compile with Debug Symbols:
    • Use gcc -g to compile the crashing program from Project 1
    • Verify debug symbols are present using file command
  2. Generate a Core Dump:
    • Run the debug-compiled program to crash
    • Confirm a core dump file is created
  3. Load into GDB:
    • Start GDB with both executable and core file
    • Observe GDB’s initial output
  4. Execute Backtrace:
    • Run the bt (backtrace) command
    • Document the output
  5. Interpret Results:
    • Identify the crash location (file, function, line)
    • Understand each frame in the backtrace
  6. Comparison Experiment:
    • Repeat with a version compiled without -g
    • Document the difference in backtrace output

3.3 Non-Functional Requirements

  • Reproducibility: Steps should be documentable and repeatable
  • Understanding: You should be able to explain each field in the backtrace output
  • Transferability: Skills should apply to any C/C++ program, not just this example

3.4 Example Usage / Output

Here is exactly what you will see when you complete this project:

# Step 1: Compile with debug symbols
$ gcc -g -o crashing_program crashing_program.c

# Step 2: Verify debug symbols are present
$ file crashing_program
crashing_program: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
BuildID[sha1]=abc123..., for GNU/Linux 3.2.0, with debug_info, not stripped

# Note: "with debug_info" and "not stripped" indicate symbols are present

# Step 3: Enable core dumps and run
$ ulimit -c unlimited
$ ./crashing_program
Segmentation fault (core dumped)

# Step 4: Find the core file
$ ls -la core*
-rw------- 1 user user 356352 Dec 29 10:30 core.crashing_program.12345

# Step 5: Load into GDB
$ gdb crashing_program core.crashing_program.12345
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
...
Reading symbols from crashing_program...
[New LWP 12345]
Core was generated by `./crashing_program'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000555555555149 in main () at crashing_program.c:4

# Step 6: Run backtrace
(gdb) bt
#0  0x0000555555555149 in main () at crashing_program.c:4
(gdb)

# Step 7: Get more detail
(gdb) bt full
#0  0x0000555555555149 in main () at crashing_program.c:4
        ptr = 0x0
(gdb)

3.5 Real World Outcome

After completing this project, you will have:

crashing_program         # Executable with debug symbols
crashing_program.c       # Original source code
core.crashing_program.X  # Core dump file
analysis_notes.txt       # Your documented findings including:
                         #   - Backtrace output with explanation
                         #   - Comparison with non-debug version
                         #   - Identified crash location

You will understand that at crashing_program.c:4 is the critical piece of information that makes post-mortem debugging possible.

4. Solution Architecture

4.1 High-Level Design

This project is about using existing tools effectively, not building new software. The “architecture” is the workflow:

┌─────────────────────────────────────────────────────────────────────────┐
│                        GDB Post-Mortem Workflow                         │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│   ┌─────────────┐    gcc -g     ┌─────────────┐                        │
│   │   Source    │──────────────▶│  Executable │                        │
│   │  (.c file)  │               │ (with debug │                        │
│   └─────────────┘               │   symbols)  │                        │
│                                 └──────┬──────┘                        │
│                                        │                               │
│                                        ▼ Run                           │
│                                 ┌─────────────┐                        │
│                                 │   CRASH!    │                        │
│                                 │  (SIGSEGV)  │                        │
│                                 └──────┬──────┘                        │
│                                        │                               │
│                                        ▼ Kernel generates              │
│                                 ┌─────────────┐                        │
│                                 │  Core Dump  │                        │
│                                 │   File      │                        │
│                                 └──────┬──────┘                        │
│                                        │                               │
│   ┌─────────────┐    load both  ┌──────┴──────┐                        │
│   │  Executable │◀─────────────▶│     GDB     │                        │
│   │ (with debug │               │             │                        │
│   │   symbols)  │               │ > bt        │                        │
│   └─────────────┘               │ > frame N   │                        │
│                                 │ > print var │                        │
│                                 └──────┬──────┘                        │
│                                        │                               │
│                                        ▼                               │
│                                 ┌─────────────┐                        │
│                                 │  Backtrace  │                        │
│                                 │   Output    │                        │
│                                 │             │                        │
│                                 │ #0 main()   │                        │
│                                 │   line 4    │                        │
│                                 └─────────────┘                        │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

4.2 Key Components

Component Role How You Interact
GCC with -g Embeds DWARF debug information in executable Compile your program
Core Dump Contains memory snapshot at crash time Generated by kernel on crash
GDB Reads executable + core, displays symbolic information Interactive commands
DWARF Info Maps addresses to source locations Used by GDB behind the scenes

4.3 Data Structures

The key “data structure” to understand is the backtrace output format:

#<frame_num> <address> in <function_name> (<arguments>) at <file>:<line>

Example:
#0  0x0000555555555149 in main () at crashing_program.c:4
│   │                     │     │    │                   │
│   │                     │     │    │                   └─ Line number
│   │                     │     │    └─ Source file name
│   │                     │     └─ Function arguments (empty here)
│   │                     └─ Function name
│   └─ Memory address of the instruction (in hexadecimal)
└─ Frame number (0 = crash site, higher = callers)

4.4 Algorithm Overview

GDB’s backtrace algorithm (simplified):

  1. Start at crash point: Read instruction pointer (RIP) from core dump
  2. Find function: Look up address in debug info to find function name
  3. Find source location: Map address to file:line using DWARF line tables
  4. Walk the stack: Follow saved base pointer to previous frame
  5. Repeat: Continue until reaching the bottom of the stack
Backtrace Algorithm (Simplified)
================================

current_frame = crash_location_from_core_dump

while current_frame is valid:
    address = get_instruction_pointer(current_frame)

    # Look up in debug info
    function = lookup_function_by_address(address)
    source_location = lookup_line_by_address(address)

    # Print this frame
    print(frame_number, address, function, source_location)

    # Move to caller's frame
    current_frame = get_saved_base_pointer(current_frame)
    frame_number += 1

5. Implementation Guide

5.1 Development Environment Setup

Ensure you have the tools from Project 1:

# Verify GCC is installed
$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

# Verify GDB is installed
$ gdb --version
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1

# Verify you can generate core dumps
$ ulimit -c
unlimited   # If it shows 0, run: ulimit -c unlimited

# Check core_pattern (should not be piped to a crash handler)
$ cat /proc/sys/kernel/core_pattern
core        # or core.%p.%e or similar

5.2 Project Structure

Create a directory for this project:

project2_gdb_backtrace/
├── crashing_program.c     # Source code (from Project 1)
├── Makefile               # Build instructions
├── run_analysis.sh        # Script to automate analysis
├── analysis_notes.md      # Your documented findings
└── cores/                 # Directory for core dumps (gitignore this)

Example Makefile:

CC = gcc
CFLAGS_DEBUG = -g -O0
CFLAGS_NODEBUG = -O0

all: debug nodebug

debug: crashing_program.c
	$(CC) $(CFLAGS_DEBUG) -o crashing_program_debug crashing_program.c

nodebug: crashing_program.c
	$(CC) $(CFLAGS_NODEBUG) -o crashing_program_nodebug crashing_program.c

clean:
	rm -f crashing_program_debug crashing_program_nodebug
	rm -f core.*

5.3 The Core Question You’re Answering

“How does GDB translate a raw memory address into a meaningful source code location, and why does this require debug symbols?”

This question drives home the key insight: debug symbols are the bridge between machine code and source code. Without them, you’re lost in a sea of hexadecimal addresses.

5.4 Concepts You Must Understand First

Before starting, verify you can answer these questions:

Concept Self-Assessment Question Where to Learn
Segmentation Fault What causes a SIGSEGV signal? CS:APP Ch. 9, Project 1
Core Dump What information does a core dump contain? man core, Project 1
Memory Addresses What does a hex address like 0x555555555149 represent? CS:APP Ch. 9
Function Call Stack Why does calling a function push onto a “stack”? CS:APP Ch. 3.7
Compilation What’s the difference between source, object, and executable? CS:APP Ch. 7

5.5 Questions to Guide Your Design

Work through these questions as you explore:

  1. Loading Phase:
    • What does GDB print when it first loads the core dump?
    • Does GDB immediately show you the crash location before you type anything?
  2. Symbol Lookup:
    • What happens if the executable doesn’t have debug symbols?
    • Can GDB still show function names without -g?
  3. Frame Navigation:
    • If the backtrace shows 3 frames, which one is the crash location?
    • What information does each frame give you?
  4. Deep vs Shallow Backtraces:
    • What does bt full show that bt doesn’t?
    • When would you use one versus the other?

5.6 Thinking Exercise

Before loading GDB, trace through this program mentally:

#include <stdio.h>

void level3(int *ptr) {
    *ptr = 42;  // What happens here if ptr is NULL?
}

void level2(int *ptr) {
    printf("In level2, ptr = %p\n", (void*)ptr);
    level3(ptr);
}

void level1(int *ptr) {
    level2(ptr);
}

int main() {
    int *ptr = NULL;
    level1(ptr);
    return 0;
}

On paper, predict:

  1. Which function will appear at frame #0 in the backtrace?
  2. How many frames will the backtrace show?
  3. What will ptr show in frame #3 (main)?
  4. What line number will GDB report for the crash?

Then compile with -g, crash it, and verify your predictions.

5.7 Hints in Layers

If you get stuck, reveal hints one at a time:

Hint 1: Compiling with Debug Symbols

The -g flag tells GCC to include debug information:

gcc -g -o my_program my_program.c

You can verify symbols are present:

file my_program
# Should say "with debug_info, not stripped"

# Or check for debug sections:
readelf -S my_program | grep debug
# Should show .debug_info, .debug_line, etc.
Hint 2: Starting GDB with a Core File

GDB takes two arguments for post-mortem debugging:

gdb <executable> <core_file>

Example:

gdb ./crashing_program core.crashing_program.1234

GDB will automatically analyze the crash and show you the location.

Hint 3: Basic Backtrace Commands

Once in GDB:

  • bt or backtrace - Show the call stack
  • bt full - Show call stack with local variables
  • bt 3 - Show only the top 3 frames
  • frame N - Select frame N for inspection
  • info frame - Show details about current frame
  • list - Show source code around current location
Hint 4: Understanding the Output

A backtrace line like:

#0  0x0000555555555149 in main () at crashing_program.c:4

Means:

  • #0 - This is frame 0 (the crash site)
  • 0x0000555555555149 - The instruction pointer address
  • in main () - The function name (parentheses show it takes no arguments worth displaying)
  • at crashing_program.c:4 - File and line number

The main () part comes from the symbol table. The crashing_program.c:4 part comes from DWARF debug info.

5.8 The Interview Questions They’ll Ask

After completing this project, you should be able to answer:

  1. “How would you debug a crash if you only had the core dump and executable?”
    • Answer: Load both into GDB with gdb <exe> <core>, run bt to get the backtrace, examine variables with print, examine memory with x
  2. “What’s the difference between a debug build and a release build?”
    • Answer: Debug builds include DWARF debug information (-g flag) and often disable optimizations (-O0). Release builds strip symbols and enable optimizations (-O2 or -O3).
  3. “Why might a backtrace show only addresses and no function names?”
    • Answer: The executable was compiled without debug symbols, or the executable was stripped after compilation. Library functions might show if the library has debug packages installed.
  4. “Explain what a stack frame is.”
    • Answer: A stack frame is a region of the call stack containing a function’s local variables, saved registers, and return address. Each function call creates a new frame. The backtrace walks through these frames.
  5. “You get a crash report showing ‘SIGSEGV at 0x0’. What does this tell you?”
    • Answer: A NULL pointer (address 0x0) was dereferenced. Something passed a NULL where a valid pointer was expected. The backtrace will show you which function tried to access address 0.

5.9 Books That Will Help

Topic Book Chapter
Debug symbols and DWARF format “The Art of Debugging with GDB” by Matloff & Salzman Chapter 1
Stack frames and calling conventions “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron Chapter 3.7
GDB commands and workflow “The Art of Debugging with GDB” by Matloff & Salzman Chapters 2-3
ELF format and symbol tables “Practical Binary Analysis” by Dennis Andriesse Chapter 2
Memory layout and the stack “Computer Systems: A Programmer’s Perspective” by Bryant & O’Hallaron Chapter 3.4-3.7

5.10 Implementation Phases

Phase 1: Basic Backtrace (30 minutes)

Goal: Get a backtrace from the Project 1 crash

Steps:

  1. Navigate to your Project 1 directory
  2. Recompile with -g: gcc -g -o crashing_program crashing_program.c
  3. Enable core dumps: ulimit -c unlimited
  4. Run the program to crash: ./crashing_program
  5. Load into GDB: gdb crashing_program core.*
  6. Type bt and observe the output

Checkpoint: You see a backtrace showing main at a specific line number.

Phase 2: Comparison Experiment (30 minutes)

Goal: Understand the difference debug symbols make

Steps:

  1. Compile without -g: gcc -o crashing_program_nodebug crashing_program.c
  2. Crash it and get a core dump (rename the previous one first)
  3. Load the non-debug version: gdb crashing_program_nodebug core.*
  4. Type bt and compare to Phase 1

Checkpoint: You see the difference - no line numbers, possibly no function names (or only ?? symbols).

Phase 3: Deeper Exploration (1-2 hours)

Goal: Learn to extract more information from a crash

Steps:

  1. Create a multi-function crashing program (see Thinking Exercise)
  2. Compile with -g and crash it
  3. Use bt full to see local variables
  4. Use frame N to navigate between frames
  5. Use print <variable> to inspect values
  6. Use list to see source code context

Checkpoint: You can navigate frames and inspect variables at each level.

Phase 4: Documentation (30 minutes)

Goal: Solidify your learning through documentation

Steps:

  1. Create analysis_notes.md
  2. Document the exact commands you used
  3. Paste your backtrace outputs with annotations
  4. Explain what each piece of information means
  5. Note the difference between debug and non-debug versions

Checkpoint: Someone else could read your notes and understand GDB backtraces.

5.11 Key Implementation Decisions

Decision Recommendation Rationale
Optimization level Use -O0 with -g Optimizations can confuse beginners (inlined functions, reordered code)
Core pattern Use simple core pattern Avoid systemd-coredump complexity for learning
GDB version Use system default Modern GDB versions all support these basics
Multiple test programs Create 2-3 different crashers See how backtraces differ for different bugs

6. Testing Strategy

Since this is an exploration project, “testing” means verifying your understanding:

Self-Tests

Test How to Verify
Debug symbols present file crashing_program shows “with debug_info”
GDB loads correctly No error messages about missing symbols
Backtrace shows source Output includes “at filename:line”
Frame navigation works frame 1 changes the context
Variable inspection works print variable shows a value

Test Cases to Try

  1. Single-function crash: 
    int main() { int *p = NULL; *p = 1; }

    Expected: 1-frame backtrace, line number visible

  2. Multi-function crash (from Thinking Exercise): Expected: 4-frame backtrace, can navigate all frames

  3. Library function crash: 
    #include <string.h>
int main() { strcpy(NULL, "test"); }

    Expected: Top frames in library, but your code is visible below

  4. No debug symbols: Compile without -g, verify backtrace is less useful

7. Common Pitfalls & Debugging

Pitfall Symptom Solution
Wrong executable loaded Backtrace shows wrong line numbers or ?? Ensure you load the same executable that created the core dump
Core dump disabled “No core dump” message Run ulimit -c unlimited before crashing
systemd intercepts cores Can’t find core file Check /var/lib/systemd/coredump/ or modify core_pattern
Executable stripped No symbols even though compiled with -g Check with file, don’t run strip
Recompiled between crash and analysis Addresses don’t match Use the exact executable that crashed
Library symbols missing Library calls show ?? Install debug packages (e.g., libc6-dbg)

Debugging Your Debugging

If GDB can’t find your symbols:

# Check if debug info is present
$ readelf -S crashing_program | grep debug
  [26] .debug_aranges    PROGBITS         0000000000000000  00001038
  [27] .debug_info       PROGBITS         0000000000000000  00001068
  [28] .debug_abbrev     PROGBITS         0000000000000000  000010f4
  [29] .debug_line       PROGBITS         0000000000000000  00001147
  [30] .debug_str        PROGBITS         0000000000000000  00001189

# If those sections are missing, you didn't compile with -g

If GDB says “No core dump”:

# Check ulimit
$ ulimit -c
0        # <-- Problem! Should be unlimited or a number

# Fix it
$ ulimit -c unlimited

# Check core_pattern
$ cat /proc/sys/kernel/core_pattern
|/usr/share/apport/apport ...  # <-- Piped to crash handler

# For simple core files:
$ echo "core.%p.%e" | sudo tee /proc/sys/kernel/core_pattern

8. Extensions & Challenges

Beginner Extensions

  • Try different crash types: Stack overflow, divide by zero, abort()
  • Use bt -full: See local variables in each frame
  • Install library debug symbols: Get line numbers for libc functions

Intermediate Extensions

  • Analyze a real project crash: Download an open-source project, build with -g, crash it
  • Explore GDB TUI mode: Try gdb -tui for a split-screen interface
  • Write a GDB command script: Automate the backtrace process

Advanced Extensions

  • Compare optimization levels: How does -O2 affect backtrace readability?
  • Learn about DWARF directly: Use objdump --dwarf to see raw debug info
  • Debug a multi-threaded crash: See how thread apply all bt works

9. Real-World Connections

Industry Applications

  • Bug Tracking Systems: Crash reports in Jira, GitHub Issues always include backtraces
  • Crash Reporting Services: Sentry, Crashlytics, Bugsnag all collect and symbolicate backtraces
  • Support Escalation: Level 1 support asks for backtraces to pass to engineers
  • Post-Incident Analysis: Production crashes are analyzed using these exact techniques
  • addr2line: Converts addresses to file:line without full GDB (part of binutils)
  • libunwind: Library for programmatic stack unwinding
  • elfutils: Alternative tools for ELF inspection
  • lldb: LLVM debugger with similar capabilities

Career Relevance

This skill is essential for:

  • Systems programmers
  • Game developers
  • Embedded engineers
  • Database developers
  • Security researchers
  • Anyone debugging native code

10. Resources

Essential Reading

  • GDB Manual: info gdb or online documentation
  • CS:APP Chapter 3: “Machine-Level Representation of Programs” - explains the stack
  • “The Art of Debugging”: Comprehensive GDB guide

Quick References

# GDB Quick Reference
gdb <exe> <core>      # Load executable and core dump
bt                    # Show backtrace
bt full               # Show backtrace with local variables
frame N               # Select frame N
info frame            # Show current frame details
print <var>           # Print variable value
list                  # Show source code
quit                  # Exit GDB

Video Resources

  • “GDB Tutorial” on YouTube by CS50
  • “Post-mortem debugging with GDB” talks from various conferences
  • LiveOverflow’s binary exploitation series (covers GDB extensively)

11. Self-Assessment Checklist

Before considering this project complete, verify:

Understanding

  • I can explain what debug symbols are and why they matter
  • I can describe what a stack frame contains
  • I understand what each field in a backtrace line means
  • I know the difference between bt and bt full
  • I can explain why the crash location is always frame #0

Practical Skills

  • I can compile a C program with debug symbols
  • I can verify that debug symbols are present
  • I can load a core dump into GDB
  • I can get a backtrace and identify the crash location
  • I can navigate between stack frames

Growth

  • I have compared debug vs non-debug backtraces
  • I have analyzed a crash with multiple stack frames
  • I am comfortable at the GDB prompt
  • I have documented my findings for future reference

12. Submission / Completion Criteria

Minimum Viable Completion

  • Compiled crashing program with -g flag
  • Generated a core dump from the debug build
  • Loaded into GDB and got a backtrace
  • Correctly identified the file and line number of the crash

Full Completion

  • Completed comparison between debug and non-debug builds
  • Analyzed a multi-function crash with frame navigation
  • Used bt full to inspect local variables
  • Documented findings in analysis notes
  • Can explain each part of a backtrace line

Excellence (Going Above & Beyond)

  • Analyzed crashes with different causes (stack overflow, etc.)
  • Installed library debug symbols and got full library backtraces
  • Created a script to automate initial crash analysis
  • Explored GDB TUI mode or other advanced features
  • Wrote a guide that could teach someone else

This guide was expanded from LEARN_LINUX_CRASH_DUMP_ANALYSIS.md. Continue to Project 3: The Memory Inspector to learn about examining memory and debugging buffer overflows.