Linux Crash Dump Analysis - Expanded Project Guides
Generated from:
LEARN_LINUX_CRASH_DUMP_ANALYSIS.md
Overview
Master the art of post-mortem debugging in Linux. This learning path takes you from analyzing a simple segmentation fault to dissecting kernel panics, teaching you the tools and techniques to find the root cause of any crash. You’ll learn to use GDB for user-space debugging, understand ELF core dump formats, diagnose complex multi-threaded issues, and eventually analyze kernel crashes using kdump and the crash utility.
What You’ll Master
After completing these projects, you will:
- Confidently use GDB and other tools for post-mortem debugging
- Understand the ELF format of core dump files
- Diagnose segmentation faults, memory corruption, and stack overflows from dumps
- Analyze crashes in complex multi-threaded applications
- Parse and understand minidump formats used by production systems
- Configure and use
kdumpfor kernel crash capture - Analyze kernel panics using the
crashutility - Build automated crash reporting infrastructure
Project Index
| # | Project | Difficulty | Time | Key Focus |
|---|---|---|---|---|
| 1 | The First Crash | Beginner | Weekend | Core dump generation, ulimit, core_pattern |
| 2 | The GDB Backtrace | Beginner | Weekend | Debug symbols, stack traces, basic GDB |
| 3 | The Memory Inspector | Intermediate | 1-2 weeks | Memory examination, buffer overflows, stack analysis |
| 4 | The Automated Crash Detective | Intermediate | 1-2 weeks | GDB scripting, Python automation, batch analysis |
| 5 | Multi-threaded Mayhem | Advanced | 1-2 weeks | Thread debugging, data races, concurrency analysis |
| 6 | Deconstructing a Stripped Binary Crash | Advanced | 1-2 weeks | Symbol stripping, disassembly, address mapping |
| 7 | The Minidump Parser | Advanced | 2-3 weeks | Breakpad/Crashpad, binary parsing, minidump format |
| 8 | Introduction to Kernel Panics | Expert | 2-3 weeks | Kernel modules, kdump, vmcore capture |
| 9 | Analyzing a Kernel Panic with crash | Expert | 1-2 weeks | crash utility, kernel debugging, kernel internals |
| 10 | Building a Centralized Crash Reporter | Master | 1 month+ | System design, crash pipelines, deduplication |
Learning Paths
Path 1: User-Space Debugging Focus (Weeks 1-4)
For developers who primarily work on applications and need to debug crashes in their own code:
Week 1: P01 → P02 (Foundations)
Week 2: P03 (Memory Analysis)
Week 3: P04 (Automation)
Week 4: P05 (Multi-threading)
Outcome: You can debug any user-space crash, including complex multi-threaded issues, and automate your analysis workflow.
Path 2: Production Systems Engineer (Weeks 1-6)
For SREs and platform engineers who need to handle crashes at scale:
Week 1: P01 → P02 (Foundations)
Week 2: P03 → P04 (Analysis & Automation)
Week 3: P05 → P06 (Advanced User-Space)
Week 4-5: P07 (Minidump Parsing)
Week 6+: P10 (Crash Reporter)
Outcome: You can build and operate crash reporting infrastructure for production systems.
Path 3: Kernel Developer Track (Weeks 1-8)
For kernel developers, driver writers, and those working on low-level systems:
Week 1: P01 → P02 (User-Space Foundations)
Week 2: P03 (Memory Analysis)
Week 3: P05 → P06 (Advanced User-Space)
Week 4-5: P08 (Kernel Panics)
Week 6-7: P09 (Crash Analysis)
Week 8: Review and practice
Outcome: You can debug both user-space and kernel crashes, write kernel modules safely, and analyze kernel panics.
Path 4: Complete Mastery (8+ weeks)
Complete all projects in order for comprehensive crash analysis expertise:
P01 → P02 → P03 → P04 → P05 → P06 → P07 → P08 → P09 → P10
Outcome: Full expertise in crash dump analysis from application crashes to kernel panics to infrastructure.
Prerequisites
Essential Prerequisites
- C Programming: Comfortable reading and writing C code, understanding pointers
- Linux Command Line: Navigation, file operations, basic shell scripting
- Process Memory Model: Basic understanding of stack, heap, and code segments
- Text Editor/IDE: Ability to write and compile C programs
Self-Assessment Questions
Before starting, you should be able to answer:
- What happens when you dereference a NULL pointer in C?
- What is the difference between the stack and the heap?
- How do you compile a C program with GCC?
- What does
ulimitdo? - What is a signal in Unix/Linux?
If you can’t answer these, consider reviewing:
- “The C Programming Language” by Kernighan & Ritchie (Chapters 1-5)
- “Computer Systems: A Programmer’s Perspective” (Chapter 1, 3)
Development Environment
Required:
- Linux system (native, WSL2, or VM)
- GCC compiler
- GDB debugger
- Python 3.x (for automation projects)
Recommended:
- Virtual machine software (VirtualBox, QEMU) for kernel projects
- Text editor with C syntax highlighting
- Terminal multiplexer (tmux/screen)
Core Concepts Overview
The Crash Analysis Landscape
┌───────────────────────────────────────────────────────────┐
│ RUNNING APPLICATION │
│ │
│ int main() {
│ char *p = NULL;
│ *p = 'a'; // CRASH! (Segmentation Fault)
│ return 0;
│ }
│
└───────────────────────────────────────────────────────────┘
│
▼ Signal (SIGSEGV)
┌───────────────────────────────────────────────────────────┐
│ LINUX KERNEL HANDLER │
│ │
│ Generates a snapshot of the process's memory state │
│ based on system settings (ulimit -c, core_pattern).
│
└───────────────────────────────────────────────────────────┘
│
▼ File is created
┌───────────────────────────────────────────────────────────┐
│ CORE DUMP FILE (e.g., core.1234) │
│ │
│ ELF format containing: │
│ • Register values (RIP, RSP, RAX, etc.) │
│ • The entire memory space (.text, .data, stack, heap) │
│ • Information about threads, signals, etc. │
│
└───────────────────────────────────────────────────────────┘
│
▼ Post-Mortem Debugging
┌───────────────────────────────────────────────────────────┐
│ ANALYSIS WITH GDB │
│ │
│ $ gdb ./my_app ./core.1234 │
│ (gdb) bt // See the call stack │
│ (gdb) frame 2 // Switch to a specific stack frame │
│ (gdb) p my_var // Print a variable's value │
│ (gdb) x/16wx $rsp // Examine memory on the stack │
│
└───────────────────────────────────────────────────────────┘
Key Terminology
| Term | Definition |
|---|---|
| Core Dump | A file containing a snapshot of a process’s memory at crash time |
| Segmentation Fault | Access to memory that the process doesn’t have permission to access |
| Debug Symbols | Metadata mapping machine code addresses to source code locations |
| Stack Frame | A section of the call stack containing a function’s local variables and return address |
| Backtrace | The sequence of function calls leading to the current point of execution |
| Kernel Panic | A fatal error in the kernel requiring system restart |
| vmcore | A kernel crash dump captured by kdump |
| Minidump | A smaller, more portable crash dump format (Google Breakpad) |
Recommended Books
| Book | Focus Area | Projects |
|---|---|---|
| “Computer Systems: A Programmer’s Perspective” - Bryant & O’Hallaron | Memory, Processes, Signals | P01-P06 |
| “The Art of Debugging with GDB” - Matloff & Salzman | GDB Usage | P02-P06 |
| “Understanding and Using C Pointers” - Reese | Memory & Pointers | P03 |
| “The Linux Programming Interface” - Kerrisk | System Calls, Threads | P05, P08 |
| “Practical Binary Analysis” - Andriesse | ELF Format, Reverse Engineering | P06, P07 |
| “Linux Device Drivers” - Corbet et al. | Kernel Modules | P08 |
| “Linux Kernel Development” - Love | Kernel Internals | P09 |
| “Designing Data-Intensive Applications” - Kleppmann | System Design | P10 |
Quick Start (First 48 Hours)
If you’re eager to get started immediately:
Hour 1-2: Set up your environment
# Install required tools
sudo apt-get update
sudo apt-get install build-essential gdb
# Verify GDB is working
gdb --version
Hour 3-4: Complete Project 1 (The First Crash)
- Write a simple crashing program
- Configure core dumps
- Generate your first core file
Hour 5-8: Complete Project 2 (The GDB Backtrace)
- Learn to compile with debug symbols
- Load a core dump in GDB
- Get your first backtrace
Weekend: Continue with Project 3 (Memory Inspector)
- Understand memory examination
- Debug a buffer overflow from a core dump
By the end of 48 hours, you’ll have the fundamental skills to analyze most user-space crashes.
Project Difficulty Progression
P10 ★★★★★
(Master)
│
P08 ★★★★ ─── P09 ★★★★
(Expert) (Expert)
│
P05 ★★★ ─── P06 ★★★ ─── P07 ★★★
(Advanced) (Advanced) (Advanced)
│
P03 ★★ ─── P04 ★★
(Intermediate) (Intermediate)
│
P01 ★ ─── P02 ★
(Beginner) (Beginner)
Success Metrics
By the end of this learning path, you should be able to:
- Generate and locate core dump files on any Linux system
- Use GDB to extract backtraces, examine memory, and inspect variables
- Debug crashes in multi-threaded applications
- Analyze crashes in stripped (production) binaries
- Understand and parse minidump file formats
- Configure kdump and capture kernel crash dumps
- Use the
crashutility to analyze kernel panics - Design and build crash reporting infrastructure
Getting Help
If you get stuck:
- Re-read the relevant section - Most answers are in the project guides
- Check the man pages -
man gdb,man core,man crash - Use GDB’s help system -
help <command>within GDB - Consult the recommended books - They provide deeper explanations
- Search for specific error messages - Stack Overflow and forums help with specific issues
Begin with Project 1: The First Crash to start your journey into crash dump analysis.