Project 2: The Crash - Crash Report Analysis

Build a crashing C program and perform post-mortem analysis with LLDB and a core file.

Quick Reference

Attribute Value
Difficulty Beginner
Time Estimate 1-2 hours
Language C (debug target), LLDB commands
Prerequisites Project 1 or equivalent LLDB basics
Key Topics Core dumps, stop reasons, register inspection

1. Learning Objectives

By completing this project, you will:

  1. Generate and load a core dump into LLDB.
  2. Identify the faulting instruction and stop reason.
  3. Inspect registers and variables to determine root cause.
  4. Produce a short crash report with actionable debugging notes.

2. Theoretical Foundation

2.1 Core Concepts

  • Crash vs. Core Dump: A crash is an event; a core dump is a snapshot of process memory and registers at the crash point.
  • Stop Reasons: LLDB stops with a reason like EXC_BAD_ACCESS or SIGSEGV, indicating invalid memory access.
  • Fault Address: The register state shows the address that caused the failure, often 0x0 for null dereferences.

2.2 Why This Matters

Most production bugs are found after the fact. Being able to analyze a core dump lets you diagnose failures without live access to the program, which is critical in production environments.

2.3 Historical Context / Background

Core dumps have existed for decades in Unix systems, originally as raw memory snapshots to be inspected with debuggers. Modern debuggers like LLDB add metadata and symbolic context via DWARF.

2.4 Common Misconceptions

  • “A core dump shows everything”: It shows a snapshot of memory and registers, not system state or logs.
  • “If it crashes, it must be the last line”: The crash line is the symptom; the cause may be earlier corruption.

3. Project Specification

3.1 What You Will Build

A minimal C program that dereferences a null pointer, producing a deterministic crash. You will then load the core file into LLDB and determine exactly why it failed.

3.2 Functional Requirements

  1. Generate a crash: Program must crash reliably.
  2. Capture a core file: Enable core dumps and locate the file.
  3. Inspect crash state: Use bt, frame, register read, and frame variable.

3.3 Non-Functional Requirements

  • Reliability: Crash must be deterministic.
  • Clarity: Use a single faulting instruction.
  • Reproducibility: Works on a clean environment with -g.

3.4 Example Usage / Output

$ ulimit -c unlimited
$ clang -g -o crash crash.c
$ ./crash
Segmentation fault (core dumped)
$ lldb ./crash -c core
(lldb) bt
* thread #1, stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: crash`crash_me at crash.c:3

3.5 Real World Outcome

You will load a core file and see LLDB highlight the exact failing line and the null pointer value. Example session:

$ lldb ./crash -c core
(lldb) bt
* thread #1, stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x000055555555513d crash`crash_me() at crash.c:3
(lldb) fr v p
(char *) p = 0x0000000000000000

4. Solution Architecture

4.1 High-Level Design

Program -> Crash -> Core dump
                 -> LLDB loads core -> Inspect stack + registers -> Root cause

4.2 Key Components

Component Responsibility Key Decisions
crash.c Trigger null deref Keep crash line obvious
Core file Snapshot at crash Enable with ulimit -c
LLDB session Post-mortem analysis Focus on stop reason and registers

4.3 Data Structures

void crash_me(void) {
    char *p = NULL;
    *p = 'A';
}

4.4 Algorithm Overview

Key Algorithm: Post-mortem Analysis

  1. Load core with lldb ./crash -c core.
  2. Run bt to find faulting frame.
  3. Inspect locals and registers.
  4. Identify invalid address and cause.

Complexity Analysis:

  • Time: O(1)
  • Space: O(1)

5. Implementation Guide

5.1 Development Environment Setup

ulimit -c unlimited
clang -g -o crash crash.c

5.2 Project Structure

project-root/
├── crash.c
└── README.md

5.3 The Core Question You’re Answering

“How do I reconstruct a crash from a core file when the process is already dead?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. Core Dumps
    • What information is stored?
    • Where does the OS write the file?
    • How do you enable/disable them?
    • Book Reference: “The Linux Programming Interface” Ch. 22
  2. Signals and Stop Reasons
    • What is SIGSEGV?
    • How does LLDB report a stop reason?
  3. Registers at Crash Time
    • Which register holds the faulting address?
    • How does LLDB display it?

5.5 Questions to Guide Your Design

Before implementing, think through these:

  1. How will you locate the core file produced by the OS?
  2. What does the stop reason tell you about the crash?
  3. Which variables should you inspect to explain the crash?

5.6 Thinking Exercise

Manual Crash Trace

Without LLDB, read the crash code and explain why it will fault. Then confirm your hypothesis by inspecting p in the core dump.

5.7 The Interview Questions They’ll Ask

Prepare to answer these:

  1. “What is a core dump, and why is it useful?”
  2. “What does SIGSEGV indicate?”
  3. “How would you debug a crash you cannot reproduce live?”

5.8 Hints in Layers

Hint 1: Use bt first

(lldb) bt

Hint 2: Check stop reason

(lldb) thread info

Hint 3: Read registers

(lldb) register read

5.9 Books That Will Help

Topic Book Chapter
Core dumps “The Linux Programming Interface” Ch. 22
Debugging “The Art of Debugging with GDB” Ch. 4-5
Signals “Advanced Programming in the Unix Environment” Ch. 10

5.10 Implementation Phases

Phase 1: Foundation (20 minutes)

Goals:

  • Enable core dumps.
  • Generate a deterministic crash.

Tasks:

  1. Write crash.c.
  2. Set ulimit -c unlimited.
  3. Run the program to produce a core file.

Checkpoint: A core file exists and matches the crash executable.

Phase 2: Core Functionality (30 minutes)

Goals:

  • Load the core in LLDB.
  • Identify the crash frame.

Tasks:

  1. Start lldb ./crash -c core.
  2. Run bt and select the faulting frame.

Checkpoint: You can see the exact line that caused the crash.

Phase 3: Polish & Edge Cases (30 minutes)

Goals:

  • Inspect registers and variables.
  • Write a short crash summary.

Tasks:

  1. Use register read and fr v.
  2. Summarize the root cause in a note.

Checkpoint: You can explain the crash in one paragraph.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Crash type Null deref vs buffer overflow Null deref Deterministic and simple
Debug data Core dump vs live attach Core dump Pure post-mortem workflow

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Crash Repro Ensure crash happens Run ./crash
Core Analysis Validate LLDB session bt, fr v, register read

6.2 Critical Test Cases

  1. Core file created: The OS produces a core file after crash.
  2. Stop reason matches SIGSEGV: LLDB shows invalid address.
  3. Variable inspection: p is 0x0.

6.3 Test Data

Fault address = 0x0

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Core dumps disabled No core file Run ulimit -c unlimited
Missing symbols No source lines Recompile with -g
Wrong core file Stack trace mismatched Use core for the same binary

7.2 Debugging Strategies

  • Verify binary/core match: target modules list to confirm UUIDs.
  • Start with bt: Always identify the failing frame first.

7.3 Performance Traps

Not applicable for this project.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a second function frame before the crash.
  • Use a conditional crash based on a command-line argument.

8.2 Intermediate Extensions

  • Cause a crash via out-of-bounds write and find it.
  • Compare LLDB output with GDB output.

8.3 Advanced Extensions

  • Analyze a crash in a stripped binary with dsymutil or split debug symbols.
  • Use image lookup -a to resolve raw addresses.

9. Real-World Connections

9.1 Industry Applications

  • Production crash triage: Post-mortem analysis on servers.
  • Mobile crash dumps: Similar workflow with symbolicated reports.
  • LLDB: https://lldb.llvm.org
  • Crashpad: https://chromium.googlesource.com/crashpad/crashpad

9.3 Interview Relevance

  • Core dumps and stop reasons are standard debugging topics.

10. Resources

10.1 Essential Reading

  • LLDB Tutorial - https://lldb.llvm.org/use/tutorial.html
  • Advanced Apple Debugging & Reverse Engineering by Derek Selander - Ch. 2-4

10.2 Video Resources

  • Crash analysis walkthrough - LLDB community videos

10.3 Tools & Documentation

  • LLDB Crash Debugging: https://lldb.llvm.org/use/remote.html

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain what a core dump contains.
  • I can interpret a stop reason like SIGSEGV.
  • I can identify the faulting instruction in LLDB.

11.2 Implementation

  • Core file loads correctly in LLDB.
  • I can inspect variables and registers at crash time.
  • I can summarize the crash cause in one paragraph.

11.3 Growth

  • I can debug a crash without rerunning the program.
  • I can teach someone else how to load a core in LLDB.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Produce a core file from crash.c.
  • Load it in LLDB and find the faulting line.
  • Identify the null pointer as the root cause.

Full Completion:

  • Inspect registers and explain how they prove the crash.
  • Add notes on how to prevent this class of bug.

Excellence (Going Above & Beyond):

  • Analyze a different crash type (buffer overflow or use-after-free).
  • Compare LLDB output to GDB and explain differences.

This guide was generated from LEARN_LLDB_DEEP_DIVE.md. For the complete learning path, see the parent directory.