Project 1: The First Crash

Learn to intentionally crash a program and capture the evidence - the foundation of all crash analysis.

Quick Reference

Attribute	Value
Difficulty	Beginner
Time Estimate	Weekend
Language	C
Prerequisites	Basic C programming, Linux command line
Key Topics	Segmentation faults, ulimit, core_pattern, ELF core dumps

---

1. Learning Objectives

By completing this project, you will:

• Understand what causes a segmentation fault at the hardware and OS level
• Write C code that reliably triggers a SIGSEGV signal
• Configure Linux to generate core dump files when processes crash
• Locate core dump files using the kernel’s core_pattern setting
• Verify that a core dump is a valid ELF file ready for analysis
• Create a reusable script to automate crash generation and verification
• Build the foundation for all subsequent crash analysis projects

---

2. Theoretical Foundation

2.1 Core Concepts

What is a Segmentation Fault?

A segmentation fault (segfault) occurs when a program attempts to access memory that it is not allowed to access. This is a protection mechanism provided by the operating system through the CPU’s Memory Management Unit (MMU).

Virtual Address Space of a Process
┌─────────────────────────────────────────────────────────────┐
│                        KERNEL SPACE                         │
│                   (Off limits to user code)                 │
├─────────────────────────────────────────────────────────────┤
│                          STACK                              │
│             (Local variables, function calls)               │
│                            ↓                                │
│                                                             │
│                      Unmapped Memory                        │
│            (Accessing here causes SIGSEGV)                  │
│                                                             │
│                            ↑                                │
│                          HEAP                               │
│                 (Dynamically allocated)                     │
├─────────────────────────────────────────────────────────────┤
│                          BSS                                │
│              (Uninitialized global data)                    │
├─────────────────────────────────────────────────────────────┤
│                         DATA                                │
│               (Initialized global data)                     │
├─────────────────────────────────────────────────────────────┤
│                         TEXT                                │
│                   (Program code - readonly)                 │
├─────────────────────────────────────────────────────────────┤
│               Reserved / NULL Page (0x0)                    │
│          (Deliberately unmapped - catches NULL              │
│           pointer dereferences)                             │
└─────────────────────────────────────────────────────────────┘

The first page of memory (around address 0x0) is intentionally left unmapped. This is a clever design decision: when a programmer forgets to initialize a pointer, it defaults to NULL (address 0). Any attempt to read from or write to this address triggers a segfault immediately, making the bug obvious rather than causing subtle corruption.

The Journey from NULL Dereference to Core Dump

When your program dereferences a NULL pointer, here is the sequence of events:

┌───────────────────────────────────────────────────────────────┐
│  1. Your Code: *ptr = 42;  (where ptr == NULL)               │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  2. CPU attempts to write to virtual address 0x0             │
│     MMU looks up page table entry for address 0x0            │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  3. Page table shows: Address 0x0 is NOT MAPPED              │
│     MMU raises a PAGE FAULT exception                        │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  4. CPU transfers control to kernel page fault handler       │
│     Kernel sees: fault in user space, invalid address        │
│     Decision: This is a protection violation, not fixable    │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  5. Kernel sends SIGSEGV signal to the process               │
│     Default action: Terminate and dump core                  │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  6. Kernel checks: Is core dumping enabled?                  │
│     - ulimit -c > 0?                                         │
│     - core_pattern configured?                               │
│     - Disk space available?                                  │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  7. If yes: Kernel writes process memory image to disk       │
│     Creates ELF core file with all register and memory state │
└─────────────────────────────┬─────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────┐
│  8. Process terminates                                       │
│     Shell displays: "Segmentation fault (core dumped)"       │
└───────────────────────────────────────────────────────────────┘

What is a Core Dump?

A core dump is a snapshot of a process’s memory at the moment it crashed. The name comes from early computing when memory was made of ferrite “cores” - tiny magnetic rings that stored bits.

A core dump file contains:

• CPU Register Values: The exact state of all registers (RIP, RSP, RAX, etc.) at the crash moment
• Memory Contents: All mapped memory regions including:
  • The program code (text segment)
  • Global variables (data/BSS segments)
  • The heap (dynamically allocated memory)
  • The stack (local variables and call history)
  • Memory-mapped files
• Process Metadata: Information about signals, threads, and file descriptors

This file is stored in ELF (Executable and Linkable Format), the same format used for executables and shared libraries on Linux.

The ulimit Command

The ulimit command controls resource limits for processes. The key setting for core dumps is:

ulimit -c [size]    # Core file size limit in blocks (512 bytes each)
ulimit -c 0         # Disable core dumps (default on many systems)
ulimit -c unlimited # Allow core dumps of any size

Why is it often disabled by default?

1. Core dumps can be large (gigabytes for big applications)
2. They may contain sensitive data (passwords, keys in memory)
3. They can fill up disk space if many crashes occur

The core_pattern Setting

The kernel’s /proc/sys/kernel/core_pattern controls how core dumps are named and where they are stored:

# Default pattern on many systems - creates file named "core" in current directory
core

# More descriptive pattern with PID
core.%p

# Full path with process name, PID, and timestamp
/var/cores/core.%e.%p.%t

# Piped to a program (used by systemd-coredump, abrt, etc.)
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

Common pattern specifiers:

• %p - PID of the crashed process
• %e - Executable filename
• %t - Time of crash (Unix timestamp)
• %u - User ID
• %h - Hostname
• %s - Signal number that caused the dump

2.2 Why This Matters

Understanding crash dump generation is the first step in post-mortem debugging - the art of figuring out what went wrong after a program has already crashed.

In production environments:

• You cannot attach a debugger to a running server
• Crashes happen at 3 AM when no one is watching
• The evidence is gone as soon as the process terminates

Core dumps preserve that evidence. They let you “freeze time” at the moment of failure and examine every detail of the program’s state.

Real-world applications:

• Web servers crash from memory corruption bugs
• Database engines crash from race conditions
• Game engines crash from physics calculation errors
• Operating systems crash (kernel panics) from driver bugs

All of these can be diagnosed if you have a core dump and know how to read it.

2.3 Historical Context

The concept of “dumping core” dates back to the 1950s and 1960s:

• 1949: EDSAC used delay-line memory, not cores
• 1950s: Ferrite core memory became standard (magnetic cores stored bits)
• 1955: First use of “core dump” to describe memory snapshots for debugging
• 1970s: Unix adopted the “core” filename for crash dumps
• 1980s: The term persisted even as core memory was replaced by semiconductor RAM
• 2000s: Linux added core_pattern for flexible naming
• 2010s: systemd-coredump added centralized core management

The name stuck despite the technology change - a testament to how fundamental the concept is.

2.4 Common Misconceptions

Misconception 1: “Core dumps are always created when a program crashes”

Reality: Core dumps are disabled by default on most modern Linux distributions. You must explicitly enable them with ulimit -c unlimited and ensure core_pattern points somewhere writable.

Misconception 2: “Any crash creates a core dump”

Reality: Only certain signals cause core dumps by default:

• SIGSEGV (segmentation fault)
• SIGABRT (abort)
• SIGFPE (floating-point exception)
• SIGBUS (bus error)
• SIGQUIT (quit from keyboard)
• SIGSYS (bad system call)

Signals like SIGKILL and SIGTERM terminate without dumping.

Misconception 3: “Core dumps and crash logs are the same thing”

Reality: Core dumps are binary snapshots of memory. They require special tools (GDB, crash) to interpret. Log files are human-readable text that the application chose to write before crashing.

Misconception 4: “Modern systems don’t use core dumps”

Reality: They absolutely do. Chrome uses Breakpad/Crashpad for minidumps. Linux distributions use systemd-coredump. Production servers use tools like abrt (Red Hat) or apport (Ubuntu). The concept has evolved but remains essential.

---

3. Project Specification

3.1 What You Will Build

You will create two components:

1. A crashing C program (crashing_program.c) - A minimal C program that reliably triggers a segmentation fault by dereferencing a NULL pointer.

2. An automation script (run_crash.sh) - A bash script that:

   • Enables core dumps for the current session
   • Compiles the C program with debug symbols
   • Runs the program and captures the crash
   • Locates the resulting core dump file
   • Verifies the core dump is a valid ELF file

3.2 Functional Requirements

1. The C program MUST crash with a segmentation fault every time it runs
2. The script MUST enable core dumps regardless of the system’s default settings
3. The script MUST compile the program with the -g flag for debug symbols
4. The script MUST detect whether a core dump was created
5. The script MUST use the file command to verify the core dump format
6. The script MUST handle the case where the core dump is created in a non-obvious location
7. The script MUST provide clear, informative output at each step

3.3 Non-Functional Requirements

• Portability: Script should work on major Linux distributions (Ubuntu, Debian, Fedora, CentOS)
• Robustness: Script should handle edge cases (missing compiler, permission issues)
• Educational: Output should explain what is happening at each step
• Cleanup: Script should offer to clean up generated files

3.4 Example Usage / Output

$ ./run_crash.sh
=== Crash Dump Generation Demo ===

[Step 1/6] Checking prerequisites...
  - GCC found: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
  - file command found: file-5.41

[Step 2/6] Configuring core dump settings...
  - Current ulimit -c: 0 (disabled)
  - Setting ulimit -c unlimited...
  - New ulimit -c: unlimited

[Step 3/6] Checking core_pattern...
  - Current pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
  - Note: systemd-coredump is handling core dumps
  - Core dumps will be stored by coredumpctl

[Step 4/6] Compiling crashing_program.c with debug symbols...
  - Command: gcc -g -o crashing_program crashing_program.c
  - Compilation successful

[Step 5/6] Running crashing_program...
  - Executing ./crashing_program
Segmentation fault (core dumped)

[Step 6/6] Verifying core dump...
  - Searching for core dump...
  - Found: /var/lib/systemd/coredump/core.crashing_progr.1000.abc123.1703865600000000.zst
  - Decompressing for verification...
  - File type: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from './crashing_program', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: './crashing_program', platform: 'x86_64'

=== SUCCESS ===
Core dump was generated and verified!
You can now analyze it with:
  gdb ./crashing_program /path/to/core

Cleanup: Remove generated files? [y/N]

3.5 Real World Outcome

When this project is complete, you will have:

1. A working demonstration of the entire crash-to-dump pipeline
2. A reusable script for generating test crashes
3. Deep understanding of why crashes sometimes produce dumps and sometimes don’t
4. The foundation needed for Project 2 (analyzing the dump with GDB)

---

4. Solution Architecture

4.1 High-Level Design

┌─────────────────────────────────────────────────────────────────┐
│                       run_crash.sh                              │
│                                                                 │
│  ┌───────────┐    ┌───────────┐    ┌───────────────────────┐   │
│  │  Check    │───▶│  Enable   │───▶│  Check core_pattern   │   │
│  │  prereqs  │    │  ulimit   │    │  (systemd or file?)   │   │
│  └───────────┘    └───────────┘    └───────────────────────┘   │
│                                               │                 │
│                                               ▼                 │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                     Compile C program                     │ │
│  │           gcc -g -o crashing_program crashing_program.c   │ │
│  └───────────────────────────────────────────────────────────┘ │
│                                               │                 │
│                                               ▼                 │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                   Execute program                         │ │
│  │                 ./crashing_program                        │ │
│  │                                                           │ │
│  │  ┌─────────────────────────────────────────────────────┐ │ │
│  │  │              crashing_program.c                      │ │ │
│  │  │                                                      │ │ │
│  │  │  int main() {                                        │ │ │
│  │  │      int *ptr = NULL;  ◄─── Pointer to address 0    │ │ │
│  │  │      *ptr = 42;        ◄─── Write to address 0      │ │ │
│  │  │      return 0;              causes SIGSEGV          │ │ │
│  │  │  }                                                   │ │ │
│  │  └─────────────────────────────────────────────────────┘ │ │
│  │                                                           │ │
│  │  Result: SIGSEGV ──▶ Kernel writes core dump             │ │
│  └───────────────────────────────────────────────────────────┘ │
│                                               │                 │
│                                               ▼                 │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                   Locate core dump                        │ │
│  │  Option A: Look for core.* in current directory          │ │
│  │  Option B: Query systemd-coredump via coredumpctl        │ │
│  └───────────────────────────────────────────────────────────┘ │
│                                               │                 │
│                                               ▼                 │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                   Verify with 'file'                      │ │
│  │           Confirm: "ELF 64-bit LSB core file"            │ │
│  └───────────────────────────────────────────────────────────┘ │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

4.2 Key Components

Component 1: The Crashing Program

A minimal C program that guarantees a segmentation fault. The code must:

• Use a NULL pointer (address 0x0)
• Attempt to write to that address
• Have no error handling that would prevent the crash

Component 2: The Shell Script

A bash script that orchestrates the entire process:

• Prerequisite check: Verify GCC and file are available
• Configuration: Enable core dumps via ulimit
• Detection: Determine how core dumps are handled on this system
• Compilation: Build the C program with debug symbols
• Execution: Run the program and capture exit status
• Verification: Find and validate the core dump file

4.3 Data Structures

The main “data structure” in this project is the ELF core file itself:

ELF Core File Structure
┌──────────────────────────────────────────────────────────────┐
│                      ELF Header                              │
│  - Magic number: 0x7f 'E' 'L' 'F'                           │
│  - Class: 64-bit (ELFCLASS64) or 32-bit                     │
│  - Type: ET_CORE (core dump)                                │
│  - Entry point: Usually 0 (not executable)                  │
│  - Program header offset                                     │
│  - Section header offset (usually 0 for core dumps)         │
└──────────────────────────────────────────────────────────────┘
│                    Program Headers                           │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ PT_NOTE: Contains process metadata                     │ │
│  │   - NT_PRSTATUS: Register state, signal info          │ │
│  │   - NT_PRPSINFO: Process name, state                  │ │
│  │   - NT_AUXV: Auxiliary vector                         │ │
│  │   - NT_FILE: Mapped files list                        │ │
│  └────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ PT_LOAD: Loadable memory segment (text)               │ │
│  │   - Virtual address, permissions (r-x)                │ │
│  │   - File offset to contents                           │ │
│  └────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ PT_LOAD: Loadable memory segment (data)               │ │
│  │   - Virtual address, permissions (rw-)                │ │
│  │   - File offset to contents                           │ │
│  └────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ PT_LOAD: Loadable memory segment (stack)              │ │
│  │   - Virtual address, permissions (rw-)                │ │
│  │   - File offset to contents                           │ │
│  └────────────────────────────────────────────────────────┘ │
│  ... more segments ...                                       │
└──────────────────────────────────────────────────────────────┘
│                    Memory Contents                           │
│  (The actual bytes from each memory region)                  │
└──────────────────────────────────────────────────────────────┘

4.4 Algorithm Overview

Step-by-step process:

1. Check prerequisites
   • Verify gcc is in PATH
   • Verify file is in PATH
   • Report versions for debugging
2. Configure core dump settings
   • Check current ulimit -c value
   • Set ulimit -c unlimited
   • Verify the change took effect
3. Determine core dump handling
   • Read /proc/sys/kernel/core_pattern
   • If starts with |, it’s piped to a program (e.g., systemd-coredump)
   • Otherwise, it’s a file path pattern
4. Compile the C program
   • Use gcc -g to include debug symbols
   • Check for compilation errors
5. Execute and capture crash
   • Run ./crashing_program
   • Save the exit status
   • Check for “(core dumped)” message
6. Locate the core dump
   • For systemd: use coredumpctl list
   • For file patterns: search in current directory or specified path
   • Handle compressed dumps (.zst, .lz4)
7. Verify the core dump
   • Run file command on the dump
   • Confirm “ELF 64-bit LSB core file” in output

---

5. Implementation Guide

5.1 Development Environment Setup

Required tools:

# On Ubuntu/Debian:
sudo apt update
sudo apt install gcc gdb file

# On Fedora/CentOS:
sudo dnf install gcc gdb file

# On Arch Linux:
sudo pacman -S gcc gdb file

# Verify installation:
gcc --version
gdb --version
file --version

Optional but recommended:

# For systemd-coredump systems:
coredumpctl --version

# For manual core dump inspection:
sudo apt install binutils  # For readelf
readelf --version

5.2 Project Structure

LEARN_LINUX_CRASH_DUMP_ANALYSIS/
├── README.md
├── P01-the-first-crash.md      # This file
├── crashing_program.c          # The C program to crash
├── run_crash.sh                # The automation script
└── core.*                      # Generated core dump (gitignored)

5.3 The Core Question You’re Answering

“What are the preconditions that must be satisfied for the Linux kernel to write a core dump file when a process crashes?”

This question drives the entire project. You’ll discover:

• The ulimit -c setting must be non-zero
• The process must have write permission to the core dump location
• Sufficient disk space must be available
• The signal must be one that generates core dumps
• No setuid/setgid bits on the executable (or special handling)
• The core_pattern determines where and how dumps are stored

5.4 Concepts You Must Understand First

Before starting, ask yourself:

1. What is a pointer in C?
   • Can you explain what int *ptr declares?
   • What is the difference between ptr and *ptr?
2. What is NULL?
   • What value does NULL typically have? (Answer: 0)
   • Why is address 0 special in Linux?
3. What is a signal in Unix?
   • Can you name a few signals? (SIGTERM, SIGKILL, SIGINT)
   • What does “sending a signal” mean?
4. What is an executable file?
   • How do you compile a C program?
   • What does gcc -o output source.c do?

If you can answer these, you’re ready to proceed.

5.5 Questions to Guide Your Design

For the C program:

• What is the simplest way to create an invalid memory access?
• Should you use malloc() and not initialize? Or just use NULL directly?
• Should you read from or write to the bad address? Does it matter?

For the script:

• How will you handle systems where ulimit -c unlimited doesn’t persist?
• What if the user doesn’t have permission to change ulimit?
• How will you detect if systemd-coredump is active?
• Where might the core dump be if core_pattern is a relative path?

5.6 Thinking Exercise

Before writing any code, trace through this scenario mentally:

1. You have a system where ulimit -c is 0 (disabled)
2. You run a program that dereferences NULL
3. What happens? Walk through each step:
   • CPU encounters invalid memory access
   • Page fault occurs
   • Kernel decides to send SIGSEGV
   • Kernel checks if it should dump core
   • With ulimit -c at 0, what happens?

Now trace the same scenario with ulimit -c unlimited:

• At which step does the behavior differ?
• What additional work does the kernel do?

5.7 Hints in Layers

Hint 1: Starting Point (Conceptual)

The crashing program is literally 5 lines of code. You need:

1. A pointer variable
2. The pointer set to an invalid address
3. An attempt to use that pointer

The script needs to manipulate shell settings and find files. Start by doing each step manually in the terminal before scripting it.

Hint 2: Next Level (More Specific)

For the C program, remember:

• NULL is defined in <stddef.h> (or just use 0)
• The simplest crash: *p = value; where p is NULL

For the script:

• ulimit -c is a shell builtin - it only affects the current shell and children
• core_pattern is read with cat /proc/sys/kernel/core_pattern
• A pattern starting with | means piped to a program

Hint 3: Technical Details (Approach)

C program structure:

// What header(s) do you need? (Think about it)
// Hint: Maybe none for the minimal version!

int main(void) {
    // Declare a pointer to int
    // Set it to NULL
    // Dereference and assign a value
    // Return (you'll never reach this)
}

Script structure:

#!/bin/bash
# Check for gcc and file commands
# Set ulimit -c unlimited
# Check core_pattern
# Compile with gcc -g
# Run the program
# Find the core file
# Verify with file command

Hint 4: Tools/Debugging (Verification)

How to verify each step manually:

# Check current ulimit
ulimit -c

# Set unlimited
ulimit -c unlimited

# Verify it changed
ulimit -c

# Check core pattern
cat /proc/sys/kernel/core_pattern

# If using systemd, check for recent dumps
coredumpctl list

# Compile manually
gcc -g -o crashing_program crashing_program.c

# Run and observe
./crashing_program
echo $?  # Exit code (139 = 128 + 11, where 11 = SIGSEGV)

# Find core (traditional systems)
ls -la core*

# Verify with file
file core.1234

5.8 The Interview Questions They’ll Ask

1. “What is a segmentation fault and what causes it?”
   • Expected: Memory access violation, typically NULL pointer dereference, use after free, or buffer overflow
2. “Why might a core dump not be generated when a program crashes?”
   • Expected: ulimit set to 0, insufficient disk space, pipe to handler that drops it, setuid binary restrictions
3. “What information is stored in a core dump file?”
   • Expected: CPU register state, process memory (stack, heap, data segments), signal information, thread info
4. “How would you enable core dumps on a production server?”
   • Expected: Set ulimit in service config, configure core_pattern, ensure disk space, consider security implications
5. “What’s the difference between SIGSEGV and SIGBUS?”
   • Expected: SIGSEGV = unmapped memory access; SIGBUS = aligned memory access violation or bad physical address
6. “If you see ‘Segmentation fault’ without ‘(core dumped)’, what do you check?”
   • Expected: ulimit -c value, core_pattern setting, disk space, directory permissions

5.9 Books That Will Help

Topic	Book	Chapter(s)
Virtual Memory and Signals	Computer Systems: A Programmer’s Perspective (Bryant & O’Hallaron)	Chapter 9
C Pointers	The C Programming Language (Kernighan & Ritchie)	Chapter 5
Unix Signals	The Linux Programming Interface (Kerrisk)	Chapters 20-22
Core Dumps	Advanced Programming in the UNIX Environment (Stevens)	Chapter 10
Shell Scripting	The Linux Command Line (Shotts)	Chapters 24-36

5.10 Implementation Phases

Phase 1: Manual Testing (30 minutes)

• Perform every step manually in the terminal
• Write a crashing C program in a text editor
• Compile and run it
• Find and verify the core dump
• Document what you did

Phase 2: Basic Script (1 hour)

• Create a script that does the manual steps
• Add echo statements to show progress
• Test on your system

Phase 3: Robust Script (2-3 hours)

• Add prerequisite checking
• Handle systemd-coredump vs. traditional core files
• Add error handling
• Test on multiple configurations

Phase 4: Polish (1 hour)

• Add cleanup functionality
• Add colored output (optional)
• Add help message
• Final testing

5.11 Key Implementation Decisions

Decision 1: How to crash?

Option A: NULL pointer dereference (simplest)

int *p = NULL;
*p = 42;

Option B: Array out of bounds

int arr[10];
arr[1000000] = 42;  // May or may not crash depending on system

Option C: Call abort()

#include <stdlib.h>
abort();  // Sends SIGABRT, not SIGSEGV

Recommendation: Use Option A. It’s the most reliable and demonstrates the most common cause of crashes.

Decision 2: How to handle systemd-coredump?

Option A: Ignore systemd, write your own core_pattern

• Requires root
• Modifies system configuration

Option B: Use coredumpctl to retrieve dumps

• Works with the system as configured
• More realistic for real-world debugging

Recommendation: Option B for portability, but document how to do Option A.

Decision 3: Should the script require root?

Option A: Require root, modify core_pattern Option B: Run as regular user, work with existing configuration

Recommendation: Option B. The script should demonstrate the concepts without requiring system modification.

---

6. Testing Strategy

Test Case 1: Verify Crash Behavior

# Compile without the script
gcc -g -o crashing_program crashing_program.c

# Run the program
./crashing_program

# Expected output:
# Segmentation fault (core dumped)

# Verify exit code
echo $?
# Expected: 139 (128 + 11, where 11 = SIGSEGV)

Test Case 2: Verify Core Dump Generation

# Enable core dumps
ulimit -c unlimited

# Run the program
./crashing_program

# Check for core file
ls -la core* 2>/dev/null || coredumpctl list | head -5

# At least one method should show a dump

Test Case 3: Verify ELF Format

# Get the core file path (varies by system)
CORE_FILE=$(ls core* 2>/dev/null | head -1)

# Or for systemd:
# coredumpctl dump -o /tmp/core

file $CORE_FILE

# Expected output contains:
# ELF 64-bit LSB core file

Test Case 4: Test Script Error Handling

# Test with missing gcc
PATH=/bin ./run_crash.sh
# Should report: GCC not found

# Test with insufficient permissions
chmod 000 crashing_program.c
./run_crash.sh
# Should report: Cannot read source file

---

7. Common Pitfalls & Debugging

Pitfall 1: “Segmentation fault” but No “(core dumped)”

Symptom: The program crashes but no core file appears.

Root Cause: ulimit -c is set to 0.

Fix:

# Check the limit
ulimit -c
# If it shows 0:
ulimit -c unlimited

Verification: Run the crashing program again and check for “(core dumped)” in the output.

Pitfall 2: Can’t Find the Core File

Symptom: “(core dumped)” appears but no file in current directory.

Root Cause: core_pattern is set to a different location or piped to systemd-coredump.

Fix:

# Check where cores go
cat /proc/sys/kernel/core_pattern

# If it starts with |, use coredumpctl:
coredumpctl list
coredumpctl dump -o ./core  # Extract latest dump

Verification: The coredumpctl list command shows your crash.

Pitfall 3: Compilation Errors

Symptom: gcc: command not found or “undefined reference”

Root Cause: GCC not installed or wrong package.

Fix:

# Ubuntu/Debian
sudo apt install build-essential

# Fedora
sudo dnf install gcc

# Arch
sudo pacman -S gcc

Verification: gcc --version shows version information.

Pitfall 4: Core Dump File is Empty or Truncated

Symptom: Core file exists but is 0 bytes or very small.

Root Cause: Insufficient disk space or ulimit too small.

Fix:

# Check disk space
df -h .

# Check ulimit (might be limited to specific size)
ulimit -c

# Set truly unlimited
ulimit -c unlimited

Verification: ls -la core* shows a file of several kilobytes or more.

Pitfall 5: Permission Denied When Writing Core

Symptom: Shell might not show “(core dumped)” if write fails.

Root Cause: Current directory is not writable, or core_pattern points to unwritable location.

Fix:

# Check directory permissions
ls -la .

# Create cores in /tmp if needed
sudo sysctl kernel.core_pattern=/tmp/core.%e.%p

Verification: Run in a directory where you have write permission.

Pitfall 6: setuid/setgid Binary Restrictions

Symptom: Core dump not generated for setuid programs.

Root Cause: Security measure to prevent sensitive data leakage.

Fix: Don’t use setuid for testing, or enable with:

# Allow core dumps for setuid (not recommended in production)
sudo sysctl fs.suid_dumpable=1

Verification: Use a regular (non-setuid) binary for this project.

---

8. Extensions & Challenges

Extension 1: Different Crash Types

Modify your crashing program to demonstrate different signals:

// SIGABRT (signal 6)
#include <stdlib.h>
abort();

// SIGFPE (signal 8) - Floating point exception
int x = 1 / 0;  // Note: Modern compilers may optimize this

// SIGBUS (signal 7) - Requires aligned access violation
// More complex to trigger reliably

Create separate test programs for each and compare the core dumps.

Extension 2: Stack Trace Preview

Extend your script to show a basic stack trace:

# After finding the core dump, add:
echo "Quick backtrace preview:"
gdb -batch -ex "bt" ./crashing_program ./core 2>/dev/null

Extension 3: Core Dump Comparison Tool

Write a script that:

1. Generates multiple crashes
2. Compares their core dump sizes
3. Extracts and compares backtraces
4. Reports differences (useful for detecting heap size variations)

Extension 4: Custom Core Pattern

Configure your own core pattern on a test VM:

# As root:
echo "/var/cores/core.%e.%p.%t" > /proc/sys/kernel/core_pattern
mkdir -p /var/cores
chmod 777 /var/cores  # For testing only!

# Test and observe the naming
./crashing_program
ls /var/cores/

Extension 5: Understand the ELF Structure

Use readelf to explore the core dump:

readelf -h core              # Show ELF header
readelf -l core              # Show program headers
readelf -n core              # Show notes (process info)

Write a summary of what you find.

---

9. Real-World Connections

Production Core Dump Management

In production environments, organizations use:

1. systemd-coredump (Linux)

• Compresses dumps automatically
• Stores in /var/lib/systemd/coredump/
• Query with coredumpctl
• Can be configured to send to remote servers

2. abrt (Red Hat/Fedora)

• Automatic Bug Reporting Tool
• Analyzes crashes, creates reports
• Can submit to bugzilla automatically

3. apport (Ubuntu)

• Creates .crash files in /var/crash
• Integrates with Ubuntu’s error tracker
• Can be configured for custom handling

4. Google Breakpad/Crashpad

• Cross-platform (Windows, macOS, Linux)
• Generates minidumps (smaller than full cores)
• Used by Chrome, Firefox, Electron apps

Example: How Chrome Handles Crashes

Chrome Process
      │
      ▼ (crash)
┌─────────────────────────────────────────┐
│           Crashpad Handler              │
│  - Catches the crash signal             │
│  - Creates minidump in memory           │
│  - Writes to /tmp/crash-reports/        │
│  - Uploads to crash.corp.google.com     │
└─────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────┐
│      Google's Crash Analysis Server     │
│  - Symbolizes the stack trace           │
│  - Groups similar crashes               │
│  - Creates bug reports automatically    │
└─────────────────────────────────────────┘

Security Considerations

Core dumps can contain:

• Passwords in memory
• Encryption keys
• Personal user data
• API tokens and secrets

Production systems often:

• Disable core dumps entirely
• Encrypt core dumps at rest
• Restrict access to core dump directories
• Automatically delete dumps after analysis
• Redact sensitive data before upload

---

10. Resources

Official Documentation

• man core - Core dump file format and configuration
• man signal - Signal types and default actions
• man ulimit - Shell resource limits
• man gdb - GNU Debugger manual

Online References

• Anatomy of a Core Dump
• How Core Dumps are Made
• ELF Specification
• systemd-coredump Documentation

Tools

• GDB - GNU Debugger for analyzing core dumps
• readelf - Display ELF file information
• file - Identify file types
• coredumpctl - Query systemd-coredump
• objdump - Display object file information

Related Projects in This Series

• Project 2: The GDB Backtrace - Use GDB to analyze the core dump you created
• Project 3: The Memory Inspector - Examine memory contents in a core dump
• Project 4: The Automated Crash Detective - Build a script to automate analysis

---

11. Self-Assessment Checklist

Before considering this project complete, verify:

Conceptual Understanding

• I can explain what a segmentation fault is
• I understand why address 0 causes a crash
• I can describe what information is in a core dump
• I know the difference between SIGSEGV, SIGABRT, and SIGBUS
• I understand why core dumps might be disabled by default

Practical Skills

• I can write C code that crashes with a segfault
• I can enable core dumps with ulimit
• I can find the core_pattern setting
• I can locate a core dump file on my system
• I can verify a core dump is a valid ELF file

Debugging Ability

• I know what to check when “(core dumped)” doesn’t appear
• I can troubleshoot missing core files
• I understand the difference between systemd and traditional core handling

Script Quality

• My script checks prerequisites before running
• My script provides informative output
• My script handles errors gracefully
• My script works on at least one Linux distribution

---

12. Submission / Completion Criteria

This project is complete when:

1. The C program (crashing_program.c) exists and:
   • Compiles without warnings with gcc -Wall -g
   • Crashes with a segmentation fault 100% of the time
   • Is minimal (under 10 lines of code)
2. The shell script (run_crash.sh) exists and:
   • Runs without errors on your system
   • Enables core dumps
   • Compiles the C program
   • Runs the program and captures the crash
   • Locates the core dump file
   • Verifies the core dump with the file command
   • Provides clear output at each step
3. You can answer:
   • Why does dereferencing NULL crash the program?
   • What does ulimit -c unlimited do?
   • Where does your system store core dumps?
   • How do you verify a file is a valid core dump?
4. You have documented:
   • Any system-specific configuration needed
   • How to run your script
   • What the expected output looks like

---

Next Steps

Once you have completed this project, you are ready for Project 2: The GDB Backtrace, where you will:

• Load the core dump you created into GDB
• Learn to read a stack backtrace
• Understand the importance of debug symbols
• Find the exact line of code that caused the crash

The core dump you generated here is your ticket to learning the most fundamental skill in crash analysis: reading a backtrace.

---

Project 1 of 10 in the Linux Crash Dump Analysis learning path.