Project 3: The Hang - Attaching to a Running Process

Create a hanging program and attach LLDB to inspect and modify it live.

Quick Reference

Attribute Value
Difficulty Intermediate
Time Estimate 1 hour
Language C (debug target), LLDB commands
Prerequisites Projects 1-2 or equivalent LLDB basics
Key Topics process attach, live inspection, detach

1. Learning Objectives

By completing this project, you will:

  1. Attach LLDB to a running process by PID.
  2. Pause execution and inspect the active thread.
  3. Modify a variable in a live process.
  4. Detach cleanly without killing the process.

2. Theoretical Foundation

2.1 Core Concepts

  • Attach vs. Launch: Attaching stops an existing process and creates a debug session without restarting it.
  • Signals and Stop Reasons: LLDB sends a stop signal (e.g., SIGSTOP) to pause the process.
  • Thread Context: In a live process, you can inspect any thread, not just the one you are interested in.

2.2 Why This Matters

Real systems hang in production or staging, and restarting them can be expensive. Live attach is the fastest way to inspect state without losing the current conditions.

2.3 Historical Context / Background

Unix debuggers have supported attach for decades. LLDB improves the experience with fast attach and rich symbol awareness in modern toolchains.

2.4 Common Misconceptions

  • “Attach kills the process”: It only stops it temporarily; detach resumes it.
  • “You cannot modify live state”: LLDB lets you change variables with expr.

3. Project Specification

3.1 What You Will Build

A C program that loops forever (a simulated hang). You will attach LLDB to it, pause execution, inspect the call stack, and update a counter variable to confirm you can modify running state.

3.2 Functional Requirements

  1. Start a long-running process: Use a loop with sleep.
  2. Attach by PID: Use process attach --pid.
  3. Inspect and modify state: Use bt, frame variable, and expr.
  4. Detach cleanly: The process should continue after detach.

3.3 Non-Functional Requirements

  • Stability: Process should keep running after detach.
  • Safety: Avoid corrupting memory; modify only simple variables.

3.4 Example Usage / Output

# Terminal 1
$ ./hang

# Terminal 2
$ pgrep hang
12345
$ lldb
(lldb) process attach --pid 12345
Process 12345 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: ... in sleep
(lldb) frame select 1
(lldb) fr v counter
(int) counter = 10
(lldb) expr counter = 100
(int) $0 = 100
(lldb) detach

3.5 Real World Outcome

You will attach to a running program, see it pause, change a variable, and detach so it keeps running. Example output:

(lldb) process attach --pid 12345
Process 12345 stopped
* thread #1, stop reason = signal SIGSTOP
(lldb) fr v counter
(int) counter = 10
(lldb) expr counter = 100
(int) $0 = 100
(lldb) detach
Process 12345 detached

4. Solution Architecture

4.1 High-Level Design

Running process -> LLDB attach -> Stop process -> Inspect/modify -> Detach -> Process resumes

4.2 Key Components

Component Responsibility Key Decisions
hang.c Simulate unresponsive loop Keep it single-threaded
LLDB attach Pause and inspect Use PID attach for precision

4.3 Data Structures

volatile int counter = 0;
while (1) { counter++; sleep(1); }

4.4 Algorithm Overview

Key Algorithm: Live Attach Debugging

  1. Identify PID with pgrep.
  2. Attach with process attach --pid.
  3. Inspect stack and locals.
  4. Modify counter with expr.
  5. Detach.

Complexity Analysis:

  • Time: O(1) for operations
  • Space: O(1)

5. Implementation Guide

5.1 Development Environment Setup

clang -g -o hang hang.c

5.2 Project Structure

project-root/
├── hang.c
└── README.md

5.3 The Core Question You’re Answering

“How do I inspect and change a running program without restarting it?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. Process IDs and Signals
    • What is a PID?
    • What does SIGSTOP do?
    • Book Reference: APUE Ch. 10
  2. Thread State
    • How does LLDB select threads?
    • Why does frame select matter?
  3. Expression Evaluation
    • How does expr modify program state?

5.5 Questions to Guide Your Design

Before implementing, think through these:

  1. What output shows the program is still running after detach?
  2. How can you verify counter changed?
  3. What does the call stack look like inside sleep?

5.6 Thinking Exercise

Write a Resume-Safe Debug Note

Write a short note describing how you would attach to a hung service, identify the blocking function, and detach without disrupting users.

5.7 The Interview Questions They’ll Ask

Prepare to answer these:

  1. “How do you attach LLDB to a running process?”
  2. “What is the difference between attaching and launching?”
  3. “How do you modify a variable in a running process?”

5.8 Hints in Layers

Hint 1: Find the PID

pgrep hang

Hint 2: Use frame select

(lldb) frame select 1

Hint 3: Confirm state change

Let the program run and reattach later to confirm counter keeps incrementing.

5.9 Books That Will Help

Topic Book Chapter
Processes and signals “Advanced Programming in the Unix Environment” Ch. 10
Debugging workflow “The Art of Debugging with GDB” Ch. 6

5.10 Implementation Phases

Phase 1: Foundation (15 minutes)

Goals:

  • Build the hanging program.
  • Run it in a separate terminal.

Tasks:

  1. Write hang.c.
  2. Compile and run.

Checkpoint: Program keeps running without exiting.

Phase 2: Core Functionality (25 minutes)

Goals:

  • Attach and inspect.
  • Find the loop frame.

Tasks:

  1. Find PID with pgrep.
  2. Attach with LLDB.
  3. Use bt and frame select.

Checkpoint: You can inspect counter.

Phase 3: Polish & Edge Cases (20 minutes)

Goals:

  • Modify state.
  • Detach safely.

Tasks:

  1. Use expr counter = 100.
  2. Detach and confirm the program continues.

Checkpoint: Program remains alive after detach.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Attach method PID vs name PID Avoid attaching to wrong process
Variable to modify Counter vs pointer Counter Low risk of corruption

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Attach Test Verify attach works process attach --pid
Live Inspect Validate stack and locals bt, fr v
Live Modify Confirm expr changes value expr counter = 100

6.2 Critical Test Cases

  1. Attach succeeds: LLDB stops the process.
  2. Stack trace visible: bt shows main and sleep.
  3. Variable modification: counter updates.

6.3 Test Data

counter: 0 -> 100

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Attaching wrong PID Stack trace is unrelated Use ps -f to confirm
No symbols No source line info Build with -g
Forgetting detach Process stays paused detach or process continue

7.2 Debugging Strategies

  • Always verify PID: use ps -fp <pid> to confirm.
  • Check threads: thread list for multi-threaded apps.

7.3 Performance Traps

Not applicable for this project.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a signal handler and inspect it in LLDB.
  • Use process interrupt instead of attach on a child process.

8.2 Intermediate Extensions

  • Create a multi-threaded loop and inspect different threads.
  • Add a busy loop to compare CPU usage with a sleeping loop.

8.3 Advanced Extensions

  • Attach to a real long-running service (local web server).
  • Automate attach and backtrace with an LLDB script.

9. Real-World Connections

9.1 Industry Applications

  • Production hangs: attach to diagnose stuck worker threads.
  • Deadlock detection: inspect lock ownership and wait states.
  • LLDB: https://lldb.llvm.org
  • strace: https://strace.io - Complementary tool for live process inspection.

9.3 Interview Relevance

  • Explaining how to debug a hanging process is a common systems interview topic.

10. Resources

10.1 Essential Reading

  • LLDB Command Guide - https://lldb.llvm.org/use/command.html

10.2 Video Resources

  • Live attach demos - LLDB community videos

10.3 Tools & Documentation

  • process attach: https://lldb.llvm.org/use/command.html#process

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain the difference between attach and launch.
  • I can interpret a stack trace from a live process.
  • I understand how expr affects live state.

11.2 Implementation

  • I can attach by PID and inspect locals.
  • I can safely detach and resume the program.
  • I can change a variable and observe the effect.

11.3 Growth

  • I can describe a safe live-debug workflow for production.
  • I can explain how to avoid accidentally killing a process.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Attach to a running process.
  • Inspect a local variable.
  • Detach without killing the process.

Full Completion:

  • Modify a variable and explain the effect.
  • Document the steps as a repeatable checklist.

Excellence (Going Above & Beyond):

  • Attach to a multi-threaded program and analyze thread states.
  • Automate attach and bt output with a custom LLDB command.

This guide was generated from LEARN_LLDB_DEEP_DIVE.md. For the complete learning path, see the parent directory.