Project 9: System Daemon with Proper Daemonization

Build a properly daemonized background service that detaches from terminal, handles signals for clean shutdown, implements single-instance locking, and logs through syslog.

Quick Reference

Attribute	Value
Difficulty	Level 3 - Advanced
Time Estimate	2-3 weeks (25-40 hours)
Language	C (primary), Rust/Go (alternatives)
Prerequisites	Signal handling (Project 6), process relationships
Key Topics	Daemon processes, sessions, setsid(), syslog, PID files

1. Learning Objectives

After completing this project, you will be able to:

Understand process relationships including sessions, process groups, and controlling terminals
Implement the double-fork daemonization pattern and explain why each step is necessary
Handle signals correctly in daemons including SIGHUP for config reload and SIGTERM for shutdown
Use syslog for daemon logging with proper priorities and facilities
Implement single-instance locking using PID files and file locking
Write production-quality daemon code that follows UNIX conventions
Debug daemon startup issues using strace, journalctl, and log analysis

2. Theoretical Foundation

2.1 Core Concepts

A daemon is a background process that:

Has no controlling terminal (can’t receive keyboard signals)
Runs as a child of init/systemd (orphaned from parent)
Has its own session and process group
Runs indefinitely until explicitly stopped

Daemon vs Regular Process

Regular Process:                    Daemon Process:
┌─────────────────────┐            ┌─────────────────────┐
│ Terminal (pts/0)    │            │ init (PID 1)        │
│                     │            │         │           │
│ ┌─────────────────┐ │            │         ▼           │
│ │ Shell (bash)    │ │            │ ┌─────────────────┐ │
│ │  Parent of your │ │            │ │ mydaemon        │ │
│ │  program        │ │            │ │  - no terminal  │ │
│ │        │        │ │            │ │  - own session  │ │
│ │        ▼        │ │            │ │  - runs forever │ │
│ │ Your Program    │ │            │ └─────────────────┘ │
│ │  - receives     │ │            │                     │
│ │    Ctrl-C       │ │            │ Cannot be:          │
│ │  - dies when    │ │            │  - interrupted by   │
│ │    terminal     │ │            │    Ctrl-C           │
│ │    closes       │ │            │  - affected by      │
│ └─────────────────┘ │            │    logout           │
└─────────────────────┘            └─────────────────────┘

The Daemonization Ritual:

The Double-Fork Dance

Step 1: First Fork
┌──────────────────┐
│ Parent (shell)   │  Parent exits immediately
│     │            │  → Returns control to shell
│     ▼            │
│   fork()         │
│     │            │
└──────┼───────────┘
       │
       ▼
┌──────────────────┐
│ Child 1          │  Now orphaned (parent died)
│ - Still has      │  Adopted by init/systemd
│   session of     │
│   parent         │
│     │            │
│   setsid()       │  Creates NEW session
│     │            │  Child becomes session leader
│     ▼            │  No controlling terminal
│   fork() again   │
└──────┼───────────┘
       │
       ▼
Step 2: Second Fork
┌──────────────────┐
│ Child 2          │  The actual daemon
│ (Grandchild)     │
│ - New session    │
│ - NOT session    │  Cannot acquire
│   leader         │  controlling terminal
│ - No controlling │  (only session leaders can)
│   terminal       │
│ - Parent is init │
└──────────────────┘

Why second fork?
Session leaders CAN acquire a controlling terminal
by opening a terminal device. The second fork
ensures the daemon is NOT a session leader,
so it can NEVER accidentally get a terminal.

2.2 Why This Matters

Real-World Usage:

Every background service you use is a daemon:

Web servers: nginx, Apache, lighttpd
Databases: PostgreSQL, MySQL, MongoDB
Schedulers: cron, systemd timers
Logging: syslog-ng, rsyslog
Container runtimes: containerd, dockerd
Network services: sshd, named (DNS)

Career Impact:

Understanding daemons is essential for:

Backend/infrastructure engineering roles
DevOps and SRE positions
Systems programming jobs
Building any long-running service
Debugging production server issues

Industry Reality:

When a web server fails to start or mysteriously stops, understanding the daemonization process helps you:

Read and interpret log files correctly
Understand why the process died
Debug PID file issues
Fix permission problems
Handle signal delivery

2.3 Historical Context

The UNIX Way (1970s-2000s): Daemons managed themselves with the double-fork pattern. Each daemon had custom code for:

Daemonization
PID file management
Signal handling
Log rotation

SysVinit Era (1980s-2010s): Init scripts in /etc/init.d/ started daemons and expected them to self-daemonize. The scripts used PID files to track daemon status.

systemd Era (2010s-present): systemd can handle daemonization itself. Modern daemons can run in foreground (Type=simple) while systemd manages backgrounding, logging, and restart. However:

Understanding the traditional pattern is still crucial
Not all systems use systemd (BSD, macOS, embedded)
Legacy systems still require self-daemonizing daemons
The concepts (sessions, process groups) still apply

2.4 Common Misconceptions

Misconception 1: “Just add & to run in background”

Reality: ./mydaemon & backgrounds the process but:

It still has a controlling terminal
It dies when the terminal closes (unless using nohup)
It’s in the shell’s session and process group

Misconception 2: “nohup is the same as daemonization”

Reality: nohup ignores SIGHUP but doesn’t:

Create a new session
Change working directory
Close file descriptors properly
Handle PID files

Misconception 3: “systemd makes daemonization obsolete”

Reality: systemd simplifies deployment but:

BSD systems don’t use systemd
macOS uses launchd (different approach)
Embedded systems often have minimal init
Understanding the concepts remains important

Misconception 4: “One fork is enough”

Reality: One fork + setsid creates a session leader that CAN acquire a controlling terminal. The second fork prevents this possibility.

3. Project Specification

3.1 What You Will Build

A daemon process with the following capabilities:

Proper daemonization using the double-fork pattern
PID file management with file locking for single-instance enforcement
Signal handling for SIGHUP (reload), SIGTERM (shutdown), SIGUSR1 (status)
Syslog integration for all logging output
Configuration file support with runtime reloading
Graceful shutdown with proper resource cleanup

3.2 Functional Requirements

Daemonization
- Fork twice to fully detach from terminal
- Create new session with setsid()
- Change working directory to /
- Reset umask to 0
- Close stdin/stdout/stderr, redirect to /dev/null
PID File
- Create PID file at configurable location (default: /var/run/mydaemon.pid)
- Lock PID file to prevent multiple instances
- Write PID to file after daemonization
- Remove PID file on clean shutdown
Signal Handling
- SIGTERM: Graceful shutdown
- SIGHUP: Reload configuration
- SIGUSR1: Write status to syslog
- SIGINT: Graceful shutdown (same as SIGTERM)
Logging
- Use syslog with configurable facility (default: LOG_DAEMON)
- Log startup, shutdown, config reload events
- Log errors with appropriate priority levels
- Include PID in log messages
Configuration
- Read config file at startup
- Support runtime reload via SIGHUP
- Validate configuration before applying
Main Loop
- Perform periodic work (configurable interval)
- Check for shutdown signal
- Handle errors gracefully

3.3 Non-Functional Requirements

Reliability
- Never crash on malformed input
- Handle all signals safely (async-signal-safe functions only)
- Recover from transient errors
Security
- Drop privileges after opening privileged resources
- Validate configuration file permissions
- Don’t follow symbolic links for PID file
Portability
- Work on Linux and BSD systems
- Handle differences in signal behavior
- Use POSIX functions where possible
Observability
- All state changes logged
- Status queryable via SIGUSR1
- Exit code indicates shutdown reason

3.4 Example Usage / Output

# 1. Start the daemon
$ ./mydaemon
mydaemon: starting (pid 12345)
mydaemon: daemonizing...
$                          # Prompt returns immediately!

# 2. Check it's running
$ ps aux | grep mydaemon
root     12346  0.0  0.1  12345  1234 ?  Ss  10:00  0:00 mydaemon

# Note: PID is 12346, not 12345 (grandchild after double-fork)

# 3. Check PID file
$ cat /var/run/mydaemon.pid
12346

# 4. Check logs
$ tail /var/log/syslog
Mar 15 10:00:00 host mydaemon[12346]: Daemon started successfully
Mar 15 10:00:00 host mydaemon[12346]: Configuration loaded from /etc/mydaemon.conf
Mar 15 10:00:01 host mydaemon[12346]: Entering main loop (interval: 60s)

# 5. Send SIGHUP to reload config
$ kill -HUP 12346
$ tail /var/log/syslog
Mar 15 10:01:00 host mydaemon[12346]: Received SIGHUP, reloading configuration
Mar 15 10:01:00 host mydaemon[12346]: Configuration reloaded successfully

# 6. Check status via SIGUSR1
$ kill -USR1 12346
$ tail /var/log/syslog
Mar 15 10:02:00 host mydaemon[12346]: Status: uptime=120s, iterations=2, last_run=10:01:00

# 7. Try to start second instance
$ ./mydaemon
mydaemon: error: already running (pid 12346)
$ echo $?
1

# 8. Stop daemon gracefully
$ kill -TERM 12346
$ tail /var/log/syslog
Mar 15 10:03:00 host mydaemon[12346]: Received SIGTERM, initiating shutdown
Mar 15 10:03:00 host mydaemon[12346]: Closing resources...
Mar 15 10:03:00 host mydaemon[12346]: Removing PID file
Mar 15 10:03:00 host mydaemon[12346]: Daemon stopped (exit code 0)

$ ps aux | grep mydaemon
# (no output - daemon is gone)

$ cat /var/run/mydaemon.pid
cat: /var/run/mydaemon.pid: No such file or directory
# (PID file removed on clean shutdown)

# 9. Command-line options
$ ./mydaemon --help
Usage: mydaemon [OPTIONS]
Options:
  -c, --config FILE   Configuration file (default: /etc/mydaemon.conf)
  -p, --pidfile FILE  PID file location (default: /var/run/mydaemon.pid)
  -f, --foreground    Don't daemonize (for debugging)
  -d, --debug         Enable debug logging
  -h, --help          Show this help message

# 10. Foreground mode for debugging
$ ./mydaemon -f -d
[DEBUG] Starting in foreground mode
[DEBUG] Configuration loaded
[DEBUG] Entering main loop
^C
[DEBUG] Caught SIGINT, shutting down
[DEBUG] Cleanup complete

3.5 Real World Outcome

When your daemon is complete, you will have:

A production-ready daemon template that can serve as the base for any background service
Syslog entries showing proper daemon lifecycle management
PID file locking that prevents accidental multiple instances
Signal handling that enables live configuration reload
Clean shutdown behavior with proper resource cleanup

4. Solution Architecture

4.1 High-Level Design

Daemon Lifecycle

┌─────────────────────────────────────────────────────────────────────────────┐
│                           STARTUP PHASE                                      │
│                                                                             │
│  Parse Args → Load Config → Daemonize → Write PID → Setup Signals → Log    │
│       │            │            │            │             │           │    │
│       ▼            ▼            ▼            ▼             ▼           ▼    │
│  [Options]    [Config]     [Double     [Lock &      [Install     [Open    │
│               struct       Fork]       Write]       Handlers]    syslog]   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
                                      │
                                      ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           MAIN LOOP                                          │
│                                                                             │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │                                                                       │  │
│  │    while (!shutdown_requested) {                                      │  │
│  │        if (reload_requested) {                                        │  │
│  │            reload_config();                                           │  │
│  │            reload_requested = 0;                                      │  │
│  │        }                                                              │  │
│  │        if (status_requested) {                                        │  │
│  │            log_status();                                              │  │
│  │            status_requested = 0;                                      │  │
│  │        }                                                              │  │
│  │        do_work();                                                     │  │
│  │        sleep(interval);  // or select/poll with timeout              │  │
│  │    }                                                                  │  │
│  │                                                                       │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                                                             │
│  Signal Flags (volatile sig_atomic_t):                                      │
│    shutdown_requested ─────────────────────► Set by SIGTERM/SIGINT         │
│    reload_requested ───────────────────────► Set by SIGHUP                 │
│    status_requested ───────────────────────► Set by SIGUSR1                │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
                                      │
                                      ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           SHUTDOWN PHASE                                     │
│                                                                             │
│  Log Shutdown → Cleanup Resources → Remove PID → Close Syslog → Exit       │
│        │               │                │              │           │        │
│        ▼               ▼                ▼              ▼           ▼        │
│   [syslog()]     [close fds,      [unlink()      [closelog()]  [exit(0)]   │
│                  free memory]      flock()]                                 │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

4.2 Key Components

1. Configuration Module

Parse config file
Validate settings
Support reload

2. Daemonization Module

Double-fork implementation
Session and process group setup
File descriptor cleanup

3. PID File Module

Create and lock PID file
Write PID after daemonization
Cleanup on shutdown

4. Signal Handler Module

Install handlers with sigaction()
Use volatile sig_atomic_t flags
Async-signal-safe operations only

5. Logging Module

Wrapper around syslog
Priority level support
Foreground mode output to stderr

6. Main Loop

Check signal flags
Perform periodic work
Handle errors

4.3 Data Structures

// Configuration structure
typedef struct {
    char config_file[PATH_MAX];
    char pid_file[PATH_MAX];
    int work_interval;         // seconds between work cycles
    int log_level;             // LOG_DEBUG, LOG_INFO, etc.
    int foreground;            // don't daemonize if true
    // Add daemon-specific config here
} config_t;

// Daemon state
typedef struct {
    config_t config;
    int pid_fd;                // File descriptor for PID file (keep open for lock)
    time_t start_time;         // When daemon started
    unsigned long iterations;  // Number of main loop iterations
    int running;               // Main loop control
} daemon_state_t;

// Signal flags (must be volatile sig_atomic_t for async-signal-safety)
static volatile sig_atomic_t shutdown_requested = 0;
static volatile sig_atomic_t reload_requested = 0;
static volatile sig_atomic_t status_requested = 0;

4.4 Algorithm Overview

Daemonization Pseudocode:

daemonize():
    # First fork
    pid = fork()
    if pid > 0:
        exit(0)  # Parent exits, returns to shell
    if pid < 0:
        error("First fork failed")

    # Create new session
    if setsid() < 0:
        error("setsid failed")

    # Second fork (prevent acquiring controlling terminal)
    pid = fork()
    if pid > 0:
        exit(0)  # First child exits
    if pid < 0:
        error("Second fork failed")

    # Now we're the grandchild (the daemon)

    # Change to safe directory
    chdir("/")

    # Reset file creation mask
    umask(0)

    # Close standard file descriptors
    close(STDIN_FILENO)
    close(STDOUT_FILENO)
    close(STDERR_FILENO)

    # Redirect to /dev/null
    open("/dev/null", O_RDWR)  # stdin  (fd 0)
    dup(0)                      # stdout (fd 1)
    dup(0)                      # stderr (fd 2)

PID File Locking Pseudocode:

create_pid_file(path):
    # Open or create PID file
    fd = open(path, O_RDWR | O_CREAT, 0644)
    if fd < 0:
        error("Cannot open PID file")

    # Try to lock exclusively (non-blocking)
    if flock(fd, LOCK_EX | LOCK_NB) < 0:
        if errno == EWOULDBLOCK:
            # Another instance is running
            # Read its PID
            read(fd, buf, sizeof(buf))
            error("Already running with PID %s", buf)
        error("Cannot lock PID file")

    # Truncate and write our PID
    ftruncate(fd, 0)
    dprintf(fd, "%d\n", getpid())

    # Keep fd open - lock is held until close or exit
    return fd

5. Implementation Guide

5.1 Development Environment Setup

# Create project structure
$ mkdir mydaemon && cd mydaemon
$ touch mydaemon.c mydaemon.h Makefile mydaemon.conf

# Create Makefile
$ cat > Makefile << 'EOF'
CC = gcc
CFLAGS = -Wall -Wextra -Werror -g -O2 -D_POSIX_C_SOURCE=200809L
LDFLAGS =

all: mydaemon

mydaemon: mydaemon.c mydaemon.h
	$(CC) $(CFLAGS) -o $@ mydaemon.c $(LDFLAGS)

clean:
	rm -f mydaemon *.o

install: mydaemon
	install -m 755 mydaemon /usr/local/sbin/
	install -m 644 mydaemon.conf /etc/

test: mydaemon
	./test_daemon.sh

.PHONY: all clean install test
EOF

# Create sample config
$ cat > mydaemon.conf << 'EOF'
# mydaemon configuration file
work_interval = 60
log_level = info
# Add daemon-specific settings here
EOF

5.2 Project Structure

mydaemon/
├── mydaemon.c        # Main daemon implementation
├── mydaemon.h        # Header with structures and declarations
├── mydaemon.conf     # Sample configuration file
├── test_daemon.sh    # Test script
├── Makefile
└── README.md

5.3 The Core Question You’re Answering

“How do you create a process that runs in the background, survives terminal logout, and behaves properly as a system service?”

This is what every web server, database, and background service must do. The daemonization ritual exists because of how UNIX process groups and controlling terminals work.

Think about:

Why does the first fork allow setsid() to work?
Why would a session leader be able to acquire a controlling terminal?
What happens to file descriptors across fork()?
Why does the parent exit immediately after forking?

5.4 Concepts You Must Understand First

1. Sessions and Controlling Terminal

What is a session? How is it different from a process group?
What is setsid() and what does it do?
What is a controlling terminal and how is it assigned?
Book Reference: “APUE” Ch. 9.5-9.6

2. The Double-Fork Pattern

Why fork twice? What does each fork accomplish?
Why can a session leader acquire a controlling terminal?
What prevents the final daemon from getting a terminal?
Book Reference: “APUE” Ch. 13.3

3. File Descriptor Cleanup

Why close stdin/stdout/stderr?
What could happen if we don’t?
Why redirect to /dev/null instead of just closing?

4. Syslog

openlog(), syslog(), closelog()
What are facilities and priorities?
How does syslog work on different systems?
Book Reference: “APUE” Ch. 13.4

5.5 Questions to Guide Your Design

Single Instance:

Where should the PID file be stored? (/var/run? /run? configurable?)
What locking mechanism? (flock vs fcntl)
What if the system crashed and left a stale PID file?

Config Reload:

How do you safely reload config without stopping the daemon?
What if the new config is invalid?
What state can be changed at runtime?

Startup Dependencies:

What if the daemon needs network or database?
How to handle resources not available at startup?
Should the daemon retry or exit?

5.6 Thinking Exercise

Trace the Double-Fork

Why does the daemonization ritual require TWO forks?

Original Process (pid 100, pgid 100, sid 100)
    │
    │ fork()
    │
    ├───────────────────────────────────────────┐
    │                                           │
    │ Parent                                    │ Child (pid 101, pgid 100, sid 100)
    │ exit(0)                                   │
    │ (Returns control to shell)                │ setsid()
    │                                           │ (Now: pid 101, pgid 101, sid 101)
                                                │ (Child IS the SESSION LEADER)
                                                │
                                                │ fork() AGAIN
                                                │
                                    ┌───────────┴───────────┐
                                    │                       │
                              Child (101)              Grandchild (102)
                              exit(0)                  pgid 101, sid 101
                              (session leader          BUT NOT session leader!
                               is gone)
                                                       Cannot EVER acquire a
                                                       controlling terminal.

                                                       This is the daemon.

Step-by-step analysis:

1. Before first fork: Process 100 is in session 100, which was created
   by your login shell. It has a controlling terminal (your terminal).

2. After first fork: Child 101 still in session 100. Parent 100 exits.
   This orphans child 101, which gets adopted by init.

3. setsid() call: Child 101 becomes session leader of NEW session 101.
   No controlling terminal (new sessions start without one).
   BUT: As session leader, 101 COULD acquire a terminal by opening one.

4. Second fork: Child 101 exits. Grandchild 102 is:
   - In session 101
   - NOT the session leader (101 was, but it exited)
   - Cannot ever acquire a controlling terminal
   - This is crucial for daemons!

Why the second fork matters:
If you open a terminal device (like /dev/tty), POSIX says you become
the controlling process of that terminal IF:
  1. You don't already have a controlling terminal, AND
  2. You are a session leader

The second fork ensures condition 2 is false, so the daemon can
never accidentally get a terminal (which would make it receive
SIGHUP when that terminal disconnects).

5.7 Hints in Layers

Hint 1: Basic Structure

Parse command line
Read config file
Daemonize (or skip if --foreground)
Write PID file
Set up signal handlers
Open syslog
Main loop
Cleanup on exit

Hint 2: Daemonize Function

void daemonize(void) {
    pid_t pid;

    // First fork
    pid = fork();
    if (pid < 0) exit(EXIT_FAILURE);
    if (pid > 0) exit(EXIT_SUCCESS);  // Parent exits

    // Create new session
    if (setsid() < 0) exit(EXIT_FAILURE);

    // Second fork
    pid = fork();
    if (pid < 0) exit(EXIT_FAILURE);
    if (pid > 0) exit(EXIT_SUCCESS);  // First child exits

    // Change directory, reset umask
    chdir("/");
    umask(0);

    // Close and redirect file descriptors
    close(STDIN_FILENO);
    close(STDOUT_FILENO);
    close(STDERR_FILENO);
    open("/dev/null", O_RDWR);  // stdin
    dup(0);  // stdout
    dup(0);  // stderr
}

Hint 3: PID File Locking

int create_pid_file(const char *path) {
    int fd = open(path, O_RDWR | O_CREAT, 0644);
    if (fd < 0) return -1;

    if (flock(fd, LOCK_EX | LOCK_NB) < 0) {
        if (errno == EWOULDBLOCK) {
            char buf[32];
            ssize_t n = read(fd, buf, sizeof(buf) - 1);
            if (n > 0) {
                buf[n] = '\0';
                fprintf(stderr, "Already running (PID %s)\n", buf);
            }
        }
        close(fd);
        return -1;
    }

    ftruncate(fd, 0);
    dprintf(fd, "%d\n", getpid());
    return fd;  // Keep open to hold lock
}

Hint 4: Signal Handling

static volatile sig_atomic_t shutdown_flag = 0;
static volatile sig_atomic_t reload_flag = 0;

void signal_handler(int signo) {
    if (signo == SIGTERM || signo == SIGINT) {
        shutdown_flag = 1;
    } else if (signo == SIGHUP) {
        reload_flag = 1;
    }
}

void setup_signals(void) {
    struct sigaction sa;
    sa.sa_handler = signal_handler;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART;

    sigaction(SIGTERM, &sa, NULL);
    sigaction(SIGINT, &sa, NULL);
    sigaction(SIGHUP, &sa, NULL);
}

5.8 The Interview Questions They’ll Ask

“Why do daemons fork twice?”
- First fork: allows setsid() (can’t call on process group leader)
- setsid(): creates new session, no controlling terminal
- Second fork: ensures daemon is NOT session leader (can’t acquire terminal)
“What happens to a process when the controlling terminal closes?”
- Kernel sends SIGHUP to session leader
- Session leader (shell) typically sends SIGHUP to foreground process group
- Processes not in foreground group may not receive signal
- nohup makes process immune to SIGHUP
“How do you prevent multiple instances of a daemon?”
- PID file with exclusive lock (flock or fcntl)
- Lock is automatically released when process exits
- Check if lock succeeds; if not, another instance is running
“What’s the difference between SIGHUP and SIGTERM for daemons?”
- SIGTERM: request graceful shutdown
- SIGHUP: traditionally means “reload configuration”
- SIGHUP originates from terminal hangup, repurposed for daemons
“How does systemd change daemon startup?”
- systemd can daemonize for you (Type=simple)
- Daemon runs in foreground, systemd handles backgrounding
- Logging goes to journal instead of syslog
- Socket activation, dependency management, etc.

5.9 Books That Will Help

Topic	Book	Chapter
Daemon processes	“APUE” by Stevens & Rago	Ch. 13
Process relationships	“APUE” by Stevens & Rago	Ch. 9
Sessions and terminals	“APUE” by Stevens & Rago	Ch. 9
Modern daemons	“The Linux Programming Interface” by Kerrisk	Ch. 37
Syslog	“APUE” by Stevens & Rago	Ch. 13.4

5.10 Implementation Phases

Phase 1: Basic Daemonization (3-4 hours)

Implement double-fork
Add setsid() and directory change
Close and redirect file descriptors
Test: verify process detaches from terminal

Phase 2: PID File Management (2-3 hours)

Create PID file with locking
Write PID after daemonization
Detect already-running instance
Test: run two instances, second should fail

Phase 3: Signal Handling (2-3 hours)

Install handlers with sigaction()
Implement SIGTERM shutdown
Implement SIGHUP reload (placeholder)
Test: send signals, verify response

Phase 4: Syslog Integration (1-2 hours)

Replace printf with syslog
Proper facility and priority
Log startup, shutdown, signals
Test: verify messages in /var/log/syslog

Phase 5: Configuration (3-4 hours)

Parse simple config file
Validate configuration
Implement SIGHUP reload
Test: change config, reload

Phase 6: Main Loop (2-3 hours)

Implement work interval
Check signal flags
Graceful shutdown
Status reporting via SIGUSR1

Phase 7: Polish (2-3 hours)

Command-line argument parsing
Foreground mode for debugging
Error messages and exit codes
Man page or README documentation

5.11 Key Implementation Decisions

Decision 1: Foreground Mode Add a -f/–foreground option that skips daemonization. Essential for:

Debugging (see output directly)
Running under systemd Type=simple
Container environments

Decision 2: Where to Write PID File

Traditional: /var/run/mydaemon.pid
Modern Linux: /run/mydaemon.pid (/var/run is often a symlink)
Non-root: /tmp or ~/.local/run/
Make configurable via command-line

Decision 3: Signal-Safe Logging In signal handlers, you cannot call syslog() (not async-signal-safe). Options:

Set a flag, log in main loop
Use write() to a dedicated log fd
Use signalfd() instead of handlers

Decision 4: Config File Format Options:

Simple key=value (easy to parse, easy to read)
INI file with sections (more structure)
JSON/YAML (requires library)
Recommendation: Start with key=value

6. Testing Strategy

6.1 Unit Tests

Test individual components:

# Test daemonization
test_daemonize() {
    ./mydaemon -f &  # Run in foreground first
    PID=$!
    sleep 1
    kill -0 $PID 2>/dev/null && echo "PASS: Process running"
    kill $PID
}

# Test PID file creation
test_pid_file() {
    ./mydaemon
    sleep 1
    PID=$(cat /var/run/mydaemon.pid)
    kill -0 $PID 2>/dev/null && echo "PASS: PID file correct"
    kill $PID
}

6.2 Integration Tests

Test complete scenarios:

#!/bin/bash
# test_daemon.sh

DAEMON=./mydaemon
PIDFILE=/tmp/test_daemon.pid
CONFIG=/tmp/test_daemon.conf

# Setup
echo "work_interval = 5" > $CONFIG

# Test 1: Basic startup and shutdown
echo "Test 1: Basic startup/shutdown"
$DAEMON -p $PIDFILE -c $CONFIG
sleep 2
if [ -f $PIDFILE ]; then
    PID=$(cat $PIDFILE)
    if kill -0 $PID 2>/dev/null; then
        echo "  PASS: Daemon started (PID $PID)"
    fi
    kill $PID
    sleep 1
    if [ ! -f $PIDFILE ]; then
        echo "  PASS: PID file removed on shutdown"
    fi
fi

# Test 2: Single instance
echo "Test 2: Single instance enforcement"
$DAEMON -p $PIDFILE -c $CONFIG
sleep 1
if ! $DAEMON -p $PIDFILE -c $CONFIG 2>&1 | grep -q "already running"; then
    echo "  FAIL: Second instance should fail"
else
    echo "  PASS: Second instance rejected"
fi
kill $(cat $PIDFILE) 2>/dev/null

# Test 3: SIGHUP reload
echo "Test 3: Configuration reload"
$DAEMON -p $PIDFILE -c $CONFIG
sleep 1
PID=$(cat $PIDFILE)
echo "work_interval = 10" > $CONFIG
kill -HUP $PID
sleep 1
# Check syslog for reload message
if grep -q "reloading configuration" /var/log/syslog; then
    echo "  PASS: Config reload logged"
fi
kill $PID

# Cleanup
rm -f $PIDFILE $CONFIG

6.3 Edge Cases to Test

Stale PID file: Kill daemon with SIGKILL (leaves PID file), then start again
Permission denied: Run as non-root, try to write to /var/run
Config file missing: Start without config file
Invalid config: Malformed config file
Signal during startup: Send SIGTERM before fully initialized
Rapid signals: Send multiple SIGHUP rapidly
Foreground mode: Verify Ctrl-C works in foreground
Long shutdown: Task running during SIGTERM

6.4 Verification Commands

# Check process details
$ ps aux | grep mydaemon
$ ps -eo pid,ppid,pgid,sid,tty,comm | grep mydaemon
#   PID   PPID  PGID   SID TTY      COMMAND
# 12346     1  12346 12346 ?        mydaemon
# Note: PPID=1 (init), SID=PGID=PID (session leader), TTY=? (no terminal)

# Check open file descriptors
$ ls -la /proc/$(cat /var/run/mydaemon.pid)/fd
# Should show:
#   0 -> /dev/null
#   1 -> /dev/null
#   2 -> /dev/null
#   3 -> /var/run/mydaemon.pid (the lock)

# Check session and process group
$ ps -o pid,ppid,sid,pgid,tty,stat,comm -p $(cat /var/run/mydaemon.pid)

# Trace system calls
$ sudo strace -p $(cat /var/run/mydaemon.pid)

# Check syslog
$ journalctl -u mydaemon  # If using systemd service
$ tail -f /var/log/syslog | grep mydaemon  # Traditional syslog

# Memory check
$ valgrind ./mydaemon -f  # Foreground mode for valgrind

7. Common Pitfalls & Debugging

Problem 1: “Daemon exits immediately”

Symptom: Daemon appears to start but process immediately disappears

Why:

No main loop (daemon falls through to exit)
Error during initialization that causes early exit
Signal delivered before main loop starts

Fix:

// Add infinite main loop
while (!shutdown_requested) {
    if (reload_requested) {
        reload_config();
        reload_requested = 0;
    }
    do_work();
    sleep(config.interval);
}

Debug:

# Run in foreground to see errors
$ ./mydaemon -f -d

# Check syslog for errors
$ tail /var/log/syslog | grep mydaemon

Problem 2: “Can’t see any log output”

Symptom: No messages in /var/log/syslog

Why:

Didn’t call openlog() or forgot to call syslog()
Wrong facility (logs going elsewhere)
syslog daemon not running
Foreground mode not redirecting to stderr

Fix:

// At startup
openlog("mydaemon", LOG_PID | LOG_NDELAY, LOG_DAEMON);

// Throughout code
syslog(LOG_INFO, "Daemon started");
syslog(LOG_ERR, "Error: %s", strerror(errno));

// In foreground mode, also write to stderr
if (foreground) {
    fprintf(stderr, "[INFO] Daemon started\n");
}

Debug:

# Check where LOG_DAEMON goes
$ grep daemon /etc/rsyslog.conf

# Try different facility
$ logger -p daemon.info "test message"
$ grep "test message" /var/log/*

Problem 3: “Zombie children”

Symptom: Defunct processes accumulate

Why: Daemon forks child processes but doesn’t wait() for them

Fix:

// Option 1: Handle SIGCHLD
void sigchld_handler(int sig) {
    (void)sig;
    while (waitpid(-1, NULL, WNOHANG) > 0);
}

// Option 2: Double-fork for each child (orphan it to init)
pid_t pid = fork();
if (pid == 0) {
    pid_t pid2 = fork();
    if (pid2 == 0) {
        // Grandchild does the work
        execl(...);
    }
    exit(0);  // Child exits immediately
}
waitpid(pid, NULL, 0);  // Wait for child (not grandchild)

Problem 4: “Files created with wrong permissions”

Symptom: Log files, PID files have unexpected permissions

Why: Inherited umask from parent shell

Fix:

// Set umask early in daemonization
umask(0);  // Clear umask, set permissions explicitly in open()

// Or set to desired default
umask(022);  // Files: 644, Dirs: 755

Problem 5: “Daemon doesn’t respond to signals”

Symptom: kill -HUP doesn’t trigger reload

Why:

Signal handler not installed
Main loop not checking flags
Signal blocked

Fix:

// Check signal mask
sigset_t current;
sigprocmask(SIG_BLOCK, NULL, &current);
if (sigismember(&current, SIGHUP)) {
    syslog(LOG_WARNING, "SIGHUP is blocked!");
}

// Verify handler installation
struct sigaction sa;
sigaction(SIGHUP, NULL, &sa);
if (sa.sa_handler == SIG_DFL || sa.sa_handler == SIG_IGN) {
    syslog(LOG_WARNING, "SIGHUP handler not installed!");
}

8. Extensions & Challenges

8.1 Easy Extensions

Privilege dropping: Start as root, drop to unprivileged user
Watchdog: Periodically touch a file to prove liveness
Graceful restart: Exec new binary without losing state
Log rotation: Respond to SIGUSR1 by reopening log files

8.2 Advanced Challenges

systemd integration: Add sd_notify() for Type=notify services
Socket activation: Accept sockets passed by systemd
Credential management: Securely load secrets at startup
Health check endpoint: HTTP server for health probes
Cluster coordination: Use PID file on shared storage

8.3 Research Topics

Supervision trees: How do Erlang/OTP supervisors work?
Containerized daemons: How does daemonization work in containers?
launchd on macOS: How does Apple’s approach differ?
Modern init systems: Compare systemd, runit, s6, OpenRC

9. Real-World Connections

9.1 Production Systems Using This

nginx: Double-fork daemonization, master/worker model
Apache httpd: Traditional daemon with prefork/worker MPMs
PostgreSQL: Postmaster daemon with forked backends
sshd: System daemon with per-connection forks
cron: Classic daemon that runs scheduled tasks
rsyslogd: The logging daemon itself is a daemon

9.2 How the Pros Do It

nginx:

Master process reads config, manages workers
Workers handle connections
SIGHUP triggers graceful reload (new workers, old finish)
SIGUSR1 reopens log files

systemd (when used):

Type=simple: systemd does the daemonization
Type=forking: daemon does traditional double-fork
Type=notify: daemon signals ready with sd_notify()

Container best practices:

Run in foreground (no daemonization)
Log to stdout/stderr (captured by container runtime)
Handle SIGTERM for graceful shutdown

9.3 Reading the Source

daemon(3) library function: BSD/Linux helper function
- https://man7.org/linux/man-pages/man3/daemon.3.html
- Simpler than manual double-fork
nginx daemon code: src/os/unix/ngx_daemon.c
- https://github.com/nginx/nginx
systemd documentation:
- https://www.freedesktop.org/software/systemd/man/daemon.html

10. Resources

10.1 Man Pages

$ man daemon          # daemon(3) library function
$ man setsid          # Create new session
$ man fork            # Process creation
$ man syslog          # Logging
$ man flock           # File locking
$ man signal          # Signal handling basics
$ man sigaction       # Reliable signal handling

10.2 Online Resources

systemd for Developers: https://0pointer.de/blog/projects/socket-activation.html
New-Style Daemons: https://www.freedesktop.org/software/systemd/man/daemon.html
The Double Fork: https://stackoverflow.com/questions/881388

10.3 Book Chapters

Book	Chapter	Topic
“APUE” by Stevens	Ch. 13	Daemon Processes
“APUE” by Stevens	Ch. 9	Process Relationships
“APUE” by Stevens	Ch. 10	Signals
“TLPI” by Kerrisk	Ch. 37	Daemons
“TLPI” by Kerrisk	Ch. 34	Process Groups and Sessions

11. Self-Assessment Checklist

Before considering this project complete, verify:

12. Submission / Completion Criteria

Your project is complete when:

Proper daemonization: Verified by ps output showing PPID=1, no TTY
Single instance: Second start attempt fails with helpful message
Signal handling: SIGTERM, SIGHUP, SIGUSR1 all work correctly
Syslog integration: All messages appear in system log
Clean shutdown: PID file removed, resources freed
Foreground mode: Works correctly for debugging

Deliverables:

mydaemon.c - Complete implementation
mydaemon.h - Header file
mydaemon.conf - Sample configuration
Makefile - Build and install targets
test_daemon.sh - Test script

Verification commands that must succeed:

# Start daemon
$ ./mydaemon
$ ps -eo pid,ppid,sid,tty,comm | grep mydaemon
12346  1  12346  ?  mydaemon

# Verify PID file
$ cat /var/run/mydaemon.pid
12346

# Verify single instance
$ ./mydaemon
Already running (PID 12346)

# Reload config
$ kill -HUP 12346
$ tail /var/log/syslog | grep "reload"
... mydaemon[12346]: Configuration reloaded

# Clean shutdown
$ kill 12346
$ ls /var/run/mydaemon.pid
ls: cannot access '/var/run/mydaemon.pid': No such file or directory