Project 4: Service Management and Logging

Install and manage a service the FreeBSD way, and read logs like an operator.

Quick Reference

Attribute	Value
Difficulty	Level 2
Time Estimate	Weekend
Main Programming Language	Shell (sh) (Alternatives: csh, Python)
Alternative Programming Languages	sh, csh, Python
Coolness Level	Level 2
Business Potential	Level 1
Prerequisites	Projects 1-3 complete
Key Topics	rc.d, service, syslog, log files

1. Learning Objectives

By completing this project, you will:

Install a lightweight service and enable it at boot.
Manage service lifecycle with service and rc scripts.
Locate and interpret service logs in FreeBSD.
Perform a failure/recovery drill using logs.

2. All Theory Needed (Per-Concept Breakdown)

Concept 1: FreeBSD Service Lifecycle (rc.d + service)

Fundamentals FreeBSD manages services through the rc system, which uses rc scripts to start and stop daemons. The service command is the standard interface for lifecycle actions like start, stop, restart, status, and onestart. Services are enabled for boot by setting variables in rc.conf, not by enabling units. This creates a clear separation between “installed,” “enabled,” and “running.” For service management, you need to understand how rc scripts are organized, how they discover configuration, and how they write PID files that status checks rely on.

In practice, write a short checklist for service lifecycle management and confirm it after each reboot. This keeps the concept concrete and prevents accidental drift between sessions.

Deep Dive into the concept Each rc script is a shell script that follows a common framework. It defines a name, optional dependencies, and commands for start/stop/status. At boot, the rc system runs these scripts in a dependency-respecting order. For third-party services, the scripts are installed under /usr/local/etc/rc.d, while base services live under /etc/rc.d. The service command resolves the location and invokes the proper action. This abstraction is important because you can treat all services the same even though their scripts are in different locations.

When you enable a service in rc.conf, the rc system will call that script at boot. But you can also start a service manually with service nginx onestart or service nginx start. The onestart option ignores rc.conf, which is useful for testing. The status command typically checks for a PID file; if the daemon does not write a PID file, status can be misleading. Understanding this subtlety helps when debugging “service is running” vs “service started but exited.”

Service configuration is typically done in files under /usr/local/etc, and the rc script often reads variables from rc.conf such as nginx_enable. The script can also accept options like nginx_flags to pass extra arguments. This means service lifecycle and service configuration are separate concerns. In practice, you manage enablement in rc.conf, configuration in the service’s config file, and runtime state with service. This triad forms the operational model for FreeBSD daemons.

Finally, rc is designed to be transparent. You can open the rc script to see exactly how a service starts, which is an educational advantage over opaque service managers. For this project, you will pick a small service (like nginx or cron for testing), enable it, start it, confirm status, and then simulate a failure to see how logs and rc work together.

Operationally, service lifecycle management is easiest to keep stable when you treat it as a small contract between configuration, tooling, and observable outputs. Write down the exact files that own the state and the commands that reveal the current truth. Then verify the contract at three points: immediately after you make the change, after a reboot, and after a deliberate disturbance such as restarting services or reloading modules. FreeBSD rewards this discipline because it rarely hides state; if something changes, it is usually in a file you control. Make a habit of collecting a before-and-after snapshot of commands and outputs so you can explain which change caused which effect.

At scale, service lifecycle management is also about failure containment. Identify what must remain available when something breaks and design a safe escape hatch. For example, keep console access for firewall changes, keep a previous boot environment for upgrades, or keep a dataset snapshot before risky edits. The same pattern applies across domains: define invariants, define the rollback path, and then only proceed when you can trigger that rollback quickly. Finally, test the failure path while the system is healthy; you learn more from a controlled rollback than from an emergency. This perspective turns the lab exercise into an operational capability you can trust on production systems.

How this fits on projects

This concept is used in Section 3.2, Section 5.4, and Section 5.10.
It is a prerequisite for P08 Firewall Basics and P09 Jail Lab.

Definitions & key terms

rc.d -> Directory containing rc scripts.
service -> Command to control services.
PID file -> File containing the process ID of a daemon.
onestart -> Starts a service without enabling it at boot.

Mental model diagram

rc.conf -> rc.d script -> daemon process -> PID file

How it works (step-by-step, with invariants and failure modes)

Install service and rc script.
Enable service in rc.conf.
Start service and create PID file.
Use service status to verify.

Invariants:

rc.conf entry must match script name.
rc script must exist and be executable.
PID file path must be correct.

Failure modes:

Missing rc.conf entry -> no boot start.
Stale PID file -> false “running” status.
Misconfigured service flags -> crash loop.

Minimal concrete example

sysrc nginx_enable="YES"
service nginx onestart
service nginx status

Common misconceptions

“service start implies enabled at boot.” -> It does not.
“status is always accurate.” -> It may be wrong if PID files are stale.

Check-your-understanding questions

What does onestart do differently from start?
Where do third-party rc scripts live?
Why can status be misleading?

Check-your-understanding answers

It starts without enabling rc.conf.
/usr/local/etc/rc.d.
If PID files are missing or stale.

Real-world applications

Managing services consistently across FreeBSD hosts.
Debugging service startup failures after upgrades.

Where you’ll apply it

Section 3.2 Functional Requirements
Section 5.10 Phase 1
Also used in: P08 Firewall Basics, P09 Jail Lab

References

“Absolute FreeBSD, 3rd Edition” (Ch. 14, 20)
FreeBSD Handbook: The rc System

Key insights Service lifecycle is explicit: install -> enable -> start -> verify.

Summary The rc system provides transparent, script-based service management.

Homework/Exercises to practice the concept

Start a service with onestart and confirm it does not start after reboot.
Identify the rc script for a package you installed.
Add a custom flag in rc.conf and observe the effect.

Solutions to the homework/exercises

onestart does not modify rc.conf, so it won’t persist.
Check /usr/local/etc/rc.d for package scripts.
Flags modify how the daemon launches.

Concept 2: System Logging and Log Interpretation (syslog)

Fundamentals FreeBSD uses syslog to collect system and service logs. Many logs are written to /var/log, including /var/log/messages and /var/log/auth.log. Services may also write their own logs under /var/log or /usr/local/var/log depending on configuration. Understanding which log files apply to which service is critical to debugging. The syslog daemon reads /etc/syslog.conf to decide where messages go. For this project, you must know how to tail logs, identify service-specific entries, and use logs to diagnose startup failures.

In practice, write a short checklist for system logging and log interpretation and confirm it after each reboot. This keeps the concept concrete and prevents accidental drift between sessions.

Deep Dive into the concept Syslog is a message routing system. Processes send log messages with a facility (like daemon, auth, kern) and a severity level (like info, warning, err). The syslog daemon reads /etc/syslog.conf and routes messages to files, remote hosts, or consoles based on facility and severity. For example, authentication messages go to /var/log/auth.log, while general daemon messages often land in /var/log/messages. Understanding this mapping is essential because it prevents you from “tailing the wrong file” when troubleshooting.

In FreeBSD, many services log through syslog by default, but some third-party services log to their own files. For instance, nginx typically logs to /var/log/nginx or /usr/local/var/log/nginx depending on configuration. This means you must check both syslog and service-specific logs. The debugging workflow is: verify service status, attempt a connection, then check logs for the exact timestamp. The combination of status and log evidence helps you distinguish between startup failures (service never launched) and runtime failures (service crashes or rejects connections).

Log hygiene also matters. Large log files can hide important signals. FreeBSD uses newsyslog to rotate logs based on /etc/newsyslog.conf. If logs grow without rotation, they can consume disk space and make troubleshooting harder. For a basic lab, you do not need to customize rotation, but you should at least know that it exists. When you later run long-lived services or jails, understanding rotation keeps your VM stable.

Most importantly, logs are your truth when you are unsure. The rc system can tell you whether a service started, but logs tell you whether it started correctly. When you intentionally break a configuration in this project, you will practice reading logs to identify the error and then restoring the service. This is the operational mindset you need for real FreeBSD work.

Operationally, system logging and log interpretation is easiest to keep stable when you treat it as a small contract between configuration, tooling, and observable outputs. Write down the exact files that own the state and the commands that reveal the current truth. Then verify the contract at three points: immediately after you make the change, after a reboot, and after a deliberate disturbance such as restarting services or reloading modules. FreeBSD rewards this discipline because it rarely hides state; if something changes, it is usually in a file you control. Make a habit of collecting a before-and-after snapshot of commands and outputs so you can explain which change caused which effect.

At scale, system logging and log interpretation is also about failure containment. Identify what must remain available when something breaks and design a safe escape hatch. For example, keep console access for firewall changes, keep a previous boot environment for upgrades, or keep a dataset snapshot before risky edits. The same pattern applies across domains: define invariants, define the rollback path, and then only proceed when you can trigger that rollback quickly. Finally, test the failure path while the system is healthy; you learn more from a controlled rollback than from an emergency. This perspective turns the lab exercise into an operational capability you can trust on production systems.

How this fits on projects

This concept is used in Section 3.2, Section 5.6, and Section 7.
It is central to P10 Update Runbook.

Definitions & key terms

syslog -> Logging daemon and protocol for message routing.
facility -> Category of log message (auth, daemon, kern).
severity -> Importance level (info, warning, err).
newsyslog -> Log rotation utility.

Mental model diagram

Service -> syslog -> /var/log/* -> newsyslog rotation

How it works (step-by-step, with invariants and failure modes)

Service emits log messages with facility and severity.
syslogd routes messages based on syslog.conf.
Logs accumulate in /var/log or service-specific locations.
newsyslog rotates logs on schedule.

Invariants:

syslogd must be running.
syslog.conf defines routing.

Failure modes:

syslogd stopped -> no system logs.
Wrong log file tailed -> missed errors.
Log rotation misconfigured -> disk fills.

Minimal concrete example

tail -n 50 /var/log/messages
newsyslog -n

Common misconceptions

“All logs are in /var/log/messages.” -> Many services use other files.
“If status says running, everything is fine.” -> Logs may show errors.

Check-your-understanding questions

What file contains authentication logs?
What does newsyslog do?
Why might a service log outside /var/log?

Check-your-understanding answers

/var/log/auth.log.
Rotates and archives log files.
Service-specific configuration may direct logs elsewhere.

Real-world applications

Incident response using logs to identify failures.
Compliance and auditing of access events.

Where you’ll apply it

Section 5.6 Thinking Exercise
Section 7 Common Pitfalls & Debugging
Also used in: P10 Update Runbook

References

“Absolute FreeBSD, 3rd Edition” (Ch. 21)
FreeBSD Handbook: Logging and newsyslog

Key insights Logs are the authoritative record of what actually happened.

Summary Understanding syslog and log locations turns failures into solvable problems.

Homework/Exercises to practice the concept

Generate a failed login and find it in auth.log.
Start a service and find its startup log entry.
Run newsyslog in dry-run mode to see rotation targets.

Solutions to the homework/exercises

auth.log records the failed attempt with timestamp and user.
/var/log/messages or service log shows the event.
newsyslog prints which logs would rotate.

3. Project Specification

3.1 What You Will Build

A FreeBSD service (e.g., nginx or a simple web server) that starts at boot and writes logs you can interpret. You will perform a failure and recovery drill using log evidence.

3.2 Functional Requirements

Service installed: package installed successfully.
Service enabled: rc.conf entry created.
Service running: service nginx status confirms process.
Logs located: identify primary log files.
Failure drill completed: break config, diagnose via logs, recover.

3.3 Non-Functional Requirements

Performance: Service starts in under 10 seconds.
Reliability: Service survives a reboot.
Usability: Logs are documented with paths and meaning.

3.4 Example Usage / Output

$ service nginx status
nginx is running as pid 2345.

$ tail -n 2 /var/log/nginx/error.log
2025/01/07 12:00:00 [notice] 2345#0: using the "kqueue" event method

3.5 Data Formats / Schemas / Protocols

rc.conf
```
nginx_enable="YES"
```

Log excerpt

2025/01/07 12:00:00 [notice] nginx started

3.6 Edge Cases

Service starts but ports are blocked by firewall.
Log file path differs from documentation.
Stale PID file causes incorrect status.

3.7 Real World Outcome

A manageable service with logs that allow you to explain any startup or runtime failure.

3.7.1 How to Run (Copy/Paste)

pkg install -y nginx
sysrc nginx_enable="YES"
service nginx start
service nginx status
tail -n 20 /var/log/nginx/error.log

3.7.2 Golden Path Demo (Deterministic)

Use a fixed service name (nginx).
Use TZ=UTC to standardize log timestamps.

3.7.3 If CLI: provide an exact terminal transcript

$ TZ=UTC service nginx status
nginx is running as pid 2345.

$ TZ=UTC tail -n 1 /var/log/nginx/error.log
2025/01/07 12:00:00 [notice] 2345#0: using the "kqueue" event method

$ echo $?
0

Failure demo (deterministic)

$ TZ=UTC mv /usr/local/etc/nginx/nginx.conf /usr/local/etc/nginx/nginx.conf.bak
$ TZ=UTC service nginx restart
Performing sanity check on nginx configuration:
nginx: [emerg] open() "/usr/local/etc/nginx/nginx.conf" failed (2: No such file or directory)

$ echo $?
1

4. Solution Architecture

4.1 High-Level Design

pkg -> rc.conf enable -> service lifecycle -> logs -> recovery

4.2 Key Components

Component	Responsibility	Key Decisions
Service package	Provides daemon	nginx or similar
rc.conf	Boot enablement	sysrc usage
Logs	Observability	/var/log vs /usr/local/var/log

4.3 Data Structures (No Full Code)

ServiceRecord
- name: "nginx"
- enabled: true
- config_path: "/usr/local/etc/nginx/nginx.conf"
- log_path: "/var/log/nginx/error.log"

4.4 Algorithm Overview

Key Algorithm: Service Recovery Drill

Break config intentionally.
Restart service and observe failure.
Read logs for exact error.
Restore config and verify running state.

Complexity Analysis:

Time: O(restart + log check)
Space: O(log size)

5. Implementation Guide

5.1 Development Environment Setup

pkg update

5.2 Project Structure

service-lab/
+-- rc.conf.snippet
+-- logs/
| +-- nginx-error.log
+-- recovery-notes.md

5.3 The Core Question You’re Answering

“How do I operate services the FreeBSD way?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

rc system and service lifecycle
syslog and log interpretation

5.5 Questions to Guide Your Design

Which service is small and predictable for testing?
Where will that service log errors?
What does a healthy status check look like?

5.6 Thinking Exercise

Failure Drill

Stop the service unexpectedly and recover it using logs.

5.7 The Interview Questions They’ll Ask

How do you enable nginx at boot on FreeBSD?
What does service onestart do?
Where is rc.conf located?
How do you check logs?
What is different from systemd?

5.8 Hints in Layers

Hint 1: Start with a simple service Choose nginx or a tiny HTTP server.

Hint 2: Enable with sysrc Avoid manual edits to rc.conf.

Hint 3: Follow logs Tail the error log after each restart.

Hint 4: Keep notes Write down the exact error and fix.

5.9 Books That Will Help

Topic	Book	Chapter
rc.conf	“Absolute FreeBSD, 3rd Edition”	Ch. 14, 20
Logging	“Absolute FreeBSD, 3rd Edition”	Ch. 21

5.10 Implementation Phases

Phase 1: Install and Enable (2-3 hours)

Goals: Install service and enable at boot. Tasks:

Install package.
Enable in rc.conf. Checkpoint: service reports running.

Phase 2: Observe Logs (2-3 hours)

Goals: Identify log files. Tasks:

Locate error and access logs.
Capture log entries for startup. Checkpoint: log paths documented.

Phase 3: Failure Drill (2-3 hours)

Goals: Break and recover service. Tasks:

Introduce config error.
Recover using logs. Checkpoint: service running again, error understood.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Service choice	nginx, lighttpd	nginx	Common and well-documented
Log source	syslog, service logs	service logs + syslog	Most complete visibility
Enable method	sysrc, manual	sysrc	Avoids syntax mistakes

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Service Tests	Verify running	`service nginx status`
Log Tests	Validate output	`tail /var/log/nginx/error.log`
Recovery Tests	Restore from failure	restart after fixing config

6.2 Critical Test Cases

Service running: status reports running after reboot.
Log presence: logs show startup message.
Recovery: service restarts after fixing config.

6.3 Test Data

Service name: nginx

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Service not enabled	Stops after reboot	Add rc.conf entry
Wrong log path	No errors visible	Check service docs
Stale PID file	Status lies	Remove PID and restart

7.2 Debugging Strategies

Check status then logs: status tells you if it started, logs tell you why it failed.
Use onestart for testing: avoid committing config too early.

7.3 Performance Traps

Excessive logging without rotation can fill disk.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a simple HTML page and verify it serves.
Document exact log paths and meaning.

8.2 Intermediate Extensions

Configure custom log format.
Use newsyslog to rotate service logs.

8.3 Advanced Extensions

Build a custom rc script for a toy service.
Centralize logs to a remote syslog server.

9. Real-World Connections

9.1 Industry Applications

Web service operations: nginx management on FreeBSD.
Incident response: log-driven debugging.

nginx: web server and reverse proxy.
syslogd: core logging daemon.

9.3 Interview Relevance

Service management without systemd.
Log analysis and operational recovery.

10. Resources

10.1 Essential Reading

“Absolute FreeBSD, 3rd Edition” by Michael W. Lucas - Ch. 14, 20-21
FreeBSD Handbook - The rc System and Logging

10.2 Video Resources

“Managing Services in FreeBSD” - community tutorials

10.3 Tools & Documentation

service(8): service lifecycle manual
syslogd(8): logging daemon manual

P03 Package Workflow - install services safely.
P08 Firewall Basics - control service exposure.

11. Self-Assessment Checklist

11.1 Understanding

I can explain the rc service model.
I can locate and interpret service logs.
I can perform a recovery drill.

11.2 Implementation

Service installed and enabled.
Logs documented.
Failure drill completed.

11.3 Growth

I can automate service enablement.
I can explain how logs guide debugging.
I can teach this workflow to others.

12. Submission / Completion Criteria

Minimum Viable Completion:

Service installed and running.
Enabled at boot.
Logs located and documented.

Full Completion:

Failure drill completed with recovery notes.
Service confirmed running after reboot.

Excellence (Going Above & Beyond):

Custom rc script created for a toy service.
Log rotation configured and validated.