Project 1: Headless Bring-Up and Provisioning Pipeline

Build a repeatable, secure, headless imaging and first-boot pipeline that takes a blank microSD and produces a reachable, locked-down Pi Zero 2 W every time.

Quick Reference

Attribute Value
Difficulty Beginner
Time Estimate 1–2 weekends
Main Programming Language Bash (Alternatives: Python, Go, Rust)
Alternative Programming Languages Python, Go, Rust
Coolness Level High
Business Potential Medium
Prerequisites Linux CLI basics, SSH basics, SD card imaging
Key Topics Boot partition, Wi-Fi provisioning, SSH hardening, first-boot verification

1. Learning Objectives

By completing this project, you will:

  1. Build a deterministic, scriptable imaging pipeline for Raspberry Pi OS.
  2. Provision Wi-Fi and SSH securely without physical access.
  3. Verify boot success using repeatable, measurable checks.
  4. Design a recovery path for failed provisioning.
  5. Explain the Pi boot chain and where your configuration changes take effect.

2. All Theory Needed (Per-Concept Breakdown)

Concept 1: Raspberry Pi Boot Partition and Early Boot Configuration

Fundamentals

The Raspberry Pi Zero 2 W boots from a microSD card that contains a small FAT boot partition and a larger root filesystem partition. The boot partition is read by the GPU firmware before the ARM CPU even starts, so configuration files placed here can change hardware behavior long before Linux exists. This means boot configuration is not just a runtime tweak; it is part of the boot chain. Understanding which files live on the boot partition, the order they are read, and how errors surface is the foundation for headless provisioning. If you modify the wrong file or introduce a syntax error, the board may fail to boot or boot with missing peripherals. Because the boot partition is plain FAT, you can edit it from any computer without special tools, making it the primary injection point for headless provisioning. Your pipeline must treat this partition as a critical, fragile control plane.

Deep Dive into the concept

The Pi boot flow begins when power is applied and the SoC boot ROM loads GPU firmware from the first partition it can understand. On Raspberry Pi OS, the boot partition is a FAT filesystem with firmware blobs (for example start.elf, fixup.dat), configuration files (config.txt, cmdline.txt), and device tree blobs (*.dtb). The GPU reads config.txt first and configures the system before the ARM cores begin executing. This is why config.txt can enable or disable interfaces, set clocks, or load overlays that change pin multiplexing. cmdline.txt is read next and passed to the kernel as a single line of space-separated parameters; it must not contain line breaks because the bootloader parses it as a literal command line. The kernel image (kernel8.img on 64-bit) and the device tree are then loaded, and control is transferred to the ARM CPU.

From a provisioning standpoint, the boot partition is also where you can place marker files that the OS consumes at first boot. For example, Raspberry Pi OS reads an empty ssh file to enable the SSH service on first boot. For Wi-Fi provisioning, a wpa_supplicant.conf file placed in the boot partition is copied into /etc/wpa_supplicant/ on first boot (exact paths differ slightly between Raspberry Pi OS releases). These are not magical files; they are part of a first-boot script that scans the boot partition for known names. That makes your pipeline stable but also brittle: misspellings, wrong file encoding, or incorrect country codes can silently break provisioning.

On newer Raspberry Pi OS releases (Bookworm), the boot files are located at /boot/firmware once Linux is running, but the external computer still sees the same boot partition when the SD card is mounted. This can cause confusion if you script the pipeline using paths like /boot on the Pi and then reuse those scripts on a laptop while imaging. A robust pipeline explicitly addresses the SD card device, mounts the boot partition, and edits the files directly with validations (e.g., checksum the file after writing, verify line endings, and validate that cmdline.txt remains a single line). It should also keep a backup of the original boot files so you can recover if the Pi fails to boot.

Failure modes are a crucial part of the mental model. If config.txt has a syntax error, the GPU may ignore later lines or boot with default settings, resulting in missing interfaces (SPI/I2C disabled). If the kernel image name does not match the config.txt setting, the Pi may show a black screen or never come up on the network. If the SD card is corrupted or the boot partition is not FAT, the boot ROM cannot load firmware and the Pi appears dead. Because the Zero 2 W is headless, your only observable signals might be the power LED behavior and the absence of network presence. This is why your pipeline must include a test or a fallback path that can be verified without physical access.

How this fit on projects

This concept powers the entire provisioning pipeline and is used directly in §3 (Project Specification) and §5.10 Phase 1. You will also reuse it later in Project 13 (boot optimization) and Project 14 (USB gadget mode) when you edit boot configuration for system services and device roles.

Definitions & key terms

  • Boot partition: The FAT filesystem that stores firmware and configuration used before Linux boots.
  • config.txt: Firmware configuration file parsed by the GPU to set hardware options.
  • cmdline.txt: Kernel command line, a single-line string passed to the Linux kernel.
  • Device Tree Blob (DTB): Hardware description used by the kernel to bind drivers.
  • First-boot scripts: OS scripts that copy provisioning files from the boot partition.

Mental model diagram (ASCII)

Power
  |
  v
Boot ROM -> GPU firmware -> config.txt -> kernel+DTB -> init -> first-boot scripts
  |
  v
Boot partition (FAT) contains firmware + provisioning files

How it works (step-by-step, with invariants and failure modes)

  1. Power stabilizes; Boot ROM locates the FAT boot partition.
  2. GPU loads firmware and parses config.txt (invariant: file must be valid text).
  3. Kernel image and DTB are loaded (failure: missing file name or wrong path).
  4. Kernel reads cmdline.txt (invariant: single line, no newline).
  5. First-boot scripts copy ssh and wpa_supplicant.conf if present.
  6. systemd starts SSH and networking (failure: Wi-Fi config invalid).

Minimal concrete example

config.txt enabling UART and I2C:

# Enable UART for early logs
enable_uart=1
# Enable I2C interface
_dtparam=i2c_arm=on

Common misconceptions

  • “config.txt is a Linux config file.” It is read before Linux exists.
  • “If it booted once it will always boot.” SD corruption and config edits can break future boots.
  • “cmdline.txt can be multi-line.” It must be a single line or boot may fail.

Check-your-understanding questions

  1. Why is the GPU responsible for reading config.txt?
  2. What happens if cmdline.txt contains a newline?
  3. Why does placing an ssh file on the boot partition enable SSH?

Check-your-understanding answers

  1. The GPU firmware runs before the ARM CPU and sets hardware state.
  2. The kernel receives a truncated or malformed command line, possibly failing to mount the root filesystem.
  3. First-boot scripts scan the boot partition for that marker and enable the service.

Real-world applications

  • Mass-provisioning fleets of kiosks or sensors.
  • Rapid recovery after SD card reimaging.
  • Secure factory floor provisioning without monitors.

Where you’ll apply it

  • This project: §3.7 (Real World Outcome), §5.10 Phase 1, §5.11 decisions.
  • Other projects: Project 13, Project 14.

References

  • Raspberry Pi boot flow documentation (Raspberry Pi documentation)
  • “How Linux Works”, Ch. 2 (boot)

Key insights

Boot reliability is determined before Linux starts; the boot partition is your control plane.

Summary

The boot partition controls firmware, kernel selection, and first-boot provisioning. By mastering it, you can script reliable headless bring-up and recover from configuration failures.

Homework/Exercises to practice the concept

  1. Identify every file in the boot partition and describe its role.
  2. Create a deliberate config.txt syntax error and observe the failure.
  3. Verify how the Pi behaves when the kernel filename is changed.

Solutions to the homework/exercises

  1. List the directory and document start.elf, fixup.dat, config.txt, cmdline.txt, and DTBs.
  2. Add a malformed line and observe missing peripheral behavior; fix by restoring the file.
  3. Rename the kernel file and confirm the Pi fails to boot, then restore it.

Concept 2: Headless Wi-Fi Provisioning and SSH Hardening

Fundamentals

Headless provisioning is the process of configuring a device’s network and secure access without a monitor or keyboard. On Raspberry Pi OS, this typically involves placing a Wi-Fi configuration file and an SSH enable marker on the boot partition so that first-boot scripts import them. The Wi-Fi configuration must specify the correct SSID, password, and regulatory country code, or the device will never join the network. SSH hardening means disabling password login, installing your public key, and ensuring that only trusted hosts can access the device. The combination of correct Wi-Fi provisioning and secure SSH is what transforms a Pi from “a board that might boot” into a reachable, safe, and manageable embedded computer.

Deep Dive into the concept

Wi-Fi provisioning on Raspberry Pi OS uses wpa_supplicant, which is a daemon that negotiates authentication with wireless access points. The configuration file includes the SSID, PSK, country code, and optional network priorities. When you place a wpa_supplicant.conf file in the boot partition, a first-boot service copies it into /etc/wpa_supplicant/ and removes it from the boot partition to reduce exposure. This process is sensitive to whitespace, quoting, and file encoding; a UTF-8 BOM or Windows line endings can break parsing. Your pipeline should explicitly validate that the file contains the correct country code (e.g., country=US) and that the network block is syntactically valid.

Once Wi-Fi comes up, SSH is the preferred remote access method. Raspberry Pi OS disables SSH by default in many releases to reduce the risk of password-based compromise. The ssh marker file in the boot partition forces SSH to enable on first boot. However, enabling SSH is not enough: you must disable password authentication and install a public key. Doing this reliably requires first boot access or pre-seeding the authorized_keys file. A common approach is to place an authorized_keys file in /home/pi/.ssh/ via a first-boot script or provisioning process. You should also change the default pi password or disable the pi user entirely if you create your own user.

Hardening extends beyond SSH keys. You should restrict the SSH daemon to key-based logins (PasswordAuthentication no), limit root login, and optionally bind to the network interface used for provisioning. Because you are headless, you need a way to confirm that access is secure without locking yourself out. This is why the pipeline should include a verification step: attempt SSH with keys, then attempt password login and expect failure. If the test fails, the pipeline should roll back or warn before you deploy multiple devices.

The network provisioning process must also consider device discovery. Most headless setups rely on mDNS (hostname.local) so you can reach the Pi without knowing its IP address. But mDNS depends on the avahi-daemon service and can be disabled in minimal configurations. Your pipeline should include a fallback: query the router DHCP lease table, scan the subnet, or use USB gadget mode (covered later). A robust provisioning pipeline therefore includes both a “primary” and “secondary” discovery path, plus a clear recovery plan when Wi-Fi credentials are wrong.

Security is the last pillar. If you script provisioning for multiple devices, your private keys must be protected, and each device should have a unique host key (generated on first boot). Your pipeline must avoid cloning a single SD card image with static SSH host keys across devices, because it creates security warnings and weakens trust. The right approach is to allow each device to generate its own host keys on first boot, then record them or verify them when you first connect.

How this fit on projects

This concept underpins §3 (Project Specification), §5.10 Phase 2, and §6 (Testing). It is also reused in Project 9 (MQTT connectivity) and Project 16 (health monitoring) where secure connectivity is required.

Definitions & key terms

  • wpa_supplicant: Linux daemon that manages Wi-Fi authentication.
  • mDNS: Multicast DNS, used for hostname.local discovery.
  • SSH key authentication: Public/private key mechanism to authenticate without passwords.
  • Host key: SSH server identity key that must be unique per device.

Mental model diagram (ASCII)

Boot partition
  ├── wpa_supplicant.conf  -> copied to /etc/wpa_supplicant/
  └── ssh (marker)         -> enables sshd

First boot -> Wi-Fi join -> DHCP -> mDNS -> SSH login

How it works (step-by-step, with invariants and failure modes)

  1. wpa_supplicant.conf is copied on first boot (invariant: valid syntax).
  2. Wi-Fi interface negotiates with AP (failure: wrong PSK or country code).
  3. DHCP assigns IP (failure: AP isolation or DHCP disabled).
  4. mDNS advertises hostname (failure: avahi disabled).
  5. SSH daemon starts (invariant: keys present; failure: disabled or misconfigured).

Minimal concrete example

wpa_supplicant.conf example:

country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="MyWiFi"
    psk="correcthorsebatterystaple"
}

Common misconceptions

  • “If the SSID is correct, it will always connect.” Country codes and 2.4GHz settings matter.
  • “SSH keys are optional for small devices.” Passwords are a high-risk default.
  • “mDNS always works.” It depends on services that can be disabled.

Check-your-understanding questions

  1. Why does Raspberry Pi OS remove wpa_supplicant.conf after first boot?
  2. What breaks if you clone an SD card image without regenerating SSH host keys?
  3. How can you recover if Wi-Fi provisioning fails?

Check-your-understanding answers

  1. To avoid leaving credentials on the boot partition where they are easily read.
  2. All devices share the same host identity, causing trust issues and security risk.
  3. Use USB gadget mode, wired Ethernet, or re-edit the boot partition on another machine.

Real-world applications

  • Provisioning fleets of headless signage players.
  • Securely enabling remote diagnostics for IoT devices.
  • Factory provisioning lines with zero manual configuration.

Where you’ll apply it

  • This project: §3.7 (Real World Outcome), §5.10 Phase 2, §6.2 tests.
  • Other projects: Project 9, Project 16.

References

  • Raspberry Pi OS networking docs
  • “Linux Basics for Hackers”, Ch. 11 (SSH)

Key insights

Secure connectivity is part of provisioning; if you can’t trust SSH, you can’t trust the device.

Summary

Headless provisioning is a controlled handoff from boot partition to network services. Wi-Fi must be correct and SSH must be hardened, or your device is either unreachable or insecure.

Homework/Exercises to practice the concept

  1. Create two Wi-Fi configs and test priority selection.
  2. Disable password authentication and confirm SSH key-only access.
  3. Verify that mDNS fails when avahi-daemon is disabled and document a fallback.

Solutions to the homework/exercises

  1. Add two network blocks with different priority values and observe connection order.
  2. Set PasswordAuthentication no in sshd_config and test with ssh -o PasswordAuthentication=yes.
  3. Stop avahi-daemon and confirm hostname.local fails; use nmap or router DHCP list.

3. Project Specification

3.1 What You Will Build

A fully scripted imaging and provisioning pipeline that:

  • flashes Raspberry Pi OS to a microSD card,
  • injects Wi-Fi and SSH configuration,
  • validates first boot via network checks,
  • and records a provisioning report for repeatability.

3.2 Functional Requirements

  1. Image Creation: Flash a base Raspberry Pi OS image to an SD card.
  2. Boot Partition Injection: Add wpa_supplicant.conf and ssh marker.
  3. Security Hardening: Install your SSH public key and disable password auth.
  4. Verification: Confirm hostname resolution, SSH login, and kernel version.
  5. Recovery Path: Provide steps to re-image or reconfigure if Wi-Fi fails.

3.3 Non-Functional Requirements

  • Performance: Provisioning should complete within 10 minutes on a typical SD card.
  • Reliability: The pipeline should succeed on three consecutive runs.
  • Usability: Single command or script with clear logs and errors.

3.4 Example Usage / Output

$ ./provision.sh --image 2025-11-10-raspios-bookworm.img --device /dev/sdX --ssid MyWiFi
[INFO] Writing image...
[INFO] Mounting boot partition...
[INFO] Injecting Wi-Fi and SSH config...
[INFO] Ejecting SD card...
[INFO] Waiting for device to appear...
[INFO] SSH OK. Kernel: 6.6.XX-v8+
[INFO] Provisioning report: reports/pi-zero2w-2026-01-01.json

3.5 Data Formats / Schemas / Protocols

Provisioning report JSON:

{
  "device_id": "pi-zero2w-01",
  "image": "2025-11-10-raspios-bookworm.img",
  "hostname": "pi-zero2w",
  "ip": "192.168.1.42",
  "kernel": "6.6.XX-v8+",
  "ssh_key_fingerprint": "SHA256:...",
  "timestamp": "2026-01-01T10:21:04Z"
}

3.6 Edge Cases

  • Incorrect Wi-Fi password or country code.
  • mDNS disabled on the device or router.
  • SD card write error or corrupted image.
  • SSH host key mismatch from previous provisioning.

3.7 Real World Outcome

You can insert a freshly provisioned SD card, power the Pi, and connect securely within minutes without a monitor.

3.7.1 How to Run (Copy/Paste)

./provision.sh \
  --image 2025-11-10-raspios-bookworm.img \
  --device /dev/sdX \
  --ssid MyWiFi \
  --psk "correcthorsebatterystaple" \
  --hostname pi-zero2w

3.7.2 Golden Path Demo (Deterministic)

Set a fixed timestamp to make logs deterministic:

export PROVISION_TIME="2026-01-01T10:21:04Z"
./provision.sh --image 2025-11-10-raspios-bookworm.img --device /dev/sdX --ssid MyWiFi --psk "correcthorsebatterystaple"

Expected output snippet:

[INFO] Provisioning time: 2026-01-01T10:21:04Z
[INFO] SSH OK. Kernel: 6.6.XX-v8+

3.7.3 Failure Demo (Deterministic)

export PROVISION_TIME="2026-01-01T10:21:04Z"
./provision.sh --image 2025-11-10-raspios-bookworm.img --device /dev/sdX --ssid MyWiFi --psk "wrongpass"

Expected output:

[ERROR] Wi-Fi association failed after 180s
[ERROR] Provisioning failed: E_WIFI_AUTH

Exit code: 12

3.7.4 CLI Exit Codes

  • 0: Success
  • 10: SD card write failure
  • 11: Boot partition mount failure
  • 12: Wi-Fi auth failure
  • 13: SSH verification failure

4. Solution Architecture

4.1 High-Level Design

Laptop/CI
   |
   v
[Flash Image] -> [Inject Boot Files] -> [Boot Device] -> [Verify via SSH] -> [Report]

4.2 Key Components

| Component | Responsibility | Key Decisions | |———–|—————-|—————| | Image Writer | Flash base OS image | Use dd vs rpi-imager CLI | | Boot Injector | Add ssh and wpa_supplicant.conf | Validate syntax, remove BOM | | Verifier | Check SSH, kernel, RSSI | Timeout strategy | | Reporter | Write JSON report | Deterministic timestamps |

4.3 Data Structures (No Full Code)

# Bash associative array for report
REPORT[hostname]="pi-zero2w"
REPORT[kernel]="6.6.XX-v8+"

4.4 Algorithm Overview

Key Algorithm: Provision-and-Verify Loop

  1. Flash image and mount boot partition.
  2. Inject configuration files.
  3. Wait for device to appear on network.
  4. Attempt SSH until timeout.
  5. Record report or emit error.

Complexity Analysis:

  • Time: O(Tboot + Tretry)
  • Space: O(1) in script, O(size of report)

5. Implementation Guide

5.1 Development Environment Setup

# On your laptop
sudo apt-get install -y openssh-client rsync

5.2 Project Structure

project-root/
├── provision.sh
├── templates/
│   └── wpa_supplicant.conf.tmpl
├── reports/
└── README.md

5.3 The Core Question You’re Answering

“How can a Pi go from a blank SD card to a secure, reachable device without a screen or keyboard?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. Boot partition layout and config.txt roles.
  2. wpa_supplicant configuration syntax and country codes.
  3. SSH public key authentication and sshd_config hardening.
  4. mDNS vs direct IP discovery.

5.5 Questions to Guide Your Design

  1. How will you verify success without a screen?
  2. How will you ensure SSH keys are installed before disabling password login?
  3. What is your recovery path if Wi-Fi fails?

5.6 Thinking Exercise

Draw the first 60 seconds after power-on and mark where configuration is read, where Wi-Fi starts, and where SSH becomes available.

5.7 The Interview Questions They’ll Ask

  1. Which files enable headless Wi-Fi on Raspberry Pi OS?
  2. Why are SSH host keys per-device, and what happens if they are cloned?
  3. How do you verify boot success without HDMI?

5.8 Hints in Layers

Hint 1: Start by flashing the image only; boot it once to verify.

Hint 2: Add the ssh marker file and confirm SSH works.

Hint 3: Add Wi-Fi config and test mDNS discovery.

Hint 4: Automate SSH hardening and verify password login is disabled.

5.9 Books That Will Help

| Topic | Book | Chapter | |—|—|—| | Boot process | Raspberry Pi User Guide | Ch. 2–3 | | Linux boot | How Linux Works | Ch. 2 | | Secure access | Linux Basics for Hackers | Ch. 11 |

5.10 Implementation Phases

Phase 1: Imaging & Boot Files (2–3 hours)

Goals: Flash image; inject ssh and Wi-Fi files. Tasks:

  1. Script SD card detection and image write.
  2. Mount boot partition and add provisioning files. Checkpoint: SD card boots and Wi-Fi attempts to connect.

Phase 2: Secure Access (2–4 hours)

Goals: Key-based SSH and hardening. Tasks:

  1. Install public key via first SSH login.
  2. Disable password auth and verify. Checkpoint: SSH key works and password login fails.

Phase 3: Verification & Reporting (2–4 hours)

Goals: Automated verification and report. Tasks:

  1. Add network discovery loop.
  2. Collect kernel version, RSSI, IP.
  3. Write report JSON with deterministic timestamp. Checkpoint: reports/*.json contains accurate info.

5.11 Key Implementation Decisions

| Decision | Options | Recommendation | Rationale | |—|—|—|—| | Imaging tool | dd / Raspberry Pi Imager CLI | dd with checksum | Simple and scriptable | | Discovery | mDNS / DHCP table scan | mDNS + fallback scan | More reliable across routers | | Hardening timing | Pre-seed keys / First login | First login + disable password | Avoid lockout |

6. Testing Strategy

6.1 Test Categories

| Category | Purpose | Examples | |—|—|—| | Unit Tests | Validate config file creation | Wi-Fi template rendering | | Integration Tests | Full provisioning run | Flash + SSH verify | | Edge Case Tests | Failure handling | Wrong SSID, no DHCP |

6.2 Critical Test Cases

  1. Correct Wi-Fi: Device connects and SSH works within 3 minutes.
  2. Wrong PSK: Script fails with exit code 12.
  3. SSH key missing: Hardening step refuses to disable password login.

6.3 Test Data

SSID=MyWiFi
PSK=correcthorsebatterystaple
HOST=pi-zero2w

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

| Pitfall | Symptom | Solution | |—|—|—| | Wrong country code | Wi-Fi never associates | Set country=US | | Windows line endings | wpa_supplicant fails | Convert to Unix LF | | Disabled SSH | Connection refused | Ensure ssh marker exists |

7.2 Debugging Strategies

  • Serial console: Enable UART to see boot logs.
  • Router DHCP list: Confirm IP assignment.

7.3 Performance Traps

  • Long retry timeouts hide real errors; set explicit timeouts and error codes.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add hostname auto-generation based on SD card label.
  • Add a provisioning summary printed to console.

8.2 Intermediate Extensions

  • Add A/B boot partition support with fallback.
  • Include public key fingerprint verification.

8.3 Advanced Extensions

  • Provision multiple devices in parallel with USB SD card readers.
  • Add encrypted Wi-Fi configuration handling.

9. Real-World Connections

9.1 Industry Applications

  • Factory provisioning: Automated imaging and validation for device fleets.
  • Field replacement: Rapid SD replacement in remote installations.
  • pi-gen: Custom Raspberry Pi OS image builder.
  • balenaOS: Fleet provisioning and device management.

9.3 Interview Relevance

  • Boot flow, Linux provisioning, and secure remote access are common embedded interview topics.

10. Resources

10.1 Essential Reading

  • “Raspberry Pi User Guide” — boot and configuration chapters.
  • “How Linux Works” — boot and init systems.

10.2 Video Resources

  • Raspberry Pi Foundation: headless setup tutorials.

10.3 Tools & Documentation

  • wpa_supplicant docs: Wi-Fi config syntax.
  • sshd_config man page: SSH hardening options.
  • Previous: None (this is Project 1).
  • Next: Project 2 — hardware I/O basics.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain the boot partition and why it matters.
  • I can explain how Wi-Fi provisioning works on first boot.
  • I can explain why SSH key auth is required.

11.2 Implementation

  • All functional requirements are met.
  • All tests pass and errors are handled.
  • Provisioning reports are deterministic.

11.3 Growth

  • I can describe the recovery path for a failed headless boot.
  • I can explain this pipeline in an interview.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Provisioning script flashes image and enables SSH.
  • Device is reachable over Wi-Fi by hostname.
  • Report JSON is generated.

Full Completion:

  • Password login disabled and SSH key verified.
  • Recovery steps documented and tested.

Excellence (Going Above & Beyond):

  • Multi-device batch provisioning with unique host keys.
  • A/B boot support with rollback.