Project 4: Hot-Reload Development Server

Build a runtime server that watches a shared library, recompiles it on change, and hot-swaps logic without restarting the host or losing state.

Quick Reference

Attribute Value
Difficulty Level 4: Expert
Time Estimate 2-4 weeks
Main Programming Language C (Alternatives: C++, Rust)
Alternative Programming Languages C++, Rust
Coolness Level Level 5: Pure Magic
Business Potential Level 4: Open Core Infrastructure
Prerequisites Dynamic loading, ABI design, file watching, build tooling
Key Topics Hot reload lifecycle, state isolation, inotify, ABI guards

1. Learning Objectives

By completing this project, you will:

  1. Build a safe hot-reload loop using dlopen/dlclose and file watchers.
  2. Separate state from code to preserve gameplay or service state across reloads.
  3. Implement ABI compatibility checks to prevent unsafe reloads.
  4. Handle stale function pointers and prevent use-after-unload errors.
  5. Create deterministic reload scenarios for testing and demos.

2. All Theory Needed (Per-Concept Breakdown)

2.1 Hot Reload Lifecycle and State Separation

Fundamentals Hot reloading means replacing code at runtime without restarting the process. The key insight is that code can be reloaded, but state must persist. Therefore, you must separate mutable state from the code that operates on it. The host process owns the state, while the plugin provides pure functions that operate on that state. When a new version of the plugin is loaded, the host swaps function pointers but retains the state memory. This pattern is the backbone of live-reload systems in games and servers.

Deep Dive into the concept At runtime, code and data live in different regions of memory. When you dlclose a plugin, the loader may unmap its code and data segments. If your state lives inside the plugin, unloading will invalidate it. That is why the host must allocate and own state that survives reloads. Typically, you define a state_t structure in a header that is shared between host and plugin. The host allocates it once, initializes it, and passes a pointer into the plugin’s tick or update function. This makes the plugin stateless with respect to allocation, which greatly reduces reload risk.

Hot reload introduces ABI stability concerns. The plugin and host must agree on the layout of state_t and the signature of the API functions. If the plugin changes the struct layout, the host may continue to pass a pointer with an outdated layout, leading to memory corruption. Therefore, the plugin should report its ABI version and the expected size of state_t. The host can then reject reloads where these do not match. This is similar to the ABI versioning approach from the plugin project, but more stringent because you are reloading into a live process.

Another issue is function pointer staleness. When you unload a library, any function pointers into it become invalid. Therefore, you must update all function pointers immediately after a reload and ensure no threads are currently executing in the old code. A safe approach is to pause the main loop during reload and ensure no worker threads call plugin functions. For advanced setups, you can implement an epoch-based system where threads check a version number before calling plugin functions.

Hot reload also requires careful management of static variables within the plugin. Static variables in the plugin are part of the plugin’s data segment and will be reset on reload. This can be surprising if the plugin assumes they persist. This is another reason to keep state outside the plugin.

How this fits in this project Your host will own a state_t structure and pass it to the plugin. The reload loop will swap the function table and continue running without resetting the state.

Definitions & key terms

  • Hot reload -> Swapping code at runtime without restarting.
  • State separation -> Keeping mutable state in the host, not the plugin.
  • Function pointer table -> A struct of function pointers that the host swaps on reload.
  • Epoch -> A version number used to coordinate reload safety.

Mental model diagram (ASCII)

Host state (persistent) ----> Plugin logic v1
          |                      |
          +-- reload --> Plugin logic v2

How it works (step-by-step, with invariants and failure modes)

  1. Host loads plugin and obtains API table.
  2. Host allocates state_t and passes it into plugin functions.
  3. File watcher detects change; host pauses loop.
  4. Host unloads old plugin and loads new plugin.
  5. Host validates ABI and swaps function pointers.
  6. Host resumes loop with same state.

Invariants: state memory remains valid; ABI version matches. Failure modes: stale function pointers, ABI mismatch, state corruption.

Minimal concrete example

typedef struct { int score; } state_t;

typedef struct {
    int api_version;
    size_t state_size;
    void (*tick)(state_t*);
} api_t;

// host owns state
state_t* state = calloc(1, sizeof(state_t));
api_t* api = plugin_get_api();
api->tick(state);

Common misconceptions

  • “Reloading code is enough.” -> State must persist outside the plugin.
  • “Static variables are safe.” -> They reset on reload.

Check-your-understanding questions

  1. Why does plugin state vanish on reload?
  2. How can the host detect a breaking ABI change?
  3. Why must you pause the main loop during reload?

Check-your-understanding answers

  1. The plugin’s data segment is unloaded with the library.
  2. Compare api_version and state_size values.
  3. To avoid calling stale function pointers while the plugin is being replaced.

Real-world applications

  • Game engines hot-swapping gameplay logic.
  • Live coding systems in creative software.

Where you’ll apply it

References

  • “Game Programming Patterns” (Nystrom), hot reload patterns.
  • “C Interfaces and Implementations” (Hanson), API boundaries.

Key insights Hot reload succeeds only when state and code are decoupled.

Summary The reload lifecycle is simple but fragile. State ownership and ABI checks are non-negotiable.

Homework/Exercises to practice the concept

  1. Write a toy host that reloads a plugin and preserves a counter.
  2. Add a new field to state_t and observe ABI breakage.
  3. Implement a version check that rejects incompatible plugins.

Solutions to the homework/exercises

  1. Use a loop that increments a counter and reloads on file change.
  2. The old plugin will interpret the struct incorrectly unless versioned.
  3. Compare api_version and state_size before swapping.

2.2 File Watching and Build Automation (inotify, polling)

Fundamentals Hot reload depends on detecting file changes quickly and reliably. On Linux, inotify provides efficient file system events. On macOS, kqueue or FSEvents serve similar roles. If you cannot use native watchers, polling the file modification time is a simpler but less efficient fallback. Once a change is detected, your server should rebuild the plugin and then reload it.

Deep Dive into the concept inotify works by creating a watch on a file or directory and receiving events when changes occur. You add a watch using inotify_add_watch, then read events from the file descriptor. Each event tells you which file changed and what kind of change happened (modified, moved, deleted). For this project, you can watch the plugin source file or the output .so file. Watching the output is simpler, but you must be careful: build systems often write a temporary file and rename it, which can produce events like IN_MOVED_TO rather than IN_MODIFY. Therefore, your watcher should handle multiple event types.

Build automation must be deterministic. If you rebuild the plugin on each change, you should use a fixed build command and consistent flags. You should also ensure that you do not attempt to reload while a build is in progress. A common approach is to wait for the build to finish successfully before reloading; if the build fails, keep the old plugin loaded and report the failure.

Polling is a fallback: check stat for mtime changes at fixed intervals. While inefficient, it is portable and easy to implement. For a simple project, you can provide both: --watch-mode=inotify and --watch-mode=poll.

How this fits in this project Your host will watch for changes and run a build command (e.g., gcc -shared -fPIC). When the build completes successfully, it triggers reload.

Definitions & key terms

  • inotify -> Linux API for file system events.
  • polling -> Periodic checking of file modification time.
  • debounce -> Wait for changes to stabilize before acting.

Mental model diagram (ASCII)

[watcher] -> change detected -> build -> reload -> resume

How it works (step-by-step, with invariants and failure modes)

  1. Start watcher on plugin output file.
  2. Event arrives -> debounce for 100ms.
  3. Run build command; check exit code.
  4. If build succeeds, reload plugin.
  5. If build fails, keep old plugin and report error.

Invariants: never reload a partially built library. Failure modes: partial writes, rebuild loops, failed builds.

Minimal concrete example

# Build command
cc -shared -fPIC -o logic.so logic.c

Common misconceptions

  • “Any file event means the build is done.” -> You must debounce and wait.
  • “Reload even if build fails.” -> That can crash if the output is invalid.

Check-your-understanding questions

  1. Why debounce file events?
  2. Why watch the output .so instead of the source?
  3. What should the system do if the build fails?

Check-your-understanding answers

  1. Build systems often emit multiple events; debouncing avoids premature reload.
  2. The output reflects successful builds and avoids source-only changes.
  3. Keep the old plugin active and report the error.

Real-world applications

  • Live-reload web servers and game engines.

Where you’ll apply it

References

  • man inotify.
  • Build system documentation (Make/CMake).

Key insights Hot reload is only as reliable as your change detection and build pipeline.

Summary File watching connects code changes to reload. Handle events carefully and reload only after successful builds.

Homework/Exercises to practice the concept

  1. Write a small watcher using inotify that prints events.
  2. Implement a polling loop with a fixed interval.
  3. Add a debounce timer and test with rapid file saves.

Solutions to the homework/exercises

  1. Use inotify_init, inotify_add_watch, and read events.
  2. Use stat and compare st_mtime.
  3. Sleep for 100-200ms after the first event.

2.3 ABI Guards and Safe Reload Boundaries

Fundamentals Hot reload is unsafe if the plugin ABI changes. The host must guard against incompatible plugins by checking version numbers, struct sizes, and required functions. A reload boundary is the point where the host safely swaps old code for new code. The boundary should ensure that no threads are executing plugin code and that all function pointers are updated.

Deep Dive into the concept An ABI guard is a runtime check that compares plugin metadata to host expectations. A common pattern is:

  • api_version (major) must match exactly.
  • api_revision (minor) can be greater or equal if the host is backward compatible.
  • state_size must match exactly, because the host has already allocated state.
  • All required function pointers must be non-NULL.

If any check fails, the host must refuse to reload and continue using the old plugin. This makes hot reload safe even when developers change structs accidentally.

The reload boundary is the point where you stop the main loop, unload the old plugin, load the new plugin, validate it, and then resume. You should also guard against in-flight calls. For single-threaded loops, simply stopping the loop is enough. For multi-threaded systems, you need synchronization: either stop worker threads or use a versioned pointer to ensure they only call valid code.

How this fits in this project Your host will define and enforce ABI guards, and implement a safe reload boundary in the main loop.

Definitions & key terms

  • ABI guard -> Runtime check that ensures compatibility.
  • Reload boundary -> Safe point to swap plugin code.
  • State size -> Expected size of persistent state struct.

Mental model diagram (ASCII)

loop
  tick() -> (reload boundary) -> swap -> tick()

How it works (step-by-step, with invariants and failure modes)

  1. Detect file change and pause loop.
  2. Load new plugin and fetch API.
  3. Validate api_version, state_size.
  4. If valid, swap function table and unload old plugin.
  5. Resume loop.

Invariants: old code must not run after unload. Failure modes: stale pointers, ABI mismatch.

Minimal concrete example

if (api->api_version != HOST_API_VERSION) {
    fprintf(stderr, "ABI mismatch\n");
    return RELOAD_REJECTED;
}

Common misconceptions

  • “Reloading can fix ABI changes.” -> ABI changes must be rejected or versioned.
  • “It’s safe to unload while threads run.” -> It is not.

Check-your-understanding questions

  1. What is the minimum ABI data you should check?
  2. Why must state_size match exactly?
  3. How do you ensure no thread calls old code?

Check-your-understanding answers

  1. Version, state size, and required function pointers.
  2. The host already allocated state based on the old size.
  3. Pause the loop or use synchronization barriers.

Real-world applications

  • Live updates in game engines and hot-patched services.

Where you’ll apply it

  • In this project: see Section 3.2 Functional Requirements and Section 5.11 Key Implementation Decisions.
  • Also used in: P05-cross-platform-c-api for ABI checks.

References

  • “C Interfaces and Implementations” (Hanson), ABI boundaries.

Key insights Hot reload is safe only when you treat ABI checks as gatekeepers.

Summary Hot reload without ABI guards is gambling. Always validate and only swap at safe boundaries.

Homework/Exercises to practice the concept

  1. Add a version field to your API and reject mismatches.
  2. Simulate a size mismatch and verify reload is rejected.
  3. Add a log message that explains why reload was rejected.

Solutions to the homework/exercises

  1. Compare api_version and return a non-zero error code.
  2. Change the struct size in the plugin and observe rejection.
  3. Print a message with expected vs actual values.

3. Project Specification

3.1 What You Will Build

A CLI server hotreload that:

  • Runs a simple simulation loop (e.g., a counter or game state).
  • Loads logic from a shared library plugin.
  • Watches for changes, rebuilds the plugin, and reloads it.
  • Preserves state and rejects incompatible ABI changes.

3.2 Functional Requirements

  1. Plugin API: Expose init, tick, and shutdown.
  2. State preservation: Host owns state_t and keeps it across reloads.
  3. File watcher: Detect changes and trigger rebuild.
  4. Reload safety: ABI checks and safe swap boundary.
  5. Determinism: Fixed tick rate and fixed seed for any random logic.

3.3 Non-Functional Requirements

  • Reliability: Never crash on bad plugin builds.
  • Usability: Clear logs showing reload success/failure.
  • Portability: Linux focus with optional polling mode.

3.4 Example Usage / Output

$ ./hotreload --watch logic.c --build "cc -shared -fPIC -o logic.so logic.c" --tick-ms 100
[server] running
[state] score=0
[watch] change detected
[build] success
[reload] swapped logic.so (api=1)
[state] score=0 (preserved)

3.5 Data Formats / Schemas / Protocols

Plugin API

typedef struct {
    int api_version;
    size_t state_size;
    void (*init)(void* state);
    void (*tick)(void* state, double dt);
    void (*shutdown)(void* state);
} logic_api_t;

logic_api_t* logic_get_api(void);

3.6 Edge Cases

  • Build fails due to compiler error.
  • Plugin ABI mismatch after reload.
  • Plugin file replaced mid-reload.

3.7 Real World Outcome

3.7.1 How to Run (Copy/Paste)

./hotreload --watch logic.c --build "cc -shared -fPIC -o logic.so logic.c" --tick-ms 100 --seed 123

3.7.2 Golden Path Demo (Deterministic)

  • State starts at score=0.
  • After 10 ticks, score is exactly 10.
  • Reload occurs; score remains 10.

3.7.3 CLI Transcript (Success + Failure)

$ ./hotreload --watch logic.c --build "cc -shared -fPIC -o logic.so logic.c" --tick-ms 100 --seed 123
[server] running
[state] score=0
[watch] change detected
[build] success
[reload] swapped logic.so (api=1)
[state] score=0 (preserved)
[exit] code=0

$ ./hotreload --watch logic.c --build "cc -shared -fPIC -o logic.so logic.c" --tick-ms 100
[watch] change detected
[build] failed: error: expected ';'
[reload] rejected, keeping old plugin
[exit] code=9

3.7.4 If CLI: Exit Codes

  • 0: success
  • 9: build failed
  • 10: ABI mismatch

4. Solution Architecture

4.1 High-Level Design

+------------------+
| host process     |
|  - state         |
|  - watch/build   |
|  - reload logic  |
+---------+--------+
          |
          v
+------------------+
| logic.so         |
|  - init/tick     |
+------------------+

4.2 Key Components

Component Responsibility Key Decisions
Watcher Detect file changes inotify + debounce
Builder Run build command capture exit code
Loader dlopen/dlsym fail fast
State Persistent data owned by host

4.3 Data Structures (No Full Code)

typedef struct {
    int score;
    double time;
} state_t;

4.4 Algorithm Overview

Key Algorithm: Reload Loop

  1. Tick loop calls plugin tick.
  2. On file change, pause loop.
  3. Build and reload plugin if build succeeds.
  4. Swap API table and resume.

Complexity Analysis:

  • Time: O(1) per tick; reload cost depends on build.
  • Space: O(1) persistent state.

5. Implementation Guide

5.1 Development Environment Setup

sudo apt-get install build-essential inotify-tools

5.2 Project Structure

hotreload/
|-- src/
|   |-- host.c
|   |-- watcher.c
|   `-- reload.c
|-- logic/
|   `-- logic.c
|-- Makefile
`-- README.md

5.3 The Core Question You’re Answering

“How do you reload code without losing state or crashing the process?”

5.4 Concepts You Must Understand First

  1. State separation and ABI stability.
  2. File watching and build automation.
  3. Safe reload boundaries.

5.5 Questions to Guide Your Design

  1. Where does state live and who owns it?
  2. How will you handle a failed rebuild?
  3. What triggers are safe for reload?

5.6 Thinking Exercise

Draw a memory map before and after reload. Which pointers become invalid?

5.7 The Interview Questions They’ll Ask

  1. “Why is hot reload hard in C?”
  2. “How do you preserve state across reloads?”
  3. “What happens if the ABI changes?”

5.8 Hints in Layers

Hint 1: Separate state and logic

Hint 2: Use function tables

Hint 3: Pause the loop while reloading

5.9 Books That Will Help

Topic Book Chapter
Dynamic loading TLPI Ch. 42
Interfaces C Interfaces and Implementations Ch. 2
Live reload patterns Game Programming Patterns Chapter on live coding

5.10 Implementation Phases

Phase 1: Foundation (3-4 days)

  • Build a host that loads a plugin and runs a loop.

Phase 2: Core Functionality (5-7 days)

  • Add file watching and rebuild triggers.
  • Implement reload boundary and ABI checks.

Phase 3: Polish & Edge Cases (3-5 days)

  • Add deterministic tests and failure handling.
  • Add logging and exit codes.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
State ownership Plugin vs host Host Persistence across reload
Watch mode inotify vs polling inotify with fallback Efficient and portable
Reload timing immediate vs debounce debounce Avoid partial builds

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests ABI checks version mismatch
Integration Tests Reload loop change logic.c and reload
Edge Case Tests Build fail invalid syntax

6.2 Critical Test Cases

  1. Build fails -> old plugin remains active.
  2. ABI mismatch -> reload rejected.
  3. Deterministic tick count preserved across reload.

6.3 Test Data

logic.c (v1)
logic.c (v2)

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Reloading mid-call Crash Pause loop and join threads
ABI mismatch Corrupted state Enforce size/version checks
Build races Partial load Debounce and wait for build success

7.2 Debugging Strategies

  • Log all reload decisions with version numbers.
  • Keep the old plugin handle until the new one is validated.

7.3 Performance Traps

  • Excessive rebuilds due to noisy file events.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a --poll mode for file watching.
  • Add a reload counter to logs.

8.2 Intermediate Extensions

  • Add a GUI or TUI to display state.
  • Add rollback to a previous plugin version.

8.3 Advanced Extensions

  • Multi-threaded hot reload with epoch-based safety.
  • Remote reload server over TCP.

9. Real-World Connections

9.1 Industry Applications

  • Game engines and live coding environments.
  • Long-running servers that need hot patches.
  • Live++ (commercial) and open-source live coding tools.

9.3 Interview Relevance

  • Demonstrates understanding of ABI, loaders, and runtime safety.

10. Resources

10.1 Essential Reading

  • “The Linux Programming Interface” (Kerrisk), Ch. 42.
  • “Game Programming Patterns” (Nystrom).

10.2 Video Resources

  • Live coding talks (game dev conferences).

10.3 Tools & Documentation

  • inotify, dlopen, dlsym.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain why state must live outside the plugin.
  • I can describe the reload boundary and ABI checks.
  • I can explain how file watching triggers reload.

11.2 Implementation

  • Reload works without losing state.
  • Build failures do not crash the host.
  • ABI mismatches are detected and rejected.

11.3 Growth

  • I can describe hot reload in an interview.
  • I documented a real reload bug and its fix.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Host loads plugin and runs a loop.
  • File watcher triggers reload.

Full Completion:

  • ABI guards and safe reload boundaries.
  • Deterministic test scenarios.

Excellence (Going Above & Beyond):

  • Multi-threaded reload safety or remote reload control.
  • Rollback to previous plugin versions.