Project 8: File I/O System

A layered file I/O subsystem with buffering, error handling, and portable file abstractions.

Quick Reference

Attribute	Value
Difficulty	Level 3 - Advanced
Time Estimate	1-2 weeks
Main Programming Language	C
Alternative Programming Languages	None
Coolness Level	Level 3 - Genuinely Clever
Business Potential	Level 2 - Micro-SaaS
Prerequisites	Pointers, struct usage, basic OS concepts
Key Topics	File descriptors, buffering, partial I/O, error handling

1. Learning Objectives

By completing this project, you will:

Build a layered file I/O API with buffered and unbuffered modes.
Handle partial reads/writes correctly and safely.
Implement consistent error reporting with errno mapping.
Demonstrate durability guarantees using fsync and atomic rename.
Produce a test suite that validates correctness under edge cases.

2. All Theory Needed (Per-Concept Breakdown)

Concept 1: File Descriptors, Buffering, and Partial I/O

Fundamentals

At the OS level, files are accessed through file descriptors (FDs), which are small integers referring to kernel-managed file objects. System calls like read and write operate on these FDs. Higher-level C APIs like fread and fwrite wrap FDs in buffered streams (FILE*). Buffering improves performance by batching I/O but can obscure when data is actually written to disk. Additionally, read and write can return partial results, so you must loop until all requested bytes are processed.

Deep Dive into the concept

A file descriptor is a handle to a kernel object that tracks file position, flags, and permissions. When you call read(fd, buf, n), the kernel attempts to read up to n bytes, but it may return fewer bytes for reasons like end-of-file, non-blocking mode, or interruptions by signals. Similarly, write may write fewer bytes than requested. Robust code must always check the return value and retry until the entire buffer is processed or an unrecoverable error occurs.

Buffering adds another layer. FILE* streams maintain a user-space buffer that reduces syscalls. This is good for performance but can cause confusion when you mix buffered and unbuffered I/O. For example, writing with fprintf and then reading with read on the same FD can lead to unexpected results because the stream has its own internal state. Professional code either uses one layer consistently or carefully flushes and synchronizes. For a custom I/O system, you must decide which layer you expose and how much control users have over buffering.

Buffering also impacts durability. When you call fwrite, the data may sit in a user-space buffer; even after fflush, it may still be in the kernel’s page cache. Only fsync or fdatasync ensures the data is flushed to disk. For systems that require correctness after crashes (like databases), you must be explicit about these steps. Your project will include a buffered layer and a raw FD layer, and you will document how data flows between them.

Partial I/O is a major source of bugs. Many developers assume a single read or write will process the full buffer, but that assumption is invalid. Your I/O system must provide read_all and write_all helpers that loop until completion, handle EINTR, and report short reads correctly. This is foundational for reliable file operations.

To deepen this topic, include an explicit discussion of file offsets and atomicity. File descriptors share a file offset; this means that if you duplicate an FD with dup, writes advance the same offset. This matters in multi-threaded code and when mixing buffered and unbuffered I/O. The O_APPEND flag forces each write to append atomically at the end of the file, which is important for logging. However, atomicity guarantees differ by OS and filesystem, especially for large writes. Your I/O layer should document these assumptions and provide options for append-only writes when appropriate. Also consider line buffering vs full buffering: interactive programs often want line-buffered output (setvbuf with _IOLBF). Finally, show how to use lseek to reposition the file offset, and make it clear that lseek is undefined for some file types (pipes, sockets). These details make your I/O system realistic and robust.

To operationalize this concept in a real codebase, create a short checklist of invariants and a set of micro-experiments. Start with a minimal, deterministic test that isolates one rule or behavior, then vary a single parameter at a time (inputs, flags, platform, or data layout) and record the outcome. Keep a table of assumptions and validate them with assertions or static checks so violations are caught early. Whenever the concept touches the compiler or OS, capture tool output such as assembly, warnings, or system call traces and attach it to your lab notes. Finally, define explicit failure modes: what does a violation look like at runtime, and how would you detect it in logs or tests? This turns abstract theory into repeatable engineering practice and makes results comparable across machines and compiler versions.

How this fits on projects

It defines the layered API in §4.1.
It drives the read_all/write_all logic in §5.2.
Also used in: Project 12: Cross-Platform Portability Layer.

Definitions & key terms

File descriptor (FD): Kernel handle to an open file.
Buffered I/O: User-space buffer that reduces syscalls.
Partial read/write: System call returns fewer bytes than requested.
fsync: System call that forces kernel buffers to disk.

Mental model diagram (ASCII)

Application -> [buffer] -> libc -> syscalls -> kernel page cache -> disk

How it works (step-by-step, with invariants and failure modes)

Open file -> FD created.
Optional buffering layer wraps FD.
Read/write requests go through buffer then syscalls.
Syscalls may return short counts; code must loop.

Invariant: All requested bytes are processed unless EOF or error. Failure mode: Assuming full reads/writes causes data loss.

Minimal concrete example

ssize_t write_all(int fd, const void *buf, size_t n) {
    size_t off = 0;
    while (off < n) {
        ssize_t r = write(fd, (const char*)buf + off, n - off);
        if (r < 0 && errno == EINTR) continue;
        if (r <= 0) return -1;
        off += (size_t)r;
    }
    return (ssize_t)off;
}

Common misconceptions

“write always writes all bytes.” → Not guaranteed.
“fflush means data is on disk.” → It only flushes to kernel.
“Mixing FILE* and FD is fine.” → Can desync buffers.

Check-your-understanding questions

Why can write return fewer bytes than requested?
What is the difference between fflush and fsync?
Why is buffering useful?
What happens if you ignore partial reads?
When should you use read_all?

Check-your-understanding answers

Kernel interruptions, non-blocking mode, or resource limits.
fflush flushes user buffer; fsync flushes to disk.
It reduces syscalls and improves performance.
You may lose data or mis-parse input.
When you need to read an exact number of bytes.

Real-world applications

Databases and loggers requiring durability guarantees.
Network file transfers and protocol implementations.

Where you’ll apply it

See §3.2 Functional Requirements for read/write helpers.
See §5.8 Hints for handling EINTR.
Also used in: Project 11: Testing and Analysis Framework.

References

“The Linux Programming Interface” — Kerrisk, file I/O chapters
POSIX read/write documentation

Key insights

Reliable file I/O means assuming syscalls are partial and buffering is layered.

Summary

File descriptors are the OS-level interface for I/O, while buffering improves performance at the cost of complexity. Handling partial reads/writes correctly is mandatory for correctness.

Homework/Exercises to practice the concept

Implement read_all and write_all for a fixed-length record.
Write a program that demonstrates partial writes by limiting buffer size.
Compare performance of buffered vs unbuffered reads.

Solutions to the homework/exercises

Loop until count bytes read or EOF.
Use a small pipe buffer or non-blocking mode.
Buffered reads should use fewer syscalls and be faster.

Concept 2: Error Handling, Durability, and Atomic Updates

Fundamentals

File operations can fail in many ways: permissions, missing files, interrupted syscalls, or disk full errors. A professional I/O system must surface these errors consistently and provide safe patterns for updates. Durable writes require fsync or fdatasync, and atomic updates typically use write-to-temp + rename. These patterns prevent partial writes from corrupting critical files.

Deep Dive into the concept

Error handling in file I/O is not just about checking return values; it’s about mapping errors into meaningful categories. errno values such as EINTR, EAGAIN, ENOENT, and EACCES must be interpreted correctly. For example, EINTR indicates a signal interruption and should be retried, while ENOENT is a real failure. Your I/O library should provide a consistent error type (enum + message) that wraps errno and makes it easy for the caller to handle.

Durability is a separate concern. Even after a successful write, data may remain in the kernel’s page cache and be lost if the system crashes. fsync forces the kernel to flush the file’s data and metadata to disk. For some use cases, fdatasync is sufficient because it flushes only data. Your project will implement a flush operation that can be configured to call fsync for durability guarantees.

Atomic updates are essential for configuration files and state files. The safest pattern is: write new contents to a temporary file in the same directory, fsync it, then rename it over the original. On POSIX systems, rename is atomic, meaning the file is always either old or new, never partially written. This pattern prevents corruption if the program crashes mid-write. You must also fsync the directory to ensure the rename is persisted.

In your I/O system, you’ll implement an atomic_write helper that uses this pattern. You will also document how durability and atomicity interact, and provide deterministic tests that simulate failure scenarios (e.g., failing after write but before rename). This creates a robust I/O layer suitable for production use.

Another way to deepen understanding is to map the concept to a small decision table: list inputs, expected outcomes, and the assumptions that must hold. Create at least one negative test that violates an assumption and observe the failure mode, then document how you would detect it in production. Add a short trade-off note: what you gain by following the rule and what you pay in complexity or performance. Where possible, instrument the implementation with debug-only checks so violations are caught early without affecting release builds. If the concept admits multiple approaches, implement two and compare them; the act of measuring and documenting the difference is part of professional practice. This habit turns theoretical understanding into an engineering decision framework you can reuse across projects.

How this fits on projects

It defines the durable write API in §3.2 and §3.7.
It informs the error enums in §3.5.
Also used in: Project 10: Modular Program Architecture.

Definitions & key terms

errno: Thread-local error code set by failed syscalls.
Durability: Guarantee that data is on stable storage.
Atomic rename: Replacement of a file without partial updates.
Temporary file: A file used as a staging area for updates.

Mental model diagram (ASCII)

write -> fsync(temp) -> rename(temp, final) -> fsync(dir)

How it works (step-by-step, with invariants and failure modes)

Write new content to temp file.
fsync temp file to ensure data is on disk.
rename temp to target (atomic).
fsync directory to persist rename.

Invariant: Target file is always valid (old or new). Failure mode: Skipping fsync can lose updates after crash.

Minimal concrete example

int atomic_write(const char *path, const void *buf, size_t n);

Common misconceptions

“A successful write means data is safe.” → It may still be in cache.
“rename isn’t atomic.” → On POSIX, it is atomic within a filesystem.
“errno is enough.” → Callers need structured error reporting.

Check-your-understanding questions

What does fsync guarantee?
Why write to a temp file before renaming?
Why must the temp file be in the same directory?
What errors should be retried automatically?
What happens if you skip directory fsync?

Check-your-understanding answers

Data and metadata are flushed to stable storage.
To avoid partial updates of the original file.
rename is only atomic within the same filesystem.
EINTR and possibly EAGAIN if non-blocking.
The rename may not persist after a crash.

Real-world applications

Configuration file updates in daemons.
Transaction logs and state snapshots.

Where you’ll apply it

See §3.7 Real World Outcome for atomic write demo.
See §7.1 Frequent Mistakes for durability pitfalls.
Also used in: Project 12: Cross-Platform Portability Layer.

References

“The Linux Programming Interface” — Kerrisk, file I/O and fsync
SQLite atomic write patterns

Key insights

Correct file I/O is not just about reading and writing; it is about durability and atomicity.

Summary

Error handling, durability, and atomic updates are essential for reliable file I/O. A professional I/O system must provide consistent error semantics and safe update patterns.

Homework/Exercises to practice the concept

Implement an atomic write helper.
Simulate a crash between write and rename and verify correctness.
Map common errno values to user-friendly messages.

Solutions to the homework/exercises

Use temp + fsync + rename + fsync dir.
Kill the process before rename; old file should remain intact.
Create a small mapping table for errno.

3. Project Specification

3.1 What You Will Build

A file I/O library with buffered and unbuffered APIs, helpers for full reads/writes, and safe atomic update functions. Includes a CLI demo program and a test suite.

3.2 Functional Requirements

Open/Close API: Wrap open, close, and FILE* streams.
Read/Write Helpers: Provide read_all and write_all.
Buffering Layer: Optional buffering with flush control.
Atomic Write: Safe update helper using temp + rename.
Error Reporting: Consistent error enum and messages.

3.3 Non-Functional Requirements

Performance: Buffered reads should reduce syscalls.
Reliability: Atomic writes guarantee file validity after crashes.
Usability: Clear API with documented error codes.

3.4 Example Usage / Output

io_file_t f = io_open("data.bin", IO_READ);
io_read_all(&f, buf, len);
io_close(&f);

3.5 Data Formats / Schemas / Protocols

Error enum:

typedef enum { IO_OK=0, IO_EOF=1, IO_EIO=-1, IO_EINVAL=-2 } io_err_t;

3.6 Edge Cases

Interrupted syscalls (EINTR).
Short reads at EOF.
Disk full during atomic write.

3.7 Real World Outcome

What you will see:

A robust I/O API with documentation.
A demo program showing buffered and unbuffered reads.
An atomic update demo with guaranteed correctness.

3.7.1 How to Run (Copy/Paste)

make
./io_demo --atomic-write config.json

3.7.2 Golden Path Demo (Deterministic)

Write a fixed file and verify contents match expected output.

3.7.3 If CLI: exact terminal transcript

$ ./io_demo --atomic-write config.json
Wrote config.json atomically
Exit: 0

Failure demo (deterministic):

$ ./io_demo --atomic-write /root/forbidden
ERROR: permission denied (EACCES)
Exit: 3

4. Solution Architecture

4.1 High-Level Design

+-------------------+
| API (io_file)      |
+---------+---------+
          |
          v
+-------------------+     +-------------------+
| buffering layer    | -->| syscalls          |
+-------------------+     +-------------------+
          |
          v
+-------------------+
| atomic write helper|
+-------------------+

4.2 Key Components

4.3 Data Structures (No Full Code)

typedef struct {
    int fd;
    FILE *fp;
    int buffered;
} io_file_t;

4.4 Algorithm Overview

Open file and initialize state.
Perform reads/writes with partial handling.
Flush/close with error checks.
Use atomic write for critical updates.

Complexity Analysis:

Time: O(n) per read/write
Space: O(buffer size)

5. Implementation Guide

5.1 Development Environment Setup

clang -std=c23 -Wall -Wextra -Werror -g

5.2 Project Structure

file-io/
├── src/
│   ├── io.c
│   ├── atomic.c
│   └── demo.c
├── include/
│   └── io.h
├── tests/
└── Makefile

5.3 The Core Question You’re Answering

“How can I make file I/O reliable, safe, and durable in C?”

5.4 Concepts You Must Understand First

File descriptors and buffering.
Partial reads/writes and retries.
Atomic updates and fsync.

5.5 Questions to Guide Your Design

What errors should be retried automatically?
How will you expose buffering options to users?
How will you ensure atomic updates are durable?

5.6 Thinking Exercise

Design a write_all loop that handles EINTR and partial writes.

5.7 The Interview Questions They’ll Ask

Why can read return fewer bytes than requested?
How do you ensure a file update is atomic?
What does fsync actually guarantee?

5.8 Hints in Layers

Hint 1: Start with raw FD helpers.
Hint 2: Add buffering as an optional layer.
Hint 3: Add atomic write using temp + rename.

5.9 Books That Will Help

| Topic | Book | Chapter | |——-|——|———| | File I/O | “The Linux Programming Interface” — Kerrisk | Ch. 4-6 |

5.10 Implementation Phases

Phase 1: Foundation (3-4 days)

Implement read_all/write_all.
Checkpoint: Tests pass for fixed-size files.

Phase 2: Core Functionality (4-5 days)

Add buffering layer and flush control.
Checkpoint: Buffered reads reduce syscalls.

Phase 3: Polish & Edge Cases (2-3 days)

Add atomic write and error mapping.
Checkpoint: Atomic update demo works.

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

Write a file and read back exact bytes.
Simulate EINTR with signal handler.
Atomic update preserves file on crash simulation.

6.3 Test Data

file contents: "hello\n"

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

Use strace/dtrace to inspect syscalls.
Add logging around error handling.

7.3 Performance Traps

Excessive fsync calls can slow throughput; only use for critical writes.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a simple file copy tool using your library.

8.2 Intermediate Extensions

Add non-blocking I/O support.

8.3 Advanced Extensions

Add memory-mapped I/O options (mmap).

9. Real-World Connections

9.1 Industry Applications

Logging systems and config management.
Data ingestion pipelines.

SQLite atomic write patterns.
libuv file I/O wrappers.

9.3 Interview Relevance

Partial I/O and durability questions are common in systems interviews.

10. Resources

10.1 Essential Reading

“The Linux Programming Interface” — Kerrisk

10.2 Video Resources

POSIX I/O talks and tutorials

10.3 Tools & Documentation

strace, dtrace, ltrace

11. Self-Assessment Checklist

11.1 Understanding

I can explain partial reads/writes.
I can describe buffering vs fsync.
I can implement atomic updates.

11.2 Implementation

I/O helpers pass tests.
Atomic write demo works.
Error reporting is consistent.

11.3 Growth

I can design a reliable file update system.
I can debug I/O issues with tracing tools.

12. Submission / Completion Criteria

Minimum Viable Completion:

read_all/write_all implemented.
Buffered I/O layer working.
Atomic write demo passes.

Full Completion:

All minimum criteria plus:
Error mapping and fsync-based durability documented.

Excellence (Going Above & Beyond):

Non-blocking and mmap support with benchmarks.