Project 4: Built-in Commands Engine

Implement built-in commands with a dispatcher and shell state updates.

Quick Reference

Attribute Value
Difficulty Level 2: Intermediate (The Developer)
Time Estimate 1 week
Main Programming Language C
Alternative Programming Languages Rust, Go, Zig
Coolness Level Level 2: Practical but Forgettable
Business Potential 1. The “Resume Gold” (Educational/Personal Brand)
Prerequisites C function pointers, environment variables, process model
Key Topics built-in dispatch, shell state, parent-only execution

1. Learning Objectives

By completing this project, you will:

  1. Explain and implement built-in dispatch in the context of a shell.
  2. Build a working built-in commands engine that matches the project specification.
  3. Design tests that validate correctness and edge cases.
  4. Document design decisions, trade-offs, and limitations.

2. All Theory Needed (Per-Concept Breakdown)

Built-ins and Shell State Management

Fundamentals Built-ins are commands executed by the shell itself rather than by a child process. They exist because some operations must modify the shell’s own state: changing directories, setting variables, altering options, or controlling jobs. If cd ran in a child, the parent shell would remain in the old directory, so built-ins are not optional. Implementing built-ins requires a dispatch table that maps command names to handler functions, and a consistent way to update shell state such as the working directory, variable table, and last exit status. A reliable built-in system makes the shell usable beyond running external programs.

Deep Dive into the concept A shell’s state is the collection of mutable data that persists across commands: current working directory, shell variables, exported environment, options (like -e or -u), job tables, and even the input history. Built-ins are the only commands allowed to mutate this state directly, because external commands run in child processes with isolated memory. This means that built-in implementations are part of the shell core, not just “special commands.” The design challenge is to make built-ins feel like normal commands while still allowing them to bypass the usual fork/exec path.

A common design is a built-in registry: an array or hash table that maps names to functions, with metadata such as “special built-in” or “can run in a child.” For example, echo can be implemented as a built-in or external command and behaves the same either way, but cd, export, unset, and exit must run in the parent. POSIX distinguishes “special built-ins” that have different error behavior in scripts; if a special built-in fails, the shell may exit in non-interactive mode. Even if you are not implementing full POSIX rules, you should define which built-ins are “stateful” and require parent execution.

Built-in argument parsing is another subtlety. Users expect built-ins to accept options that differ from external commands. For instance, export VAR=1 both sets a variable and marks it for export. read reads from stdin and assigns variables. ulimit modifies resource limits. These commands blur the line between command execution and environment management. Therefore, you should structure built-ins to access a shared shell context object rather than operating on globals scattered across the codebase.

Error handling and exit status must be consistent. Built-ins should set $? the same way external commands do: 0 on success, non-zero on failure. They should print errors to stderr and return a status code to the main loop. This keeps control flow (&&, ||, if) consistent. If a built-in fails inside a pipeline, you need to decide whether to run it in a child process; some shells execute built-ins in subshells when in a pipeline to preserve pipeline semantics, while others implement hybrid strategies. Your project should document this behavior.

Finally, built-ins are an extensibility point. A clean built-in API allows you to add custom commands later (like help, alias, history, or even plugin commands). The built-in system should handle help text, argument parsing errors, and consistent usage output. Over time, this built-in layer becomes the control panel of your shell.

How this fits on projects Built-ins are essential for any shell that manipulates its own environment, manages jobs, or provides interactive features beyond external commands.

Definitions & key terms

  • Built-in: A command implemented inside the shell process.
  • Shell state: Persistent data owned by the shell (cwd, vars, jobs).
  • Special built-in: POSIX term for built-ins with special error rules.
  • Dispatcher: Mapping of names to built-in handlers.

Mental model diagram

input word -> lookup in built-ins table
   | yes
   v
run handler -> mutate shell state -> set $?
   | no
   v
fork/exec external command

How it works (step-by-step)

  1. Parse command and identify the command name.
  2. Look up name in built-in registry.
  3. If built-in and requires parent, run handler in shell process.
  4. If built-in can run in child (pipeline), decide execution context.
  5. Update $? and any shell state changes.

Minimal concrete example

struct builtin { const char *name; int (*fn)(struct shell*, int, char**); };

Common misconceptions

  • “Built-ins are optional” -> without them, cd and export cannot work.
  • “All built-ins can run in children” -> many must run in parent.
  • “Built-ins don’t need exit status” -> they must set $?.

Check-your-understanding questions

  1. Why can’t cd be an external command?
  2. What happens if export runs in a child process?
  3. How should built-ins behave inside pipelines?

Check-your-understanding answers

  1. Because only the parent can change the shell’s directory.
  2. The export would not persist in the parent shell.
  3. Either run in a subshell or document special handling; status must be consistent.

Real-world applications

  • bash, dash, zsh built-in command sets.
  • Job control commands like fg and bg.

Where you’ll apply it

  • In this project: see §3.2 Functional Requirements and §5.4 Concepts.
  • Also used in: None

References

  • POSIX Shell Command Language (special built-ins).
  • “The Linux Programming Interface” (environment manipulation).

Key insights Built-ins are the only way for a shell to change its own state.

Summary A solid built-in system turns a launcher into a real shell with stateful behavior.

Homework/Exercises to practice the concept

  1. Implement cd, pwd, and exit as built-ins.
  2. Add a built-in registry with metadata for parent-only commands.
  3. Make built-ins set $? like external commands.

Solutions to the homework/exercises

  1. Use chdir() and update a shell context.
  2. Store flags in the registry and enforce them at execution time.
  3. Return status codes from built-ins and store in shell state.

Environment Variables, Export, and Scoping

Fundamentals Shells maintain two related variable spaces: shell variables (internal to the shell) and environment variables (exported to child processes). The export built-in marks a variable so it appears in the envp array passed to execve(). Shell variables can exist without being exported, and assignments can be local to a single command invocation (VAR=1 cmd). Understanding how variables are stored, expanded, and inherited is essential for scripting and predictable behavior.

Deep Dive into the concept A shell typically stores variables in a dictionary-like structure mapping names to values, plus metadata indicating whether a variable is exported. When the shell executes an external command, it must produce an environment array (list of KEY=VALUE strings) that includes only exported variables. This array is passed to execve() and becomes the child’s environment. Internal-only variables are not visible to children but still participate in expansions like $VAR within the shell. This separation enables fine-grained control: you can have variables for scripting that do not leak into subcommands.

Assignments have special behavior. An assignment preceding a command (FOO=bar cmd) should not permanently change the shell’s variables; it should only affect the environment of that single command. Shells implement this by creating a temporary environment overlay for the child: in the parent, the variable table is not mutated permanently, but in the child, the environment array includes the temporary assignment. This is subtle and often mishandled in toy shells, so it’s worth getting right. For purely shell-level assignment (FOO=bar), the variable should be set in the shell table and optionally exported if the export built-in is used.

Scoping becomes more complex with functions and scripts. Many shells allow “local” variables inside functions that override globals and disappear on function return. This requires a scope stack: when entering a function, push a new scope; on return, pop it. Exported variables usually remain global, but some shells allow local exported variables as well. If you are not implementing full scoping, you should at least provide a consistent model and document it.

Expansion and quoting interact with variables. In double quotes, $VAR expands but does not undergo field splitting; in unquoted contexts, the expanded value is subject to field splitting and globbing. This means the variable system must interact with the expansion engine, not just store strings. When a variable is unset, some shells expand it to an empty string, others treat it as an error if set -u is enabled. These option-dependent behaviors are part of shell state management.

Finally, the environment is not the only inherited state. The shell may propagate other properties, like the working directory and open file descriptors. But variables are the most visible interface for configuring child processes, so correctness here is crucial. If you drop variables or incorrectly export them, commands like PATH and HOME will not work, making the shell feel broken.

How this fits on projects Variable handling appears in assignment parsing, expansion, built-ins, and execution environment construction.

Definitions & key terms

  • Shell variable: Variable stored in the shell, not exported by default.
  • Environment variable: Exported variable passed to child processes.
  • Export: Marking a variable for inheritance.
  • Scope stack: Data structure for nested variable scopes.

Mental model diagram

shell vars (internal) + exported vars -> envp[] -> execve

How it works (step-by-step)

  1. Parse assignments and decide if they are temporary or permanent.
  2. Update shell variable table and export flags.
  3. On exec, build envp from exported variables.
  4. Apply temporary assignments only in the child environment.
  5. On function entry/exit, push/pop variable scopes if supported.

Minimal concrete example

FOO=1
export BAR=2
FOO=3 cmd   # child sees FOO=3, BAR=2; parent keeps FOO=1

Common misconceptions

  • “All variables are exported” -> only exported ones reach children.
  • “Temporary assignment changes shell” -> it should not persist.
  • “Unset equals empty string” -> depends on shell options.

Check-your-understanding questions

  1. Why does FOO=1 cmd not permanently set FOO?
  2. How do exported variables differ from shell variables?
  3. What happens to local variables after a function returns?

Check-your-understanding answers

  1. The assignment is applied only to the child environment.
  2. Exported variables are included in envp for execve().
  3. They are discarded when the scope stack is popped.

Real-world applications

  • Configuring tools via PATH, HOME, EDITOR.
  • Build systems that set environment flags.
  • Scripts that pass secrets via environment variables.

Where you’ll apply it

References

  • POSIX Shell Command Language (environment and variable rules).
  • “The Linux Programming Interface” (environment handling).

Key insights The environment is a filtered view of shell variables, not a separate universe.

Summary Variable management is about scoping, export rules, and correct inheritance into child processes.

Homework/Exercises to practice the concept

  1. Implement a variable table with export flags.
  2. Support temporary assignments for single commands.
  3. Add export and unset built-ins with correct behavior.

Solutions to the homework/exercises

  1. Store name/value pairs with a boolean exported flag.
  2. Build a temporary envp array for the child process.
  3. Remove or mark variables and rebuild envp on exec.

Exit Status, Wait Semantics, and Truthiness

Fundamentals Shells treat exit status as the primary truth value of command execution. An exit status is an integer reported when a process terminates; by convention, 0 means success and non-zero means failure. The shell reads this status via waitpid() and exposes it as $?. This is not just metadata: control flow operators (&&, ||, if, while) all use exit status to decide what to do next. A correct shell must distinguish normal exit from signal termination, map errors to standard codes like 126 and 127, and propagate the right status through pipelines and compound commands. This concept links process control with scripting semantics and is essential for correctness.

Deep Dive into the concept When a child exits, the kernel encodes the reason in a status word returned by waitpid(). The shell must interpret this correctly: WIFEXITED(status) indicates normal completion and WEXITSTATUS(status) yields the actual exit code (0–255). WIFSIGNALED(status) indicates termination by a signal, and WTERMSIG(status) gives the signal number. Many shells represent signal termination as 128 + signal, so a process killed by SIGINT (2) yields status 130. This is not just a convention; scripts rely on these distinctions. For example, build scripts may treat 127 as “command not found” and fail fast, while they may retry on other codes.

Exit status also interacts with pipelines. POSIX allows the status of a pipeline to be either the status of the last command or a composite status, and some shells provide options (like pipefail) to change this behavior. If you are building a minimal shell, you should document your choice and implement it consistently. For job control, foreground pipelines typically update $? when the last process in the pipeline completes, while background pipelines update job tables but do not immediately change $?.

The concept of “truthiness” in shell scripting is inverted relative to many languages: success is 0 (true), and failure is non-zero (false). This can surprise programmers coming from C-like languages where 0 is false. In shell, if cmd; then ... fi runs the then branch if cmd exits with 0. Control operators && and || are just syntactic sugar over this truth model: a && b runs b only if a succeeded, and a || b runs b only if a failed. A shell that gets this wrong will make scripts behave unpredictably.

Exit status also carries subtle edge cases. For instance, built-ins must set $? explicitly; functions return their last command’s exit status unless overridden by return. If the shell itself exits due to a fatal error, it should return a meaningful exit status to its parent (often 2 for syntax errors in POSIX). If a child is stopped (SIGTSTP), the status indicates stoppage rather than exit; a job control shell must track that and resume it later.

How this fits on projects You will use exit status to decide control flow, report failures, and mirror real shell behavior in every project that executes commands.

Definitions & key terms

  • Exit status: Integer code set by a process at termination.
  • WIFEXITED/WEXITSTATUS: Macros to inspect normal exit codes.
  • WIFSIGNALED/WTERMSIG: Macros to inspect signal termination.
  • Truthiness: In shell, 0 is true and non-zero is false.

Mental model diagram

child exits -> kernel encodes status -> waitpid() -> shell updates $?
           |                               |
           +-- signal? -> 128 + signal     +-- normal? -> exit code

How it works (step-by-step)

  1. Parent calls waitpid() for a child PID.
  2. Inspect WIFEXITED vs WIFSIGNALED.
  3. Map signal termination to 128 + signal.
  4. Store the code in $? and in job metadata.
  5. For pipelines, choose consistent policy (last command or pipefail).

Minimal concrete example

int status;
waitpid(pid, &status, 0);
if (WIFEXITED(status)) {
shell_status = WEXITSTATUS(status);
} else if (WIFSIGNALED(status)) {
shell_status = 128 + WTERMSIG(status);
}

Common misconceptions

  • “Non-zero is true” -> in shell, non-zero is false.
  • “Signal termination has no exit code” -> shells map signals to 128+N.
  • “Pipelines always return last status” -> depends on policy.

Check-your-understanding questions

  1. Why does false && echo ok not print anything?
  2. What should $? be when a process is killed by SIGKILL (9)?
  3. How does a shell decide the status of a pipeline?

Check-your-understanding answers

  1. Because false exits non-zero, so the && short-circuits.
  2. 128 + 9 = 137 in most shells.
  3. By policy: often the last command, optionally pipefail.

Real-world applications

  • Shell scripting control flow.
  • CI pipelines that interpret exit codes.
  • Service supervisors that restart failed programs.

Where you’ll apply it

References

  • POSIX Shell Command Language (exit status rules).
  • “Advanced Programming in the UNIX Environment” (waitpid).

Key insights Exit status is the shell’s core truth value, not a minor detail.

Summary Correct exit status handling ties process control to scripting semantics and determines whether higher-level logic behaves correctly.

Homework/Exercises to practice the concept

  1. Write a program that runs false and prints its exit status.
  2. Send SIGINT to a child and print the resulting $? equivalent.
  3. Build a tiny pipeline and decide how you will compute its status.

Solutions to the homework/exercises

  1. Use fork/exec and waitpid to capture the status.
  2. Use kill(child, SIGINT) and map to 128+signal.
  3. Record the last child’s status, or implement pipefail.

3. Project Specification

3.1 What You Will Build

A built-in registry and handlers for commands like cd, pwd, exit, and export.

Included:

  • Core feature set described above
  • Deterministic CLI behavior and exit codes

Excluded:

  • No full scripting semantics; focus on core built-ins.

3.2 Functional Requirements

  1. Requirement 1: Maintain a built-in table with names and handler functions.
  2. Requirement 2: Implement cd, pwd, exit, export, and unset.
  3. Requirement 3: Ensure built-ins run in the parent when required.
  4. Requirement 4: Provide consistent error messages and exit status.
  5. Requirement 5: Expose a help or list command for built-ins.

3.3 Non-Functional Requirements

  • Performance: Interactive latency under 50ms for typical inputs; pipeline setup should scale linearly.
  • Reliability: No crashes on malformed input; errors reported clearly with non-zero status.
  • Usability: Clear prompts, deterministic behavior, and predictable error messages.

3.4 Example Usage / Output

$ ./mysh
mysh> pwd
/home/user
mysh> cd /tmp
mysh> pwd
/tmp
mysh> export MY_VAR=hello
mysh> /bin/sh -c 'echo $MY_VAR'
hello
mysh> exit 0
$ echo $?
0

3.5 Data Formats / Schemas / Protocols

  • Shell state struct storing cwd, vars, and last status.

3.6 Edge Cases

  • cd with missing HOME
  • Invalid variable names
  • Exit with non-numeric status

3.7 Real World Outcome

This is the exact behavior you should be able to demonstrate.

3.7.1 How to Run (Copy/Paste)

  • make
  • ./mysh

3.7.2 Golden Path Demo (Deterministic)

$ ./mysh
mysh> pwd
/home/user
mysh> cd /tmp
mysh> pwd
/tmp
mysh> export MY_VAR=hello
mysh> /bin/sh -c 'echo $MY_VAR'
hello
mysh> exit 0
$ echo $?
0

3.7.3 Failure Demo (Deterministic)

$ ./mysh
mysh> not_a_command
mysh> echo $?
127

4. Solution Architecture

4.1 High-Level Design

[Input] -> [Parser/Lexer] -> [Core Engine] -> [Executor/Output]

4.2 Key Components

Component Responsibility Key Decisions
Builtin Table Maps names to handlers Allows easy extension.
Shell State Stores cwd, vars, last status Centralize mutable state.
Validator Checks args and variable names Prevents invalid state.

4.4 Data Structures (No Full Code)

struct ShellState { char *cwd; int last_status; /* vars map */ };

4.4 Algorithm Overview

Key Algorithm: Dispatch

  1. Lookup command name
  2. Invoke handler
  3. Set status

Complexity Analysis:

  • Time: O(1) average with hash table
  • Space: O(1) average with hash table

5. Implementation Guide

5.1 Development Environment Setup

# install dependencies (if any)
# build
make

5.2 Project Structure

project-root/
├── src/
│   ├── main.c
│   ├── lexer.c
│   └── executor.c
├── tests/
│   └── test_basic.sh
├── Makefile
└── README.md

5.3 The Core Question You’re Answering

Which commands must run inside the shell, and why?

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. Execution environment
  2. Special built-ins
  3. Environment variables

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

The “cd in a pipeline” Problem

What should cd /tmp | cat do?

  • Should the parent change directories?
  • How does a real shell behave?

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

Hint 1: Use a struct table

struct builtin { const char *name; int (*fn)(int, char**); };

Hint 2: Resolve before fork Check built-in name and execute in parent if matched.

Hint 3: Save/restore fds for redirections Built-ins should respect redirections and restore fd state.

Hint 4: Special built-ins Treat exit, cd, export, unset specially in scripts.

5.9 Books That Will Help

Topic Book Chapter
Built-ins “Advanced Programming in the UNIX Environment” Ch. 4, 8
Environment “The Linux Programming Interface” Ch. 6
POSIX built-ins POSIX Shell Command Language Built-in sections

5.10 Implementation Phases

Phase 1: Foundation (2-3 days)

Goals:

  • Define data structures and interfaces
  • Build a minimal end-to-end demo

Tasks:

  1. Implement the core data structures
  2. Build a tiny CLI or harness for manual tests

Checkpoint: A demo command runs end-to-end with clear logging.

Phase 2: Core Functionality (1 week)

Goals:

  • Implement full feature set
  • Validate with unit tests

Tasks:

  1. Implement core requirements
  2. Add error handling and edge cases

Checkpoint: All functional requirements pass basic tests.

Phase 3: Polish & Edge Cases (2-4 days)

Goals:

  • Harden for weird inputs
  • Improve UX and documentation

Tasks:

  1. Add edge-case tests
  2. Document design decisions

Checkpoint: Deterministic golden demo and clean error output.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Parsing depth Minimal vs full Incremental Start small, expand safely
Error policy Silent vs verbose Verbose Debuggability for learners

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests Test individual components Tokenizer, matcher, env builder
Integration Tests Test component interactions Full command lines
Edge Case Tests Handle boundary conditions Empty input, bad args

6.2 Critical Test Cases

  1. Golden Path: Run the canonical demo and verify output.
  2. Failure Path: Provide invalid input and confirm error status.
  3. Stress Path: Run repeated commands to detect leaks or state corruption.

6.3 Test Data

input: echo hello
output: hello

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Misordered redirection Output goes to wrong place Apply redirections left-to-right
Leaked file descriptors Commands hang waiting for EOF Close unused fds in parent/child
Incorrect exit status &&/|| behave wrong Use waitpid macros correctly

7.2 Debugging Strategies

  • Trace syscalls: Use strace/dtruss to verify fork/exec/dup2 order.
  • Log state transitions: Print parser states and job table changes in debug mode.
  • Compare with dash: Run the same input in a reference shell.

7.3 Performance Traps

  • Avoid O(n^2) behavior in hot paths like line editing.
  • Minimize allocations inside the REPL loop.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a help built-in with usage docs.
  • Add colored prompt themes.

8.2 Intermediate Extensions

  • Add a simple profiling mode for command timing.
  • Implement a which built-in using PATH lookup.

8.3 Advanced Extensions

  • Add programmable completion or plugin system.
  • Add a scriptable test harness with golden outputs.

9. Real-World Connections

9.1 Industry Applications

  • Build systems: shells orchestrate compilation and test pipelines.
  • DevOps automation: scripts manage deployments and infrastructure.
  • bash: The most common interactive shell.
  • dash: Minimal POSIX shell often used as /bin/sh.
  • zsh: Feature-rich interactive shell.

9.3 Interview Relevance

  • Process creation and lifecycle questions.
  • Parsing and system programming design trade-offs.

10. Resources

10.1 Essential Reading

  • “Advanced Programming in the UNIX Environment” by W. Richard Stevens - focus on the chapters relevant to this project.
  • “Advanced Programming in the UNIX Environment” - process control and pipes.

10.2 Video Resources

  • Unix process model lectures (any OS course).
  • Compiler front-end videos for lexing/parsing projects.

10.3 Tools & Documentation

  • strace/dtruss: inspect syscalls.
  • man pages: fork, execve, waitpid, pipe, dup2.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain the core concept without notes.
  • I can trace a command through my subsystem.
  • I understand at least one key design trade-off.

11.2 Implementation

  • All functional requirements are met.
  • All critical tests pass.
  • Edge cases are handled cleanly.

11.3 Growth

  • I documented lessons learned.
  • I can explain this project in an interview.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Core feature works for the golden demo.
  • Errors are handled with non-zero status.
  • Code is readable and buildable.

Full Completion:

  • All functional requirements met.
  • Tests cover edge cases and failures.

Excellence (Going Above & Beyond):

  • Performance benchmarks and clear documentation.
  • Behavior compared against a reference shell.