← Back to all projects

LEARN LISP DEEP DIVE

Learn Lisp: From S-expressions to Macro Magic

Goal: Deeply understand the Lisp family of languages—from its foundational “code as data” philosophy to the powerful macro system that makes it one of the most expressive programming paradigms ever created.

Why Learn Lisp?

Lisp is not just a language; it’s an idea. It’s the second-oldest high-level programming language still in common use, and its core concepts have influenced almost every other language you’ve ever used. Learning Lisp is like seeing the source code of programming language design itself.

Most languages have a strict distinction between “code” and “data.” Lisp’s core idea—homoiconicity—erases that line. Code is just a data structure (a list), which means you can write programs that build other programs. This is Lisp’s superpower.

After completing these projects, you will:

  • Understand the world in terms of S-expressions.
  • Fluently use functional programming concepts like map, filter, and reduce.
  • Write powerful macros that extend the language to fit your problem domain.
  • Appreciate the power of a REPL-driven, interactive development workflow.
  • Understand how Lisp’s unique features (like the condition system) differ from mainstream languages.

Core Concept Analysis

The Lisp Philosophy: Code is Data

┌─────────────────────────────────────────────────────────────────────────┐
│                           LISP SOURCE CODE                              │
│                                                                         │
│   (defun factorial (n)
│     (if (<= n 1)
│         1
│         (* n (factorial (- n 1)))))
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼ The Lisp Reader (parser)
┌─────────────────────────────────────────────────────────────────────────┐
│                         DATA STRUCTURE (a list)                         │
│                                                                         │
│   (:DEFUN :FACTORIAL (:N)
│     (:IF (:<= :N 1)
│         1
│         (:* :N (:FACTORIAL (:- :N 1)))))
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
                                 │
          ┌──────────────────────┼──────────────────────┐
          ▼                      ▼                      ▼
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│    EVALUATION    │  │    MANIPULATION  │  │   CODE GENERATION│
│   (The REPL)     │  │  (Your Program)  │  │     (Macros)     │
│                  │  │                  │  │                  │
│ • Execute it     │  │ • Treat as data  │  │ • Transform it   │
│ • Run the logic  │  │ • Analyze it     │  │ • Create new code│
│                  │  │ • Modify it      │  │ • Build a DSL    │
└──────────────────┘  └──────────────────┘  └──────────────────┘

Key Concepts Explained

1. S-expressions (Symbolic Expressions)

The fundamental syntax of Lisp. An S-expression is either an atom (like a number 42 or a symbol MY-VAR) or a list of other S-expressions enclosed in parentheses.

  • (+ 1 2) is a list containing the symbol + and the numbers 1 and 2.
  • When evaluated, this becomes a function call. This is “prefix notation”.

2. Lists and cons cells

The core data structure. A list is a chain of cons cells.

  • (cons 'a 'b) creates a cell (a . b).
  • car gets the first item in a cell. (car '(a . b)) is a.
  • cdr gets the rest of the cell. (cdr '(a . b)) is b.
  • The list (1 2 3) is really (1 . (2 . (3 . nil))). nil is the empty list.

3. eval and the REPL

The Read-Eval-Print Loop is the heart of Lisp development.

  • Read: The reader turns your text into a data structure (S-expression).
  • Eval: The evaluator takes that data structure and executes it according to Lisp’s rules.
  • Print: The printer displays the result.

4. Macros

Macros are functions that run at compile time. They take code (as a data structure) as input and produce new code (as a data structure) as output. This new code is then evaluated.

A simplified when macro:

(defmacro my-when (condition &body body)
  `(if ,condition
       (progn
         ,@body)))
  • The backquote ` means “don’t evaluate this yet, treat it as a template.”
  • The comma , means “evaluate this part inside the template.”
  • The comma-at ,@ means “evaluate this and splice its contents into the list.”

When you write (my-when (> x 0) (print "Positive")), the macro expands it into (if (> x 0) (progn (print "Positive"))) before it is ever executed.

5. The Condition System

More powerful than traditional exceptions. When an error occurs, the condition system signals a “condition”. Code higher up the stack can define “handlers” to deal with it, or interactive “restarts” that allow the program to resume from the error with corrected input.

Project List

The following 12 projects will guide you from the basic syntax to the most powerful and unique features of Lisp.

Project 1: Build an S-expression Parser

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Python, C, Rust
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Parsing / Core Language Mechanics
  • Software or Tool: Your own Lisp parser
  • Main Book: “Structure and Interpretation of Computer Programs” (SICP) by Abelson, Sussman, and Sussman

What you’ll build: A program that takes a string like "(+ 10 (* 2 3))" and turns it into a nested list data structure: (:+, 10, (:*, 2, 3)).

Why it teaches Lisp: This is the first step. You cannot understand Lisp until you understand that Lisp code is a data structure. Building the reader yourself forces you to internalize this homoiconic nature.

Core challenges you’ll face:

  • Tokenization → maps to splitting the string into (, ), and atoms
  • Recursive parsing → maps to handling nested lists by calling the parser on itself
  • Handling different data types → maps to distinguishing between numbers, strings, and symbols

Key Concepts:

  • S-expressions: “Practical Common Lisp” Ch. 3 - Peter Seibel
  • Recursion: “The Little Schemer” - Friedman & Felleisen
  • Parsing: “Crafting Interpreters” Ch. 4 - Robert Nystrom (for the general idea)

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic programming concepts (loops, conditionals, recursion).

Real world outcome:

$ ./s-expr-parser "(+ 10 (* 2 3))"
Parsing: (+ 10 (* 2 3))
Result:
(
  +
  10
  (
    *
    2
    3
  )
)

Implementation Hints:

  1. Write a tokenize function that takes a string and returns a list of tokens. For example, "(+ 1 2)" becomes ['(', '+', '1', '2', ')'].
  2. Write a parse function that takes this list of tokens.
  3. Inside parse, if the current token is (, start a new list. Then, call parse recursively for each element until you hit a matching ).
  4. If the token is an atom (not ( or )), try to convert it to a number. If you can’t, treat it as a symbol.
  5. Your main parse function should return two things: the parsed expression and the rest of the unparsed tokens.

Learning milestones:

  1. Parsing atoms → You can convert strings to numbers and symbols.
  2. Parsing a flat list → You can handle a single level of parentheses.
  3. Parsing nested lists → You have mastered the recursive nature of S-expressions.

Project 2: Create a Lisp Interpreter (A “lis.py”)

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Python, Scheme, Go
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Interpreters / Compilers / Language Design
  • Software or Tool: Your own Lisp!
  • Main Book: “Structure and Interpretation of Computer Programs” (SICP) - Chapter 4

What you’ll build: A program that can evaluate the data structures from Project 1. It will be a simple Lisp interpreter capable of handling basic arithmetic, variables, and function definitions.

Why it teaches Lisp: This is the eval part of the REPL. It demystifies how Lisp runs. You’ll implement the core eval/apply loop, which is the heart of every Lisp system, and gain a deep appreciation for the language’s elegant simplicity.

Core challenges you’ll face:

  • The eval/apply loop → maps to the core execution model of Lisp
  • Managing environments → maps to how variables and functions are scoped
  • Handling special forms vs. functions → maps to understanding that if, defun, etc., don’t evaluate all their arguments
  • Implementing functions and closures → maps to first-class functions and lexical scope

Key Concepts:

  • Evaluation: “(How to Write a (Lisp) Interpreter in Python)” - Peter Norvig
  • Environments: SICP, Chapter 3.2
  • Special Forms: “Practical Common Lisp” Ch. 5 - Peter Seibel

Difficulty: Intermediate Time estimate: 1-2 weeks

  • Prerequisites: Project 1. Solid understanding of recursion and data structures (like hash maps for the environment).

Real world outcome: You’ll have a working REPL for your own tiny Lisp dialect.

> (defvar x 10)
=> X
> (+ x 5)
=> 15
> (defun square (n) (* n n))
=> SQUARE
> (square x)
=> 100

Implementation Hints:

Your eval function is a giant if/else or cond block:

  • If expression is a number/string: return it.
  • If expression is a symbol: look it up in the environment.
  • If expression is a list:
    • The car (first element) determines what to do.
    • If car is a special form (like if, defun, quote): Handle it specially. For example, for if, you eval the condition, and then eval only one of the branches.
    • If car is a function name: Evaluate all the other elements of the list (the arguments), then apply the function to the evaluated arguments.

Your environment can be a hash map (or an association list) that maps symbols to values. For functions, you’ll need to handle nested scopes correctly (lexical scoping).

Learning milestones:

  1. You can evaluate basic arithmetic → Core eval/apply loop works.
  2. You can define and use variables → Environment management is working.
  3. You can define and call your own functions → You’ve implemented closures and lexical scope.
  4. Special forms like if and quote work → You understand the difference between standard evaluation and special evaluation.

Project 3: A Macro-Powered HTML DSL

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Clojure, Racket
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Metaprogramming / Domain Specific Languages (DSLs)
  • Software or Tool: A web-templating engine
  • Main Book: “On Lisp” by Paul Graham - Chapters 7-10

What you’ll build: A set of Lisp macros that allow you to write HTML using Lisp S-expressions. Instead of writing <p class="foo">Hello</p>, you’ll write (:p :class "foo" "Hello").

Why it teaches Lisp: This is it. This is the Lisp superpower. This project is your first real taste of metaprogramming. You are no longer just writing code; you are writing code that writes code, extending the language to create your own mini-language for generating HTML.

Core challenges you’ll face:

  • Writing your first macro → maps to understanding defmacro
  • Using backquote, comma, and comma-at → maps to mastering the macro templating system ( ,,@`)
  • Generating code, not values → maps to the fundamental difference between functions and macros
  • Handling nested tags → maps to recursive macro expansion

Key Concepts:

  • Macros: “Practical Common Lisp” Ch. 8 - Peter Seibel
  • Quasiquotation (Backquote): “On Lisp” Ch. 7 - Paul Graham
  • DSLs: Martin Fowler’s article on “Domain Specific Languages”

Difficulty: Intermediate Time estimate: Weekend Prerequisites: Project 2 (or a solid grasp of Lisp basics).

Real world outcome: Your Lisp REPL becomes an HTML generator.

CL-USER> (macroexpand-1
           '(:html
             (:head (:title "My Page"))
             (:body
              (:h1 "Welcome!")
              (:p "This is my page."))))

=> (CONCATENATE 'STRING
      "<HTML>"
      (CONCATENATE 'STRING
        "<HEAD>"
        (CONCATENATE 'STRING "<TITLE>" "My Page" "</TITLE>")
        "</HEAD>")
      (CONCATENATE 'STRING
        "<BODY>"
        (CONCATENATE 'STRING "<H1>" "Welcome!" "</H1>")
        (CONCATENATE 'STRING "<P>" "This is my page." "</P>")
        "</BODY>")
      "</HTML>")
T

And running the code would output the full HTML string.

Implementation Hints:

Start with a single tag macro.

(defmacro with-tag (name &body body)
  `(progn
     (format t "<~A>" ,name)
     ,@body
     (format t "</~A>" ,name)))

This is a naive start. A better approach is to write a single macro, say :tag, that generates the code for any given tag name and its content. This involves processing the arguments to separate attributes from the body content. Use recursion to process the body, which will contain more calls to your macros.

The key is that (:p "hello") doesn’t return a string. It expands into Lisp code that, when run, will return a string.

Learning milestones:

  1. You can generate a single tag → You’ve written a basic macro.
  2. You can nest tags → You understand recursive macro expansion.
  3. You can add attributes → You are parsing arguments within a macro.
  4. You start thinking of problems as “what kind of language would solve this best?” → You’ve internalized the Lisp philosophy.

Project 4: Text-Based Adventure Game (Zork-like)

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Scheme
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: State Management / Data Structures
  • Software or Tool: A simple game engine
  • Main Book: “Land of Lisp” by Conrad Barski

What you’ll build: A classic text adventure game where you can move between rooms, pick up items, and interact with your environment using commands like (walk north), (look), and (pickup sword).

Why it teaches Lisp: It’s a fantastic exercise in data representation. How do you represent the game world, the player’s inventory, the connections between rooms? In Lisp, the answer is often “with lists!” You’ll get very comfortable with property lists (plists), association lists (alists), and manipulating them. It also highlights REPL-driven development.

Core challenges you’ll face:

  • Representing the game world → maps to choosing the right Lisp data structures (plists, structs)
  • Parsing user commands → maps to processing list-based input
  • Managing game state → maps to understanding how and when to modify data
  • Creating a game loop → maps to the core of REPL-driven development

Key Concepts:

  • Data Structures: “Practical Common Lisp” Ch. 10 & 11 - Peter Seibel
  • REPL-Driven Development: The interactive loop of read, eval, print.
  • Global Variables (Dynamic Scope): Using *special-variables* to hold global state like the player’s location.

Difficulty: Beginner Time estimate: 1-2 weeks Prerequisites: Basic Lisp syntax.

Real world outcome: A running game in your terminal.

You are in the grand hall. There is a door to the NORTH.
A rusty key is on the floor.
> (walk north)
The door is locked.
> (pickup key)
You pick up the rusty key.
> (walk north)
You are in the treasury. It is full of gold!

Implementation Hints: Represent your world using lists of property lists.

(defvar *world*
  '((living-room (description "A cozy room.")
                 (exits (north dining-room) (west kitchen)))
    (dining-room (description "A long table with dusty plates.")
                 (exits (south living-room)))
    ...))

Your game loop will:

  1. Print the description of the current room.
  2. Read a command from the user (which is already an S-expression!).
  3. Evaluate that command, which will be a function call like (walk 'north).
  4. Update the game state (e.g., change the player’s location).
  5. Repeat.

Learning milestones:

  1. You can describe a room → You’ve built your data structures.
  2. You can move between rooms → You’re manipulating game state.
  3. You can pick up and use items → You’re managing nested state (player inventory).
  4. You add a new command without restarting the program → You’ve embraced REPL-driven development.

Project 5: Symbolic Mathematics Differentiator

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Scheme, Racket
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Symbolic Computation / AI / Recursion
  • Software or Tool: A symbolic algebra system
  • Main Book: “Structure and Interpretation of Computer Programs” (SICP) - Chapter 2.3.2

What you’ll build: A function that takes a mathematical expression in prefix S-expression format, and returns the S-expression for its derivative.

Why it teaches Lisp: This is a classic AI problem and a perfect demonstration of “code as data.” The formula (+ x 3) is not just code to be run; it’s a data structure (:+, :X, 3) that your program can analyze and transform. You’ll implement the rules of differentiation by recursively transforming this data structure.

Core challenges you’ll face:

  • Representing expressions as data → maps to S-expressions are trees
  • Implementing differentiation rules → maps to recursive data transformation
  • Simplifying the resulting expression → maps to more data transformation
  • Dispatching on operator type → maps to using cond or case to handle different parts of the expression tree

Key Concepts:

  • Symbolic Programming: SICP, Chapter 2.3
  • Data-Directed Programming: Using data (the operator symbol) to direct program flow.
  • Tree Recursion: Recursively processing a tree data structure.

Difficulty: Intermediate Time estimate: Weekend Prerequisites: Understanding of Lisp data structures and recursion. Basic calculus knowledge.

Real world outcome: Your Lisp can now do calculus symbolically.

> (differentiate '(* x x) 'x)
=> (+ (* 1 X) (* X 1))

;; After adding a simplifier
> (differentiate '(* x x) 'x)
=> (* 2 X)

> (differentiate '(+ (* 2 x) 5) 'x)
=> 2

Implementation Hints: Write a function (deriv expression variable). It will be a cond that checks the type of expression:

  • If expression is a number, the derivative is 0.
  • If expression is a symbol:
    • If it’s the same as variable, the derivative is 1.
    • Otherwise, it’s 0.
  • If expression is a list:
    • Check the operator (the car of the list).
    • If it’s +, the derivative is (list '+ (deriv (second expression) variable) (deriv (third expression) variable)). This should be constructed as (+, (deriv ...), (deriv ...)).
    • If it’s *, apply the product rule: f'g + fg'. This will be a new list: (list '+ (list '* (deriv f) g) (list '* f (deriv g))).
    • Implement rules for -, /, and ^.

Learning milestones:

  1. You can differentiate constants and variables → Base cases of recursion work.
  2. You can differentiate sums and products → You are recursively building new data structures.
  3. You implement a simplifier → You are performing another pass of data transformation (e.g., (+ x 0) becomes x, (* x 1) becomes x).
  4. You realize any rule-based system can be modeled this way → The power of symbolic programming clicks.

Project 6: A Rule-Based Expert System (like MYCIN)

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Prolog (for comparison), Scheme
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: AI / Logic Programming / Pattern Matching
  • Software or Tool: A simple inference engine
  • Main Book: “Paradigms of Artificial Intelligence Programming” (PAIP) by Peter Norvig

What you’ll build: A simple forward-chaining inference engine. You’ll define a set of facts and a set of rules. The system will use the rules to infer new facts until no more inferences can be made.

Why it teaches Lisp: This project dives deep into pattern matching and symbolic processing. You’ll write code that searches for patterns in your “knowledge base” and triggers actions. It’s a great way to learn how to build complex logic systems and will likely lead you to build your own match macro.

Core challenges you’ll face:

  • Representing facts and rules → maps to using S-expressions for logical representation
  • The pattern-matching algorithm → maps to unifying patterns with facts
  • The inference loop (forward-chaining) → maps to repeatedly applying rules to derive new facts
  • Handling variables in patterns → maps to binding values during matching

Key Concepts:

  • Pattern Matching: PAIP, Chapter 6
  • Logic Programming: The core idea of Prolog, implemented in Lisp.
  • Rule-Based Systems: PAIP, Chapter 7

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Project 5, strong grasp of recursion and list manipulation.

Real world outcome: A working “thinking machine” that can solve simple logic puzzles.

;; Defining facts
(add-fact '(Socrates is a man))

;; Defining a rule
(add-rule '((?x is a man) -> (?x is mortal)))

;; Running the engine
> (run-inference)
Applying rule: ((?X IS A MAN) -> (?X IS MORTAL))
  with bindings: ((?X . SOCRATES))
New fact: (SOCRATES IS MORTAL)

> (query '(Socrates is mortal))
=> T

Implementation Hints:

  • Facts: A list of lists, e.g., `’((socrates is a man) (plato is a man))’.
  • Rules: A list of rule structures, e.g., '((rule1 (if (and (?x is a man))) (then (?x is mortal))))'. Note the ?` denoting a variable.
  • Matcher: A function (match pattern fact bindings) that tries to match a pattern like (?x is a man) with a fact like (socrates is a man). If it succeeds, it returns a new set of bindings, e.g., `((?x . socrates))’.
  • Engine Loop:
    1. Iterate through all rules.
    2. For each rule, try to match its antecedents (the if part) against all known facts.
    3. If a rule’s antecedents are all satisfied, use the bindings to instantiate its consequents (the then part) and add them as new facts.
    4. Repeat until a full pass produces no new facts.

Learning milestones:

  1. You can match literal patterns → The basic matching logic works.
  2. You can match patterns with variables → You’ve implemented unification.
  3. The inference engine can apply one rule → Forward chaining works for a single step.
  4. The engine runs until completion → The full inference loop is complete.

Project 7: Build a Simple Object System

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Scheme
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Object-Oriented Programming / Language Internals
  • Software or Tool: A mini-CLOS (Common Lisp Object System)
  • Main Book: “The Art of the Metaobject Protocol” by Gregor Kiczales et al.

What you’ll build: A simplified object system using only functions and closures. You’ll create a defclass-like macro and a send-message function to interact with objects.

Why it teaches Lisp: It shows that OOP is not a fundamental language feature, but a pattern that can be implemented. You’ll see how Lisp’s functional nature (especially closures) can be used to build an object system from scratch, giving you a much deeper understanding of both Lisp and OOP.

Core challenges you’ll face:

  • Encapsulating state → maps to using closures to create private variables
  • Implementing methods → maps to a dispatch table or cond inside the closure
  • Handling inheritance → maps to chaining environments or message-passing
  • Creating an elegant syntax → maps to writing macros (defclass, make-instance) to hide the implementation details

Key Concepts:

  • Closures: SICP, Chapter 3.2
  • Message Passing: The Smalltalk/Alan Kay model of OOP.
  • Metaobject Protocol (MOP): The idea that the object system itself is an object you can modify.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Strong understanding of functions as first-class citizens and closures.

Real world outcome: You can define and use objects with encapsulated state in your Lisp.

> (defclass counter () ((count :initform 0)))
> (defmethod increment ((c counter))
    (incf (slot-value c 'count)))
> (defmethod get-count ((c counter))
    (slot-value c 'count))

> (defvar my-counter (make-instance 'counter))
> (increment my-counter)
> (get-count my-counter)
=> 1

(Your implementation might use a syntax like (send-message my-counter 'increment) instead of generic functions).

Implementation Hints: A simple object is just a function (a closure) that acts as a dispatcher.

(defun make-counter ()
  (let ((count 0))
    (lambda (message)
      (cond ((eq message 'increment) (incf count))
            ((eq message 'get-count) count)
            (t (error "Unknown message"))))))

(let ((c1 (make-counter)))
  (funcall c1 'increment)   ; Call the closure with the message
  (funcall c1 'get-count))  ; Returns 1

Your job is to wrap this core idea in macros (defclass, defmethod) to make it look like a built-in feature. For inheritance, when an object receives a message it doesn’t understand, it can pass it along to its “parent” object.

Learning milestones:

  1. You can create an object with private state → You’ve successfully used a closure for encapsulation.
  2. The object can respond to messages → You’ve implemented a dispatch mechanism.
  3. You’ve written a defclass macro → You’re building a user-friendly API for your system.
  4. You’ve implemented single inheritance → You’ve managed to chain object behaviors.

Project 8: An Interactive Debugger with Restarts

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 4: Expert
  • Knowledge Area: Error Handling / Language Internals / Debugging
  • Software or Tool: A simplified Common Lisp debugger
  • Main Book: “Practical Common Lisp” - Chapter 19 (for using it), “On Lisp” (for implementing it)

What you’ll build: Implement a simplified version of the Common Lisp Condition System. You will create signal, handler-bind, and restart-case macros.

Why it teaches Lisp: This is one of Lisp’s most unique and powerful features, and what sets it apart from traditional try/catch exception handling. Building it forces you to understand the separation of detecting an error from handling it. You will see why Lisp is considered one of the best environments for building robust, long-running systems.

Core challenges you’ll face:

  • Managing the handler stack → maps to dynamically scoped handlers
  • Signaling vs. throwing → maps to the difference between raising a non-local control transfer and simply notifying
  • Finding and invoking restarts → maps to letting the lower-level code offer recovery options to the higher-level code
  • Implementing the macros → maps to complex metaprogramming to manage the dynamic environment

Key Concepts:

  • Condition System: “Practical Common Lisp” Ch. 19
  • Dynamic Scope: In contrast to lexical scope, variables are looked up in the call stack.
  • Non-Local Exits: The mechanism behind throw/catch that underpins the system.

Difficulty: Expert Time estimate: 2-3 weeks

  • Prerequisites: Project 7, deep understanding of macros, scope, and function calling conventions.

Real world outcome: You can handle errors in a way that allows for interactive recovery.

(defun parse-number-or-retry (str)
  (restart-case (parse-integer str)
    (use-value (v) ; A restart named 'use-value'
      v)
    (retry () ; A restart named 'retry'
      (parse-number-or-retry str))))

(defun robust-add (a b)
  (handler-bind ((parse-error
                   (lambda (c)
                     (format *query-io* "Invalid number found. What to do?~%)
                     (invoke-restart 'use-value 0)))) ; Handle error by providing 0
    (+ (parse-number-or-retry a) (parse-number-or-retry b))))

> (robust-add "10" "foo")
Invalid number found. What to do?
=> 10 ; ("foo" was replaced with 0)

Implementation Hints:

  • Use a special (dynamically scoped) variable, say *handler-stack*.
  • handler-bind is a macro that pushes a new handler onto this stack and then executes its body. The handler is a function that will be called if a condition is signaled.
  • signal is a function that traverses *handler-stack* looking for a handler that matches the signaled condition. If it finds one, it calls it.
  • restart-case is a macro that makes a set of recovery functions (restarts) available. invoke-restart then searches for and calls one of these. This requires another special variable to hold available restarts.

This is a very advanced project that touches the deepest parts of what makes Lisp’s environment so powerful.

Learning milestones:

  1. signal can find and invoke a handler → You have basic condition signaling.
  2. handler-bind correctly manages the dynamic stack of handlers → You’ve mastered dynamic scope.
  3. restart-case can offer recovery options → You have separated the error detection from the recovery policy.
  4. invoke-restart can jump back into the context of the error and continue → You have built a truly interactive and resumable error handling system.

Project 9: Emacs Lisp Productivity Script

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Emacs Lisp
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Scripting / Editor Customization / Practical Lisp
  • Software or Tool: Emacs
  • Main Book: “An Introduction to Programming in Emacs Lisp” by Robert J. Chassell

What you’ll build: A custom interactive function for Emacs that automates a repetitive task. For example, a function that takes the current line, treats it as a task, and appends it to an org-mode file with a timestamp.

Why it teaches Lisp: This is Lisp in the wild! Emacs is a “Lisp machine” disguised as a text editor. You’ll learn Lisp in a live, practical, and highly interactive environment. You will manipulate buffers, text, and the editor’s state, all using Lisp.

Core challenges you’ll face:

  • Interacting with the Emacs environment → maps to learning the Emacs Lisp API for buffers, windows, and text
  • Writing an interactive function → maps to using (interactive) to get input from the user/editor
  • Manipulating text programmatically → maps to functions for searching, inserting, and deleting text
  • Configuring Emacs to load your script → maps to understanding the Emacs initialization process

Key Concepts:

  • Emacs Lisp Basics: “An Introduction to Programming in Emacs Lisp”
  • Interactive Functions: The (interactive) form.
  • Buffer-Local Variables: How Emacs manages state for different files.

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic familiarity with Emacs as a text editor.

Real world outcome: You press a key combination (e.g., M-x log-task), and the current line in any file is automatically added to your to-do list, saving you time every day.

;; In your init.el or a separate file
(defun log-current-line-as-task ()
  "Take the current line and append it as a task to my tasks.org file."
  (interactive)
  (let ((task (thing-at-point 'line t))) ; Get current line text
    (with-current-buffer (find-file-noselect "~/tasks.org")
      (goto-char (point-max))
      (insert (format "\n* TODO %s\n  SCHEDULED: <%s>" 
                      (trim-string task)
                      (format-time-string "%Y-%m-%d"))))))

(global-set-key (kbd "C-c l") 'log-current-line-as-task)

Pressing Ctrl-c l now runs your Lisp code.

Learning milestones:

  1. You write a function that prints “Hello World” in the minibuffer → You’ve run your first piece of Elisp.
  2. You can programmatically insert text into the current buffer → You’re manipulating the editor state.
  3. You create an interactive function bound to a key → You have extended the editor with a new command.
  4. You find yourself writing small Elisp snippets to automate everything → You are thinking in Lisp.

Project 10: A Lisp in Lisp (Self-Hosting Interpreter)

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Scheme (for its minimalism)
  • Alternative Programming Languages: Common Lisp
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 4: Expert
  • Knowledge Area: Compilers / Bootstrapping / Metaprogramming
  • Software or Tool: A self-hosting Lisp
  • Main Book: “Lisp in Small Pieces” by Christian Queinnec

What you’ll build: Take the Lisp interpreter you built in Project 2 and rewrite it in its own language. You’ll define a minimal subset of your Lisp that is powerful enough to interpret itself.

Why it teaches Lisp: This is a classic, mind-bending exercise in bootstrapping and reflection. It proves the Turing-completeness of your language and demonstrates the ultimate expression of “code as data.” It forces you to think about what the absolute minimal core of a language is.

Core challenges you’ll face:

  • Defining the minimal core language → maps to what are the essential primitives (if, quote, cons, car, cdr, lambda)
  • The bootstrapping problem → maps to how do you run the interpreter before it can interpret itself?
  • Implementing eval and apply in your own Lisp → maps to deep recursion and symbolic manipulation
  • Debugging a compiler with itself → maps to a uniquely challenging and rewarding debugging experience

Key Concepts:

  • Metacircular Evaluation: SICP, Chapter 4.1
  • Bootstrapping: The process of writing a compiler in the language it compiles.
  • Language Core vs. Library: Understanding what must be primitive vs. what can be defined in the language itself.

Difficulty: Expert Time estimate: 1 month+ Prerequisites: Project 2. A deep and comfortable understanding of evaluation models.

Real world outcome: You have two versions of your interpreter: one written in a host language (e.g., Python) and one written in your own Lisp (meta-lisp.lisp). You can use the Python one to run meta-lisp.lisp, which can then interpret other Lisp programs. You have reached a new level of enlightenment.

# The host interpreter (in Python) runs the Lisp-based interpreter
$ python host_lisp.py meta_lisp.lisp

# Now the meta-interpreter is running, and you can give it code
> (in meta-lisp) (+ 1 2)
=> 3

Implementation Hints: The metacircular evaluator is famously short and elegant. The core of eval will look something like this (in Lisp itself):

(defun eval (expr env)
  (cond ((self-evaluating? expr) expr)
        ((variable? expr) (lookup-variable-value expr env))
        ((quoted? expr) (text-of-quotation expr))
        ((assignment? expr) (eval-assignment expr env))
        ((definition? expr) (eval-definition expr env))
        ((if? expr) (eval-if expr env))
        ((lambda? expr) (make-procedure (lambda-parameters expr)
                                        (lambda-body expr)
                                        env))
        ((application? expr)
         (apply (eval (operator expr) env)
                (list-of-values (operands expr) env)))
        (t (error "Unknown expression type -- EVAL" expr))))

This is taken directly from SICP. Your task is to implement all the helper functions (self-evaluating?, lookup-variable-value, etc.) in your mini-Lisp. You’ll start by running this code with your host interpreter (e.g., the one from Project 2).

Learning milestones:

  1. You define the eval function in your own Lisp → You have written down the specification of your language.
  2. The host interpreter can run your meta-interpreter to evaluate a simple expression like (+ 1 2) → The bootstrap works for a single step.
  3. The meta-interpreter can evaluate a user-defined function → lambda and apply are working correctly in the meta-level.
  4. You understand that your evaluator is just another Lisp program → You have ascended.

Project 11: A Pattern Matching Library

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Common Lisp
  • Alternative Programming Languages: Scheme
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Metaprogramming / DSLs / Compilers
  • Software or Tool: A language extension
  • Main Book: “Let Over Lambda” by Doug Hoyte

What you’ll build: A powerful match macro that brings ML-style pattern matching to Lisp. It will allow you to destructure nested data structures cleanly.

Why it teaches Lisp: It’s a masterclass in writing complex macros. You will be writing a macro that not only generates code but also needs to parse a “mini-language” (the patterns), bind variables, and construct potentially complex conditional logic, all at compile time.

Core challenges you’ll face:

  • Parsing the pattern syntax → maps to macros as parsers
  • Generating code for destructuring → maps to using let, when, and to bind and check values
  • Handling variable bindings → maps to creating new variable bindings from inside the macro
  • Supporting guards and complex patterns → maps to extending your mini-language

Key Concepts:

  • Advanced Macros: “Let Over Lambda”
  • Code Generation: Your macro will be a small compiler.
  • Destructuring: A common feature in functional languages.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Project 3, very comfortable with macros.

Real world outcome: You can refactor complex, nested if/cond expressions into clean, declarative patterns.

;; Before
(defun process-command (cmd)
  (if (and (listp cmd) (eq (first cmd) 'move))
      (let ((x (second cmd))
            (y (third cmd)))
        (format t "Moving to ~A, ~A" x y))
      (if (and (listp cmd) (eq (first cmd) 'attack))
          (let ((target (second cmd)))
            (format t "Attacking ~A" target))
          (format t "Unknown command"))))

;; After
(defun process-command (cmd)
  (match cmd
    (`(move ,x ,y) (format t "Moving to ~A, ~A" x y))
    (`(attack ,target) (format t "Attacking ~A" target))
    (_ (format t "Unknown command"))))

Implementation Hints: Your match macro will take an expression and a series of (pattern body) clauses. It will expand into a series of nested cond or if statements.

For a clause ((move ,x ,y) (format …))`, the macro should generate code that looks roughly like:

  1. Is the value a list?
  2. Is its car the symbol move?
  3. Does it have exactly 3 elements?
  4. If all are true, create a let binding for x to (second value) and y to (third value), and then execute the body.
  5. If not, try the next clause.

The underscore _ is a “don’t care” pattern that should always match.

Learning milestones:

  1. You can match on literal atoms and lists → Basic pattern matching works.
  2. You can bind variables in patterns → Your macro is generating let bindings.
  3. You can handle nested patterns → Your macro logic is recursive.
  4. You add support for _ and guards (e.g., ((move ,x ,y) :when (> x 0))`) → You are building a feature-rich language extension.

Project 12: A Lazy Language with Thunks

  • File: LEARN_LISP_DEEP_DIVE.md
  • Main Programming Language: Scheme
  • Coolness Level: Level 5: Pure Magic (Super Cool)
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 5: Master
  • Knowledge Area: Language Design / Compilers / Functional Programming
  • Software or Tool: A lazy interpreter
  • Main Book: “Structure and Interpretation of Computer Programs” (SICP) - Chapter 4.2

What you’ll build: Modify your Lisp interpreter (from Project 2) to be lazy, like Haskell. Arguments to functions won’t be evaluated until they are actually used. This will allow you to work with infinite data structures.

Why it teaches Lisp: It demonstrates the power of Lisp’s malleability. You can fundamentally change the evaluation strategy of the language itself. It forces a deep understanding of the relationship between the evaluator, functions, and arguments, and it’s a direct gateway to understanding advanced functional programming concepts.

Core challenges you’ll face:

  • Delaying evaluation → maps to creating “thunks” (a zero-argument function that holds an expression to be evaluated later)
  • Forcing evaluation → maps to calling the thunk when the value is needed
  • Memoization → maps to storing the result of a thunk so it’s only evaluated once
  • Changing the apply logic → maps to apply now passes thunks instead of values

Key Concepts:

  • Lazy Evaluation: SICP, Chapter 4.2
  • Thunks: The data structure representing a delayed computation.
  • Infinite Data Structures (Streams): “SICP” Chapter 3.5

Difficulty: Master Time estimate: 1 month+

  • Prerequisites: Project 2 and Project 10. You need to be extremely comfortable with how eval and apply work.

Real world outcome: Your Lisp can now handle infinite lists.

;; Define an infinite list of integers starting from n
(defun integers-from (n)
  (cons n (integers-from (+ n 1))))

;; Create the infinite list of all positive integers
(defvar all-integers (integers-from 1))

;; Take the first 10 elements
> (take 10 all-integers)
=> (1 2 3 4 5 6 7 8 9 10)

;; Without lazy evaluation, `(integers-from 1)` would loop forever.

Implementation Hints:

  • The core idea is to change what happens when a function is called. Instead of evaluating the arguments before calling the function, you wrap each argument expression in a lambda to create a thunk.
  • Your apply function will now pass these thunks to the function body.
  • Inside the function body, when a variable is accessed, you must “force” the thunk to get its value.
  • The force function checks if a thunk has already been evaluated (memoization). If so, it returns the stored value. If not, it calls the thunk, stores the result, and then returns it.
  • Special forms like if are critical. They must force the condition, but not the branches, maintaining laziness.

Learning milestones:

  1. You have implemented delay and force → You have the core mechanics of lazy evaluation.
  2. Function applications are now lazy → You have successfully modified your evaluator.
  3. You can define and manipulate an infinite stream → The power of lazy evaluation is unlocked.
  4. You understand the trade-offs between lazy and eager evaluation → You have a deep, practical understanding of language design choices.

Summary

Project Difficulty Time Main Concept Main Language
1. S-expression Parser Beginner Weekend Homoiconicity Common Lisp
2. Lisp Interpreter Intermediate 1-2 weeks Evaluation Common Lisp
3. HTML DSL Intermediate Weekend Macros / DSLs Common Lisp
4. Text Adventure Game Beginner 1-2 weeks Data Structures Common Lisp
5. Symbolic Differentiator Intermediate Weekend Symbolic AI Common Lisp
6. Expert System Advanced 1-2 weeks Logic Programming Common Lisp
7. Simple Object System Advanced 1-2 weeks Closures / MOP Common Lisp
8. Interactive Debugger Expert 2-3 weeks Condition System Common Lisp
9. Emacs Lisp Script Beginner Weekend Practical Scripting Emacs Lisp
10. Lisp in Lisp Expert 1 month+ Metacircular Eval Scheme
11. Pattern Matching Advanced 1-2 weeks Advanced Macros Common Lisp
12. Lazy Language Master 1 month+ Lazy Evaluation Scheme

```