Project 5: Calling Convention Visualizer

A tool that visualizes argument passing and stack layout for SysV and Windows x64 calls.

Quick Reference

Attribute Value
Difficulty Level 4
Time Estimate 2 weeks
Main Programming Language Python or C (Alternatives: Rust, Go)
Alternative Programming Languages Rust, Go
Coolness Level Level 4
Business Potential 1
Prerequisites Control Flow, Stack, and Calling Conventions, Control Flow, Stack, and Calling Conventions
Key Topics Control Flow, Stack, and Calling Conventions, Control Flow, Stack, and Calling Conventions

1. Learning Objectives

By completing this project, you will:

  1. Explain why calling convention visualizer reveals key x86-64 behaviors.
  2. Build a deterministic tool with clear, inspectable output.
  3. Validate correctness against a golden reference output.
  4. Connect the tool output to ABI and architecture rules.
  5. The ABI is the glue between functions and binaries.

2. All Theory Needed (Per-Concept Breakdown)

Control Flow, Stack, and Calling Conventions

Fundamentals Control flow in x86-64 is built from instruction pointer changes: calls, returns, jumps, and conditional branches. The stack provides a structured way to save state, pass arguments, and return values. Calling conventions define the contract between caller and callee: which registers hold arguments, which registers must be preserved, how the stack is aligned, and where return values appear. The two dominant 64-bit ABIs are System V AMD64 (used by Linux and macOS) and the Windows x64 convention. They share ideas but differ in register usage, red zone rules, and stack shadow space. These conventions are defined in official ABI documents and OS documentation. (Sources: System V AMD64 ABI, Microsoft x64 calling convention docs)

Deep Dive The stack is a contiguous region of memory that grows downward. Every call pushes a return address and often creates a stack frame for local storage and saved registers. A function prologue typically adjusts the stack pointer, saves callee-saved registers, and sets up a frame pointer (optional). The epilogue reverses these steps and returns to the caller. This is a convention, not a requirement; optimized code can omit a frame pointer or use a leaf function that never touches the stack. Still, understanding the standard layout is critical for debugging, unwinding, and ABI interoperability.

System V AMD64 ABI defines the primary calling convention for UNIX-like systems. The first six integer or pointer arguments are passed in registers; additional arguments are passed on the stack. Return values are placed in a designated register. The stack must be aligned to a 16-byte boundary at call sites. The ABI also defines a red zone: a small region below the stack pointer that is not touched by signal handlers, allowing leaf functions to use it without adjusting the stack. This subtle rule impacts code generation and is why some stack probes are optional in user code. The ABI also specifies which registers are caller-saved and callee-saved. Caller-saved registers can be clobbered by the callee; callee-saved must be preserved across the call. Understanding these rules allows you to read disassembly and reconstruct the calling context.

Windows x64 uses a different register assignment and requires a 32-byte shadow space (home space) on the stack for the first four arguments, even if they are passed in registers. The callee can use this space to spill arguments, and the caller must allocate it. Windows x64 does not define a red zone, so leaf functions cannot safely use space below the stack pointer. This means that calling convention errors between platforms are common sources of crashes when interfacing with mixed environments or when porting low-level code.

Control flow also includes indirect calls and jumps, which use register or memory operands. These are common in virtual dispatch, function pointers, and dynamic linking. Understanding the ABI rules helps you determine whether a given register contains a valid function pointer or an argument. The stack alignment rule is critical for SIMD operations; misalignment can cause crashes or performance penalties. The ABI is therefore both a functional contract and a performance contract.

Unwinding is another piece of the puzzle. Debuggers and exception handlers need to reconstruct call stacks after a crash or during stack walking. This uses metadata (DWARF on UNIX, PDB on Windows) that describes how to restore registers and adjust the stack. Even if you are not writing that metadata, you must follow the ABI so that compilers and tools can generate it correctly.

How this fits on projects

  • Projects 5 and 6 are focused on calling conventions, stack layout, and unwinding.
  • Projects 7-8 rely on accurate control flow interpretation to map syscalls and exceptions.

Definitions & key terms

  • Calling convention: Contract between caller and callee.
  • Callee-saved: Registers preserved by the callee.
  • Caller-saved: Registers that may be clobbered by the callee.
  • Red zone: Stack space below SP reserved for leaf functions (SysV).
  • Shadow space: Stack space reserved for argument spilling (Windows).

Mental model diagram

CALLER STACK (higher addresses)
+----------------------------+
| argN (stack)               |
| ...                        |
| return address             | <-- pushed by call
| saved regs / locals        |
+----------------------------+
| red zone (SysV only)       |
+----------------------------+ <-- current SP

CALLER REGISTERS (SysV)
ARG1..ARG6 -> REG_A..REG_F
RETURN      -> REG_RET

How it works

  1. Caller places arguments in registers/stack per ABI.
  2. Caller ensures stack alignment and reserves shadow space (Windows).
  3. Call instruction pushes return address and jumps to callee.
  4. Callee saves required registers and sets up locals.
  5. Callee returns value in return register.
  6. Callee restores registers and returns to caller.

Invariants and failure modes:

  • Invariant: Stack alignment at call boundaries.
  • Failure: Misalignment breaks SIMD usage or ABI compliance.
  • Invariant: Callee-saved registers are preserved.
  • Failure: Clobbered callee-saved registers corrupt callers.

Minimal concrete example (pseudo-assembly, not real code)

# PSEUDOCODE ONLY
CALLER:
  ARG_REG1 = VAL1
  ARG_REG2 = VAL2
  ALIGN_STACK_16
  CALL FUNC_X
  RESULT = RET_REG

Common misconceptions

  • “Calling conventions are optional.” They are required for interoperability.
  • “Stack alignment only matters for performance.” It can break correctness.
  • “Red zone exists everywhere.” It is SysV-only and not on Windows.

Check-your-understanding questions

  1. Why does Windows require shadow space?
  2. Why do ABIs define caller-saved vs callee-saved registers?
  3. What breaks if stack alignment is wrong?

Check-your-understanding answers

  1. It gives the callee guaranteed spill space for register arguments.
  2. It establishes a clear contract and enables efficient codegen.
  3. SIMD operations and ABI compliance can fail or crash.

Real-world applications

  • ABI debugging and crash analysis
  • Interfacing assembly with C/C++
  • Reverse engineering function boundaries

Where you will apply it Projects 5, 6, 7

References

  • System V AMD64 ABI Draft 0.99.7
  • Microsoft x64 calling convention documentation
  • “The Art of 64-Bit Assembly, Volume 1” by Randall Hyde - Ch. 4-6

Key insights ABIs are the social contract that makes assembly code usable across compilers and OSes.

Summary Control flow and the stack are simple ideas, but ABIs make them reliable and interoperable.

Homework/Exercises to practice the concept

  • Draw a stack frame for a function with 2 arguments and 3 locals.
  • Identify which registers must be preserved in SysV and Windows.

Solutions to the homework/exercises

  • Include return address, saved registers, locals, and alignment padding.
  • SysV preserves specific callee-saved regs; Windows preserves its own set.

    Control Flow, Stack, and Calling Conventions

Fundamentals Control flow in x86-64 is built from instruction pointer changes: calls, returns, jumps, and conditional branches. The stack provides a structured way to save state, pass arguments, and return values. Calling conventions define the contract between caller and callee: which registers hold arguments, which registers must be preserved, how the stack is aligned, and where return values appear. The two dominant 64-bit ABIs are System V AMD64 (used by Linux and macOS) and the Windows x64 convention. They share ideas but differ in register usage, red zone rules, and stack shadow space. These conventions are defined in official ABI documents and OS documentation. (Sources: System V AMD64 ABI, Microsoft x64 calling convention docs)

Deep Dive The stack is a contiguous region of memory that grows downward. Every call pushes a return address and often creates a stack frame for local storage and saved registers. A function prologue typically adjusts the stack pointer, saves callee-saved registers, and sets up a frame pointer (optional). The epilogue reverses these steps and returns to the caller. This is a convention, not a requirement; optimized code can omit a frame pointer or use a leaf function that never touches the stack. Still, understanding the standard layout is critical for debugging, unwinding, and ABI interoperability.

System V AMD64 ABI defines the primary calling convention for UNIX-like systems. The first six integer or pointer arguments are passed in registers; additional arguments are passed on the stack. Return values are placed in a designated register. The stack must be aligned to a 16-byte boundary at call sites. The ABI also defines a red zone: a small region below the stack pointer that is not touched by signal handlers, allowing leaf functions to use it without adjusting the stack. This subtle rule impacts code generation and is why some stack probes are optional in user code. The ABI also specifies which registers are caller-saved and callee-saved. Caller-saved registers can be clobbered by the callee; callee-saved must be preserved across the call. Understanding these rules allows you to read disassembly and reconstruct the calling context.

Windows x64 uses a different register assignment and requires a 32-byte shadow space (home space) on the stack for the first four arguments, even if they are passed in registers. The callee can use this space to spill arguments, and the caller must allocate it. Windows x64 does not define a red zone, so leaf functions cannot safely use space below the stack pointer. This means that calling convention errors between platforms are common sources of crashes when interfacing with mixed environments or when porting low-level code.

Control flow also includes indirect calls and jumps, which use register or memory operands. These are common in virtual dispatch, function pointers, and dynamic linking. Understanding the ABI rules helps you determine whether a given register contains a valid function pointer or an argument. The stack alignment rule is critical for SIMD operations; misalignment can cause crashes or performance penalties. The ABI is therefore both a functional contract and a performance contract.

Unwinding is another piece of the puzzle. Debuggers and exception handlers need to reconstruct call stacks after a crash or during stack walking. This uses metadata (DWARF on UNIX, PDB on Windows) that describes how to restore registers and adjust the stack. Even if you are not writing that metadata, you must follow the ABI so that compilers and tools can generate it correctly.

How this fits on projects

  • Projects 5 and 6 are focused on calling conventions, stack layout, and unwinding.
  • Projects 7-8 rely on accurate control flow interpretation to map syscalls and exceptions.

Definitions & key terms

  • Calling convention: Contract between caller and callee.
  • Callee-saved: Registers preserved by the callee.
  • Caller-saved: Registers that may be clobbered by the callee.
  • Red zone: Stack space below SP reserved for leaf functions (SysV).
  • Shadow space: Stack space reserved for argument spilling (Windows).

Mental model diagram

CALLER STACK (higher addresses)
+----------------------------+
| argN (stack)               |
| ...                        |
| return address             | <-- pushed by call
| saved regs / locals        |
+----------------------------+
| red zone (SysV only)       |
+----------------------------+ <-- current SP

CALLER REGISTERS (SysV)
ARG1..ARG6 -> REG_A..REG_F
RETURN      -> REG_RET

How it works

  1. Caller places arguments in registers/stack per ABI.
  2. Caller ensures stack alignment and reserves shadow space (Windows).
  3. Call instruction pushes return address and jumps to callee.
  4. Callee saves required registers and sets up locals.
  5. Callee returns value in return register.
  6. Callee restores registers and returns to caller.

Invariants and failure modes:

  • Invariant: Stack alignment at call boundaries.
  • Failure: Misalignment breaks SIMD usage or ABI compliance.
  • Invariant: Callee-saved registers are preserved.
  • Failure: Clobbered callee-saved registers corrupt callers.

Minimal concrete example (pseudo-assembly, not real code)

# PSEUDOCODE ONLY
CALLER:
  ARG_REG1 = VAL1
  ARG_REG2 = VAL2
  ALIGN_STACK_16
  CALL FUNC_X
  RESULT = RET_REG

Common misconceptions

  • “Calling conventions are optional.” They are required for interoperability.
  • “Stack alignment only matters for performance.” It can break correctness.
  • “Red zone exists everywhere.” It is SysV-only and not on Windows.

Check-your-understanding questions

  1. Why does Windows require shadow space?
  2. Why do ABIs define caller-saved vs callee-saved registers?
  3. What breaks if stack alignment is wrong?

Check-your-understanding answers

  1. It gives the callee guaranteed spill space for register arguments.
  2. It establishes a clear contract and enables efficient codegen.
  3. SIMD operations and ABI compliance can fail or crash.

Real-world applications

  • ABI debugging and crash analysis
  • Interfacing assembly with C/C++
  • Reverse engineering function boundaries

Where you will apply it Projects 5, 6, 7

References

  • System V AMD64 ABI Draft 0.99.7
  • Microsoft x64 calling convention documentation
  • “The Art of 64-Bit Assembly, Volume 1” by Randall Hyde - Ch. 4-6

Key insights ABIs are the social contract that makes assembly code usable across compilers and OSes.

Summary Control flow and the stack are simple ideas, but ABIs make them reliable and interoperable.

Homework/Exercises to practice the concept

  • Draw a stack frame for a function with 2 arguments and 3 locals.
  • Identify which registers must be preserved in SysV and Windows.

Solutions to the homework/exercises

  • Include return address, saved registers, locals, and alignment padding.
  • SysV preserves specific callee-saved regs; Windows preserves its own set.

3. Project Specification

3.1 What You Will Build

A tool that visualizes argument passing and stack layout for SysV and Windows x64 calls.

Why this teaches x86-64: The ABI is the glue between functions and binaries.

Included:

  • Deterministic CLI output for a fixed input
  • Clear mapping between inputs and architectural meaning
  • A small test suite with edge cases

Excluded:

  • Full compiler or full disassembler coverage
  • Production-grade UI or packaging

3.2 Functional Requirements

  1. Deterministic Output: Same input yields identical output.
  2. Architecture-Aware: Output references ABI/ISA rules where relevant.
  3. Validation Mode: Provide a compare mode against a golden output.

3.3 Non-Functional Requirements

  • Performance: Fast enough for small inputs and interactive use.
  • Reliability: Handles malformed inputs with clear errors.
  • Usability: Outputs are readable and documented.

3.4 Example Usage / Output

$ x64abi --sysv --args 8 --locals 3

CALL LAYOUT (SysV)
ARG_REGS: A1->REG_A  A2->REG_B  A3->REG_C  A4->REG_D  A5->REG_E  A6->REG_F
STACK_ARGS: ARG7, ARG8
STACK ALIGNMENT: 16-byte at call
RED ZONE: enabled

$ x64abi --win64 --args 8 --locals 3

CALL LAYOUT (Win64)
ARG_REGS: A1->REG_C  A2->REG_D  A3->REG_E  A4->REG_F
STACK_ARGS: ARG5..ARG8
SHADOW SPACE: 32 bytes reserved
RED ZONE: disabled

3.5 Data Formats / Schemas / Protocols

  • Input format: line-oriented text or hex bytes (documented in README)
  • Output format: stable, human-readable report with labeled fields

3.6 Edge Cases

  • Empty input or missing fields
  • Invalid numeric values or malformed hex
  • Inputs that exercise maximum/minimum bounds

3.7 Real World Outcome

This section is your golden reference. Match it exactly.

3.7.1 How to Run (Copy/Paste)

  • Build: (if needed) make or equivalent
  • Run: P05-calling-convention-visualizer with sample input
  • Working directory: project root

3.7.2 Golden Path Demo (Deterministic)

Run with the provided demo input and confirm output matches the transcript.

3.7.3 If CLI: exact terminal transcript

$ x64abi --sysv --args 8 --locals 3

CALL LAYOUT (SysV)
ARG_REGS: A1->REG_A  A2->REG_B  A3->REG_C  A4->REG_D  A5->REG_E  A6->REG_F
STACK_ARGS: ARG7, ARG8
STACK ALIGNMENT: 16-byte at call
RED ZONE: enabled

$ x64abi --win64 --args 8 --locals 3

CALL LAYOUT (Win64)
ARG_REGS: A1->REG_C  A2->REG_D  A3->REG_E  A4->REG_F
STACK_ARGS: ARG5..ARG8
SHADOW SPACE: 32 bytes reserved
RED ZONE: disabled

4. Solution Architecture

4.1 High-Level Design

INPUT -> PARSER -> MODEL -> RENDERER -> REPORT

4.2 Key Components

Component Responsibility Key Decisions
Parser Turn input into structured records Strict vs permissive parsing
Model Apply ISA/ABI rules Deterministic state transitions
Renderer Produce readable output Stable formatting

4.4 Data Structures (No Full Code)

  • Record: holds one instruction/event with decoded fields
  • State: represents register/flag or address state
  • Report: list of formatted output lines

4.4 Algorithm Overview

Key Algorithm: Parse and Evaluate

  1. Parse input into records.
  2. Apply rules to update state.
  3. Render the state and summary output.

Complexity Analysis:

  • Time: O(n) over input records
  • Space: O(n) for report output

5. Implementation Guide

5.1 Development Environment Setup

# Ensure basic tools are installed
# build-essential or clang, plus objdump/readelf if needed

5.2 Project Structure

project-root/
├── src/
│   ├── main.*
│   ├── parser.*
│   └── model.*
├── tests/
│   └── test_cases.*
└── README.md

5.3 The Core Question You’re Answering

How do different ABIs move arguments and manage the stack, and why does it matter?

5.4 Concepts You Must Understand First

  1. Calling conventions
    • Which registers are used for arguments?
    • Book Reference: “The Art of 64-Bit Assembly, Volume 1” - Ch. 4-6
  2. Stack alignment
    • Why does the stack need 16-byte alignment?
    • Book Reference: “Computer Systems: A Programmer’s Perspective” - Ch. 3

5.5 Questions to Guide Your Design

  1. Visualization
    • How will you represent the stack frame in output?
    • How will you show preserved vs volatile registers?
  2. Rules engine
    • How will you encode ABI rules for each platform?
    • How will you handle variadic or mixed argument types?

5.6 Thinking Exercise

ABI Walkthrough

Sketch the register and stack layout for a function with 8 integer arguments on both SysV and Windows.

Questions to answer:

  • Which arguments spill to the stack?
  • Where is alignment padding needed?

5.7 The Interview Questions They’ll Ask

  1. “How do SysV and Windows x64 calling conventions differ?”
  2. “What is the red zone and why does it exist?”
  3. “Why does Windows require shadow space?”
  4. “Which registers are callee-saved?”
  5. “How do variadic arguments affect the ABI?”

5.8 Hints in Layers

Hint 1: Starting Point Create a static table for each ABI: arg registers, caller-saved, callee-saved.

Hint 2: Next Level Compute the stack layout with alignment rules and show a diagram.

Hint 3: Technical Details Model the stack as a list of slots with offsets from SP.

Hint 4: Tools/Debugging Cross-check with ABI documentation examples and known compiler output.

5.9 Books That Will Help

Topic Book Chapter
ABI fundamentals “The Art of 64-Bit Assembly, Volume 1” Ch. 4-6
Stack layout “Computer Systems: A Programmer’s Perspective” Ch. 3

5.10 Implementation Phases

Phase 1: Foundation (2-3 days)

Goals:

  • Parse input format
  • Produce a minimal output Tasks:
    1. Define input grammar and example files.
    2. Implement a minimal parser and renderer. Checkpoint: Golden output matches a small input.

Phase 2: Core Functionality (1 week)

Goals:

  • Implement full rule set
  • Add validation and errors Tasks:
    1. Implement rule engine for core cases.
    2. Add error handling for invalid inputs. Checkpoint: All core tests pass.

Phase 3: Polish & Edge Cases (2-3 days)

Goals:

  • Add edge-case coverage
  • Improve output readability Tasks:
    1. Add edge-case tests.
    2. Refine output formatting and summary. Checkpoint: Output matches golden transcript for all cases.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Input format Text, JSON Text Easiest to audit and diff
Output format Plain text, JSON Plain text Matches CLI tooling

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests Validate parsing and rule application Valid/invalid inputs
Integration Tests End-to-end output comparison Golden transcripts
Edge Case Tests Stress unusual inputs Empty input, max values

6.2 Critical Test Cases

  1. Minimal Input: One record, verify output.
  2. Boundary Values: Largest/smallest values.
  3. Malformed Input: Ensure clean error messages.

6.3 Test Data

INPUT: sample_min.txt
EXPECTED: matches golden transcript

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Wrong assumptions Output mismatches Re-read ABI/ISA rules
Off-by-one parsing Missing fields Add explicit length checks
Ambiguous output Hard to verify Add labels and separators

Project-specific pitfalls

Problem 1: “Arguments are mapped to wrong registers”

  • Why: Mixing SysV and Windows rules.
  • Fix: Keep separate rule tables and tests for each ABI.
  • Quick test: Use a known 4-arg example and verify output.

7.2 Debugging Strategies

  • Golden diffing: Use diff to compare outputs line by line.
  • State logging: Print intermediate state after each step.

7.3 Performance Traps

  • Avoid over-optimizing; correctness and determinism matter most.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a new input case and golden output
  • Add a summary line with counts

8.2 Intermediate Extensions

  • Add JSON output mode
  • Add validation warnings for suspicious inputs

8.3 Advanced Extensions

  • Support additional ABI or instruction variants
  • Integrate with a real binary to collect inputs

9. Real-World Connections

9.1 Industry Applications

  • Profilers and tracers: Use similar decoding and state models.
  • Security analysis: Use precise ABI knowledge to interpret crashes.
  • objdump: reference tool for binary inspection.
  • llvm-objdump: LLVM-based disassembly and inspection.

9.3 Interview Relevance

  • ABI and calling conventions are common systems interview topics.
  • Explaining decoding and linking demonstrates low-level fluency.

10. Resources

10.1 Essential Reading

  • Intel 64 and IA-32 Architectures Software Developer’s Manual - ISA reference
  • System V AMD64 ABI Draft 0.99.7 - calling convention rules

10.2 Video Resources

  • Vendor and university lectures on x86-64 and ABIs (search official channels)