← Back to all projects

LEARN CHAOS ENGINEERING

In 2011, Netflix was migrating to AWS. They realized that in the cloud, failure is a given. Instead of trying to prevent every possible failure (which is impossible in distributed systems), they decided to embrace it. They created **Chaos Monkey**—a tool that randomly kills production instances.

Learn Chaos Engineering: From Zero to Chaos Master

Goal: Deeply understand Chaos Engineering by building a “Simian Army” clone. You will learn how to systematically prove that a system can withstand unexpected failures in production by building the very tools that break it. You’ll master failure injection (network, process, disk, cloud), observability, and the philosophy of building resilient, self-healing systems from first principles.

Why Chaos Engineering Matters

In 2011, Netflix was migrating to AWS. They realized that in the cloud, “failure is a given.” Instead of trying to prevent every possible failure (which is impossible in distributed systems), they decided to embrace it. They created Chaos Monkey—a tool that randomly kills production instances.

If you know your servers will die at random, you are forced to build systems that don’t care when they do.

Chaos Engineering is NOT “breaking things in production.” It is experimentation. It is the discipline of performing experiments on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

By mastering this, you unlock:

  • The mindset of “Failure as a First-Class Citizen.”
  • Deep knowledge of Linux Internals (how to manipulate traffic, processes, and resources).
  • Advanced Observability skills (how do you know the system is “healthy” if it’s currently failing?).
  • Distributed Systems Mastery (understanding timeouts, retries, and circuit breakers).

Core Concept Analysis

1. The Chaos Loop: The Scientific Method

Chaos Engineering follows a strict scientific loop. It is not random destruction.

1. DEFINE STEADY STATE
   (What does "normal" look like? e.g., 200ms latency, 0.01% error rate)
          │
          ▼
2. FORM HYPOTHESES
   ("If we kill one database node, the cluster will re-elect a leader in <5s")
          │
          ▼
3. INJECT FAILURE (The "Monkey")
   (Kill a process, add 500ms latency, drop 10% of packets)
          │
          ▼
4. ANALYZE IMPACT
   (Did the steady state hold? Did the blast radius expand?)
          │
          ▼
5. IMPROVE & REPEAT
   (Fix the bottleneck and automate the test)

2. Failure Domains and Blast Radius

Failure injection must be controlled. You start with the smallest possible “Blast Radius” and expand only when confidence is high.

[ BLAST RADIUS HIERARCHY ]

Level 5: Region / Cloud Provider Failure (Chaos Kong)
Level 4: Availability Zone Failure (Chaos Gorilla)
Level 3: Network Partition / DNS Failure
Level 2: Service / Container / Instance Failure (Chaos Monkey)
Level 1: Local Resource Pressure (CPU, RAM, Disk, Latency)

3. Failure Injection Techniques

How do we actually “break” things programmatically?

  • Process Level: SIGKILL, SIGSTOP, SIGTERM.
  • Network Level: iproute2 (tc - traffic control), iptables, ebpf.
  • Resource Level: Cgroups (limiting CPU/RAM), I/O stress, Disk exhaustion.
  • Application Level: Proxying requests to inject HTTP 500s or timeouts.

Concept Summary Table

Concept Cluster What You Need to Internalize
Steady State The measurable output of a system that indicates it is providing business value. If this drops, the experiment failed.
Blast Radius The subset of users or infrastructure affected by an experiment. Keeping this small is the “Safety First” rule.
Hypothesis A specific, measurable prediction. “If X happens, the system will respond by doing Y, and users won’t notice Z.”
Failure Injection The mechanism of introducing “harm.” It must be automated, reversible, and targeted.
Observability The ability to see inside the system. Without high-cardinality metrics, Chaos Engineering is just guessing.

Deep Dive Reading by Concept

This section maps each concept from above to specific book chapters for deeper understanding. Read these before or alongside the projects to build strong mental models.

Foundation & Philosophy

Concept Book & Chapter
Principles of Chaos “Chaos Engineering” by Casey Rosenthal & Nora Jones — Ch. 1: “The Discipline of Chaos Engineering”
The Mindset “Site Reliability Engineering” by Beyer, Jones, Petoff, Murphy — Ch. 1: “Introduction”
Reliability Patterns “Release It!” by Michael Nygard — Ch. 4: “Stability Patterns” (Circuit Breakers, Bulkheads)

Technical Execution

Concept Book & Chapter
Network Manipulation “Linux System Programming” by Robert Love — Ch. 10: “Signals” & Ch. 12: “Networking”
Resource Constraints “How Linux Works” by Brian Ward — Ch. 15: “Containers and Virtualization” (Cgroups/Namespaces)
Observability “Distributed Systems Observability” by Cindy Sridharan — Ch. 2: “The Three Pillars”

Essential Reading Order

For maximum comprehension, read in this order:

  1. The Why (Week 1):
    • Chaos Engineering (Rosenthal) Ch. 1-2.
    • Release It! (Nygard) Ch. 4-5.
  2. The How (Week 2):
    • How Linux Works (Ward) Ch. 15.
    • Linux System Programming (Love) Ch. 10.

Project 1: The Process Sniper (Basic Chaos Monkey)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go
  • Alternative Programming Languages: C, Python, Rust
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Systems Programming / Process Management
  • Software or Tool: Linux procfs, POSIX Signals
  • Main Book: “Linux System Programming” by Robert Love

What you’ll build: A CLI tool that targets a specific process group or “label” and randomly kills a percentage of those processes at a configurable interval.

Why it teaches Chaos Engineering: This is the foundation. You learn that systems aren’t just “on” or “off.” You’ll see how supervisors (like systemd or Docker) react when a process vanishes. It teaches you how to map a logical “service” to physical “Process IDs” (PIDs).

Core challenges you’ll face:

  • Finding target processes → maps to scanning /proc or using syscalls.
  • Signal handling → maps to understanding the difference between SIGTERM and SIGKILL.
  • Randomization logic → maps to statistical failure injection.
  • Safety valves → maps to ensuring you don’t kill the chaos tool itself or critical system processes.

Key Concepts:

  • POSIX Signals: “Linux System Programming” Ch. 10 - Robert Love
  • Process Lifecycle: “How Linux Works” Ch. 3 - Brian Ward

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic shell knowledge.

Real World Outcome

You’ll have a tool named sniper. When you run 5 instances of a “worker” app, the sniper will pick one and kill it every 30 seconds. You will see if your “orchestrator” (like a shell script loop or systemd) brings it back.

Example Output:

$ ./sniper --target "worker-app" --probability 0.2 --interval 10s

[2024-12-28 10:00:00] Target: worker-app | Strategy: Random Kill
[2024-12-28 10:00:00] Found 5 instances: [1201, 1202, 1203, 1204, 1205]
[2024-12-28 10:00:10] Dice roll: 0.15 < 0.20. KILLING PID 1203.

The Core Question You’re Answering

“How does my system behave when a component simply vanishes without warning?”

Before you write any code, sit with this question. Does the system hang? Does a load balancer detect the death immediately, or does it keep sending requests into the void for 30 seconds?

Concepts You Must Understand First

Stop and research these before coding:

  1. The /proc Filesystem
    • How can you find all PIDs belonging to a specific executable name?
    • Book Reference: “How Linux Works” Ch. 3 - Brian Ward
  2. Signals (SIGKILL vs SIGTERM)
    • Why do we use SIGKILL (9) for chaos instead of SIGTERM (15)?
    • Book Reference: “Linux System Programming” Ch. 10 - Robert Love

Questions to Guide Your Design

Before implementing, think through these:

  1. Filtering
    • How will you ensure your tool doesn’t accidentally kill PID 1 or your own SSH session?
  2. Scheduling
    • Is it better to use a fixed interval or a jittered interval? Why?

Thinking Exercise

The “Ghost Request” Trace

Imagine a client sends a request to a server. Midway through processing, the sniper kills the server process.

  1. What happens to the TCP connection?
  2. Does the client get a “Connection Reset” or does it “Timeout”?

The Interview Questions They’ll Ask

  1. “What is the difference between an ungraceful shutdown and a graceful one?”
  2. “How would you implement a ‘Safety Valve’ to prevent a chaos tool from killing 100% of a fleet?”

Hints in Layers

Hint 1: Finding Processes On Linux, look at the directory /proc. Every sub-directory that is a number is a PID.

Hint 2: Sending the Kill In Go, use os.FindProcess(pid) then process.Signal(syscall.SIGKILL).

Hint 3: The Probability Engine Generate a random float between 0.0 and 1.0. If it’s less than your --probability, pull the trigger.

Project 2: The Latency Injector (Latency Monkey)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go / Shell
  • Alternative Programming Languages: C (using libnetfilter_queue)
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Networking / Linux Traffic Control
  • Software or Tool: tc (Traffic Control), netem (Network Emulator)
  • Main Book: “How Linux Works” by Brian Ward (Networking section)

What you’ll build: A tool that wraps the Linux tc command to inject controlled latency, jitter, and packet loss into specific network interfaces or IP ranges.

Why it teaches Chaos Engineering: In distributed systems, “Slow is the new Down.” A server that takes 30 seconds to reply is often more dangerous than one that is dead, because it ties up resources.

Core challenges you’ll face:

  • Interfacing with Kernel Qdiscs → maps to learning how Linux queues packets.
  • Targeting specific traffic → maps to using tc filters.

Real World Outcome

A tool that allows you to say: “Make all outgoing traffic to the database (port 5432) have an extra 500ms of latency.”

Example Output:

$ ./latency-monkey --target-port 5432 --latency 500ms

[INFO] Applying 500ms latency to egress traffic on eth0:5432

Project 3: The Disk Devourer (Resource Chaos)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Python / Go
  • Alternative Programming Languages: C
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Filesystems / Resource Management
  • Software or Tool: fallocate, dd
  • Main Book: “The Linux Programming Interface” by Michael Kerrisk

What you’ll build: A process that fills up disk space or exhausts disk I/O bandwidth to see how applications handle “Disk Full” or “I/O Wait” conditions.

Why it teaches Chaos Engineering: Most developers assume write() always succeeds. This project forces you to see what happens when it doesn’t.

Project 4: The Container Saboteur (Docker Monkey)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Python, Rust
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Containerization / Orchestration
  • Software or Tool: Docker API, Containerd
  • Main Book: “How Linux Works” by Brian Ward (Containers section)

What you’ll build: A tool that talks to the Docker Daemon to randomly pause, restart, or limit the CPU of containers with specific labels.

Why it teaches Chaos Engineering: In modern systems, containers are the unit of failure. This teaches you about the container lifecycle and how orchestrators (like K8s) react to missing containers.

Core challenges you’ll face:

  • Docker API Interaction → maps to using the Docker SDK or raw sockets.
  • Cgroup Manipulation → maps to understanding how CPU/RAM limits are enforced.

Real World Outcome

A docker-monkey that periodically “pauses” the payment-service container. You’ll observe if the order-service times out or crashes.

Project 5: The DNS Poisoner (Discovery Chaos)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go
  • Alternative Programming Languages: C, Python
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 4: Expert
  • Knowledge Area: Networking / DNS
  • Software or Tool: iptables, dnstap
  • Main Book: “TCP/IP Illustrated, Volume 1” by W. Richard Stevens

What you’ll build: A tool that intercepts DNS queries and occasionally returns incorrect IP addresses or “NXDOMAIN” errors.

Why it teaches Chaos Engineering: Most apps assume DNS is a constant. This project breaks that assumption.

Real World Outcome

Your application logs UnknownHostException for its database. You verify if it recovers automatically when DNS is fixed.

Project 6: The HTTP Chaos Proxy (The “Toxiproxy” Clone)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, Node.js
  • Coolness Level: Level 5: Pure Magic
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Web Development / Proxy Servers
  • Software or Tool: net/http (Go)
  • Main Book: “Release It!” by Michael Nygard

What you’ll build: A Layer 7 proxy that sits between your app and an API. It injects “toxics” (503 errors, 2s delays) via an API.

Why it teaches Chaos Engineering: This is Application Level Chaos. It forces you to test Retries and Circuit Breakers in your code.

Real World Outcome

You point your frontend at this proxy. You trigger a “Toxic” and see a “Retry” spinner appear on your website.

Project 7: The Steady-State Monitor (The “Watchman”)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go / Python
  • Alternative Programming Languages: JavaScript (Node)
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Observability / Monitoring
  • Software or Tool: Prometheus, Grafana API
  • Main Book: “Distributed Systems Observability” by Cindy Sridharan

What you’ll build: A service that monitors “Steady State” metrics (e.g., HTTP 200 rate) and provides an “Automatic Abort” signal to other chaos monkeys if the steady state drops below a threshold.

Why it teaches Chaos Engineering: Chaos Engineering is not about breaking things; it’s about learning. If an experiment hurts the system more than expected, you must stop immediately. This is the “Safety Valve.”

Core challenges you’ll face:

  • Metric Aggregation → maps to querying Prometheus/TimescaleDB.
  • Threshold Logic → maps to defining what “broken” looks like.
  • The “Kill Switch” Pattern → maps to distributed state management (stopping all monkeys at once).

Key Concepts:

  • SLIs/SLOs: Service Level Indicators vs Objectives.
  • Error Budgets: Knowing when you can afford to run chaos experiments.
  • Observability Pipelines: How data flows from the app to your monitor.

Difficulty: Intermediate Time estimate: 1 week

Project 8: The Chaos Orchestrator (Simian Army Core)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Rust, Python
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Distributed Systems / Scheduling
  • Software or Tool: Etcd, Redis, or PostgreSQL
  • Main Book: “Chaos Engineering” by Rosenthal & Jones

What you’ll build: A central management plane that coordinates various “monkeys.” It allows you to schedule experiments (“Run Network Chaos every Tuesday at 2 PM”) and ensures only one experiment runs at a time.

Why it teaches Chaos Engineering: This is where individual tools become a “System.” You learn about scheduling, concurrency control, and logging the history of experiments.

Project 9: The Game Day Dashboard (Visualization)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: React / Go
  • Alternative Programming Languages: Vue, Svelte
  • Coolness Level: Level 5: Pure Magic
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Frontend / UX
  • Software or Tool: D3.js, Tailwind CSS
  • Main Book: “Principles of Chaos Engineering” (Online)

What you’ll build: A visual dashboard that shows your system’s architecture (nodes/edges) and lights up specific parts in red when a chaos experiment is active. It should overlay “Steady State” metrics in real-time.

Why it teaches Chaos Engineering: It makes failure “visible.” This is crucial for “Game Days” where teams sit together to watch the system fail and recover.

Project 10: The Cloud Terminator (Chaos Gorilla)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Python (Boto3)
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 5. The “Industry Disruptor”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Cloud Infrastructure
  • Software or Tool: AWS SDK, GCP SDK
  • Main Book: “Chaos Engineering” by Rosenthal & Jones

What you’ll build: A tool that uses cloud provider APIs to simulate larger-scale failures: “Delete all EC2 instances in Availability Zone ‘us-east-1a’” or “Modify Security Groups to block all cross-region traffic.”

Why it teaches Chaos Engineering: It moves the focus from “one server” to “one data center.” This tests your multi-AZ and multi-region failover logic.

Project 11: The BPF Failure Injector (Modern Chaos)

  • File: LEARN_CHAOS_ENGINEERING.md
  • Main Programming Language: C / Go
  • Alternative Programming Languages: Rust (Aya)
  • Coolness Level: Level 5: Pure Magic
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 5: Master
  • Knowledge Area: Kernel Programming / eBPF
  • Software or Tool: eBPF, libbpf, bpftrace
  • Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: An eBPF program that attaches to kernel functions like tcp_v4_connect or sys_read and injects errors (e.g., -ECONNREFUSED or -EIO) into specific processes without actually stopping the process.

Why it teaches Chaos Engineering: This is the “God Mode” of failure injection. You aren’t breaking the system; you’re lying to the application about what the kernel is doing. It’s incredibly precise and has almost zero overhead.

Core challenges you’ll face:

  • Writing BPF Programs → maps to restricted C code for the JIT compiler.
  • Helper Functions → maps to using bpf_override_return to inject errors.
  • Safety → maps to ensuring you don’t crash the host kernel.

Recommendation

If you are a beginner: Start with Project 1 (Process Sniper) and Project 3 (Disk Devourer). These give you instant feedback with simple code and teach you the core “Loop” of Chaos Engineering.

If you want to master Networking: Go for Project 2 (Latency Injector) and Project 5 (DNS Poisoner). These are the “bread and butter” of distributed systems reliability.

If you want to be a Kernel Wizard: Jump straight to Project 11 (BPF Failure Injector). It is the most technically impressive project on this list and will teach you more about the Linux kernel than any other project.

Final Overall Project: The “Simian Army Suite”

What you’ll build: A fully integrated Chaos Engineering platform that combines your Orchestrator, Monitoring, and at least three failure injection “Monkeys” into a single, cohesive system.

Success Criteria:

  1. You can define an experiment in a YAML file: “Target Service A, Inject 500ms Latency, for 10 minutes, abort if Success Rate < 95%”.
  2. The system executes the experiment across multiple nodes/containers.
  3. The Monitoring service automatically detects a breach and halts the experiment.
  4. You produce a “Post-Mortem Report” PDF automatically showing the metrics during the failure.

Why this is the ultimate test: This requires you to handle concurrency, networking, APIs, and the human side of reliability (the report). It is a production-grade tool you can actually use in your career.

Summary

This learning path covers Chaos Engineering through 11 hands-on projects, taking you from basic process management to kernel-level failure injection.

# Project Name Main Language Difficulty Time Estimate
1 Process Sniper Go Beginner Weekend
2 Latency Injector Go/Shell Advanced 1 Week
3 Disk Devourer Python/Go Intermediate Weekend
4 Container Saboteur Go Advanced 1 Week
5 DNS Poisoner Go Expert 2 Weeks
6 HTTP Chaos Proxy Go Advanced 1-2 Weeks
7 Steady-State Monitor Go/Python Intermediate 1 Week
8 Chaos Orchestrator Go Advanced 2 Weeks
9 Game Day Dashboard React/Go Intermediate 1 Week
10 Cloud Terminator Go Advanced 1 Week
11 BPF Failure Injector C/Go Master 1 Month+

For beginners: Start with projects #1, #3, #7. For intermediate: Focus on projects #2, #4, #6, #9. For advanced: Focus on projects #5, #8, #10, #11.

Expected Outcomes

After completing these projects, you will:

  • Understand exactly how failure propagates through distributed systems.
  • Master Linux traffic control, process signals, and the /proc filesystem.
  • Be able to design and implement robust “Steady State” monitoring.
  • Have a portfolio of tools that demonstrate deep systems and reliability engineering skills.
  • Be prepared to lead “Game Days” and reliability initiatives at any scale.

You’ll have built a working “Simian Army” suite that demonstrates deep understanding of Chaos Engineering from first principles.