LEARN BPF EBPF LINUX

Learn BPF/eBPF: From Zero to Kernel Ninja

Goal: Deeply understand BPF (Berkeley Packet Filter) and its modern evolution eBPF (extended BPF)—the technology that’s revolutionizing Linux observability, networking, and security by allowing you to run sandboxed programs directly in the kernel.

Why BPF/eBPF Matters

Every time you use tcpdump, run a container, profile an application, or use modern observability tools like Datadog or Cilium, BPF is working under the hood. It’s arguably the most significant advancement in Linux kernel technology in the past decade.

eBPF lets you:

Observe everything: See every system call, network packet, disk I/O, and function call
React at kernel speed: Process millions of events per second with nanosecond latency
Stay safe: The BPF verifier ensures your code can’t crash the kernel
Deploy instantly: No kernel recompilation, no module loading, no reboots

After completing these projects, you will:

Understand how BPF programs execute inside the Linux kernel
Write BPF programs in C, attach them to kernel hooks, and communicate with userspace
Build production-grade observability, networking, and security tools
Know when and how to use bpftrace, BCC, libbpf, and language-specific libraries
Understand the BPF verifier, JIT compilation, and safety guarantees

Core Concept Analysis

What is BPF?

┌─────────────────────────────────────────────────────────────────┐
│                         USER SPACE                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │
│  │  bpftrace   │  │    BCC      │  │   libbpf    │              │
│  │ (one-liners)│  │(Python/Lua) │  │  (C/Go/Rust)│              │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘              │
│         │                │                │                      │
│         └────────────────┼────────────────┘                      │
│                          │ bpf() syscall                         │
├──────────────────────────┼──────────────────────────────────────┤
│                          ▼                   KERNEL SPACE        │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                    BPF SUBSYSTEM                          │   │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐                │   │
│  │  │ Verifier │→ │   JIT    │→ │  Maps    │                │   │
│  │  │ (safety) │  │(compile) │  │ (data)   │                │   │
│  │  └──────────┘  └──────────┘  └──────────┘                │   │
│  └──────────────────────────────────────────────────────────┘   │
│                          │                                       │
│         ┌────────────────┼────────────────────┐                 │
│         ▼                ▼                    ▼                 │
│  ┌────────────┐  ┌────────────┐  ┌────────────────────┐        │
│  │  kprobes   │  │ tracepoints│  │    XDP/TC          │        │
│  │ (function  │  │ (stable    │  │(network packets)   │        │
│  │   entry)   │  │  events)   │  │                    │        │
│  └────────────┘  └────────────┘  └────────────────────────────│
│                                                                  │
│  [Syscalls]  [Scheduler]  [Filesystem]  [Network]  [Memory]    │
└─────────────────────────────────────────────────────────────────┘

The BPF Virtual Machine

BPF programs run in a register-based virtual machine inside the kernel:

Component	Description
11 Registers	R0-R10, 64-bit each. R0 = return value, R1-R5 = function args, R10 = frame pointer
512-byte Stack	Limited stack space forces careful programming
BPF Maps	Key-value stores shared between kernel and userspace
Helper Functions	Kernel-provided functions BPF programs can call
Verifier	Static analyzer that ensures program safety before loading
JIT Compiler	Translates BPF bytecode to native machine code

Program Types and Attach Points

Type	Attach Point	Use Case
`BPF_PROG_TYPE_KPROBE`	Any kernel function	Tracing function calls
`BPF_PROG_TYPE_TRACEPOINT`	Stable kernel events	Reliable observability
`BPF_PROG_TYPE_XDP`	Network interface (earliest)	High-speed packet processing
`BPF_PROG_TYPE_SOCKET_FILTER`	Socket	Packet filtering
`BPF_PROG_TYPE_TC`	Traffic Control	Packet modification
`BPF_PROG_TYPE_LSM`	Linux Security Module	Security enforcement
`BPF_PROG_TYPE_PERF_EVENT`	CPU performance counters	Profiling
`BPF_PROG_TYPE_CGROUP_*`	cgroups	Container policies

BPF Maps (Data Structures)

Map Type	Description	Use Case
`BPF_MAP_TYPE_HASH`	Hash table	Counting, lookups
`BPF_MAP_TYPE_ARRAY`	Fixed-size array	Per-CPU data, configs
`BPF_MAP_TYPE_PERF_EVENT_ARRAY`	Event buffer	Sending events to userspace
`BPF_MAP_TYPE_RINGBUF`	Ring buffer	High-performance events
`BPF_MAP_TYPE_LRU_HASH`	LRU hash table	Connection tracking
`BPF_MAP_TYPE_STACK_TRACE`	Stack traces	Profiling

The Development Ecosystem

Evolution of BPF Development:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

2015-2018: BCC (BPF Compiler Collection)
├── Python/Lua frontends
├── In-memory LLVM compilation
├── Easy to use, but runtime dependencies
└── Still great for learning and scripting

2019-2020: bpftrace
├── High-level tracing language (like awk for tracing)
├── One-liners for quick analysis
├── Built on libbpf
└── Perfect for ad-hoc tracing

2020-present: libbpf + CO-RE
├── Compile Once, Run Everywhere
├── No runtime dependencies (except kernel)
├── BTF (BPF Type Format) for portability
├── Production-grade tools
└── Recommended for serious projects

Language bindings:
├── C: libbpf (native)
├── Go: cilium/ebpf, libbpfgo
├── Rust: aya, libbpf-rs
└── Python: bcc (deprecated for new code)

Project List

Projects are ordered from fundamental understanding to advanced implementations. Each project builds on concepts from previous ones.

Project 1: Hello World Tracer (Your First BPF Program)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: bpftrace (DSL)
Alternative Programming Languages: C with BCC, Python with BCC
Coolness Level: Level 2: Practical but Forgettable
Business Potential: 1. The “Resume Gold”
Difficulty: Level 1: Beginner
Knowledge Area: BPF Basics / Tracing
Software or Tool: bpftrace
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A series of one-liner BPF programs that trace system activity—counting system calls, printing function arguments, and measuring latencies.

Why it teaches BPF: Before writing complex C programs, you need to understand what BPF can see. bpftrace lets you explore kernel attach points interactively. This is how Brendan Gregg, the author of most BPF tracing tools, does his exploratory work.

Core challenges you’ll face:

Understanding probe types → maps to kprobe vs tracepoint vs uprobe
Reading kernel data structures → maps to accessing arguments and return values
Aggregating data in-kernel → maps to BPF maps for counting and histograms
Filtering events efficiently → maps to predicates and early returns

Key Concepts:

Probe Types: “BPF Performance Tools” Chapter 2 - Brendan Gregg
bpftrace Language: bpftrace Reference Guide
Tracepoints vs Kprobes: “Learning eBPF” Chapter 7 - Liz Rice
Linux Tracing Overview: Brendan Gregg’s Linux Tracing Page

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic Linux command line, understanding of system calls

Real world outcome:

# Count syscalls by process
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
Attaching 1 probe...
^C

@[sshd]: 127
@[bash]: 234
@[chrome]: 4521
@[firefox]: 8932

# Trace file opens with path
$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%s opened %s\n", comm, str(args->filename));
}'
bash opened /etc/passwd
vim opened /home/user/.vimrc
python opened /usr/lib/python3.10/os.py

# Histogram of read sizes
$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read /args->ret > 0/ {
    @bytes = hist(args->ret);
}'
@bytes:
[1]                  234 |@@@@                                    |
[2, 4)               512 |@@@@@@@@@                               |
[4, 8)              1024 |@@@@@@@@@@@@@@@@@@                       |
[8, 16)             2341 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
[16, 32)            1892 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         |

Implementation Hints:

bpftrace uses a domain-specific language inspired by awk and DTrace:

probe_type:probe_name /filter/ { action }

Questions to guide your exploration:

What probes are available? Run sudo bpftrace -l 'tracepoint:syscalls:*'
What arguments does each probe have? Run sudo bpftrace -lv tracepoint:syscalls:sys_enter_read
How do you access the current process name? (hint: comm)
How do you build a histogram? (hint: hist() and lhist())
How do you count by key? (hint: @mapname[key] = count();)

Start with these exercises:

Count how many times each syscall is called
Print every time a specific binary runs (execve)
Measure time spent in a specific function
Build a latency histogram for disk reads

Learning milestones:

You can list available probes → You understand kernel instrumentation points
You can read syscall arguments → You understand how BPF accesses kernel data
You build histograms in-kernel → You understand BPF maps and aggregations
You filter and format output → You’re ready for more complex programs

Project 2: System Call Monitor (Your First C BPF Program)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 2. The “Micro-SaaS / Pro Tool”
Difficulty: Level 2: Intermediate
Knowledge Area: BPF Programming / System Calls
Software or Tool: libbpf, clang, bpftool
Main Book: “Learning eBPF” by Liz Rice

What you’ll build: A complete BPF program in C that monitors system calls, counts them by type and process, and displays real-time statistics—essentially your own version of strace but with kernel-level efficiency.

Why it teaches BPF: This forces you to understand the complete BPF development workflow: writing BPF C code, compiling with clang, loading with libbpf, and communicating between kernel and userspace via maps.

Core challenges you’ll face:

Setting up the BPF development environment → maps to clang, libbpf, kernel headers
Writing BPF-compatible C code → maps to verifier constraints, helper functions
Defining and using BPF maps → maps to data sharing between kernel and userspace
Loading and attaching programs → maps to libbpf skeleton, bpf() syscall

Key Concepts:

BPF Program Structure: “Learning eBPF” Chapter 3 - Liz Rice
BPF Maps: “Learning eBPF” Chapter 4 - Liz Rice
libbpf-bootstrap: GitHub libbpf-bootstrap
CO-RE and BTF: Andrii Nakryiko’s Blog on CO-RE

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: C programming, Project 1 completed, basic understanding of Linux syscalls

Real world outcome:

$ sudo ./syscall_monitor
PID     COMM            SYSCALL         COUNT   RATE/s
1234    nginx           read            15234   1523.4
1234    nginx           write           12892   1289.2
1234    nginx           epoll_wait       8923    892.3
5678    postgres        read            45123   4512.3
5678    postgres        futex           23891   2389.1
9012    chrome          mmap             3421    342.1
9012    chrome          munmap           2198    219.8

[Top syscalls last 10 seconds]
read:        123,456 (45.2%)
write:        89,123 (32.7%)
epoll_wait:   34,567 (12.7%)
futex:        15,234 (5.6%)
mmap:         10,234 (3.8%)

Implementation Hints:

Your project will have two parts:

BPF program (runs in kernel): Attaches to tracepoints, counts syscalls
Userspace loader (runs as normal program): Loads BPF, reads maps, displays stats

BPF program structure (conceptual):

// Define a map to store counts
BPF_MAP(syscall_count, hash, key=struct {pid, syscall_id}, value=u64)

// Attach to syscall entry tracepoint
SEC("tracepoint/raw_syscalls/sys_enter")
int trace_syscall(struct trace_event_raw_sys_enter *ctx) {
    // Get current PID and syscall number
    // Look up or create map entry
    // Increment count
    return 0;
}

Questions to guide your implementation:

How does libbpf-bootstrap structure projects?
What’s the difference between SEC() and __attribute__((section()))?
How do you read the current PID? (hint: bpf_get_current_pid_tgid())
How do you update a map atomically? (hint: __sync_fetch_and_add())
How does userspace iterate over map entries?

Use libbpf-bootstrap as your starting template—it handles the build system complexity.

Learning milestones:

You compile a BPF program successfully → You understand the toolchain
The verifier accepts your program → You understand BPF constraints
You read data from maps in userspace → You understand BPF communication
You display real-time statistics → You’ve built a complete observability tool

Project 3: Process Execution Logger (execsnoop Clone)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 3. The “Service & Support” Model
Difficulty: Level 2: Intermediate
Knowledge Area: Process Tracing / Security Auditing
Software or Tool: libbpf, perf_buffer/ring_buffer
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A tool that logs every process execution on the system—showing the command, arguments, parent process, return code, and timing. This is your own version of execsnoop from BCC.

Why it teaches BPF: This project introduces event streaming from kernel to userspace using perf buffers or ring buffers. You’ll also learn to capture variable-length data (command arguments) and correlate entry/exit events.

Core challenges you’ll face:

Capturing variable-length arguments → maps to reading from userspace memory
Streaming events to userspace → maps to perf_buffer vs ring_buffer
Correlating entry and exit → maps to storing state between probes
Handling high event rates → maps to performance optimization

Key Concepts:

Perf Buffers: “BPF Performance Tools” Chapter 2.5 - Brendan Gregg
Ring Buffers: Andrii Nakryiko on Ring Buffers
Reading User Memory: “Learning eBPF” Chapter 6 - Liz Rice
execsnoop Analysis: Brendan Gregg’s execsnoop

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Project 2 completed, understanding of fork/exec

Real world outcome:

$ sudo ./execsnoop
TIME      PID    PPID   RET  COMM             ARGS
14:23:01  12345  1234     0  bash             bash -c echo hello
14:23:01  12346  12345    0  echo             echo hello
14:23:02  12347  1234     0  curl             curl -s https://api.example.com
14:23:03  12348  12347    0  sh               sh -c date
14:23:03  12349  12348    0  date             date
14:23:05  12350  1        -2  badcmd           badcmd --flag  # ENOENT

[Filters: PID, PPID, command pattern, failed-only]

Implementation Hints:

This tool needs to trace two events:

sys_enter_execve - Capture the arguments before execution
sys_exit_execve - Capture the return code

Data flow:

sys_enter_execve:
  ├── Read filename from user memory (bpf_probe_read_user_str)
  ├── Read argv array (tricky: array of pointers!)
  ├── Store in temporary map keyed by pid_tgid
  └── (don't send yet, wait for exit)

sys_exit_execve:
  ├── Look up stored entry data
  ├── Add return code
  ├── Send complete event via ring buffer
  └── Delete map entry

Questions to guide your implementation:

Why can’t you just read argv directly in BPF?
What’s the maximum string length you can read?
How do you handle processes that execve but never return (they become the new process)?
What happens if your ring buffer fills up?
How do you format the output nicely with variable-length arguments?

Learning milestones:

You capture exec events with arguments → You understand userspace memory reading
You stream events via ring buffer → You understand kernel→userspace communication
You show entry and exit together → You understand state management across probes
You handle edge cases gracefully → You’re building production-quality tools

Project 4: File Access Auditor (opensnoop Clone)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 3. The “Service & Support” Model
Difficulty: Level 2: Intermediate
Knowledge Area: Filesystem Tracing / Security
Software or Tool: libbpf
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A tool that monitors all file opens on the system, showing which processes access which files, with filtering capabilities. This is essential for security auditing, debugging, and understanding application behavior.

Why it teaches BPF: File operations are fundamental. This project teaches you to trace the VFS layer, understand file descriptors, and filter events efficiently in kernel space.

Core challenges you’ll face:

Tracing the right syscalls → maps to open, openat, openat2
Resolving file descriptors to paths → maps to fd_install, dentry walking
Filtering efficiently → maps to in-kernel vs userspace filtering
Handling flags and modes → maps to interpreting syscall arguments

Key Concepts:

File Operations in Linux: “The Linux Programming Interface” Chapter 4 - Michael Kerrisk
VFS Layer: “Understanding the Linux Kernel” Chapter 12 - Bovet & Cesati
opensnoop Implementation: BCC opensnoop

Difficulty: Intermediate Time estimate: 1 week Prerequisites: Project 3 completed

Real world outcome:

$ sudo ./fileaudit
PID    COMM           FD   FLAGS          PATH
1234   nginx           5   O_RDONLY       /var/www/index.html
1234   nginx           6   O_WRONLY       /var/log/nginx/access.log
5678   python          3   O_RDWR         /tmp/data.json
5678   python          4   O_CREAT|O_EXCL /tmp/lockfile
9012   sshd            5   O_RDONLY       /etc/passwd

# Filter by path pattern
$ sudo ./fileaudit --path "/etc/*"
$ sudo ./fileaudit --pid 1234 --flags "O_WRONLY|O_CREAT"

Implementation Hints:

Key decision: Use tracepoints for stability or kprobes for more data?

Tracepoint approach (recommended):

tracepoint/syscalls/sys_enter_openat - Get filename, flags
tracepoint/syscalls/sys_exit_openat - Get return value (fd or error)

Filtering strategies:

Option 1: Filter in userspace
  - Simpler BPF code
  - Higher overhead (send all events)
  - More flexible filtering

Option 2: Filter in kernel
  - More complex BPF code
  - Lower overhead (only matching events)
  - Requires map-based filter rules

Questions to guide your implementation:

What’s the difference between open, openat, and openat2?
How do you decode flags (O_RDONLY, O_CREAT, etc.)?
How do you handle relative paths (need to track working directory)?
What’s the performance impact of tracing all opens on a busy system?

Learning milestones:

You trace file opens with paths → You understand file syscall tracing
You decode flags correctly → You understand syscall argument parsing
You implement efficient filtering → You understand kernel-space optimization
You handle errors and edge cases → You’re building robust tools

Project 5: Network Packet Counter (Basic XDP)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Rust (aya), Go (cilium/ebpf)
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 4. The “Open Core” Infrastructure
Difficulty: Level 3: Advanced
Knowledge Area: Networking / XDP
Software or Tool: libbpf, iproute2
Main Book: “Learning eBPF” by Liz Rice

What you’ll build: An XDP (eXpress Data Path) program that counts network packets by protocol, source/destination, and size—processing packets at the earliest possible point in the network stack.

Why it teaches BPF: XDP is where BPF really shines—processing millions of packets per second. This introduces network programming with BPF, packet parsing, and the unique challenges of the XDP environment.

Core challenges you’ll face:

Understanding XDP attach modes → maps to native, offload, generic
Parsing packet headers safely → maps to bounds checking, verifier requirements
Counting at high speed → maps to per-CPU maps, atomic operations
Handling different protocols → maps to Ethernet, IP, TCP, UDP

Key Concepts:

XDP Architecture: “Learning eBPF” Chapter 8 - Liz Rice
Packet Parsing: XDP Tutorial
Network Headers: “TCP/IP Illustrated Volume 1” - Stevens
XDP Actions: Julia Evans XDP Blog

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Projects 1-4 completed, basic networking knowledge (TCP/IP)

Real world outcome:

$ sudo ./xdp-counter eth0
Interface: eth0 (XDP mode: native)

PROTOCOL    PACKETS      BYTES         PPS       BPS
─────────────────────────────────────────────────────────
TCP         1,234,567    1.2 GB      45,123    156 Mbps
UDP           234,567  234.5 MB      12,456     23 Mbps
ICMP            1,234    123 KB          45      4 Kbps
Other             567     56 KB          12      1 Kbps
─────────────────────────────────────────────────────────
TOTAL       1,470,935    1.5 GB      57,636    179 Mbps

[Top Sources]
192.168.1.100:     523,456 packets
10.0.0.50:         234,123 packets
172.16.0.1:        123,456 packets

Implementation Hints:

XDP programs receive raw Ethernet frames and must parse headers carefully:

XDP Context:
┌─────────────────────────────────────────────────┐
│ data (start of packet)                          │
│ ┌─────────────┬────────────┬──────────────────┐ │
│ │  Ethernet   │    IP      │   TCP/UDP/ICMP   │ │
│ │   Header    │   Header   │      Header      │ │
│ │  (14 bytes) │ (20+ bytes)│   (varies)       │ │
│ └─────────────┴────────────┴──────────────────┘ │
│ data_end (end of packet)                        │
└─────────────────────────────────────────────────┘

Critical rule: Every pointer access must be bounds-checked!

// BAD: Verifier will reject
struct ethhdr *eth = data;
if (eth->h_proto == ETH_P_IP) ...  // REJECTED: no bounds check

// GOOD: Explicit bounds check
struct ethhdr *eth = data;
if (data + sizeof(*eth) > data_end)
    return XDP_PASS;
if (eth->h_proto == ETH_P_IP) ...  // OK: bounds verified

XDP return values:

XDP_PASS: Continue to network stack
XDP_DROP: Silently drop packet
XDP_TX: Send back out same interface
XDP_REDIRECT: Send to another interface

Questions to guide your implementation:

How do you attach an XDP program? (ip link set dev eth0 xdp obj ...)
What happens if you have a bug and return wrong values?
How do you handle VLAN-tagged packets?
Why use per-CPU maps for counters?

Learning milestones:

XDP program loads and attaches → You understand the XDP lifecycle
You parse Ethernet headers safely → You understand verifier constraints
You count packets by protocol → You understand packet classification
You achieve high throughput → You understand XDP performance

Project 6: TCP Connection Tracker (tcpconnect Clone)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 3. The “Service & Support” Model
Difficulty: Level 3: Advanced
Knowledge Area: Networking / TCP State
Software or Tool: libbpf
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A tool that tracks TCP connections in real-time—showing new connections, connection states, and detecting connection issues like refused connections or timeouts.

Why it teaches BPF: TCP connection tracking requires understanding kernel networking internals. You’ll trace multiple kernel functions and correlate events to build a complete picture of connection lifecycle.

Core challenges you’ll face:

Finding the right trace points → maps to tcp_v4_connect, inet_csk_accept
Extracting socket information → maps to sock, inet_sock structures
Tracking connection state → maps to TCP state machine
Correlating client and server views → maps to connect vs accept

Key Concepts:

TCP State Machine: “TCP/IP Illustrated Volume 1” Chapter 18 - Stevens
Linux TCP Implementation: “The Linux Programming Interface” Chapter 58-61 - Kerrisk
tcpconnect/tcpaccept: BCC Tools

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Projects 1-5 completed, TCP/IP knowledge

Real world outcome:

$ sudo ./tcptrack
TIME       TYPE     PID    COMM         SADDR:SPORT      DADDR:DPORT       LAT(ms)
14:23:01   CONNECT  1234   curl         192.168.1.10:45678  93.184.216.34:443   23.5
14:23:01   ACCEPT   5678   nginx        0.0.0.0:443      192.168.1.50:34567    0.1
14:23:02   CONNECT  1234   python       127.0.0.1:45679   127.0.0.1:5432        0.2
14:23:02   CLOSE    1234   curl         192.168.1.10:45678  93.184.216.34:443     -
14:23:05   REFUSED  9012   wget         192.168.1.10:45680  10.0.0.50:8080        -

[Active connections: 156]
[Connection rate: 23.4/s]
[Failure rate: 0.5%]

Implementation Hints:

Key kernel functions to trace:

Outgoing connections:
  tcp_v4_connect()      → Connection initiated
  tcp_v4_connect() ret  → Success/failure

Incoming connections:
  inet_csk_accept()     → New connection accepted

Connection lifecycle:
  tcp_set_state()       → State changes (ESTABLISHED, CLOSE_WAIT, etc.)
  tcp_close()           → Connection closing

Latency measurement:
  Store timestamp at connect entry
  Calculate delta at connect return

Extracting socket info (conceptual):

Given a struct sock *sk:
  - Source IP: sk->__sk_common.skc_rcv_saddr
  - Dest IP: sk->__sk_common.skc_daddr
  - Source port: sk->__sk_common.skc_num
  - Dest port: sk->__sk_common.skc_dport

CO-RE approach: Use bpf_core_read() to read kernel structures portably.

Questions to guide your implementation:

How do you distinguish IPv4 from IPv6?
What’s the difference between kprobe and fentry?
How do you measure connection latency?
What information is available at accept() time vs connect() time?

Learning milestones:

You trace connect calls with addresses → You understand socket tracing
You show both client and server sides → You understand full connection lifecycle
You measure connection latency → You understand timing across probes
You track connection states → You understand TCP state machine from kernel view

Project 7: Function Latency Histogram (funclatency Clone)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 2. The “Micro-SaaS / Pro Tool”
Difficulty: Level 3: Advanced
Knowledge Area: Performance Profiling
Software or Tool: libbpf
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A tool that measures the latency distribution of any kernel or userspace function, displaying results as a histogram. This is essential for performance debugging.

Why it teaches BPF: This project teaches you to use both kprobes (kernel functions) and uprobes (userspace functions), time measurements with nanosecond precision, and in-kernel histogram building.

Core challenges you’ll face:

Attaching to arbitrary functions → maps to dynamic kprobe/uprobe attachment
High-precision timing → maps to bpf_ktime_get_ns()
Building histograms in-kernel → maps to log2 bucketing, BPF maps
Supporting userspace functions → maps to uprobes, symbol resolution

Key Concepts:

Function Tracing: “BPF Performance Tools” Chapter 4 - Brendan Gregg
uprobes: “Learning eBPF” Chapter 7 - Liz Rice
Histogram Building: Brendan Gregg on Histograms

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Projects 1-6 completed

Real world outcome:

# Kernel function latency
$ sudo ./funclatency vfs_read
Tracing vfs_read... Hit Ctrl-C to end.
^C
     nsecs           : count    distribution
       256 -> 511    : 1234    |@@@                               |
       512 -> 1023   : 4567    |@@@@@@@@@@                        |
      1024 -> 2047   : 8901    |@@@@@@@@@@@@@@@@@@@@              |
      2048 -> 4095   : 12345   |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
      4096 -> 8191   : 8234    |@@@@@@@@@@@@@@@@@@@               |
      8192 -> 16383  : 3456    |@@@@@@@@                          |
     16384 -> 32767  : 1234    |@@@                               |
     32768 -> 65535  : 456     |@                                 |
     65536 -> 131071 : 123     |                                  |

# Userspace function latency
$ sudo ./funclatency /usr/lib/libc.so.6:malloc
$ sudo ./funclatency /usr/bin/python:PyDict_GetItem

Implementation Hints:

Histogram bucket calculation:

For log2 histogram with power-of-2 buckets:
  bucket = log2(value)

BPF has no log2, but you can use bit operations:
  bucket = 64 - __builtin_clzll(value)  // count leading zeros

Timing pattern:

Entry probe (kprobe/uprobe):
Get current timestamp
Store in map keyed by (pid_tgid)

Exit probe (kretprobe/uretprobe):
Look up entry timestamp
Calculate delta = now - entry
Increment histogram bucket
Delete entry from map

Dynamic attachment: The userspace loader needs to:

Resolve function symbol to address
Create kprobe/uprobe at runtime
Attach BPF program to the probe

Questions to guide your implementation:

How do you handle functions that don’t return (longjmp, exit)?
What happens with recursive functions?
How do you resolve symbols for userspace functions?
What’s the overhead of tracing a hot function?

Learning milestones:

You time kernel functions accurately → You understand BPF timing
You build histograms in-kernel → You understand in-kernel aggregation
You support userspace functions → You understand uprobes
You handle edge cases → You’re building production-quality tools

Project 8: CPU Stack Profiler (profile Clone)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 2. The “Micro-SaaS / Pro Tool”
Difficulty: Level 3: Advanced
Knowledge Area: Performance Profiling / CPU Analysis
Software or Tool: libbpf, perf_event
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A sampling profiler that captures stack traces at regular intervals to show where CPU time is being spent. Output in a format suitable for flame graph generation.

Why it teaches BPF: This project combines BPF with perf events for CPU sampling, teaches you about stack unwinding, and produces data for visualization. This is the foundation of production profilers.

Core challenges you’ll face:

CPU sampling with perf events → maps to PERF_TYPE_SOFTWARE, PERF_COUNT_SW_CPU_CLOCK
Capturing kernel and user stacks → maps to bpf_get_stack(), stack traces
Symbol resolution → maps to /proc/kallsyms, DWARF, frame pointers
Flame graph generation → maps to folded stacks format

Key Concepts:

CPU Profiling: “BPF Performance Tools” Chapter 6 - Brendan Gregg
Flame Graphs: Brendan Gregg Flame Graphs
Stack Walking: “BPF Performance Tools” Chapter 2.7 - Brendan Gregg
perf_event Integration: “Learning eBPF” Chapter 7 - Liz Rice

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Projects 1-7 completed

Real world outcome:

$ sudo ./stackprof -p 1234 -d 10  # Profile PID 1234 for 10 seconds
Profiling PID 1234 for 10 seconds at 99 Hz...

Collected 990 samples

# Output folded stacks for flame graph
$ sudo ./stackprof -p 1234 -d 10 -f > stacks.folded
$ flamegraph.pl stacks.folded > profile.svg

Top functions by sample count:
  pthread_mutex_lock      234  (23.6%)
  __GI___libc_read        198  (20.0%)
  __memcpy_avx_unaligned  156  (15.8%)
  process_request         123  (12.4%)
  parse_json               89   (9.0%)

Implementation Hints:

Sampling architecture:

┌─────────────────────────────────────────────────────┐
│ perf_event (CPU timer)                              │
│   triggers BPF program at 99 Hz (per CPU)          │
└──────────────────┬──────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────┐
│ BPF Program                                         │
│   1. Get kernel stack trace (bpf_get_stack)        │
│   2. Get user stack trace                           │
│   3. Store in stack_traces map                      │
│   4. Increment count for this stack                 │
└──────────────────┬──────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────┐
│ Userspace                                           │
│   1. Read stack counts from map                     │
│   2. Resolve addresses to symbols                   │
│   3. Output in folded stack format                  │
└─────────────────────────────────────────────────────┘

Why 99 Hz?: Avoids lock-step with system timers, gives ~1% overhead.

Folded stack format:

processname;func1;func2;func3 count
nginx;ngx_http_process_request;ngx_http_parse_uri;memcpy 45

Symbol resolution challenges:

Kernel: /proc/kallsyms
Userspace: Need frame pointers OR DWARF unwinding
JIT (Java, Node.js): Need special handling

Questions to guide your implementation:

Why sample at 99 Hz instead of 100 Hz?
What’s the difference between kernel and user stacks?
How do you handle missing frame pointers?
How do you limit overhead on production systems?

Learning milestones:

You sample CPU at regular intervals → You understand perf_event + BPF
You capture stack traces → You understand stack unwinding
You resolve symbols → You understand address→symbol mapping
You generate flame graphs → You can visualize performance data

Project 9: Memory Allocation Tracer (memleak Clone)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 2. The “Micro-SaaS / Pro Tool”
Difficulty: Level 4: Expert
Knowledge Area: Memory Debugging / Leak Detection
Software or Tool: libbpf, uprobes
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A tool that tracks memory allocations and deallocations, identifying memory leaks by finding allocations that are never freed, along with their stack traces.

Why it teaches BPF: Memory tracking requires correlating allocation and deallocation events across time, maintaining state for potentially millions of outstanding allocations. This teaches advanced map usage and performance optimization.

Core challenges you’ll face:

Tracing malloc/free → maps to libc uprobes, USDT probes
Tracking outstanding allocations → maps to hash maps keyed by address
Handling high allocation rates → maps to sampling, filtering
Stack trace for each allocation → maps to bpf_get_stack, storage efficiency

Key Concepts:

Memory Allocation: “The Linux Programming Interface” Chapter 7 - Kerrisk
memleak Tool: BCC memleak
USDT Probes: “BPF Performance Tools” Chapter 12 - Brendan Gregg

Difficulty: Expert Time estimate: 2-3 weeks Prerequisites: Projects 1-8 completed, understanding of memory allocation

Real world outcome:

$ sudo ./memleak -p 1234 --top 10
Tracing memory allocations in PID 1234...

[After 60 seconds]

Top 10 outstanding allocations by total size:

  4096 bytes in 1024 allocations from:
    main+0x45 [/app/myapp]
    create_buffer+0x23 [/app/myapp]
    process_request+0x67 [/app/myapp]
    handle_connection+0x89 [/app/myapp]

  2048 bytes in 512 allocations from:
    json_parse+0x34 [libcjson.so]
    parse_response+0x56 [/app/myapp]
    handle_api+0x78 [/app/myapp]

Outstanding: 1536 allocations, 6144 bytes total
Rate: 234 allocs/sec, 456 frees/sec

Implementation Hints:

Functions to trace:

Allocations (uprobes on libc):
  malloc(size)         → returns ptr
  calloc(nmemb, size)  → returns ptr
  realloc(ptr, size)   → returns new_ptr
  mmap(...)            → returns ptr

Deallocations:
  free(ptr)
  munmap(ptr, len)
  realloc(ptr, 0)      → acts as free

State management:

On allocation:
Record: address → {size, timestamp, stack_trace}
Store in hash map

On free:
Look up address
Delete from map
(Optional: track total freed)

Periodically / On exit:
Remaining entries = potential leaks
Group by stack trace
Sort by size

Performance considerations:

High-allocation programs can have millions of allocs/sec
Consider sampling (only trace 1% of allocations)
Use LRU maps if memory is a concern
Stack traces are expensive—maybe only capture on suspected leaks

Questions to guide your implementation:

How do you handle realloc (both allocation and deallocation)?
What about allocations in libraries (not just the main program)?
How do you handle programs with custom allocators?
What’s the memory overhead of tracking all allocations?

Learning milestones:

You track malloc/free pairs → You understand allocation tracing
You identify leaking allocations → You understand state correlation
You show stack traces for leaks → You understand allocation context
You handle high-volume allocation → You understand performance optimization

Project 10: XDP Packet Firewall (DDoS Protection)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Rust (aya)
Coolness Level: Level 5: Pure Magic
Business Potential: 4. The “Open Core” Infrastructure
Difficulty: Level 4: Expert
Knowledge Area: Networking / Security / XDP
Software or Tool: libbpf, XDP
Main Book: “Learning eBPF” by Liz Rice

What you’ll build: A high-performance packet filtering firewall using XDP that can block malicious traffic, rate-limit connections, and protect against DDoS attacks—all at line rate.

Why it teaches BPF: This combines XDP packet processing with dynamic rule management. You’ll build a system that can drop millions of packets per second while maintaining manageable rules from userspace.

Core challenges you’ll face:

Rule storage and lookup → maps to LPM trie maps for CIDR, hash maps for IPs
Rate limiting → maps to token bucket algorithm in BPF
Dynamic rule updates → maps to map updates from userspace
Statistics and logging → maps to counters, ring buffers

Resources for key challenges:

Key Concepts:

XDP Deep Dive: “Learning eBPF” Chapter 8 - Liz Rice
LPM Trie Maps: Cloudflare on LPM
Rate Limiting: Token Bucket Algorithm
DDoS Mitigation: Cloudflare XDP

Difficulty: Expert Time estimate: 3-4 weeks Prerequisites: Projects 1-9 completed (especially Project 5)

Real world outcome:

$ sudo ./xdp-firewall eth0

XDP Firewall loaded on eth0 (native mode)

# Add rules via CLI
$ sudo ./xdp-fw add block 192.168.1.100
Rule added: BLOCK 192.168.1.100/32

$ sudo ./xdp-fw add block 10.0.0.0/8
Rule added: BLOCK 10.0.0.0/8

$ sudo ./xdp-fw add ratelimit 0.0.0.0/0 --pps 10000
Rule added: RATE_LIMIT 0.0.0.0/0 @ 10000 pps

# Show statistics
$ sudo ./xdp-fw stats
RULE                        MATCHED     DROPPED     PASSED
────────────────────────────────────────────────────────────
BLOCK 192.168.1.100/32     12,345,678  12,345,678         0
BLOCK 10.0.0.0/8               45,678      45,678         0
RATE_LIMIT 0.0.0.0/0       98,765,432   1,234,567  97,530,865

Total: 98.7M packets, 1.3M dropped (1.3%), 14.2 Mpps

Implementation Hints:

Architecture:

Userspace CLI                    BPF Program (XDP)
┌────────────┐                  ┌────────────────────────────┐
│ Add rule   │──────────────────│ Parse packet headers       │
│ Del rule   │   (map updates)  │ ↓                          │
│ Show stats │◄─────────────────│ Lookup src IP in rules map │
│            │   (map reads)    │ ↓                          │
└────────────┘                  │ If BLOCK: return XDP_DROP  │
                                │ If RATE_LIMIT: check tokens│
                                │ If PASS: return XDP_PASS   │
                                │ ↓                          │
                                │ Update stats counters      │
                                └────────────────────────────┘

Map types needed:

Rules (LPM Trie for CIDR matching):
  key: {prefix_len, ip_address}
  value: {action, rate_limit_pps}

Rate limit state (Hash):
  key: {source_ip}
  value: {tokens, last_update_ns}

Statistics (Per-CPU Array):
  key: rule_index
  value: {packets_matched, packets_dropped, bytes}

Rate limiting in BPF:

Token bucket algorithm:
Get current time
Calculate tokens to add since last update
If tokens >= 1: allow packet, decrement tokens
If tokens < 1: drop packet

Questions to guide your implementation:

How do you handle IPv6?
What about TCP vs UDP vs ICMP different rules?
How do you avoid lock contention on shared state?
What’s the performance impact of more rules?

Learning milestones:

You block IPs with a simple rule → You understand XDP packet decisions
You implement CIDR matching → You understand LPM trie maps
You add rate limiting → You understand stateful packet processing
You achieve millions of drops/sec → You understand XDP performance

Project 11: Container Syscall Filter (seccomp-bpf)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C
Alternative Programming Languages: Rust, Go
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 4. The “Open Core” Infrastructure
Difficulty: Level 4: Expert
Knowledge Area: Security / Containers / Sandboxing
Software or Tool: seccomp-bpf, libseccomp
Main Book: “Container Security” by Liz Rice

What you’ll build: A syscall filtering system using seccomp-bpf that restricts what system calls a process can make, with policy generation based on observed behavior—similar to how Docker and Kubernetes apply security profiles.

Why it teaches BPF: This is where BPF began (Berkeley Packet Filter for syscalls!). You’ll understand how containers are sandboxed, write cBPF programs, and see the security model that protects containers.

Core challenges you’ll face:

Understanding cBPF vs eBPF → maps to seccomp uses classic BPF
Policy specification → maps to allow lists, deny lists, argument filtering
Profile generation → maps to tracing to build profiles
Error handling → maps to SECCOMP_RET_ actions*

Key Concepts:

seccomp-bpf: Linux Kernel seccomp Docs
Container Security: “Container Security” Chapter 8 - Liz Rice
libseccomp: libseccomp Library

Difficulty: Expert Time estimate: 2-3 weeks Prerequisites: Projects 1-6 completed, understanding of system calls

Real world outcome:

# Generate a profile by observing a program
$ sudo ./seccomp-profile --trace /usr/bin/myapp -- --some-args
Tracing syscalls for 30 seconds...

Observed syscalls:
  read, write, open, close, mmap, mprotect, brk
  rt_sigaction, rt_sigprocmask, ioctl, access
  execve, exit_group, futex, clock_gettime

Saved profile to myapp.seccomp.json

# Apply the profile
$ sudo ./seccomp-sandbox --profile myapp.seccomp.json -- /usr/bin/myapp
[myapp runs sandboxed]

# Attempt to use blocked syscall
[myapp] Killed: syscall ptrace blocked by seccomp filter

Implementation Hints:

seccomp-bpf filter structure:

┌─────────────────────────────────────────┐
│ struct seccomp_data {                   │
│   int   nr;        // syscall number    │
│   __u32 arch;      // architecture      │
│   __u64 instruction_pointer;            │
│   __u64 args[6];   // syscall arguments │
│ }                                       │
└─────────────────────────────────────────┘

BPF program examines this and returns:
  SECCOMP_RET_ALLOW    - Allow syscall
  SECCOMP_RET_KILL     - Kill process
  SECCOMP_RET_ERRNO    - Return error
  SECCOMP_RET_TRACE    - Notify tracer
  SECCOMP_RET_LOG      - Log and allow

Policy approaches:

Deny-list (block specific syscalls):
  - Easy to implement
  - Dangerous: miss one and you're vulnerable

Allow-list (permit specific syscalls):
  - More secure
  - Requires knowing all needed syscalls
  - Can use tracing to generate

Profile generation:

Use eBPF to trace all syscalls of target process
Build list of observed syscalls
Generate seccomp filter that allows only those
Apply filter to sandboxed execution

Questions to guide your implementation:

Why does seccomp use classic BPF, not eBPF?
How do you handle architecture differences (x86 vs ARM)?
What about syscall argument filtering?
How do you handle dynamically-loaded libraries that need more syscalls?

Learning milestones:

You block a syscall and see EPERM → You understand basic seccomp
You generate profiles from tracing → You understand policy generation
You sandbox a real application → You understand practical container security
You handle edge cases → You understand production security

Project 12: SSL/TLS Key Logger (Encrypted Traffic Visibility)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 5: Pure Magic
Business Potential: 3. The “Service & Support” Model
Difficulty: Level 4: Expert
Knowledge Area: Security / Cryptography / Debugging
Software or Tool: libbpf, uprobes, OpenSSL
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A tool that uses uprobes to intercept SSL/TLS encryption functions and capture pre-encryption plaintext, enabling visibility into encrypted traffic for debugging—without needing certificates.

Why it teaches BPF: This demonstrates the power of uprobes for userspace tracing. You’ll hook into library functions, understand how encryption works, and see why eBPF is both powerful and potentially dangerous.

Core challenges you’ll face:

Finding the right functions → maps to SSL_read, SSL_write in OpenSSL
Reading encrypted/plaintext data → maps to buffer parameters, return values
Supporting multiple libraries → maps to OpenSSL, BoringSSL, GnuTLS
Handling high throughput → maps to ring buffers, filtering

Key Concepts:

uprobe Tracing: “Learning eBPF” Chapter 7 - Liz Rice
SSL/TLS Internals: “Serious Cryptography” Chapter 15 - Aumasson
sslsniff Tool: BCC sslsniff

Difficulty: Expert Time estimate: 2-3 weeks Prerequisites: Projects 1-7 completed, basic TLS understanding

Real world outcome:

$ sudo ./sslsniff -p 1234  # PID of curl or browser
Tracing SSL/TLS in PID 1234...

FUNC       DIRECTION  LEN    DATA
────────────────────────────────────────────────────────────────
SSL_write  →          156    GET / HTTP/1.1\r\nHost: example.com\r\n...
SSL_read   ←          1234   HTTP/1.1 200 OK\r\nContent-Type: text/html...
SSL_write  →          89     GET /api/data HTTP/1.1\r\nAuth: Bearer eyJ...
SSL_read   ←          567    {"status":"ok","data":[{"id":1,"name":"...

# Save to file in SSLKEYLOGFILE format for Wireshark
$ sudo ./sslsniff --keylog > sslkeys.log

Implementation Hints:

Functions to trace:

OpenSSL:
  SSL_write(ssl, buf, num) → Returns bytes written
    - buf contains plaintext being sent
  SSL_read(ssl, buf, num)  → Returns bytes read
    - buf contains decrypted received data

At function entry: buf pointer is available
At function return: we know how many bytes were used

Dual-probe pattern:

uprobe/SSL_write:
  1. Save buf pointer and size to map (key = pid_tgid)

uretprobe/SSL_write:
  1. Look up saved buf pointer
  2. Read 'return value' bytes from buf
  3. Send to userspace via ring buffer

Library detection:

Different apps use different libraries:
  - OpenSSL: libssl.so
  - BoringSSL: libcrypto.so (different symbols)
  - GnuTLS: libgnutls.so
  - NSS: libnss3.so

Need to detect which library is loaded and attach to correct functions

Security considerations:

This tool can read all TLS traffic on a system!
Requires root (CAP_BPF)
Should only be used for debugging YOUR OWN applications
Great for understanding why eBPF requires privileges

Learning milestones:

You capture plaintext from SSL_write → You understand uprobe data access
You show both directions → You understand request/response correlation
You handle multiple SSL libraries → You understand library detection
You understand the security implications → You understand eBPF power and responsibility

Project 13: DNS Traffic Monitor (Protocol Parsing)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Go (cilium/ebpf), Rust (aya)
Coolness Level: Level 3: Genuinely Clever
Business Potential: 3. The “Service & Support” Model
Difficulty: Level 3: Advanced
Knowledge Area: Networking / Protocol Analysis
Software or Tool: libbpf, XDP or TC
Main Book: “BPF Performance Tools” by Brendan Gregg

What you’ll build: A DNS traffic monitor that parses DNS packets, extracts queries and responses, and provides visibility into DNS behavior—useful for security monitoring and debugging.

Why it teaches BPF: This combines XDP/TC packet access with application-layer protocol parsing. You’ll learn to safely parse complex protocol structures in the constrained BPF environment.

Core challenges you’ll face:

Parsing DNS packet format → maps to header, questions, answers sections
Extracting domain names → maps to label format, compression pointers
Handling various record types → maps to A, AAAA, CNAME, MX, TXT
Correlating queries and responses → maps to transaction ID tracking

Key Concepts:

DNS Protocol: “TCP/IP Illustrated Volume 1” Chapter 14 - Stevens
DNS RFC: RFC 1035 (Domain Implementation)
Packet Parsing in BPF: Datadog eBPF Guide

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Projects 1-6 completed, basic DNS knowledge

Real world outcome:

$ sudo ./dnsmon eth0
Monitoring DNS traffic on eth0...

TIME       TYPE  SRC              DST              QUERY                    RESP
───────────────────────────────────────────────────────────────────────────────────
14:23:01   Q     192.168.1.10     8.8.8.8          www.google.com          -
14:23:01   R     8.8.8.8          192.168.1.10     www.google.com          142.250.80.46
14:23:02   Q     192.168.1.10     8.8.8.8          api.github.com          -
14:23:02   R     8.8.8.8          192.168.1.10     api.github.com          140.82.113.5
14:23:05   Q     192.168.1.10     8.8.8.8          malware.bad.com         -
14:23:05   R     8.8.8.8          192.168.1.10     malware.bad.com         NXDOMAIN

[Queries: 1,234 | Responses: 1,230 | NXDOMAIN: 4]

Implementation Hints:

DNS packet structure:

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                      ID                       |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    QDCOUNT                    |
|                    ANCOUNT                    |
|                    NSCOUNT                    |
|                    ARCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                   Questions                   |
|                   Answers                     |
|                   Authority                   |
|                   Additional                  |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Domain name format:

www.google.com encoded as:
  03 w w w 06 g o o g l e 03 c o m 00
  ↑ len    ↑ len           ↑ len    ↑ null terminator

Parsing challenges in BPF:

Loops are limited (must unroll or bound)
Can’t follow compression pointers arbitrarily
Must bounds-check every access
Limited stack space for storing extracted names

Questions to guide your implementation:

How do you detect DNS traffic (port 53)?
How do you handle DNS over TCP (different framing)?
What about DNS over HTTPS (DoH) or DNS over TLS (DoT)?
How do you handle truncated responses?

Learning milestones:

You extract query domain names → You understand DNS encoding
You parse response records → You understand record types
You correlate queries and responses → You understand stateful parsing
You handle edge cases → You understand protocol robustness

Project 14: BPF-based Load Balancer (L4 XDP)

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: C (libbpf)
Alternative Programming Languages: Rust (aya)
Coolness Level: Level 5: Pure Magic
Business Potential: 5. The “Industry Disruptor”
Difficulty: Level 5: Master
Knowledge Area: Networking / Load Balancing / Infrastructure
Software or Tool: libbpf, XDP
Main Book: “Learning eBPF” by Liz Rice

What you’ll build: A Layer 4 load balancer using XDP that distributes incoming TCP connections across backend servers using consistent hashing—the foundation of technologies like Cilium and Katran.

Why it teaches BPF: This is production-grade eBPF. You’ll implement connection tracking, NAT, consistent hashing, and health checking—all at XDP speeds. This is how Facebook, Cloudflare, and Netflix handle millions of connections.

Core challenges you’ll face:

Connection tracking → maps to conntrack maps, 5-tuple hashing
Consistent hashing → maps to Maglev hashing or similar
NAT and checksum updates → maps to rewriting headers correctly
Health checking → maps to backend state management

Resources for key challenges:

Key Concepts:

XDP Load Balancing: “Learning eBPF” Chapter 8 - Liz Rice
Consistent Hashing: Maglev: A Fast and Reliable Software Network Load Balancer
Katran Architecture: Facebook Katran Blog

Difficulty: Master Time estimate: 4-6 weeks Prerequisites: All previous projects completed, deep networking knowledge

Real world outcome:

$ sudo ./xdp-lb eth0 --vip 10.0.0.100:80

XDP Load Balancer running
VIP: 10.0.0.100:80

$ sudo ./xdp-lb backend add 192.168.1.10:8080 --weight 3
$ sudo ./xdp-lb backend add 192.168.1.11:8080 --weight 2
$ sudo ./xdp-lb backend add 192.168.1.12:8080 --weight 1

Backends:
  192.168.1.10:8080  weight=3  health=OK   conns=15234
  192.168.1.11:8080  weight=2  health=OK   conns=10156
  192.168.1.12:8080  weight=1  health=OK   conns=5078

Statistics:
  Total connections: 30,468
  Active connections: 234
  Packets/sec: 145,678
  Throughput: 1.2 Gbps
  Drop rate: 0.001%

Implementation Hints:

Load balancer architecture:

Incoming Packet                      Outgoing Packet
┌────────────────┐                  ┌────────────────┐
│ DST: VIP:80    │                  │ DST: Backend   │
│ SRC: Client    │   ──────────►    │ SRC: Client    │
└────────────────┘                  └────────────────┘
        │                                   │
        ▼                                   │
┌───────────────────────────────────────────┼─────┐
│ XDP Load Balancer                         │     │
│                                           │     │
│ 1. Parse packet headers                   │     │
│ 2. Check if DST = VIP                     │     │
│ 3. Look up existing connection ──────────►│     │
│    (if found, use same backend)           │     │
│ 4. Else: hash(src_ip, src_port, dst_port) │     │
│    → select backend from consistent hash  │     │
│ 5. Store connection mapping               │     │
│ 6. Rewrite DST IP/Port                    │     │
│ 7. Update checksums                       │     │
│ 8. XDP_TX (send out)                      │     │
└─────────────────────────────────────────────────┘

Key data structures:

Connection table (LRU Hash):
  key: {src_ip, src_port, dst_port, protocol}
  value: {backend_idx, timestamp}

Backend table (Array):
  key: backend_index
  value: {ip, port, weight, health_status}

Consistent hash table (Array):
  key: hash_bucket
  value: backend_index

Checksum updates: When you change IP/port, you must update:

IP header checksum
TCP/UDP checksum

BPF has helpers: bpf_csum_diff(), incremental checksum updates.

Questions to guide your implementation:

How does consistent hashing ensure the same client always hits the same backend?
What happens when a backend fails?
How do you handle connection draining?
What about direct server return (DSR)?

Learning milestones:

You forward packets to backends → You understand XDP packet modification
You implement consistent hashing → You understand load distribution
You track connections → You understand stateful load balancing
You handle backend failures → You understand production requirements

Project 15: eBPF-based Observability Agent

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: Go (cilium/ebpf)
Alternative Programming Languages: Rust (aya), C (libbpf)
Coolness Level: Level 4: Hardcore Tech Flex
Business Potential: 4. The “Open Core” Infrastructure
Difficulty: Level 4: Expert
Knowledge Area: Observability / Distributed Systems
Software or Tool: cilium/ebpf, Prometheus, Grafana
Main Book: “Learning eBPF” by Liz Rice

What you’ll build: A complete observability agent that uses eBPF to collect metrics (CPU, memory, network, disk), traces (function calls, syscalls), and events—exposing them via Prometheus metrics and structured logs.

Why it teaches BPF: This ties together everything: multiple BPF program types, various maps, userspace integration, and production concerns like performance and reliability.

Core challenges you’ll face:

Multiple data sources → maps to combining tracepoints, kprobes, XDP
Metric aggregation → maps to in-kernel vs userspace aggregation
Configuration management → maps to dynamic program loading
Production reliability → maps to error handling, resource limits

Key Concepts:

Observability Pipelines: “Learning eBPF” Chapter 9 - Liz Rice
Prometheus Metrics: Prometheus Best Practices
Production eBPF: How Netflix Uses eBPF

Difficulty: Expert Time estimate: 4-6 weeks Prerequisites: Projects 1-10 completed, familiarity with Go

Real world outcome:

$ sudo ./ebpf-agent --config agent.yaml
eBPF Observability Agent v1.0.0
Loading BPF programs...
  ✓ syscall_counter (tracepoint)
  ✓ tcp_tracker (kprobe)
  ✓ file_monitor (kprobe)
  ✓ net_stats (XDP)

Metrics server: http://localhost:9090/metrics
Log output: /var/log/ebpf-agent/

# Prometheus metrics
$ curl localhost:9090/metrics
# HELP ebpf_syscalls_total Total system calls by type
# TYPE ebpf_syscalls_total counter
ebpf_syscalls_total{syscall="read"} 1234567
ebpf_syscalls_total{syscall="write"} 987654
ebpf_syscalls_total{syscall="openat"} 123456

# HELP ebpf_tcp_connections Active TCP connections
# TYPE ebpf_tcp_connections gauge
ebpf_tcp_connections{direction="outgoing"} 234
ebpf_tcp_connections{direction="incoming"} 567

# HELP ebpf_network_bytes_total Network bytes by direction
# TYPE ebpf_network_bytes_total counter
ebpf_network_bytes_total{direction="rx"} 12345678901
ebpf_network_bytes_total{direction="tx"} 9876543210

Implementation Hints:

Agent architecture:

┌─────────────────────────────────────────────────────┐
│                  eBPF Agent                          │
│                                                      │
│  ┌──────────────────────────────────────────────┐   │
│  │           BPF Program Manager                 │   │
│  │  ┌─────────┐ ┌─────────┐ ┌─────────┐        │   │
│  │  │ Syscall │ │   TCP   │ │   Net   │        │   │
│  │  │ Counter │ │ Tracker │ │  Stats  │ ...    │   │
│  │  └────┬────┘ └────┬────┘ └────┬────┘        │   │
│  │       │           │           │              │   │
│  │  ┌────┴───────────┴───────────┴────┐        │   │
│  │  │         BPF Maps (shared)       │        │   │
│  │  └────────────────┬────────────────┘        │   │
│  └───────────────────┼──────────────────────────┘   │
│                      │                              │
│  ┌───────────────────┴──────────────────────────┐   │
│  │           Metrics Collector                   │   │
│  │  - Read maps periodically                     │   │
│  │  - Aggregate and label                        │   │
│  │  - Expose via Prometheus                      │   │
│  └──────────────────────────────────────────────┘   │
│                                                      │
│  ┌──────────────────────────────────────────────┐   │
│  │           Event Processor                     │   │
│  │  - Read from ring buffers                     │   │
│  │  - Enrich with context                        │   │
│  │  - Send to log destination                    │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

Production considerations:

Resource management:
  - Limit map sizes
  - Bound event rates
  - Monitor CPU overhead

Error handling:
  - Program load failures
  - Map operation errors
  - Attach point unavailable

Configuration:
  - Which programs to load
  - Filtering rules
  - Export destinations

Why Go with cilium/ebpf?

Excellent library support
Good for long-running daemons
Easy Prometheus integration
Strong typing for BPF interactions

Learning milestones:

You load multiple BPF programs → You understand program management
You export metrics to Prometheus → You understand observability integration
You handle configuration → You understand production agents
You deploy and monitor the agent → You’ve built a real observability tool

Project Comparison Table

Project	Difficulty	Time	Depth of Understanding	Fun Factor
1. Hello World Tracer	⭐	Weekend	⚡⚡	🎮🎮🎮
2. System Call Monitor	⭐⭐	1-2 weeks	⚡⚡⚡	🎮🎮🎮
3. Process Execution Logger	⭐⭐	1-2 weeks	⚡⚡⚡	🎮🎮🎮🎮
4. File Access Auditor	⭐⭐	1 week	⚡⚡⚡	🎮🎮🎮
5. Network Packet Counter	⭐⭐⭐	1-2 weeks	⚡⚡⚡⚡	🎮🎮🎮🎮
6. TCP Connection Tracker	⭐⭐⭐	2 weeks	⚡⚡⚡⚡	🎮🎮🎮🎮
7. Function Latency Histogram	⭐⭐⭐	1-2 weeks	⚡⚡⚡⚡	🎮🎮🎮
8. CPU Stack Profiler	⭐⭐⭐	2 weeks	⚡⚡⚡⚡⚡	🎮🎮🎮🎮🎮
9. Memory Allocation Tracer	⭐⭐⭐⭐	2-3 weeks	⚡⚡⚡⚡⚡	🎮🎮🎮🎮
10. XDP Packet Firewall	⭐⭐⭐⭐	3-4 weeks	⚡⚡⚡⚡⚡	🎮🎮🎮🎮🎮
11. Container Syscall Filter	⭐⭐⭐⭐	2-3 weeks	⚡⚡⚡⚡⚡	🎮🎮🎮🎮
12. SSL/TLS Key Logger	⭐⭐⭐⭐	2-3 weeks	⚡⚡⚡⚡	🎮🎮🎮🎮🎮
13. DNS Traffic Monitor	⭐⭐⭐	2 weeks	⚡⚡⚡⚡	🎮🎮🎮🎮
14. BPF-based Load Balancer	⭐⭐⭐⭐⭐	4-6 weeks	⚡⚡⚡⚡⚡	🎮🎮🎮🎮🎮
15. eBPF Observability Agent	⭐⭐⭐⭐	4-6 weeks	⚡⚡⚡⚡⚡	🎮🎮🎮🎮🎮

Recommended Learning Path

Your Starting Point

Based on typical backgrounds:

If you’re a Linux admin/SRE: Start with Project 1 (bpftrace one-liners) → You’ll get immediate value and understand what BPF can observe.

If you’re a systems programmer: Start with Project 2 (C with libbpf) → Jump straight into the development workflow.

If you’re into networking: Start with Project 5 (XDP) → XDP is where BPF really shines for networking.

If you’re into security: Start with Project 11 (seccomp) → Understand how containers are sandboxed.

Recommended Sequence

Phase 1: Foundation (1-2 weeks)
└── Project 1: Hello World → Get comfortable with tracing concepts

Phase 2: Core Development (4-6 weeks)
├── Project 2: System Call Monitor → Learn the development workflow
├── Project 3: Process Execution Logger → Learn event streaming
└── Project 4: File Access Auditor → Solidify tracing skills

Phase 3: Networking & Performance (6-8 weeks)
├── Project 5: Network Packet Counter → Enter the XDP world
├── Project 6: TCP Connection Tracker → Network protocol tracing
├── Project 7: Function Latency → Performance measurement
└── Project 8: CPU Profiler → Stack traces and flame graphs

Phase 4: Advanced Topics (6-10 weeks)
├── Project 9: Memory Tracer → Complex state management
├── Project 10: XDP Firewall → Production XDP
├── Project 11: Syscall Filter → Security enforcement
└── Project 12: SSL Tracer → Userspace tracing power

Phase 5: Integration & Production (4-8 weeks)
├── Project 13: DNS Monitor → Protocol parsing
├── Project 14: Load Balancer → Production infrastructure
└── Project 15: Observability Agent → Tie it all together

Final Overall Project: Production eBPF Security Platform

File: LEARN_BPF_EBPF_LINUX.md
Main Programming Language: Go (cilium/ebpf) + C (BPF programs)
Alternative Programming Languages: Rust (aya) + C
Coolness Level: Level 5: Pure Magic
Business Potential: 5. The “Industry Disruptor”
Difficulty: Level 5: Master
Knowledge Area: Security / Observability / Infrastructure
Software or Tool: libbpf, cilium/ebpf, Kubernetes
Main Book: “Learning eBPF” by Liz Rice + “Container Security” by Liz Rice

What you’ll build: A complete eBPF-based security platform that combines:

Runtime detection: Monitor system calls, file access, network connections
Policy enforcement: Block suspicious activities in real-time
Container awareness: Understand container/pod context
Network visibility: See all network flows, DNS queries, TLS connections
Anomaly detection: Baseline normal behavior, alert on deviations

Why this is the ultimate BPF project: This is what companies like Falco, Tetragon, Cilium, and Tracee are building. You’ll combine everything you’ve learned into a production-grade security tool.

Core challenges you’ll face:

Multi-program coordination → Combine tracers, enforcers, network monitors
Container/Kubernetes integration → Correlate events with pods/containers
Real-time policy engine → Evaluate rules efficiently in kernel space
Scalability → Handle thousands of containers, millions of events
False positive reduction → Balance detection sensitivity

What you’ll build (minimum viable product):

┌─────────────────────────────────────────────────────────────┐
│                    eBPF Security Platform                    │
│                                                              │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ Detection Engine                                         │ │
│  │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐ │ │
│  │ │ Syscall  │ │  File    │ │ Network  │ │  Process    │ │ │
│  │ │ Monitor  │ │ Monitor  │ │ Monitor  │ │  Monitor    │ │ │
│  │ └──────────┘ └──────────┘ └──────────┘ └─────────────┘ │ │
│  └─────────────────────────────────────────────────────────┘ │
│                              │                               │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ Enrichment Layer                                         │ │
│  │ - Container/Pod lookup                                   │ │
│  │ - User/Group resolution                                  │ │
│  │ - Parent process chain                                   │ │
│  └─────────────────────────────────────────────────────────┘ │
│                              │                               │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ Policy Engine                                            │ │
│  │ - Detection rules (YAML-defined)                         │ │
│  │ - Enforcement actions (alert, block, kill)               │ │
│  │ - Severity classification                                │ │
│  └─────────────────────────────────────────────────────────┘ │
│                              │                               │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ Output Layer                                             │ │
│  │ - Prometheus metrics                                     │ │
│  │ - Structured logs (JSON)                                 │ │
│  │ - Alert webhooks                                         │ │
│  │ - SIEM integration                                       │ │
│  └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Real world outcome:

$ sudo ./ebpf-security --config security.yaml
eBPF Security Platform v1.0.0

Loading detection rules...
  ✓ detect_reverse_shell (high)
  ✓ detect_crypto_mining (medium)
  ✓ detect_container_escape (critical)
  ✓ detect_sensitive_file_access (medium)

Status: ACTIVE
Monitored containers: 156
Events/sec: 45,678
Alerts (last hour): 3

# Example detection
[ALERT] HIGH - Reverse shell detected
  Container: webapp-frontend-7d8f9
  Pod: webapp/webapp-frontend
  Process: /bin/bash
  Network: 10.0.0.45:443 → 185.234.12.56:4444
  Parent: python → gunicorn → bash
  Rule: detect_reverse_shell
  Action: BLOCKED (killed process)

Difficulty: Master Time estimate: 2-3 months Prerequisites: All 15 projects completed

Learning milestones:

Detection works for basic scenarios → You understand event correlation
Container context is enriched → You understand platform integration
Policies are evaluated in kernel → You understand efficient enforcement
You catch real attacks in a test environment → You’ve built security tooling
You can deploy to a cluster → You’re building production software

Essential Resources

Books (Primary)

Book	Author	Best For
Learning eBPF	Liz Rice	Comprehensive introduction, development workflow
BPF Performance Tools	Brendan Gregg	Performance analysis, tracing techniques
Linux Observability with BPF	Calavera & Fontana	Practical examples, quick start
The Linux Programming Interface	Michael Kerrisk	Syscalls, kernel interfaces
TCP/IP Illustrated, Volume 1	W. Richard Stevens	Networking foundations

Online Resources

Resource	URL	Description
eBPF.io	ebpf.io	Official documentation and tutorials
libbpf-bootstrap	GitHub	Project templates for getting started
BCC Repository	GitHub	Tools and examples
bpftrace	GitHub	One-liner tracing
XDP Tutorial	GitHub	Hands-on XDP learning
Brendan Gregg’s Blog	brendangregg.com	Performance analysis insights
Nakryiko’s Blog	nakryiko.com	libbpf and CO-RE deep dives

Video Resources

Resource	Description
eBPF Summit	Annual conference, excellent talks
Liz Rice eBPF Talks	Practical introductions
Brendan Gregg Talks	Performance deep dives

Summary

#	Project	Main Language	Knowledge Area
1	Hello World Tracer	bpftrace	BPF Basics / Tracing
2	System Call Monitor	C (libbpf)	BPF Programming / System Calls
3	Process Execution Logger	C (libbpf)	Process Tracing / Security Auditing
4	File Access Auditor	C (libbpf)	Filesystem Tracing / Security
5	Network Packet Counter	C (libbpf)	Networking / XDP
6	TCP Connection Tracker	C (libbpf)	Networking / TCP State
7	Function Latency Histogram	C (libbpf)	Performance Profiling
8	CPU Stack Profiler	C (libbpf)	Performance Profiling / CPU Analysis
9	Memory Allocation Tracer	C (libbpf)	Memory Debugging / Leak Detection
10	XDP Packet Firewall	C (libbpf)	Networking / Security / XDP
11	Container Syscall Filter	C	Security / Containers / Sandboxing
12	SSL/TLS Key Logger	C (libbpf)	Security / Cryptography / Debugging
13	DNS Traffic Monitor	C (libbpf)	Networking / Protocol Analysis
14	BPF-based Load Balancer	C (libbpf)	Networking / Load Balancing / Infrastructure
15	eBPF Observability Agent	Go (cilium/ebpf)	Observability / Distributed Systems
Final	Production eBPF Security Platform	Go + C	Security / Observability / Infrastructure

Getting Started Checklist

Before starting Project 1:

Linux system (kernel 5.4+ recommended, 5.15+ for all features)
Install bpftrace: apt install bpftrace or dnf install bpftrace
Install BCC tools: apt install bpfcc-tools
Install development tools: apt install clang llvm libbpf-dev
Verify BPF works: sudo bpftrace -e 'BEGIN { printf("Hello, BPF!\n"); exit(); }'
Read Chapter 1 of “Learning eBPF” by Liz Rice

You’re ready to become a kernel ninja! 🥷

Generated for deep understanding of BPF/eBPF in Linux