Project 7: Function Latency Histogram (funclatency Clone)

A tool that measures the latency distribution of any kernel or userspace function, displaying results as a histogram. This is essential for performance debugging.

Quick Reference

Attribute Value
Primary Language C (libbpf)
Alternative Languages Go (cilium/ebpf), Rust (aya)
Difficulty Level 3: Advanced
Time Estimate 1-2 weeks
Knowledge Area Performance Profiling
Tooling libbpf
Prerequisites Projects 1-6 completed

What You Will Build

A tool that measures the latency distribution of any kernel or userspace function, displaying results as a histogram. This is essential for performance debugging.

Why It Matters

This project teaches you to use both kprobes (kernel functions) and uprobes (userspace functions), time measurements with nanosecond precision, and in-kernel histogram building.

Core Challenges

  • Attaching to arbitrary functions → maps to dynamic kprobe/uprobe attachment
  • High-precision timing → maps to bpf_ktime_get_ns()
  • Building histograms in-kernel → maps to log2 bucketing, BPF maps
  • Supporting userspace functions → maps to uprobes, symbol resolution

Key Concepts

  • Function Tracing: “BPF Performance Tools” Chapter 4 - Brendan Gregg
  • uprobes: “Learning eBPF” Chapter 7 - Liz Rice
  • Histogram Building: Brendan Gregg on Histograms

Real-World Outcome

# Kernel function latency
$ sudo ./funclatency vfs_read
Tracing vfs_read... Hit Ctrl-C to end.
^C
     nsecs           : count    distribution
       256 -> 511    : 1234    |@@@                               |
       512 -> 1023   : 4567    |@@@@@@@@@@                        |
      1024 -> 2047   : 8901    |@@@@@@@@@@@@@@@@@@@@              |
      2048 -> 4095   : 12345   |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
      4096 -> 8191   : 8234    |@@@@@@@@@@@@@@@@@@@               |
      8192 -> 16383  : 3456    |@@@@@@@@                          |
     16384 -> 32767  : 1234    |@@@                               |
     32768 -> 65535  : 456     |@                                 |
     65536 -> 131071 : 123     |                                  |

# Userspace function latency
$ sudo ./funclatency /usr/lib/libc.so.6:malloc
$ sudo ./funclatency /usr/bin/python:PyDict_GetItem

Implementation Guide

  1. Reproduce the simplest happy-path scenario.
  2. Build the smallest working version of the core feature.
  3. Add input validation and error handling.
  4. Add instrumentation/logging to confirm behavior.
  5. Refactor into clean modules with tests.

Milestones

  • Milestone 1: Minimal working program that runs end-to-end.
  • Milestone 2: Correct outputs for typical inputs.
  • Milestone 3: Robust handling of edge cases.
  • Milestone 4: Clean structure and documented usage.

Validation Checklist

  • Output matches the real-world outcome example
  • Handles invalid inputs safely
  • Provides clear errors and exit codes
  • Repeatable results across runs

References

  • Main guide: LEARN_BPF_EBPF_LINUX.md
  • “BPF Performance Tools” by Brendan Gregg