Project 5: Live Packet Capture Dashboard

A live packet dashboard that summarizes protocol distribution, top talkers, and captures recent packets.

Quick Reference

Attribute Value
Difficulty Level 3: Advanced
Time Estimate 1 week
Main Programming Language Bash + Python
Alternative Programming Languages Go, Rust
Coolness Level Level 4: Hardcore
Business Potential 3. Service & Support
Prerequisites Basic Linux CLI
Key Topics Packet Capture, BPF, and Observability

1. Learning Objectives

By completing this project, you will:

  1. Build the core tool described in the project and validate output against a golden transcript.
  2. Explain how the tool maps to the Linux networking layer model.
  3. Diagnose at least one real or simulated failure using the tool’s output.

2. All Theory Needed (Per-Concept Breakdown)

This section includes every concept required to implement this project successfully.

Packet Capture, BPF, and Observability

Fundamentals Packet capture is the closest thing you have to ground truth in networking: it reveals what actually traversed the interface, not what a tool inferred. tcpdump is the canonical CLI packet analyzer on Linux, and it relies on libpcap’s filter language (pcap-filter) to select which packets to capture. Filters are essential because capturing “everything” is noisy, expensive, and often unsafe. The skill here is translating a hypothesis into a filter: “show only SYN packets to port 443,” or “show DNS responses larger than 512 bytes.” When you can ask precise questions and capture precise evidence, you move from guesswork to proof.

Deep Dive Packet capture is powerful precisely because it bypasses abstraction. Tools like ss and ip summarize state, but they infer behavior from kernel structures. tcpdump captures actual packets and prints key fields: timestamps, addresses, ports, flags, sequence numbers, and lengths. Those fields are enough to reconstruct protocol behavior. A three-way handshake is visible as SYN, SYN-ACK, ACK. A reset is visible as RST. Loss or retransmission is visible as repeated sequence numbers or missing ACKs. In other words, packet capture is not just “packets,” it is narrative evidence.

Filters make capture usable. The libpcap language supports protocol qualifiers (tcp, udp, icmp), host and network selectors, port selectors, and even byte-level offsets. That means you can express questions like “show all TCP SYN packets from 203.0.113.9” or “show DNS responses with the TC bit set.” The filters run in the kernel, so they reduce overhead and keep captures focused. That is critical on busy servers, where unfiltered capture can drop packets or distort performance. Good operators always constrain scope: the smallest time window, the narrowest filter, and the minimal payload inspection needed to answer the question.

Interpreting output requires protocol literacy. TCP flags reveal connection lifecycle. Sequence and acknowledgment numbers show ordering and loss. Window sizes hint at flow control. UDP lacks a state machine, so you focus on port pairs and timing. ICMP messages often explain failures: “Destination Unreachable” or “Packet Too Big” are not noise — they are direct explanations from the network. If you see an incoming SYN in tcpdump but no SYN_RECV in ss, you know the packet was dropped before socket handling. That simple correlation often pinpoints firewall or routing errors in minutes.

Packet capture also intersects with performance and privacy. On high-throughput links, capturing payloads can be expensive. Many teams capture headers only or truncate payloads to reduce risk. Some environments require explicit approval for packet capture because it can contain sensitive data. The right approach is to capture the minimum necessary data and to document why you captured it. This is part of professional network hygiene: evidence gathering should not become a liability.

Observability is broader than packets. When a link flaps, dmesg records the driver event. When a firewall drops traffic, journalctl may record it if logging is enabled. By correlating packet capture with logs and socket state, you can produce a complete causal chain: “carrier dropped, ARP failed, SYNs were dropped, no socket established.” This multi-source correlation is the difference between “it seems broken” and “here is the exact failure sequence.” That is the standard expected in production incident reports.

Finally, be aware of capture artifacts. Offloads can make captured checksums appear wrong, even when packets are valid. Promiscuous mode affects what you see. Capturing on a bridge or veth interface can show duplicate or transformed packets. These artifacts are not bugs; they are features of modern networking stacks. The expert skill is to recognize them, adjust the capture point or filter, and interpret results in context. This chapter trains that discipline so your packet evidence is both correct and actionable.

How this fit on projects

  • Live Packet Capture Dashboard (Project 5)
  • Network Log Analyzer (Project 10)
  • Real-Time Network Security Monitor (Project 15)

Definitions & key terms

  • pcap filter: Expression language used by libpcap/tcpdump to select packets.
  • Capture scope: The time window and filter criteria that bound a capture.
  • Kernel ring buffer: In-memory log of kernel messages (dmesg).

Mental model diagram

Packet -> kernel -> tcpdump (filtered)
         |                 |
         |                 +-- evidence (flags, ports, timing)
         +-- logs (dmesg/journalctl)

How it works (step-by-step, invariants, failure modes)

  1. Apply BPF filter in kernel.
  2. Capture matching packets.
  3. Interpret headers and flags.
  4. Correlate with socket state and logs. Invariants: filters are applied before capture; tcpdump output is time ordered. Failure modes: capturing too much, missing packets due to filter mistakes.

Minimal concrete example Packet transcript (simplified):

12:00:01 IP 192.0.2.10.52341 > 198.51.100.20.443: Flags [S]
12:00:01 IP 198.51.100.20.443 > 192.0.2.10.52341: Flags [S.]
12:00:01 IP 192.0.2.10.52341 > 198.51.100.20.443: Flags [.]

Common misconceptions

  • “tcpdump shows everything.” (It only shows what you filter and what the NIC sees.)
  • “If tcpdump sees a packet, the app must see it.” (Firewall or routing can still drop it.)

Check-your-understanding questions

  1. Why is filtering in the kernel important?
  2. How can you tell a TCP handshake from tcpdump output?
  3. What kind of evidence would prove a firewall drop?

Check-your-understanding answers

  1. It reduces overhead and prevents excessive capture.
  2. SYN, SYN-ACK, ACK sequence appears in order.
  3. Incoming SYN seen in tcpdump, no SYN_RECV in ss, plus firewall log entry.

Real-world applications

  • Security investigations, performance debugging, and protocol verification.

Where you’ll apply it Projects 5, 10, 15.

References

  • tcpdump description (packet capture and filter expression).
  • pcap filter language (libpcap).
  • dmesg description (kernel ring buffer).
  • journalctl description (systemd journal).

Key insights Packets are the final authority; all other tools are interpretations.

Summary You can now capture targeted traffic and correlate it with logs and socket state to build evidence-backed diagnoses.

Homework/Exercises to practice the concept

  • Write three BPF filters for (a) DNS, (b) HTTPS, (c) TCP SYN only.
  • Sketch a timeline that aligns tcpdump output with socket states.

Solutions to the homework/exercises

  • DNS: udp port 53; HTTPS: tcp port 443; SYN only: tcp[tcpflags] & tcp-syn != 0.
  • A SYN observed should be followed by SYN_RECV in ss; if not, a drop occurred before socket handling.

3. Project Specification

3.1 What You Will Build

A live packet dashboard that summarizes protocol distribution, top talkers, and captures recent packets.

3.2 Functional Requirements

  1. Core data collection: Gather the required system/network data reliably.
  2. Interpretation layer: Translate raw outputs into human-readable insights.
  3. Deterministic output: Produce stable, comparable results across runs.
  4. Error handling: Detect missing privileges, tools, or unsupported interfaces.

3.3 Non-Functional Requirements

  • Performance: Runs in under 5 seconds for baseline mode.
  • Reliability: Handles missing data sources gracefully.
  • Usability: Output is readable without post-processing.

3.4 Example Usage / Output

$ sudo ./packetdash.sh -i eth0

LIVE PACKET CAPTURE
Runtime: 00:05:32

Traffic rate:
  packets/s: 1247
  bytes/s: 892 KB

Protocol split:
  TCP 78%
  UDP 18%
  ICMP 2%

Top talkers:
  192.168.1.100 -> 142.250.80.46 (443)
  10.0.0.5 -> 192.168.1.100 (22)

Recent packets:
  14:32:01 TCP 192.168.1.100:52341 -> 142.250.80.46:443 SYN

3.5 Data Formats / Schemas / Protocols

  • Input: CLI tool output, kernel state, or service logs.
  • Output: A structured report with sections and summarized metrics.

3.6 Edge Cases

  • Missing tool binaries or insufficient permissions.
  • Interfaces or hosts that return no data.
  • Transient states (link flaps, intermittent loss).

3.7 Real World Outcome

$ sudo ./packetdash.sh -i eth0

LIVE PACKET CAPTURE
Runtime: 00:05:32

Traffic rate:
  packets/s: 1247
  bytes/s: 892 KB

Protocol split:
  TCP 78%
  UDP 18%
  ICMP 2%

Top talkers:
  192.168.1.100 -> 142.250.80.46 (443)
  10.0.0.5 -> 192.168.1.100 (22)

Recent packets:
  14:32:01 TCP 192.168.1.100:52341 -> 142.250.80.46:443 SYN

3.7.1 How to Run (Copy/Paste)

$ ./run-project.sh [options]

3.7.2 Golden Path Demo (Deterministic)

Run the tool against a known-good target and verify every section of the output matches the expected format.

3.7.3 If CLI: provide an exact terminal transcript

$ sudo ./packetdash.sh -i eth0

LIVE PACKET CAPTURE
Runtime: 00:05:32

Traffic rate:
  packets/s: 1247
  bytes/s: 892 KB

Protocol split:
  TCP 78%
  UDP 18%
  ICMP 2%

Top talkers:
  192.168.1.100 -> 142.250.80.46 (443)
  10.0.0.5 -> 192.168.1.100 (22)

Recent packets:
  14:32:01 TCP 192.168.1.100:52341 -> 142.250.80.46:443 SYN

4. Solution Architecture

4.1 High-Level Design

[Collector] -> [Parser] -> [Analyzer] -> [Reporter]

4.2 Key Components

Component Responsibility Key Decisions
Collector Gather raw tool output Which tools to call and with what flags
Parser Normalize raw text/JSON Text vs JSON parsing strategy
Analyzer Compute insights Thresholds and heuristics
Reporter Format output Stable layout and readability

4.3 Data Structures (No Full Code)

  • InterfaceRecord: name, state, addresses, stats
  • RouteRecord: prefix, gateway, interface, metric
  • Observation: timestamp, source, severity, message

4.4 Algorithm Overview

Key Algorithm: Evidence Aggregation

  1. Collect raw outputs from tools.
  2. Parse into normalized records.
  3. Apply interpretation rules and thresholds.
  4. Render the final report.

Complexity Analysis:

  • Time: O(n) over number of records
  • Space: O(n) to hold parsed records

5. Implementation Guide

5.1 Development Environment Setup

# Install required tools with your distro package manager

5.2 Project Structure

project-root/
├── src/
│   ├── main
│   ├── collectors/
│   └── formatters/
├── tests/
└── README.md

5.3 The Core Question You’re Answering

“What packets are actually crossing this interface right now?”

5.4 Concepts You Must Understand First

  1. Packet structure
    • Ethernet/IP/TCP header fields.
    • Book Reference: “TCP/IP Illustrated, Vol 1” - Ch. 2-4
  2. BPF filters
    • Qualifiers and boolean logic.
    • Book Reference: “Practical Packet Analysis” - Ch. 4
  3. Capture ethics and scope
    • Minimize data and protect privacy.
    • Book Reference: “The Practice of Network Security Monitoring” - Ch. 5

5.5 Questions to Guide Your Design

  1. How will you avoid capturing sensitive payloads?
  2. Which statistics should be rolling vs cumulative?
  3. How will you detect port scans from SYN patterns?

5.6 Thinking Exercise

Given these packets:

SYN from 203.0.113.9 to ports 20,21,22,23 within 2 seconds

Question: What does this pattern indicate?

5.7 The Interview Questions They’ll Ask

  1. “How do you capture only DNS traffic?”
  2. “What is the BPF filter for SYN packets?”
  3. “How would you detect a port scan using tcpdump?”
  4. “Why should you avoid capturing payloads?”
  5. “What is the difference between tcpdump and Wireshark?”

5.8 Hints in Layers

Hint 1: Use -n -l for parse-friendly output. Hint 2: Start with broad filters, then narrow. Hint 3: Count unique ports per source to detect scans. Hint 4: Limit capture duration to avoid overload.

5.9 Books That Will Help

Topic Book Chapter
Packet capture “Practical Packet Analysis” Ch. 1-6
BPF filters “The Practice of Network Security Monitoring” Ch. 5

5.10 Implementation Phases

Phase 1: Foundation (1-2 days)

  • Define outputs and parse a single tool.
  • Produce a minimal report.

Phase 2: Core Functionality (3-5 days)

  • Add remaining tools and interpretation logic.
  • Implement stable formatting and summaries.

Phase 3: Polish & Edge Cases (2-3 days)

  • Handle missing data and failure modes.
  • Add thresholds and validation checks.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Parsing format Text vs JSON JSON where available More stable parsing
Output layout Table vs sections Sections Readability for humans
Sampling One-shot vs periodic One-shot + optional loop Predictable runtime

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests Validate parsing Parse fixed tool output samples
Integration Tests Validate tool calls Run against a lab host
Edge Case Tests Handle failures Missing tool, no permissions

6.2 Critical Test Cases

  1. Reference run: Output matches golden transcript.
  2. Missing tool: Proper error message and partial report.
  3. Permission denied: Clear guidance for sudo or capabilities.

6.3 Test Data

Input: captured command output
Expected: normalized report with correct totals

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Wrong interface Empty output Verify interface names
Missing privileges Permission errors Use sudo or capabilities
Misparsed output Wrong stats Prefer JSON parsing

7.2 Debugging Strategies

  • Re-run each tool independently to compare raw output.
  • Add a verbose mode that dumps raw data sources.

7.3 Performance Traps

  • Avoid tight loops without sleep intervals.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add colored status markers.
  • Export report to a file.

8.2 Intermediate Extensions

  • Add JSON output mode.
  • Add baseline comparison.

8.3 Advanced Extensions

  • Add multi-host aggregation.
  • Add alerting thresholds.

9. Real-World Connections

9.1 Industry Applications

  • SRE runbooks and on-call diagnostics.
  • Network operations monitoring.
  • tcpdump / iproute2 / nftables
  • mtr / iperf3

9.3 Interview Relevance

  • Demonstrates evidence-based debugging and tool mastery.

10. Resources

10.1 Essential Reading

  • Primary book listed in the main guide.
  • Relevant RFCs and tool manuals.

10.2 Video Resources

  • Conference talks on Linux networking and troubleshooting.