Project 10: Latency and Jitter Measurement Toolkit

Instrument your RTOS to measure interrupt latency, task jitter, and worst-case timing under load.

Quick Reference

Attribute Value
Difficulty Advanced
Time Estimate 1-2 weeks
Main Programming Language C
Alternative Programming Languages Rust, Ada
Coolness Level Very High
Business Potential High
Prerequisites Projects 1-9, SysTick, preemption, timers
Key Topics Latency measurement, jitter statistics, GPIO timing, DWT cycle counter

1. Learning Objectives

By completing this project, you will:

  1. Measure interrupt latency using GPIO and cycle counters.
  2. Quantify task jitter and worst-case response time.
  3. Build a report summarizing timing performance under load.
  4. Understand how instrumentation affects real-time behavior.

2. All Theory Needed (Per-Concept Breakdown)

2.1 Latency, Jitter, and Worst-Case Timing

Fundamentals

Latency is the delay from an event to the system response, while jitter is the variation in that delay. Worst-case execution time (WCET) is the maximum time a task or ISR can take. In real-time systems, worst-case timing matters more than average performance because missing a deadline can cause system failure. Measuring these metrics allows you to validate whether your RTOS meets its real-time guarantees.

Additional fundamentals: Timing metrics are only meaningful when you define the event boundary clearly. For example, is latency measured to ISR entry or to task completion? That definition changes the number and its interpretation. Establishing clear definitions prevents confusion and makes results comparable across systems.

Deep Dive into the concept

Latency and jitter define system responsiveness. For interrupts, latency includes the time to finish the current instruction, save context, and enter the ISR. For tasks, response time includes scheduling delay plus execution time. Jitter arises from variability in scheduling, interrupts, and cache/memory access. In a simple microcontroller without cache, jitter is often dominated by interrupt masking and competing ISRs. In complex systems, DMA, bus contention, and memory wait states add variability.

Worst-case analysis is difficult because it requires considering all possible interference. In practice, engineers measure worst-case behavior under stress by injecting heavy ISR loads, long critical sections, or high task counts. The goal is to bound the maximum delay. You should design experiments that intentionally stress the system, such as adding a high-frequency ISR or long critical sections, and then measure how latency changes. This provides confidence that the RTOS can still meet deadlines.

Another important distinction is between interrupt latency and response time. Interrupt latency is purely hardware and ISR entry overhead, while response time includes the actual work done before a task can proceed. For example, a high-priority task might be READY but delayed by a critical section. Measuring both helps you identify whether delays come from ISR overhead or scheduler contention.

Finally, instrumentation itself can change timing. GPIO toggles and cycle counter reads add overhead. You must account for this by measuring the instrumentation cost and subtracting it where possible. The best practice is to keep instrumentation minimal and deterministic, and to document the overhead in your timing report.

Additional depth: Timing analysis often distinguishes between interrupt latency and end-to-end response time. For example, an interrupt might be serviced quickly, but the actual work might be delayed if the system is busy or if the work is deferred to a lower-priority task. When you design your measurement toolkit, you should decide which metric you care about for each scenario. For a hard real-time control loop, interrupt latency is critical; for a logging task, response time may be more relevant. Your reporting should explicitly label the metrics so there is no ambiguity.

Another useful concept is the distribution of latency, not just min and max. Two systems might have the same worst-case latency but very different distributions. A system with occasional rare spikes might be acceptable for some applications but not for others. You can extend your measurement toolkit with a simple histogram or percentile calculation (e.g., 99th percentile) to provide more insight. Even if you do not implement the histogram, document the idea so the reader understands how to go beyond min/max.

Finally, worst-case measurements must be reproducible. You should define a fixed test configuration (e.g., a specific ISR load, specific task set, fixed tick rate) and document it in the report. This ensures that future measurements can be compared. Reproducibility is a core principle of real-time engineering: without it, timing guarantees are guesses. Your project should encourage a disciplined approach to measurement.

One more practical angle is to separate "measurement context" from "system context." If you measure latency while UART logging is enabled, you are measuring a system that includes UART overhead. If you want to characterize the kernel, you should run measurements with logging deferred and with peripherals disabled, then explicitly add the overhead back in as a separate experiment. This creates a layered understanding: baseline kernel latency, latency under I/O load, and latency under stress. This layered approach is common in safety reviews and makes your results more credible because you can explain why worst-case numbers increased.

How this fit on projects

Latency and jitter metrics are collected in Sec. 3.1 and Sec. 3.7.2 and reference time services from Project 2 and Project 5.

Definitions & key terms

  • Latency: Time from event to response.
  • Jitter: Variation in response time.
  • WCET: Worst-case execution time.
  • Response time: Time from event to task completion.

Mental model diagram (ASCII)

Event -> [Latency] -> ISR entry -> Task ready -> [Scheduling delay] -> Task runs

How it works (step-by-step, with invariants and failure modes)

  1. Event occurs (interrupt or tick).
  2. Measure time to ISR entry (latency).
  3. Measure task response delay.
  4. Compute jitter as max-min across samples.
  5. Failure mode: instrumentation adds unbounded overhead.

Minimal concrete example

void ISR(void) {
    GPIOA_BSRR = (1u << 5); // mark entry
    // minimal ISR work
    GPIOA_BSRR = (1u << (5 + 16));
}

Common misconceptions

  • “Average latency is enough” -> real-time requires worst-case bounds.
  • “Jitter only matters for high-frequency tasks” -> even slow loops can fail if jitter is large.
  • “Instrumentation is free” -> it always adds overhead.

Check-your-understanding questions

  1. Why is worst-case latency more important than average latency?
  2. How do you calculate jitter from measurements?
  3. What factors can increase ISR latency?

Check-your-understanding answers

  1. Deadlines are violated by worst-case delays, not average.
  2. Jitter is max latency minus min latency over a sample set.
  3. Interrupt masking, higher-priority ISRs, and long critical sections.

Real-world applications

  • Motor control loops with strict timing.
  • Audio processing with jitter sensitivity.

Where you’ll apply it

  • This project: Sec. 3.1, Sec. 3.7.2, Sec. 5.10 Phase 2.
  • Also used in: Project 2.

References

  • “Real-Time Concepts for Embedded Systems” by Qing Li, Ch. 16.
  • ARM Cortex-M timing documentation.

Key insights

Real-time guarantees are only meaningful when worst-case latency and jitter are measured and documented.

Summary

Latency, jitter, and WCET are the metrics that define RTOS correctness under time constraints.

Homework/Exercises to practice the concept

  1. Measure ISR latency under idle and under heavy load.
  2. Compute jitter over 1000 samples.
  3. Identify the largest contributor to latency in your system.

Solutions to the homework/exercises

  1. Use GPIO pulses and scope measurements.
  2. Record min and max latency and compute difference.
  3. Compare results with and without long critical sections.

2.2 Instrumentation with GPIO and DWT Cycle Counter

Fundamentals

Instrumentation is the act of adding measurement points to your system. GPIO toggles are a simple way to visualize timing on a scope or logic analyzer, while the Data Watchpoint and Trace (DWT) unit provides cycle-accurate timestamps. Together, these tools let you measure latency and jitter without complex external equipment.

Additional fundamentals: Instrumentation is part of the system, not outside it. Any measurement changes timing, so you must keep instrumentation minimal and consistent. The goal is to learn the system’s behavior, not the behavior of your logging code. Even a single extra branch or print can skew results, so keep measurement paths fixed and auditable.

Deep Dive into the concept

GPIO instrumentation is straightforward: set a pin high at an event start and low at event end. The pulse width equals the event duration, and the rising edge marks the start time. This is extremely reliable because it bypasses UART logging and uses hardware timing. The limitation is that it only provides external visibility, and you need a scope or logic analyzer.

The DWT cycle counter is an internal CPU counter that increments every core cycle. On Cortex-M3/M4/M7, you can enable it by setting DEMCR and DWT->CTRL. Reading DWT->CYCCNT gives cycle-accurate timestamps. This allows you to measure durations in software and log them later. The overhead of reading the cycle counter is small and deterministic, making it useful for measuring very short intervals. However, you must ensure the DWT is available on your MCU and that it is enabled in debug configuration.

Combining GPIO and DWT gives you cross-validation. Use GPIO for external measurement and DWT for internal logs. If they match, you can trust your results. If they differ, the instrumentation itself may be influencing timing. A good practice is to measure the overhead of your instrumentation by toggling a pin or reading DWT in a tight loop and subtracting that baseline.

Instrumentation should be designed to minimize interference. Avoid printing in ISRs. Instead, store timestamps in a buffer and let a low-priority task format and output them. This avoids perturbing the very timing you are trying to measure. You should also ensure that your instrumentation code is deterministic, without branches or variable-length loops.

Additional depth: GPIO instrumentation is excellent for external validation, but it has limitations. It provides only a few channels (limited pins) and requires external equipment. DWT-based instrumentation provides internal visibility and can capture many events, but it depends on debug features that may be disabled in production. A robust measurement strategy uses both: GPIO for ground truth and DWT for detailed internal timing. When you compare them, you can estimate measurement overhead and correct for it.

Cycle-accurate timing also depends on clock stability. If the CPU clock changes due to power management or PLL drift, your cycle-based measurements may not reflect real time accurately. For this reason, you should also record the clock configuration and, if possible, include a known reference (such as a hardware timer) to validate cycle counts. This is another reason why GPIO measurements remain valuable.

Finally, storing large numbers of samples can itself consume memory and CPU. A ring buffer is a good choice because it bounds memory usage. You can also downsample or compute running statistics in real time to avoid storing every sample. For example, you can update min/max and sum incrementally and only store a small window of samples for histogram analysis. These techniques keep the measurement system lightweight and reduce its impact on the RTOS.

Another practical improvement is to timestamp both the event and the completion of the associated task, then compute the delta. For example, record a DWT timestamp in the ISR when the event occurs, store it in a queue, and then record another timestamp when the consumer task processes it. This gives you end-to-end response time rather than just ISR latency. It also reveals scheduling delays caused by priority choices or critical sections. Keeping these two timestamps in the same sample structure allows you to compute both latency and response time from the same data set, which is helpful when you are trying to explain why a task missed a deadline.

How this fit on projects

GPIO timing pulses are used in Sec. 3.7.2 to measure ISR latency. DWT timestamps are used for jitter analysis and reported in Sec. 3.7.2 and Sec. 6.2.

Definitions & key terms

  • DWT: Data Watchpoint and Trace unit with cycle counter.
  • CYCCNT: Cycle counter register.
  • DEMCR: Debug Exception and Monitor Control Register.
  • Instrumentation overhead: Time added by measurement itself.

Mental model diagram (ASCII)

Event start -> GPIO high -> event end -> GPIO low
DWT->CYCCNT timestamps stored in buffer

How it works (step-by-step, with invariants and failure modes)

  1. Enable DWT cycle counter.
  2. On event start, read CYCCNT and optionally set GPIO high.
  3. On event end, read CYCCNT and set GPIO low.
  4. Store timestamps in buffer; print later in task.
  5. Failure mode: logging in ISR increases latency.

Minimal concrete example

void dwt_init(void) {
    CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk;
    DWT->CYCCNT = 0;
    DWT->CTRL |= DWT_CTRL_CYCCNTENA_Msk;
}

Common misconceptions

  • “GPIO is too slow for timing” -> it is fast and hardware-based.
  • “DWT always available” -> some MCUs disable it or omit it.
  • “Printing in ISR is fine” -> it changes timing.

Check-your-understanding questions

  1. Why is GPIO toggling a reliable timing method?
  2. How do you enable the DWT cycle counter?
  3. How do you reduce instrumentation overhead?

Check-your-understanding answers

  1. It uses hardware pin transitions measurable by a scope.
  2. Set TRCENA in DEMCR and enable CYCCNT in DWT->CTRL.
  3. Keep instrumentation minimal and defer logging to tasks.

Real-world applications

  • Profiling ISR duration in embedded drivers.
  • Measuring control loop jitter in robotics.

Where you’ll apply it

  • This project: Sec. 3.7.2, Sec. 6.2.
  • Also used in: Project 2.

References

  • ARM documentation on DWT and CYCCNT.
  • Embedded profiling techniques in RTOS literature.

Key insights

Accurate measurements require minimal, deterministic instrumentation.

Summary

GPIO pulses and DWT cycle counts provide complementary, reliable timing measurements when used carefully.

Homework/Exercises to practice the concept

  1. Measure the overhead of a GPIO toggle and a DWT read.
  2. Build a ring buffer of timestamp pairs and dump them over UART.
  3. Compare GPIO-measured pulse widths to DWT-based durations.

Solutions to the homework/exercises

  1. Toggle in a tight loop and compute average cycle cost.
  2. Store timestamps and print from a low-priority task.
  3. Differences should be small and consistent after calibration.

3. Project Specification

3.1 What You Will Build

A measurement toolkit that captures ISR latency and task jitter using GPIO pulses and DWT cycle counters, producing a report of worst-case timing under load.

3.2 Functional Requirements

  1. Latency Measurement: Measure ISR entry latency and ISR duration.
  2. Jitter Statistics: Compute min, max, and jitter values.
  3. Reporting: Output timing report over UART.

3.3 Non-Functional Requirements

  • Performance: Measurement overhead < 5% of ISR duration.
  • Reliability: Measurements repeatable under identical conditions.
  • Usability: Clear report with units and test conditions.

3.4 Example Usage / Output

[latency] min=1.2us max=3.4us jitter=2.2us
[jitter]  task=control min=0.3ms max=1.0ms

3.5 Data Formats / Schemas / Protocols

  • Report lines: [metric] key=value ....

3.6 Edge Cases

  • DWT not available -> fallback to GPIO only.
  • UART logging slows system -> defer logging to idle task.
  • Measurement overhead skews results.

3.7 Real World Outcome

You will produce a clear timing report that quantifies interrupt latency and task jitter under controlled load, with reproducible measurements.

3.7.1 How to Run (Copy/Paste)

make clean all
make flash

Exit codes:

  • make: 0 success, 2 build failure.
  • openocd: 0 flash success, 1 connection failure.

3.7.2 Golden Path Demo (Deterministic)

  1. Enable measurement mode and run for 60 seconds.
  2. Collect 1000 ISR latency samples.
  3. Print a report with min/max/jitter.

3.7.3 Failure Demo (Deterministic)

  1. Add UART prints inside ISR.
  2. Observe latency spikes and increased jitter.
  3. Move logging to task and verify improved measurements.

4. Solution Architecture

4.1 High-Level Design

ISR -> GPIO/DWT timestamps -> buffer -> task report

4.2 Key Components

Component Responsibility Key Decisions
measure.c Timestamp capture GPIO vs DWT
report.c Compute statistics Sample window size
demo.c Load generation ISR rate and task workload

4.4 Data Structures (No Full Code)

typedef struct {
    uint32_t start;
    uint32_t end;
} sample_t;

4.4 Algorithm Overview

Key Algorithm: Jitter Stats

  1. For each sample, compute duration.
  2. Track min, max, sum.
  3. Jitter = max - min.

Complexity Analysis:

  • Time: O(n) per report.
  • Space: O(n) for sample buffer.

5. Implementation Guide

5.1 Development Environment Setup

brew install arm-none-eabi-gcc openocd

5.2 Project Structure

rtos-p10/
|-- src/
|   |-- measure.c
|   |-- report.c
|   `-- main.c
|-- include/
`-- Makefile

5.3 The Core Question You’re Answering

“How can I prove that my RTOS meets real-time timing guarantees?”

5.4 Concepts You Must Understand First

  1. Latency/jitter metrics and worst-case timing.
  2. GPIO and DWT instrumentation techniques.

5.5 Questions to Guide Your Design

  1. How will you separate measurement overhead from real latency?
  2. What load will you use to stress the system?
  3. How large should your sample buffer be?

5.6 Thinking Exercise

Design a test that increases ISR load by 2x and predict the jitter change.

5.7 The Interview Questions They’ll Ask

  1. “How do you measure interrupt latency on real hardware?”
  2. “What is jitter and why does it matter?”
  3. “How do you ensure measurement tools do not skew results?”

5.8 Hints in Layers

Hint 1: Start with GPIO Use a scope for immediate visibility.

Hint 2: Add DWT Enable cycle counter for precise values.

Hint 3: Defer logging Store samples in buffer, print later.

5.9 Books That Will Help

Topic Book Chapter
Timing analysis “Real-Time Concepts for Embedded Systems” Ch. 16
Cortex-M timing “The Definitive Guide to ARM Cortex-M” Ch. 9

5.10 Implementation Phases

Phase 1: Instrumentation (3-4 days)

Goals: Enable GPIO and DWT measurements. Tasks: Add toggle hooks, enable DWT. Checkpoint: Basic latency samples collected.

Phase 2: Statistics (2-3 days)

Goals: Compute jitter stats. Tasks: Implement min/max tracking and report formatting. Checkpoint: Report prints with correct units.

Phase 3: Stress Tests (2-3 days)

Goals: Measure under load. Tasks: Add heavy ISR or task load. Checkpoint: Worst-case latency reported.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Measurement method GPIO vs DWT Both Cross-validation
Sample storage Ring buffer vs log Ring buffer Bounded memory usage

6. Testing Strategy

6.1 Test Categories

| Category | Purpose | Examples | |——————|—————————–|———————————–| | Unit Tests | Stats computation | Min/max/jitter math | | Integration Tests | Full measurement pipeline | ISR -> buffer -> report | | Edge Case Tests | DWT unavailable | GPIO-only fallback |

6.2 Critical Test Cases

  1. ISR latency measured with and without load.
  2. Jitter computed correctly across sample set.
  3. Reporting task runs without affecting ISR timing.

6.3 Test Data

Sample count: 1000
Expected jitter under idle: < 5 us

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

| Pitfall | Symptom | Solution | |—————————|————————–|———————————–| | Logging in ISR | Large latency spikes | Defer to task | | Too few samples | Unreliable statistics | Increase sample count | | DWT not enabled | All zeros in timestamps | Enable DEMCR and CYCCNT |

7.2 Debugging Strategies

  • Verify GPIO pulse width with scope.
  • Compare DWT-based timing with GPIO-based timing.

7.3 Performance Traps

  • Large reporting intervals can overflow buffers; bound sample count.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add histogram bins for latency distribution.
  • Add CLI command to trigger measurement run.

8.2 Intermediate Extensions

  • Measure context switch time and include in report.
  • Add per-task jitter statistics.

8.3 Advanced Extensions

  • Implement trace export for offline analysis.
  • Add automated regression tests for timing.

9. Real-World Connections

9.1 Industry Applications

  • Timing validation for safety-critical systems.
  • Performance tuning of RTOS kernels.
  • Percepio Tracealyzer: Professional RTOS tracing tool.
  • FreeRTOS+Trace: Open-source tracing support.

9.3 Interview Relevance

  • Timing analysis and instrumentation are common embedded interview topics.

10. Resources

10.1 Essential Reading

  • “Real-Time Concepts for Embedded Systems” by Qing Li.
  • ARM Cortex-M DWT documentation.

10.2 Video Resources

  • Embedded timing measurement tutorials.

10.3 Tools & Documentation

  • Logic analyzer and oscilloscope.
  • GDB for enabling DWT.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain latency vs jitter.
  • I understand how to use GPIO and DWT for timing.

11.2 Implementation

  • Timing measurements are repeatable.
  • Reports include min/max/jitter values.

11.3 Growth

  • I can design experiments to stress real-time behavior.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • ISR latency measured and reported.
  • Jitter statistics computed.

Full Completion:

  • Stress test results documented.
  • Instrumentation overhead quantified.

Excellence (Going Above & Beyond):

  • Trace export for offline analysis.
  • Automated timing regression tests.