Project 2: USB HID from Scratch (Minimal Keyboard Device)

Build a minimal USB HID keyboard device that enumerates correctly and types a fixed string, using a bare HID report descriptor and interrupt IN endpoint.

Quick Reference

Attribute Value
Difficulty Level 3: Intermediate
Time Estimate 1-2 weeks
Main Programming Language C (Alternatives: Rust with TinyUSB)
Alternative Programming Languages Rust, C++
Coolness Level Level 4: Wizard
Business Potential Level 2: Embedded Consultant
Prerequisites C, basic MCU flashing, USB basics
Key Topics HID report descriptors, USB enumeration, interrupt IN

1. Learning Objectives

By completing this project, you will:

  1. Explain the role of USB descriptors and how enumeration works.
  2. Write a minimal HID report descriptor for a keyboard.
  3. Implement an interrupt IN endpoint that sends key reports.
  4. Validate device behavior using host tools (lsusb, usbmon, Wireshark).
  5. Produce a deterministic, repeatable typing demo on two OSes.

2. All Theory Needed (Per-Concept Breakdown)

2.1 HID Report Descriptors and Usage Tables

Fundamentals

USB HID is a class specification that lets devices describe their input and output reports to the host. A keyboard is not defined by vendor-specific drivers; instead, it uses a HID report descriptor that tells the host what data will be sent. The descriptor is a small program written in HID items (Usage Page, Usage, Report Size, Report Count, Input/Output/Feature) that defines the layout of a report in bits and bytes. For a keyboard, a common report includes modifiers (Ctrl/Shift/Alt), reserved bits, and up to six keycodes. The host uses this descriptor to parse interrupt IN transfers into key events. Without a correct descriptor, the host cannot interpret your data, even if the device enumerates successfully.

Deep Dive into the Concept

A HID report descriptor is a byte stream of tagged items. Each item encodes a type (Main, Global, Local), a tag (e.g., Usage Page, Logical Min/Max), and a payload. Global items set state (e.g., report size) that affects subsequent items. Local items (Usage) label a specific control. Main items (Input, Output, Feature) terminate a field definition. For a boot keyboard, the descriptor typically declares a Generic Desktop Usage Page, a Keyboard usage, and then defines inputs: 8 bits of modifier keys, 1 byte reserved, and 6 bytes of keycodes. The host uses the descriptor to understand that the first byte is modifiers and the next six bytes are keycodes.

The key subtlety is that the report descriptor must match the actual report data you send. If you send 8 bytes but the descriptor expects 7, the host ignores or misinterprets the data. For example, if you want media keys, you must include a Consumer Page section in the descriptor. The descriptor is not a “header”; it is part of the device identity. When the device enumerates, the host requests the descriptor during the Get_Descriptor sequence, parses it once, and uses it for the lifetime of the connection. Therefore, errors in the descriptor appear as “device enumerates but doesn’t type”.

Usage tables are the taxonomy for HID data. Keyboard keys are in the Keyboard/Keypad usage page, with usages like 0x04 for ‘A’. Modifiers are treated as individual bits. The order of these bits matters (Left Ctrl, Left Shift, etc.). When you press ‘A’, your report sets one keycode byte to 0x04 and leaves modifiers as zero. When you release, all keycode slots are zero. The host sees a transition and generates a key event. This is why you must always send a “release” report after a press, or the OS will think the key is held.

Report descriptors are compact but rigid. Every field is described with Report Size and Report Count. If you declare 6 keycode slots and send 10, the host ignores the extra bytes. If you declare an output report (for LEDs like Caps Lock), the host will send data to your OUT endpoint or control endpoint. Many beginners forget to implement the LED output handling, which is why caps lock LEDs fail to work on some devices. In a minimal device you can safely ignore LED output, but you should at least accept the OUT report or the host might retry and stall.

For this project, you will write a minimal descriptor that is compatible with the HID boot protocol. This allows the keyboard to work in BIOS/UEFI and early boot environments. The boot keyboard report is fixed: 8 bytes in total. You’ll learn how to encode it and how to send it reliably. Understanding the descriptor is a transferable skill: it applies to mice, gamepads, and custom HID devices.

Additional practical considerations: treat the report descriptor as a testable artifact. Keep a known-good descriptor in version control and compare byte-for-byte when you change the report format. Build a small host-side parser script that prints each field from an incoming report so you can validate modifiers and key slots deterministically. Also consider output reports: many hosts will send LED state (Caps Lock, Num Lock). Even if you do not drive LEDs, accept and acknowledge the OUT report to avoid host retries. If you plan to support media keys later, reserve a second report ID early; retrofitting report IDs after release can break host caches. Finally, document your ASCII-to-keycode mapping and explicitly handle Shift requirements so that the typed string is deterministic across keyboard layouts.

How this fits on projects

This concept is the core of Project 2 and appears again in Project 10 (firmware from scratch) and Project 11 (production-ready firmware variants).

Definitions & key terms

  • HID report descriptor: A bytecode describing the structure of HID reports.
  • Usage page: A category of controls (e.g., Keyboard/Keypad).
  • Report size/count: Bit width and number of fields per report item.
  • Boot protocol: Fixed-format report for BIOS/UEFI compatibility.

Mental model diagram (ASCII)

HID Report (8 bytes)
[mods][res][k1][k2][k3][k4][k5][k6]
 8b    8b   8b   8b   8b   8b   8b   8b

Descriptor defines each field's size and meaning.

How it works (step-by-step, with invariants and failure modes)

  1. Device provides HID report descriptor during enumeration.
  2. Host parses descriptor and expects reports matching that layout.
  3. Device sends interrupt IN packets with report bytes.
  4. Host interprets report and generates key events.

Invariant: the report bytes must match descriptor length and semantics. Failure modes include mismatched lengths, incorrect usage pages, and missing key releases.

Minimal concrete example

uint8_t report[8] = {0};
report[2] = 0x04; // 'A'
usb_hid_send(report, 8);
memset(report, 0, 8); // release
usb_hid_send(report, 8);

Common misconceptions

  • “Descriptors are optional”: HID requires them; the host depends on them.
  • “Any report length works”: Length must match descriptor.
  • “No need to send release”: Without release, keys stick.

Check-your-understanding questions

  1. Why does the host need a report descriptor?
  2. What happens if you send 7 bytes but the descriptor says 8?
  3. Why is the boot keyboard format still useful?

Check-your-understanding answers

  1. It tells the host how to interpret raw bytes into key events.
  2. The host ignores or mis-parses the report, causing no input.
  3. It ensures compatibility with BIOS/UEFI and simple HID stacks.

Real-world applications

  • USB keyboards, barcode scanners, and custom HID devices.

Where you’ll apply it

  • In this project: §3.2 Functional Requirements and §4.2 Key Components.
  • Also used in: Project 10, Project 11.

References

  • “USB Complete” by Jan Axelson, Ch. 3-6.
  • HID Usage Tables (USB-IF).

Key insights

If the descriptor is wrong, everything else can be perfect and the host will still ignore you.

Summary

HID report descriptors are the contract between your device and the host. A correct, minimal descriptor enables a keyboard to work everywhere without drivers.

Homework/Exercises to practice the concept

  1. Write a report descriptor that supports 2-key rollover instead of 6.
  2. Add a consumer control (volume up) to the report.

Solutions to the homework/exercises

  1. Reduce Report Count to 2 and adjust report length accordingly.
  2. Add a Consumer Page section with appropriate Usage and Input item.

2.2 USB Enumeration and Interrupt IN Transfers

Fundamentals

USB is host-driven: the host is in control of enumeration, configuration, and data transfer. When a device is connected, the host resets the bus, requests descriptors (device, configuration, interface, endpoint, HID), and then sets a configuration. Only after configuration can the device send interrupt IN reports. A keyboard uses an interrupt IN endpoint, which the host polls at a fixed interval (typically 1 ms). The device does not push data; it responds when polled. This polling model is essential to understanding latency and why USB keyboards are reliable. If your firmware does not respond to polls or stalls an endpoint, the host will consider the device misbehaving.

Deep Dive into the Concept

Enumeration is a handshake between the host and the device. After reset, the device is in the default state with address 0. The host asks for the device descriptor to learn vendor ID, product ID, and capabilities. It then assigns an address and requests the configuration descriptor, which contains one or more interfaces and endpoints. For a HID keyboard, the configuration includes an interface descriptor with class 0x03 (HID), a HID descriptor that points to the report descriptor, and an interrupt IN endpoint descriptor that defines packet size and polling interval. This hierarchy matters because the host uses it to decide which driver to bind. If your descriptors are inconsistent, the host might load a generic driver incorrectly or refuse to configure the device.

Interrupt IN endpoints are not interrupts in the CPU sense; they are periodic polled transfers. The host sends an IN token at the configured interval, and the device responds with data or NAK (no data). If you always respond with data, the host will interpret repeated key presses. The correct behavior is to send a report only when state changes, or send identical reports while a key is held depending on the OS handling. Most keyboards send a report every poll, which includes the current state of all keys. This ensures consistent behavior even if the host misses a packet.

The polling interval is part of the endpoint descriptor and is typically 1 ms for full-speed keyboards. This sets an upper bound on latency: a keypress is observed on the next poll. If your firmware’s scan loop is slower than the poll interval, you can add latency. If your scan loop is faster, the poll interval becomes the dominant factor. Understanding this timing path is crucial for later projects where you add features that might slow scanning.

Control requests also matter. HID devices receive class-specific requests like GET_REPORT, SET_REPORT, SET_IDLE, and SET_PROTOCOL. A minimal device must handle at least SET_IDLE and SET_PROTOCOL. If you ignore them, some hosts will still work, but others will retry or stall. A good minimal implementation acknowledges these requests with appropriate responses. In TinyUSB, many of these are handled by the stack; in bare-metal code, you must implement them yourself.

In this project, you will use a small USB stack (TinyUSB or vendor HAL) to focus on the descriptor and report logic. But you still need to understand enumeration because when things fail, the host logs and descriptor dumps are your main debugging tools. The difference between a device that enumerates but does not type and a device that fails enumeration entirely is almost always in the descriptors or endpoint configuration.

Additional enumeration notes: endpoint 0 (control) is stateful and must respond quickly. If you NAK or stall control transfers during enumeration, some hosts will retry aggressively or give up. Make sure you handle SET_ADDRESS, GET_DESCRIPTOR, and SET_CONFIGURATION in the exact sequence. For full-speed devices, the interrupt IN endpoint packet size is typically 8 bytes for boot keyboards; if you choose a larger size, verify that the host still accepts it. Also remember that the host polls on its schedule, not yours. Your firmware should prepare the next report in advance and return immediately when polled. If you implement your own stack, keep USB interrupt handlers minimal and move any heavy logic into the main loop to avoid missing packets.

How this fits on projects

This concept is central to Project 2 and reappears in Project 10 (USB stack integration) and Project 11 (production firmware variants).

Definitions & key terms

  • Enumeration: The process by which the host identifies and configures a USB device.
  • Endpoint: A unidirectional data channel; keyboards use interrupt IN.
  • Polling interval: How often the host requests data from the device.
  • HID class requests: SET_IDLE, SET_PROTOCOL, GET_REPORT, SET_REPORT.

Mental model diagram (ASCII)

Host        Device
----        ------
Reset  ---> default state
GetDesc -> device desc
SetAddr -> address assigned
GetDesc -> config desc
GetDesc -> HID report desc
SetConf -> device configured
IN token -> report (every 1 ms)

How it works (step-by-step, with invariants and failure modes)

  1. Device powers up, waits for host reset.
  2. Host requests descriptors and assigns an address.
  3. Host sets configuration and starts polling interrupt IN.
  4. Device responds to IN tokens with current report.

Invariant: Endpoint descriptors must match actual endpoint behavior. Failure modes include incorrect packet size, missing endpoint, or stalling control requests.

Minimal concrete example

// TinyUSB callback
void tud_hid_report_complete_cb(uint8_t inst, uint8_t const* report, uint16_t len) {
    // queue next report if needed
}

Common misconceptions

  • “Interrupt endpoints push data”: The host always polls; device responds.
  • “Set idle doesn’t matter”: Some hosts require it for keyboards.
  • “Enumeration errors are random”: They are almost always descriptor mismatches.

Check-your-understanding questions

  1. Why does the host poll an interrupt IN endpoint?
  2. What is the difference between device and configuration descriptors?
  3. What happens if the endpoint packet size doesn’t match the report size?

Check-your-understanding answers

  1. USB is host-driven; polling ensures bandwidth scheduling.
  2. The device descriptor is global; the configuration describes interfaces and endpoints.
  3. The host may truncate or reject reports, leading to missing key events.

Real-world applications

  • All USB HID keyboards and mice.
  • USB barcode scanners and foot pedals.

Where you’ll apply it

  • In this project: §3.7 Real World Outcome and §6 Testing Strategy.
  • Also used in: Project 10.

References

  • “USB Complete” by Jan Axelson, Ch. 2-4.
  • USB 2.0 specification (enumeration overview).

Key insights

USB HID reliability comes from host-driven polling and strict descriptor contracts.

Summary

Enumeration and interrupt IN transfers are the backbone of a USB keyboard. Once you understand the descriptor hierarchy and polling model, debugging becomes systematic instead of mysterious.

Homework/Exercises to practice the concept

  1. Use lsusb -v to inspect an existing keyboard’s descriptors.
  2. Change the polling interval in your endpoint descriptor and observe latency.

Solutions to the homework/exercises

  1. You should see a HID interface with interrupt IN endpoint and a report descriptor length.
  2. A longer interval increases worst-case latency; a shorter interval increases host polling frequency.

3. Project Specification

3.1 What You Will Build

A minimal USB HID keyboard device that:

  • Enumerates as a standard keyboard on macOS, Linux, and Windows.
  • Sends a fixed string (e.g., “HELLO”) once on connect or on a button press.
  • Responds correctly to SET_IDLE and SET_PROTOCOL requests.
  • Provides a valid report descriptor that matches the report data.

3.2 Functional Requirements

  1. Descriptors: Provide device, configuration, interface, HID, and endpoint descriptors.
  2. Report logic: Send press and release reports for each character.
  3. Polling handling: Send reports only when polled by the host.
  4. Debugging hooks: Output enumeration logs via serial or LED.

3.3 Non-Functional Requirements

  • Performance: Respond to IN tokens within 1 ms.
  • Reliability: Enumerate successfully on at least two OSes.
  • Usability: Simple build and flash commands documented.

3.4 Example Usage / Output

$ lsusb | grep "Custom HID Keyboard"
Bus 001 Device 012: ID 1209:0001 Custom HID Keyboard

3.5 Data Formats / Schemas / Protocols

HID report (boot keyboard, 8 bytes):

[mods][res][k1][k2][k3][k4][k5][k6]

3.6 Edge Cases

  • Missing key release reports (stuck keys).
  • Host requests report before your stack is ready.
  • Descriptor length mismatch.

3.7 Real World Outcome

Your device enumerates and types a fixed string reliably.

3.7.1 How to Run (Copy/Paste)

# build and flash
make
make flash

# verify on host
lsusb | grep "Custom HID Keyboard"

3.7.2 Golden Path Demo (Deterministic)

  • Use a fixed startup delay of 1 second after enumeration.
  • Device types “HELLO” once and then goes idle.

3.7.3 If CLI: exact terminal transcript

$ lsusb | grep "Custom HID Keyboard"
Bus 001 Device 012: ID 1209:0001 Custom HID Keyboard
exit_code=0

$ sudo cat /sys/kernel/debug/usb/usbmon/1u | head -n 3
ffff88007a2a1c00 0.000000 S Ii:1:012:1 0:8 0200000000000000
ffff88007a2a1c00 0.001000 C Ii:1:012:1 0:8 0200000000000000
exit_code=0

$ lsusb -v -d 1209:0001 | grep -A2 "HID Report"
  HID Report Descriptor: (length 63)
exit_code=0

$ make flash
error: device not found
exit_code=3

4. Solution Architecture

4.1 High-Level Design

┌──────────────┐   ┌──────────────┐   ┌──────────────┐
│ USB Stack    │--▶│ HID Class    │--▶│ Report Engine│
└──────────────┘   └──────────────┘   └──────────────┘
        │                    │                 │
        v                    v                 v
  Descriptors          Class Requests      Key Report FIFO

4.2 Key Components

Component Responsibility Key Decisions
Descriptors Define device and HID identity Boot keyboard format
HID Class Handle class requests Support SET_IDLE, SET_PROTOCOL
Report Engine Encode key presses/releases Fixed string sequence
USB Transport Send interrupt IN reports 1 ms polling interval

4.3 Data Structures (No Full Code)

typedef struct {
    uint8_t mods;
    uint8_t reserved;
    uint8_t keys[6];
} hid_kb_report_t;

4.4 Algorithm Overview

Key Algorithm: Type String

  1. Convert ASCII to keycodes + modifiers.
  2. Send press report.
  3. Send release report.
  4. Repeat for each character.

Complexity Analysis:

  • Time: O(n) for n characters
  • Space: O(1)

5. Implementation Guide

5.1 Development Environment Setup

# example using TinyUSB
make
make flash

5.2 Project Structure

usb-hid-keyboard/
├── src/
│   ├── main.c
│   ├── usb_descriptors.c
│   └── hid_kb.c
├── include/
│   └── hid_kb.h
├── Makefile
└── README.md

5.3 The Core Question You’re Answering

“How does a keyboard describe itself to the host and deliver keypresses without a driver?”

5.4 Concepts You Must Understand First

  1. HID report descriptors: The report is the contract.
  2. USB enumeration: Descriptor hierarchy and configuration.
  3. Interrupt IN polling: Host-driven transfers and timing.

5.5 Questions to Guide Your Design

  1. Will you use boot protocol or report protocol?
  2. How will you map ASCII to keycodes reliably?
  3. How will you confirm descriptor correctness?

5.6 Thinking Exercise

Write a report descriptor for only two keys and one modifier. What bytes change when you press Shift + A?

5.7 The Interview Questions They’ll Ask

  1. What is the role of a HID report descriptor?
  2. Why does USB use host-driven polling?
  3. Why must you send a release report?

5.8 Hints in Layers

Hint 1: Start from TinyUSB examples Use the keyboard demo descriptor and shrink it.

Hint 2: Validate with lsusb -v Check report length and endpoint descriptors.

Hint 3: Send press then release Alternate non-zero and zero reports.

Hint 4: Add a delay Insert 5-10 ms between characters so the host sees them.

5.9 Books That Will Help

Topic Book Chapter
USB fundamentals “USB Complete” Ch. 1-6
Embedded timing “Making Embedded Systems” Ch. 4-5
C structures “The C Programming Language” Ch. 6

5.10 Implementation Phases

Phase 1: Enumeration (2-3 days)

Goals: device enumerates and shows correct descriptors

Tasks:

  1. Define device and configuration descriptors.
  2. Implement report descriptor and verify via lsusb -v.

Checkpoint: Host recognizes device as HID keyboard.

Phase 2: Typing (2-3 days)

Goals: send key press and release reports

Tasks:

  1. Implement ASCII-to-keycode mapping.
  2. Send reports on a timer or button press.

Checkpoint: Device types “HELLO” once reliably.

Phase 3: Robustness (2-3 days)

Goals: handle class requests, multi-OS checks

Tasks:

  1. Implement SET_IDLE and SET_PROTOCOL.
  2. Test on a second OS.

Checkpoint: Works on Linux + Windows/macOS.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
USB stack TinyUSB vs vendor HAL TinyUSB clean HID examples
Report format Boot vs NKRO Boot simplest compatibility
Trigger Auto on connect vs button Button safer for testing

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Descriptor Tests Validate descriptors lsusb -v output
Functional Tests Verify typing OS text editor
Edge Case Tests Handling missing release stuck key scenario

6.2 Critical Test Cases

  1. Press + release: each key must send a release report.
  2. Enumeration: device appears as HID keyboard with correct VID/PID.
  3. Polling: host receives reports at 1 ms interval when active.

6.3 Test Data

Test string: HELLO
Expected: exactly one typed string, no repeats

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Descriptor mismatch Enumerates but no typing Compare to known-good descriptor
Missing release report Keys stick Always send zero report
Incorrect endpoint size Random failures Match report length

7.2 Debugging Strategies

  • Use usbmon/Wireshark: verify interrupt IN payloads.
  • Serial logging: print enumeration events and request IDs.

7.3 Performance Traps

Blocking delays in the USB ISR path can cause missed polls. Keep report generation outside interrupt context.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a button to trigger typing instead of auto-run.
  • Add Caps Lock LED handling.

8.2 Intermediate Extensions

  • Add media key support with Consumer Page.
  • Support NKRO reports.

8.3 Advanced Extensions

  • Implement a composite device (keyboard + mouse).
  • Add vendor-specific HID feature reports.

9. Real-World Connections

9.1 Industry Applications

  • Custom USB keyboards and input devices.
  • HID-based industrial controllers and scanners.
  • TinyUSB HID examples.
  • LUFA keyboard demo.

9.3 Interview Relevance

  • Demonstrates USB descriptor knowledge.
  • Shows embedded timing and protocol reasoning.

10. Resources

10.1 Essential Reading

  • “USB Complete” by Jan Axelson - Ch. 1-6.
  • USB HID Usage Tables - Keyboard/Keypad page.

10.2 Video Resources

  • USB enumeration deep dives (conference talks).

10.3 Tools & Documentation

  • lsusb, usbmon, Wireshark USB dissector.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain how enumeration works step-by-step.
  • I can describe the HID report descriptor structure.
  • I can explain why the host polls the device.

11.2 Implementation

  • Device enumerates as a HID keyboard.
  • Press/release reports are correct.
  • Works on two OSes.

11.3 Growth

  • I can debug descriptor errors using host tools.
  • I can extend the report for media keys.

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Enumerates as HID keyboard.
  • Types a fixed string once.
  • Release reports sent.

Full Completion:

  • All minimum criteria plus:
  • Works on two OSes.
  • Class requests handled.

Excellence (Going Above & Beyond):

  • Composite HID device.
  • NKRO report support.