XiaoZhi AI Robot on ESP32-S3: Full-Stack Voice Assistant Engineering

Goal: Build a complete mental model of how an ESP32-S3 voice robot works end-to-end, from microphones and I2S DMA all the way to wake words, networking, and speaking back. You will understand how to design low-latency audio pipelines, coordinate multiple real-time tasks, and ship a responsive UI that makes the device feel alive. By the end, you will be able to build your own XiaoZhi-class voice assistant, integrate it with smart-home systems, and reason about performance, memory, and reliability tradeoffs like a production firmware engineer.

Introduction

XiaoZhi AI (xiaozhi-esp32) is an open-source, MCP-based voice chatbot project for ESP32 devices. The project’s own README lists offline wake word support via ESP-SR, streaming ASR + LLM + TTS, Opus audio, and two transport options (WebSocket or MQTT+UDP). It also highlights device-side MCP control for hardware actions (speaker, LED, servo, GPIO), plus display, battery, and multilingual support. The current XiaoZhi codebase supports multiple ESP32 chips, including the ESP32-S3.

This guide turns that system into a structured learning journey. You will build:

  • A real UI on a circular display (Project 1)
  • A glitch-free audio capture and playback pipeline (Project 2)
  • A networked “dumb” chatbot that streams audio to a server (Project 3)
  • A Home Assistant-compatible voice satellite using ESPHome (Project 4)
  • A full-stack XiaoZhi clone with full-duplex audio, Opus compression, and streaming state control (Project 5)

Scope: We focus on ESP32-S3 firmware architecture, audio pipelines, wake word processing, low-latency networking, embedded UI, and system integration. We do not cover PCB design or advanced acoustic tuning beyond what is necessary to build a stable product-grade prototype.

Big Picture Architecture

[Mic Array] -> [I2S DMA] -> [Ring Buffer] -> [AFE: VAD + Wake Word]
                                     |                 |
                                     |                 v
                                     |          [State Machine]
                                     |                 |
                                     v                 v
                             [Opus Encoder] -> [WebSocket/TLS] -> [ASR/LLM/TTS]
                                     ^                 |
                                     |                 v
[Speaker] <- [I2S DMA] <- [Opus Decoder] <- [Audio Stream Response]
                                     |
                                     v
                               [LVGL UI]

How to Use This Guide

  1. Read the Primer once. The mini-book sections explain the core concepts that each project depends on.
  2. Pick a path. Choose a learning path based on your background (see “Recommended Learning Paths”).
  3. Build projects in order. Each project assumes concepts and skills from the ones before it.
  4. Keep a lab notebook. Record hardware wiring, I2S config values, buffer sizes, and timing measurements.
  5. Instrument everything. Add logs for audio latency, queue depths, and buffer underflows.

Prerequisites & Background

Essential Prerequisites (Must Have)

  • C/C++ fundamentals (pointers, structs, memory layout)
  • Basic embedded development (GPIO, serial logs, flashing)
  • Basic Linux CLI (installing toolchains, serial monitor)

Helpful But Not Required

  • Familiarity with FreeRTOS or task scheduling
  • Audio basics (sample rate, bit depth, PCM)
  • Networking basics (TCP/IP, HTTP)

Self-Assessment Questions

  • Can you explain the difference between SRAM, Flash, and PSRAM?
  • Have you configured a peripheral using a C struct before?
  • Can you describe how audio sample rate affects latency and bandwidth?
  • Have you ever debugged a timing-related bug (race, buffer underrun)?

Development Environment Setup

  • Firmware: ESP-IDF (recommended), plus optional Arduino or PlatformIO for quick display tests
  • Host Tools: Python 3, CMake/Ninja, Git, serial monitor
  • Optional: ESPHome for the Home Assistant project
  • Hardware:
    • ESP32-S3 dev board with PSRAM
    • I2S microphone (or PDM mic + codec)
    • I2S DAC or amplifier + speaker
    • Round SPI display (or any small TFT)
    • Buttons or touch input for wake/sleep

Time Investment

  • Project 1: 1-2 days
  • Project 2: 3-5 days
  • Project 3: 2-4 days
  • Project 4: 2-4 days
  • Project 5: 2-4 weeks

Important Reality Check

You will spend time debugging timing issues, buffer underruns, and unstable Wi-Fi behavior. This is expected. Building real-time audio on a microcontroller is hard. The payoff is deep understanding.

Big Picture / Mental Model

Hardware Layer
  - ESP32-S3 SoC, PSRAM, I2S peripherals, display bus

Firmware Layer
  - FreeRTOS tasks, queues, event loop, state machine

Audio Layer
  - I2S DMA, ring buffers, pre-processing (VAD/AEC)

Network Layer
  - WebSocket/TLS streams, retries, QoS decisions

AI Layer
  - ASR -> LLM -> TTS pipeline (local or cloud)

UX Layer
  - LVGL UI, LEDs, sounds, animations, user feedback

Theory Primer

1) ESP32-S3 SoC Architecture and Memory Hierarchy

Fundamentals

The ESP32-S3 is a dual-core Xtensa LX7 MCU running up to 240 MHz with 512 KB on-chip SRAM and 384 KB ROM. It integrates 2.4 GHz Wi-Fi (802.11 b/g/n) and Bluetooth 5 (LE), includes SIMD/vector extensions, and supports external flash/PSRAM over SPI, Dual SPI, Quad SPI, Octal SPI, QPI, and OPI interfaces. The datasheet notes two I2S interfaces, a dedicated DMA controller for streaming peripherals, and a rich peripheral set (SPI, I2C, UART, PWM, RMT, ADC, DAC, SD/MMC host, TWAI). Understanding the ESP32-S3 memory layout and peripheral capabilities is crucial because your audio buffers, UI framebuffers, and network stacks all compete for the same resources.

Deep Dive into the Concept

When building a real-time voice assistant on the ESP32-S3, the real constraint is not raw CPU speed but memory locality, bandwidth, and predictable timing. The ESP32-S3 gives you a hybrid memory model: some fast internal SRAM, some slower external PSRAM, and executable code stored in external flash. Each of those areas has different latency and cache behavior. For short, timing-sensitive tasks such as audio ISR work or ring-buffer read/write, you want to stay in internal SRAM (and preferably IRAM for code) to avoid cache misses. For large, bulk data like audio buffers or UI framebuffers, you want PSRAM. The trick is that your DMA engines, cache policies, and peripheral drivers need to be configured so that the data you are streaming is stable and coherent. If you store your audio ring buffer in PSRAM but the DMA engine expects linear, aligned, non-cached memory, you may see rare audio glitches or missing frames.

The ESP32-S3 can attach external PSRAM with high-speed octal SPI, giving you large buffers. That makes it possible to run Opus and a GUI at the same time. But PSRAM is slower than internal SRAM, and reads can stall the CPU. The typical strategy is to keep small, latency-sensitive data structures in internal SRAM (queues, event flags, task stacks) while placing large buffers in PSRAM. You can also pin certain tasks to cores. For example, you can dedicate core 0 to audio capture/playback and core 1 to networking and UI. This is a common pattern in ESP-IDF: use task priorities and core affinity to reduce jitter, and keep the audio path as deterministic as possible.

The ESP32-S3 also includes an Ultra Low Power (ULP) co-processor and USB OTG. The ULP can handle ultra-low-power tasks, but in a voice assistant it is more common to stay in active mode because microphones and wake-word engines require constant processing. The USB OTG interface is useful for debugging, flashing, or future designs where a USB audio device is desirable. For the XiaoZhi-style use case, the core system architecture revolves around three shared resources: memory bandwidth, Wi-Fi bandwidth, and CPU scheduling. The more you optimize memory access (align buffers, reduce copies, use DMA), the more headroom you have for wake-word detection and Opus encoding.

The ESP32-S3 feature set indicates why it is a popular voice platform: dual-core LX7, 2.4 GHz Wi-Fi, Bluetooth LE, and vector acceleration for signal processing. The vector instructions allow faster MFCC feature extraction or small neural wake word models. Those details translate into practical benefits: lower CPU usage per audio frame, more time for networking, and less risk of buffer underrun. Even the peripheral selection matters: I2S gives you clean audio sampling, SPI or QSPI gives you fast display updates, and RMT/PWM can drive LED feedback or audio alerts.

In summary, the ESP32-S3 hardware defines the boundaries of what you can build. If you want a smooth voice assistant, you must treat memory placement and peripheral usage as first-class design decisions. This concept shows up in almost every project: the UI project uses PSRAM for framebuffers, the audio project relies on DMA buffers, and the full-stack clone requires careful task placement to maintain low latency and full-duplex audio.

How This Fits in the Projects

  • Project 1 uses PSRAM and display buffers for LVGL.
  • Project 2 uses DMA buffers and I2S in real time.
  • Project 5 depends on careful memory planning for full duplex.

Definitions & Key Terms

  • SRAM: Fast on-chip memory with low latency.
  • PSRAM: External pseudo-static RAM; larger but slower.
  • IRAM: Instruction RAM; time-critical code may be placed here.
  • DMA: Direct Memory Access; peripherals read/write memory without CPU.
  • Vector Instructions: CPU extensions for SIMD-style DSP.

Mental Model Diagram

Fast (low latency)                  Large (high capacity)
+-------------------+               +---------------------+
| Internal SRAM     |<--cache-->    | External PSRAM      |
| 512 KB            |               | multi-MB            |
+-------------------+               +---------------------+
        ^                                    ^
        |                                    |
   DMA, task stacks                   Audio buffers,
   ring indexes                       UI framebuffers

How It Works (Step by Step)

  1. Boot ROM loads a small loader from flash.
  2. Flash is memory-mapped; code executes with cache.
  3. Internal SRAM holds critical stacks and queues.
  4. DMA engines stream audio into ring buffers.
  5. External PSRAM holds large audio and UI buffers.

Minimal Concrete Example

// Allocate a large audio buffer in PSRAM
uint8_t *audio_buf = heap_caps_malloc(64 * 1024, MALLOC_CAP_SPIRAM);
// Allocate a small queue in internal SRAM
QueueHandle_t q = xQueueCreate(8, sizeof(int));

Common Misconceptions

  • “PSRAM is as fast as SRAM.” It is not; expect more latency.
  • “Caches always fix it.” Cache misses still cause stalls.

Check-Your-Understanding Questions

  1. Why do audio ring buffers often live in PSRAM while indices live in SRAM?
  2. What happens if a DMA buffer is placed in cached memory without care?

Check-Your-Understanding Answers

  1. The large buffer fits in PSRAM, while ring indices need fast access.
  2. The DMA engine may read stale data or cause corruption due to cache.

Real-World Applications

  • Smart speakers and voice assistants
  • Audio streaming devices
  • Edge AI inference on MCU

Where You Will Apply It

Projects 1, 2, 5

References

  • Espressif ESP32-S3 product page (feature overview, memory, connectivity).
  • ESP32-S3 Datasheet (I2S interfaces, DMA, peripherals, ULP/USB).
  • ESP32 series comparison table (RAM/ROM totals).

Key Insight

Memory placement is a performance feature, not an afterthought.

Summary

The ESP32-S3 is a powerful microcontroller for voice systems, but its memory hierarchy forces tradeoffs. You must design with SRAM, PSRAM, and DMA constraints in mind to avoid jitter and glitches.

Homework / Exercises

  1. Draw your own memory map for a 5-second audio buffer at 16 kHz mono.
  2. Estimate how much RAM LVGL needs for a 240x240 RGB display.

Solutions

  1. 16 kHz * 2 bytes * 5 seconds = 160,000 bytes (~156 KB).
  2. One full frame: 2402402 bytes (RGB565) = 115,200 bytes.

2) ESP-IDF + FreeRTOS Concurrency and Event-Driven Firmware

Fundamentals

The ESP-IDF framework is built around FreeRTOS, providing tasks, queues, semaphores, and event groups to coordinate work. A voice assistant is a multi-task system: audio capture, wake-word inference, network streaming, UI updates, and state management must happen concurrently without blocking each other. You solve this by building a small state machine and connecting tasks with queues or ring buffers. The key idea is that each subsystem owns its timing, and the firmware orchestrates them through non-blocking events. This architecture is what separates a responsive voice robot from a glitchy prototype. You will rely on FreeRTOS primitives to separate real-time deadlines from best-effort work, and to reason about worst-case execution time in a small system.

Deep Dive into the Concept

Concurrency is the heart of embedded voice systems. The ESP32-S3 has two LX7 cores, and ESP-IDF exposes them through FreeRTOS. You can pin tasks to cores, assign priorities, and use queues to transfer data. Audio capture is high-priority, periodic, and sensitive to delay. Network streaming is bursty, and UI updates are lower priority but still critical for user feedback. A well-designed firmware ensures that the audio pipeline never blocks on network or UI tasks.

A good pattern is three main pipelines: audio input, network processing, and audio output. Audio input runs at a fixed sampling rate and writes to a ring buffer. A wake word or VAD task consumes that buffer and triggers a state change. When the system transitions to “listening” or “streaming”, a network task begins reading audio frames from the ring buffer and sending them to the server. When a response arrives, a decoder task writes to an output buffer, and the playback task reads from it and drives I2S. Each of these tasks should be able to run independently. The glue is a small state machine that reacts to events like “wake word detected”, “server connected”, or “playback complete”.

The ESP-IDF event loop (esp_event) allows you to broadcast state changes to multiple subsystems. For example, when the network handshake completes, the UI task can update the screen to show “listening”, and the audio output task can prepare to play an audio cue. Using event groups, you can block a low-priority task until a condition is true without busy waiting. When a buffer underrun occurs, you can signal an error event and recover gracefully (for example, by inserting silence instead of crashing).

Task priorities matter. The I2S DMA interrupt or audio task should be highest priority. The wake word inference can be slightly lower. Networking is usually medium, while UI can be lower. A common mistake is to allow the UI or logging tasks to run too often, causing jitter in audio capture. The solution is to throttle logging, batch UI updates, and reduce console output during streaming. Another important design is time budgeting: if you have 10 ms audio frames, your processing pipeline must finish within that 10 ms window. Any task that exceeds its slot will cause buffer backlog or underrun.

Robust systems also monitor themselves. Use stack high-water marks to confirm your tasks are not starving, and measure queue depths to detect backlog trends. Periodic tasks should use vTaskDelayUntil to keep timing stable, and shared resources should use mutexes to avoid priority inversion. If you ignore these details, the system may appear stable in short tests but fail under long runs or noisy networks.

To scale to full duplex audio (listening while speaking), you must separate input and output pipelines and ensure they do not block each other. This is where ring buffers and double-buffering become essential. Use dedicated queues for each pipeline, and ensure backpressure does not propagate into the capture path. If a network delay happens, you should drop frames or compress them rather than blocking the audio capture task.

How This Fits in the Projects

  • Project 2 uses a simple two-task pipeline (capture -> playback).
  • Project 3 introduces network tasks.
  • Project 5 requires full-duplex and complex state transitions.

Definitions & Key Terms

  • Task: A FreeRTOS thread of execution.
  • Queue: A FIFO for passing messages between tasks.
  • Event Group: A bitmask for multi-condition synchronization.
  • State Machine: A model of states and transitions that control system behavior.

Mental Model Diagram

Audio Input Task ---> Ring Buffer ---> Wake Word Task ---> State Machine
                                         |                        |
                                         v                        v
                                    Network Task            UI Task
                                         |
                                         v
                               Decoder/Playback Task ---> I2S DAC

How It Works (Step by Step)

  1. Audio task reads I2S frames and pushes into ring buffer.
  2. Wake word task reads frames and signals a state change.
  3. State machine enables network streaming.
  4. Network task sends frames, receives responses.
  5. Playback task decodes and plays audio output.

Minimal Concrete Example

xTaskCreatePinnedToCore(audio_capture_task, "audio_in", 4096, NULL, 10, NULL, 0);
xTaskCreatePinnedToCore(network_task, "net", 8192, NULL, 6, NULL, 1);
xTaskCreatePinnedToCore(ui_task, "ui", 4096, NULL, 3, NULL, 1);

Common Misconceptions

  • “Two cores means no timing problems.” Poor priorities still cause glitches.
  • “Logging is free.” Serial logging can block and cause audio dropouts.

Check-Your-Understanding Questions

  1. Why is a ring buffer better than a queue for audio frames?
  2. What happens if a low-priority task blocks a high-priority task?

Check-Your-Understanding Answers

  1. Ring buffers allow continuous streaming with less overhead.
  2. The system may miss audio deadlines, causing underruns.

Real-World Applications

  • Voice assistants
  • Real-time audio devices
  • Low-latency streaming systems

Where You Will Apply It

Projects 2, 3, 5

References

  • ESP-IDF tasking and event-driven design concepts in the official ESP-IDF guide (general framework context).

Key Insight

Concurrency is the difference between “works once” and “works every time”.

Summary

FreeRTOS provides the building blocks to build concurrent firmware. You must design task priorities and data flows intentionally to keep audio real-time.

Homework / Exercises

  1. Sketch a state machine for “idle -> wake -> streaming -> speaking -> idle”.
  2. Define task priorities and explain why you chose them.

Solutions

  1. Idle waits for wake word; wake transitions to streaming; speaking interrupts streaming; end returns to idle.
  2. Audio capture highest, wake word high, network medium, UI low.

3) Real-Time Audio I/O with I2S, DMA, and Ring Buffers

Fundamentals

The ESP32-S3 includes two standard I2S interfaces that can operate in master or slave mode, in full- or half-duplex, and with 8/16/24/32-bit sample widths. The I2S hardware supports TDM and PDM modes and has a dedicated DMA controller, which is why it is the backbone of continuous audio capture and playback. The I2S driver in ESP-IDF can be configured for sample rate, bit depth, channel format, and buffer sizes, and the documentation explicitly warns that sample rates above 48 kHz are not recommended because they can introduce glitches or noise. That is why most voice systems target 16 kHz (wake word/ASR) or 48 kHz (higher fidelity playback).

Deep Dive into the Concept

I2S is a serial bus protocol for streaming audio samples. It separates audio clock (BCLK), word select (LRCLK), and data. On ESP32-S3, I2S is typically used in master mode, generating BCLK and LRCLK to drive an I2S microphone or codec. The DMA engine continuously writes samples into memory buffers, which you then read from a ring buffer for processing. This is the foundation of stable, low-latency audio.

For voice assistants, the typical audio format is 16 kHz, 16-bit, mono PCM. This is compatible with many wake word engines and speech recognition pipelines. The ESP-SR WakeNet model, for example, assumes 16 kHz mono 16-bit input frames. If you sample at 48 kHz, you must downsample or risk unnecessary bandwidth and CPU load. The DMA buffers should be sized so that the CPU can keep up: a typical approach is to use 10-20 ms frames. For 16 kHz audio, 10 ms equals 160 samples (320 bytes at 16-bit). If you configure DMA to deliver 160 or 320 sample blocks, you can process each block in a predictable window.

The ring buffer pattern avoids contention between producer and consumer. The I2S DMA ISR or task writes into a ring buffer, and a consumer task reads and processes it. If the consumer falls behind, the ring buffer fills and can drop old frames; this is often better than blocking the audio capture task. The tradeoff is that you may lose audio during network stalls, but you preserve real-time behavior. For full duplex audio, you need two ring buffers (input and output) and separate tasks to keep them flowing.

Clock configuration matters. The ESP32 I2S driver supports several clock sources; for high accuracy you can enable APLL. Poor clocking leads to drift and incompatibility with some codecs. In a voice assistant, you want consistent timing so that wake word detection and network packetization stay aligned. Also, if you are using PDM microphones, you must set correct PDM clock parameters or you will get distorted audio. The ESP-IDF docs note that certain sample rates and configurations can lead to noise, which is why tuning the I2S clock is not optional.

Buffer sizing is your latency control knob. Larger buffers reduce underruns but increase latency. Smaller buffers reduce latency but increase risk of underruns. The right balance depends on your network path. In Project 2, you will experiment with buffer sizes to hear the difference. In Project 5, you will build a full duplex system where the overall latency is the sum of capture frame size, encode frame size, network latency, decode buffering, and playback frame size.

Finally, I2S is not just about audio. It is about deterministic throughput. When you log every audio frame or run UI animations on the same core, you risk breaking timing. Use dedicated tasks, keep logging minimal, and consider pinning audio tasks to a single core. The difference between a clean 10 ms pipeline and a jittery one is often a single blocking call or a serial printf in the wrong place.

How This Fits in the Projects

  • Project 2 is entirely about I2S and DMA.
  • Project 5 relies on precise buffer management for full duplex.

Definitions & Key Terms

  • I2S: Inter-IC Sound, a serial bus for audio.
  • DMA: Hardware memory transfers without CPU.
  • Ring Buffer: Circular buffer for streaming data.
  • Frame: A fixed-time block of audio samples.

Mental Model Diagram

I2S Peripheral --> DMA Buffer --> Ring Buffer --> Audio Processing
                           ^
                           |
                         ISR

How It Works (Step by Step)

  1. Configure I2S with sample rate and bit depth.
  2. Start DMA; hardware fills buffers.
  3. Copy or reference DMA buffers into a ring buffer.
  4. Consumer task reads frames and processes or streams them.

Minimal Concrete Example

i2s_config_t cfg = {
  .mode = I2S_MODE_MASTER | I2S_MODE_RX,
  .sample_rate = 16000,
  .bits_per_sample = 16,
  .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
  .communication_format = I2S_COMM_FORMAT_I2S,
  .dma_buf_count = 8,
  .dma_buf_len = 160
};

Common Misconceptions

  • “Higher sample rate is always better.” It increases bandwidth and CPU load.
  • “DMA means no CPU usage.” You still need to manage buffers and processing.

Check-Your-Understanding Questions

  1. Why do you want 10-20 ms frames for voice capture?
  2. What happens when the ring buffer overflows?

Check-Your-Understanding Answers

  1. It balances latency with manageable processing windows.
  2. You drop older frames to preserve real-time capture.

Real-World Applications

  • VoIP devices
  • Smart speakers
  • Any streaming audio product

Where You Will Apply It

Projects 2 and 5

References

  • ESP-IDF I2S driver and DMA overview.
  • ESP-SR audio input format expectations.
  • ESP32-S3 Datasheet (I2S full/half duplex, TDM/PDM, DMA support).

Key Insight

Audio fidelity and latency are controlled by buffer sizes and DMA flow, not by magic.

Summary

I2S and DMA provide the backbone for continuous audio streaming. Your design choices about frame size, buffer depth, and task scheduling determine whether the system is stable or glitchy.

Homework / Exercises

  1. Calculate bandwidth for 16 kHz mono 16-bit PCM.
  2. Experiment with buffer sizes and observe latency.

Solutions

  1. 16,000 samples/sec * 2 bytes = 32,000 bytes/sec (~31.25 KB/s).
  2. Smaller buffers reduce latency but increase dropouts.

4) Speech Front-End: Wake Word, VAD, and AEC

Fundamentals

A voice robot must detect when to listen. This is usually done with a wake word engine combined with voice activity detection (VAD) and sometimes acoustic echo cancellation (AEC). Espressif’s WakeNet wake word engine is a neural-network-based model designed for low-power MCUs. The ESP-SR documentation states that WakeNet supports up to five wake words, uses MFCC features, and expects 16 kHz, mono, signed 16-bit audio in 30 ms frames. These components are all about reducing false triggers while keeping latency low, which is why tuning thresholds and frame sizes matters as much as the model itself.

Deep Dive into the Concept

The speech front-end is the guardrail between always-on microphones and expensive downstream processing. It answers a simple question: “Is the user talking to me?” The wake word engine processes incoming audio in fixed frames (typically 20-30 ms), extracts features like MFCCs, and runs a lightweight neural network to detect target phrases. In the ESP-SR WakeNet model, each frame is 30 ms at 16 kHz, and the engine expects 16-bit mono PCM. That means your capture pipeline must deliver stable 16 kHz audio or your wake word detection will be unreliable.

Voice Activity Detection (VAD) is often paired with wake word detection. VAD is a simpler algorithm that detects whether audio contains speech-like energy. This helps save bandwidth by only streaming audio when speech is detected. It also helps decide when to stop recording after the user finishes speaking. Some systems use VAD as a pre-filter to reduce wake word false positives. In practice, a wake word pipeline might be: raw audio -> noise suppression -> VAD -> wake word -> trigger event.

Espressif’s Audio Front-End (AFE) framework bundles multiple algorithms that matter in noisy rooms: Acoustic Echo Cancellation (AEC), Noise Suppression (NS/NSNET), Voice Activity Detection (VAD), and Blind Source Separation (BSS) for dual-mic setups. This matters for XiaoZhi-class devices because the speaker and microphone are physically close, and echo or fan noise can destroy wake word accuracy unless you clean the signal early.

Acoustic Echo Cancellation (AEC) becomes critical when the device is speaking and listening at the same time. Without AEC, the microphone hears the device’s own speaker output and can falsely trigger wake words or distort the outgoing audio stream. AEC works by subtracting a model of the speaker output from the microphone input. On microcontrollers, AEC often needs to be tuned carefully or offloaded to a more capable audio chip. If you cannot do full AEC, a simpler strategy is “barge-in” rules: pause the speaker output when the microphone detects loud input, or lower speaker volume during listening windows.

The ESP-SR project provides customization workflows for WakeNet models, but they are not “push-button.” This informs an important design decision: for a prototype, use pre-trained wake words and focus on end-to-end latency; for production, invest in a robust wake word training pipeline. Real environments are noisy: fans, keyboards, and echoing rooms all raise the false-positive rate, so your pipeline must be tuned for the environment where the device will actually live.

The front-end also determines your latency budget. A 30 ms frame implies at least 30 ms of delay before you can trigger the wake word. Add buffering and it grows. To keep the user experience responsive, you should aim for a total wake word trigger latency under 200 ms if possible. You can get close to this by keeping your audio pipeline tight, minimizing queues, and ensuring wake word inference runs at high priority.

Finally, the speech front-end is where you balance power usage and responsiveness. An always-on wake word engine keeps the CPU active, consuming power. The ESP32-S3 includes vector instructions that can accelerate DSP, reducing CPU cycles. This is one reason the S3 is a strong platform for wake-word-driven applications.

How This Fits in the Projects

  • Project 3 uses VAD to reduce streaming.
  • Project 5 uses wake word and barge-in for full duplex.

Definitions & Key Terms

  • Wake Word: A trigger phrase that activates the assistant.
  • VAD: Voice Activity Detection.
  • AEC: Acoustic Echo Cancellation.
  • MFCC: Mel-Frequency Cepstral Coefficients.

Mental Model Diagram

Audio Frames -> Noise Suppression -> VAD -> WakeNet -> Trigger Event
                                  ^
                                  |
                               AEC Loop

How It Works (Step by Step)

  1. Capture 16 kHz PCM audio frames.
  2. Extract MFCC features for each frame.
  3. Run wake word model on features.
  4. If confidence > threshold, trigger state change.

Minimal Concrete Example

if (wakenet_detect(frame, &score) && score > THRESHOLD) {
  xEventGroupSetBits(evt_group, EVT_WAKE);
}

Common Misconceptions

  • “Wake word is just keyword spotting.” It is a full ML pipeline.
  • “AEC is optional.” Full duplex audio without AEC is fragile.

Check-Your-Understanding Questions

  1. Why do wake word engines prefer fixed frame sizes?
  2. What happens if you feed 48 kHz audio into a 16 kHz wake word model?

Check-Your-Understanding Answers

  1. Fixed frames simplify feature extraction and inference.
  2. The model will receive incorrect features and fail.

Real-World Applications

  • Smart speakers
  • Hands-free assistants
  • Wearables with voice input

Where You Will Apply It

Projects 3 and 5

References

  • ESP-SR WakeNet model requirements and MFCC pipeline.
  • ESP-SR WakeNet customization and model workflow docs.
  • ESP-SR Audio Front-End (AEC/NS/VAD/BSS) documentation.

Key Insight

Wake word accuracy is a system property, not just a model property.

Summary

The speech front-end decides when to listen, when to stop, and how to avoid echo. It is the gatekeeper for low-latency, always-on voice interaction.

Homework / Exercises

  1. Implement a simple energy-based VAD and compare it to a wake word trigger.
  2. Measure the trigger latency for a 30 ms frame pipeline.

Solutions

  1. Energy VAD detects speech but cannot identify specific phrases.
  2. Minimum latency is about 30 ms plus processing time.

5) Low-Latency Networking and Streaming (WebSocket + Wi-Fi)

Fundamentals

WebSocket provides full-duplex, bidirectional communication over a single TCP connection. RFC 6455 defines an opening HTTP Upgrade handshake followed by a framed message protocol on top of TCP. This makes it ideal for streaming audio in real time without repeated HTTP requests. On the ESP32, Wi-Fi power-saving modes can introduce latency; the ESP-IDF Wi-Fi guide notes that disabling modem-sleep (WIFI_PS_NONE) increases power consumption but minimizes delay in receiving Wi-Fi data. TCP itself also introduces head-of-line blocking, so you must size buffers carefully and keep the connection healthy.

Deep Dive into the Concept

Voice assistants are primarily constrained by latency. The audio path is only part of the story; the network path can add hundreds of milliseconds if not designed carefully. WebSocket is a good fit because it keeps a persistent TCP connection open. This eliminates repeated HTTP handshakes and allows both sides to push messages at any time. For audio streaming, you typically send binary frames containing raw PCM or Opus-compressed data. The server can send back TTS audio frames or JSON control messages in the same connection.

The WebSocket handshake uses HTTP headers and then upgrades to the WebSocket protocol. Once established, each message is framed with a small header that includes opcode and payload length. This is efficient enough for small audio packets (e.g., 20 ms of Opus frames). You should choose a frame size that matches your audio frame size to minimize buffering. If you send very large frames, you increase latency; if you send very small frames, you increase overhead and CPU load.

Wi-Fi power-saving modes can introduce jitter. In ESP-IDF, modem-sleep mode allows the station to sleep between beacons. This saves power but can add delay equal to the beacon interval. The docs note that disabling power saving (WIFI_PS_NONE) reduces delay but increases power consumption. For a voice assistant, latency is usually more important than power, so many designs disable power saving while streaming audio and re-enable it when idle.

Connection management is another major design point. If the connection drops, you need to reconnect quickly without blocking the audio pipeline. A robust design uses a network task that handles reconnection and exposes connection state to the rest of the system through events. When connection is lost, the UI should show “offline” and the system should gracefully stop streaming without crashing.

Security is also critical. WebSocket is typically run over TLS (wss://). TLS adds handshake cost and CPU usage, but for privacy you should treat it as mandatory. You can reduce the impact by keeping the connection open and reusing sessions, or by performing connection setup during boot while the device is idle. For a local-only assistant (Project 5 extension), you may decide to run without TLS on a trusted LAN, but this is a conscious tradeoff.

In a voice assistant, the network pipeline is often the bottleneck. If your network path is slow, you may need to compress audio (Opus), reduce sample rate, or implement buffering strategies. The overall latency budget is the sum of capture frame size, encoding, network RTT, decoding, and playback buffer. In practice, the network portion is the largest variable. This is why Project 3 focuses on streaming a “dumb” chatbot first: it isolates the network path from full system complexity.

Practical tuning matters. You may need to disable Nagle’s algorithm (TCP_NODELAY) to avoid packet coalescing delays, and you should measure RSSI because weak signal strength translates to retransmissions. A small jitter buffer on playback can smooth network variance, but too much buffering defeats the purpose of real-time interaction. The key is to measure and adjust based on real data, not assumptions.

How This Fits in the Projects

  • Project 3 streams audio to a server and receives responses.
  • Project 5 uses WebSocket for full duplex audio + control.

Definitions & Key Terms

  • WebSocket: Full-duplex communication over TCP.
  • Handshake: HTTP upgrade to establish WebSocket.
  • Frame: WebSocket message unit (binary or text).
  • WIFI_PS_NONE: ESP-IDF setting to disable power-saving.

Mental Model Diagram

Audio Frames -> Opus -> WebSocket Binary Frames -> Server
                                   ^
                                   |
                         Control / JSON / State

How It Works (Step by Step)

  1. Connect to Wi-Fi and establish TCP.
  2. Perform WebSocket handshake.
  3. Stream audio frames as binary messages.
  4. Receive TTS or control messages.
  5. Reconnect on errors without blocking audio capture.

Minimal Concrete Example

// Pseudocode
ws_send_binary(audio_frame, len);
ws_recv(msg, &len);

Common Misconceptions

  • “WebSocket is just HTTP.” It upgrades to a framing protocol.
  • “Power save is always good.” It can add 100+ ms latency.

Check-Your-Understanding Questions

  1. Why is WebSocket better than repeated HTTP POSTs for streaming?
  2. When should you disable Wi-Fi power saving?

Check-Your-Understanding Answers

  1. It avoids repeated handshakes and allows low-latency duplex traffic.
  2. During active streaming or low-latency interactions.

Real-World Applications

  • Live audio chat
  • IoT device control
  • Streaming telemetry

Where You Will Apply It

Projects 3 and 5

References

  • RFC 6455 WebSocket Protocol.
  • ESP-IDF Wi-Fi power-saving behavior.

Key Insight

Network latency dominates user perception; optimize the network path first.

Summary

WebSocket and Wi-Fi configuration determine whether your voice assistant feels responsive or sluggish. Manage connections, tune power-saving, and frame audio carefully.

Homework / Exercises

  1. Measure WebSocket RTT on your network.
  2. Compare power-saving enabled vs disabled latency.

Solutions

  1. Use ping messages or measure server echo time.
  2. Power-saving adds jitter; disabling reduces delay.

6) Opus Codec and Audio Compression

Fundamentals

Opus is an open, royalty-free audio codec standardized by the IETF as RFC 6716. It is designed for interactive speech and audio, supporting bitrates from 6 kb/s to 510 kb/s, sampling rates from 8 kHz to 48 kHz, and frame sizes from 2.5 ms to 60 ms. RFC 6716 specifies the standard frame durations (2.5, 5, 10, 20, 40, or 60 ms), and the Opus reference design supports both constant and variable bitrate modes. Opus also supports packet loss concealment and optional in-band forward error correction, which helps keep speech intelligible on lossy Wi-Fi links.

Deep Dive into the Concept

Compression is what makes real-time audio practical on small devices. Raw PCM audio at 16 kHz mono 16-bit is about 32 KB/s, which is manageable but still significant for Wi-Fi and cloud processing. Opus can reduce that to 8-24 kb/s with minimal quality loss for speech, drastically lowering bandwidth and latency sensitivity. For a voice assistant, the ideal Opus configuration is usually narrowband or wideband speech mode with 20 ms frames. Shorter frames (e.g., 10 ms) reduce latency but increase overhead; longer frames improve compression but add delay.

Opus is hybrid: it combines linear prediction (good for speech) and MDCT (good for music). This makes it flexible for different content. In a voice assistant, you can configure Opus for speech-only, which optimizes for intelligibility rather than music fidelity. Another important parameter is constant bitrate (CBR) vs variable bitrate (VBR). For a consistent network load and easier buffer sizing, CBR is often preferred, but VBR can improve quality at the same average bitrate. For streaming over WebSocket, predictable packet sizes simplify buffer management.

The Opus frame size you choose directly impacts user experience. A 20 ms frame means you add at least 20 ms of algorithmic latency. If you use 60 ms frames, you may save bandwidth but the assistant will feel sluggish. A good rule is to target 20 ms for both encoding and playback buffering, and keep total pipeline latency under 300 ms. For full duplex (Project 5), the effect is even more obvious: if playback latency is too high, the user hears a delayed response and might interrupt, causing barge-in handling to fail.

Opus also requires CPU cycles. On the ESP32-S3, you may not want to run Opus encode and decode simultaneously unless you carefully optimize. The S3’s vector instructions help with DSP workloads, but Opus is still heavy. In practice, you can offload encoding to the server for the “dumb chatbot” project and only decode on the device. For the full clone project, you will do both, which is why task priority and memory placement become critical.

Another subtlety is packet loss. Opus is designed to handle some loss, but if your Wi-Fi connection drops packets, your audio will degrade. This is why a stable network and good buffer strategy are essential. You can add forward error correction (FEC) or redundancy, but that increases bitrate. The better approach on ESP32 is to keep a short jitter buffer on playback and drop late packets rather than stalling.

Opus exposes configuration controls like complexity, signal type, and bitrate that directly affect CPU usage. On the ESP32-S3, lower complexity settings can reduce CPU load at the cost of some quality. The art is to pick a quality level that preserves intelligibility while staying inside your CPU budget. When you test, measure both CPU usage and subjective audio clarity. Compression is only useful if it does not destabilize the rest of the system.

Finally, remember that Opus supports different sampling rates. If your microphone runs at 16 kHz and your speaker can do 48 kHz, you might need sample rate conversion. That introduces CPU cost. Keep things simple by running the entire pipeline at 16 kHz if possible. Only use higher rates if you have a strong reason.

How This Fits in the Projects

  • Project 5 uses Opus for both upstream and downstream audio.

Definitions & Key Terms

  • Opus: IETF standard audio codec (RFC 6716).
  • Frame Size: Duration of audio encoded in one Opus packet.
  • CBR/VBR: Constant vs Variable Bitrate.
  • Algorithmic Latency: Delay introduced by the codec itself.

Mental Model Diagram

PCM -> Opus Encoder -> Network -> Opus Decoder -> PCM

How It Works (Step by Step)

  1. Buffer PCM samples into fixed-size frames.
  2. Encode each frame into Opus packets.
  3. Transmit packets over WebSocket.
  4. Decode packets back into PCM for playback.

Minimal Concrete Example

// Pseudocode: encode 20ms frame at 16kHz mono
opus_encode(enc, pcm, 320, packet, max_packet);

Common Misconceptions

  • “Opus always adds too much latency.” It can be as low as 2.5 ms.
  • “Higher bitrate always means better speech.” Above a point, returns diminish.

Check-Your-Understanding Questions

  1. Why does frame size affect latency?
  2. Why might you choose CBR for a microcontroller?

Check-Your-Understanding Answers

  1. Each frame must be collected before encoding.
  2. Predictable bandwidth simplifies buffer sizing and scheduling.

Real-World Applications

  • VoIP and conferencing
  • Game voice chat
  • Smart speakers

Where You Will Apply It

Project 5

References

  • Opus codec overview and feature ranges.
  • RFC 6716 Opus standard definition.
  • Opus reference documentation (bitrate, frame size, sample rate ranges).

Key Insight

Compression is the only way to make real-time voice scalable on embedded hardware.

Summary

Opus provides a flexible, low-latency codec tuned for speech. Your frame size and bitrate choices directly shape user experience.

Homework / Exercises

  1. Calculate bitrate for 16 kHz PCM vs Opus at 24 kb/s.
  2. Experiment with 10 ms vs 20 ms Opus frames and note latency.

Solutions

  1. PCM ~32 KB/s (~256 kb/s), Opus 24 kb/s is 10x smaller.
  2. 10 ms frames are faster but higher overhead.

7) Embedded UI and Feedback with LVGL

Fundamentals

LVGL (Light and Versatile Graphics Library) is a free and open-source embedded GUI library designed for microcontrollers. It provides widgets, animations, and styles with a low memory footprint, and it can run on any MCU with minimal dependencies. LVGL supports multi-display setups, input devices, and partial-buffer rendering. The official LVGL display docs recommend using buffers as small as 1/10 of the screen when using partial render mode, and note that the flush_cb must copy rendered pixels to the display. LVGL uses a tick counter for timing, so you must provide a periodic tick increment (usually from a hardware timer) and call the handler regularly to process animations and input.

Deep Dive into the Concept

A voice assistant is more than audio. Users need feedback: listening, thinking, speaking, offline, error. That feedback is what turns a firmware demo into a product. LVGL makes this possible with a structured UI system. In LVGL, you define screens, widgets, and styles. The rendering pipeline relies on a display driver that provides a flush callback. When LVGL draws, it writes pixels into a buffer and then asks your driver to push them to the display over SPI or parallel bus.

The key challenge is buffer size vs performance. A full frame buffer for a 240x240 RGB565 display is ~115 KB. That is too large for internal SRAM, so you typically place it in PSRAM and use partial buffers (for example, 1/10 of the screen). LVGL explicitly supports partial buffers and even single-buffer rendering. This is why LVGL is viable on ESP32-S3. You can allocate two small buffers in PSRAM and alternate them (double buffering), which reduces tearing and gives smoother animations.

UI timing also matters. If you call lv_timer_handler too often, you steal CPU cycles from audio. If you call it too rarely, UI animations are choppy. A common pattern is to run LVGL updates at 20-30 FPS, which is good enough for UI feedback. Since the voice assistant is not a full-screen animation device, you can limit updates to state transitions and simple animations (e.g., pulsing ring while listening). That keeps UI overhead low.

Another design challenge is thread safety. LVGL is not thread-safe by default; it expects all UI calls to run in a single task or under a lock. In a multi-task system, you should create a UI task that owns LVGL and other tasks send messages (via a queue) to request UI updates. This isolates UI complexity and prevents race conditions.

The display driver is hardware-specific. On ESP32-S3, you will often use SPI displays with libraries like ESP-IDF’s lcd driver. The performance of the UI depends on SPI clock rates and DMA. You should configure DMA for display transfers to avoid blocking the CPU. If the display is round, you may need to handle rotation or masking to avoid drawing outside the visible area.

Timing is a hidden complexity. LVGL expects a steady tick (for example, 1 ms). If your tick is irregular, animations will stutter and input handling will feel inconsistent. The LVGL docs also explain that with a single buffer, LVGL must wait for lv_display_flush_ready() before drawing again; with two buffers and DMA, rendering and flushing can overlap. You also need to ensure the flush callback is non-blocking or short, otherwise LVGL will hold a global lock and starve other tasks. The fix is to use DMA to push pixels asynchronously and signal completion with a semaphore. This pattern allows UI updates without compromising the audio pipeline.

The UI is also where you communicate system state. The wake word engine can trigger a “listening” animation, the network task can trigger a “thinking” animation, and the playback task can trigger a “speaking” animation. If the network disconnects, show a warning icon. These transitions must be fast and reliable, which is why UI events should be triggered from the central state machine rather than from individual tasks.

How This Fits in the Projects

  • Project 1 builds the UI foundation.
  • Project 5 integrates UI state with audio pipeline.

Definitions & Key Terms

  • LVGL: Embedded GUI library in C.
  • Flush Callback: Driver hook to send pixels to display.
  • Frame Buffer: Memory region holding pixels.

Mental Model Diagram

LVGL Widgets -> Draw Buffer -> Flush Callback -> SPI Display

How It Works (Step by Step)

  1. Initialize LVGL and display driver.
  2. Allocate draw buffers in PSRAM.
  3. Create widgets and styles.
  4. Call lv_timer_handler periodically.
  5. Update UI based on system events.

Minimal Concrete Example

lv_obj_t *label = lv_label_create(lv_scr_act());
lv_label_set_text(label, "Listening...");

Common Misconceptions

  • “UI is low priority so it doesn’t matter.” UI feedback drives UX.
  • “LVGL needs huge RAM.” It can run with partial buffers.

Check-Your-Understanding Questions

  1. Why does LVGL support partial buffers?
  2. Why should UI updates be event-driven?

Check-Your-Understanding Answers

  1. It allows rendering on low-memory systems.
  2. It avoids jitter and keeps UI consistent with state.

Real-World Applications

  • Smart devices with small displays
  • Wearables
  • IoT dashboards

Where You Will Apply It

Projects 1 and 5

References

  • LVGL introduction and key features.
  • LVGL display/renderer docs (flush_cb, buffer sizing, one/two buffer modes).

Key Insight

A voice assistant feels alive only if it can show its state visually.

Summary

LVGL provides a lightweight way to build a modern UI on ESP32-S3. Buffering and update rates must be tuned to avoid stealing time from audio tasks.

Homework / Exercises

  1. Calculate the RAM needed for a full 240x240 RGB565 frame.
  2. Implement a UI queue that updates a label text.

Solutions

  1. 2402402 = 115,200 bytes.
  2. Use a FreeRTOS queue to send strings to the UI task.

Glossary

  • AFE: Audio Front End; pre-processing like VAD and AEC.
  • Barge-In: Interrupting playback when the user speaks.
  • DMA: Peripheral-driven memory transfer.
  • Frame: Fixed-duration block of audio samples.
  • I2S: Inter-IC Sound protocol for audio streaming.
  • LVGL: Embedded GUI library.
  • Opus: Low-latency audio codec (RFC 6716).
  • PSRAM: External RAM for large buffers.
  • MCP: Model Context Protocol; a device-side control interface for tools and actions.
  • Wake Word: Trigger phrase for activation.

Why XiaoZhi AI on ESP32-S3 Matters

Voice assistants are a gateway into real-world embedded systems. They force you to solve the hardest problems at once: real-time audio, low-latency networking, UI feedback, and multi-tasking. On top of that, IoT adoption is exploding. IoT Analytics reports 18.5 billion connected IoT devices in 2024 and expects growth to 21.1 billion by the end of 2025. This means the skills you learn here apply to a massive and growing ecosystem.

The XiaoZhi AI platform demonstrates how open-source voice assistants can be built on affordable hardware. The xiaozhi-esp32 project emphasizes offline wake word detection (ESP-SR), streaming ASR/LLM/TTS, and device-side control via MCP. That combination makes it an ideal case study for anyone learning embedded AI systems.

Cloud vs Edge Voice Assistants

Cloud Model                          Edge/Local Model
+-------------------+               +-------------------+
| Mic -> Cloud ASR  |               | Mic -> Local ASR  |
| Cloud LLM -> TTS  |               | Local LLM -> TTS  |
| High latency      |               | Low latency       |
| Privacy risks     |               | Private by design |
+-------------------+               +-------------------+

Concept Summary Table

Concept What You Must Internalize Used In Projects
ESP32-S3 Architecture Memory hierarchy, PSRAM tradeoffs, peripheral set 1, 2, 5
FreeRTOS Concurrency Tasks, queues, event-driven design 2, 3, 5
I2S + DMA Audio Real-time capture, buffering, timing 2, 5
Speech Front-End Wake word, VAD, AEC basics 3, 5
Networking WebSocket streaming, Wi-Fi power tradeoffs 3, 5
Opus Codec Bitrate/frame size/latency balance 5
Embedded UI LVGL rendering, buffers, UI state feedback 1, 5

Project-to-Concept Map

Project ESP32-S3 FreeRTOS I2S/DMA Speech FE Network Opus UI
1. The Eye X           X
2. The Parrot X X X        
3. Dumb Chatbot X X X X X    
4. HA Satellite X   X X X    
5. Full Clone X X X X X X X

Deep Dive Reading by Concept

Concept Best Book Chapters (from your library) Why This Matters
ESP32-S3 Architecture Making Embedded Systems (Ch. 1-3), Code (Ch. 1-6) Builds intuition for memory/peripheral constraints
FreeRTOS Concurrency Making Embedded Systems (Ch. 4-6), Computer Systems: A Programmer’s Perspective (Ch. 12) Task scheduling and performance thinking
Audio + I2S Making Embedded Systems (Ch. 8), Computer Organization and Design (Ch. 4) I/O pipelines and DMA mental model
Speech Front-End Algorithms (Ch. 1-2), Computer Networks (Ch. 1) Feature extraction and data flow thinking
Networking TCP/IP Illustrated (Ch. 1-3), Computer Networks (Ch. 1-4) Protocols and latency reasoning
Opus Codec Algorithms in C (Parts 1-2), Computer Systems (Ch. 2) Compression tradeoffs and performance
Embedded UI Making Embedded Systems (Ch. 7), Code Complete (Ch. 5-6) UI state management and robustness

Quick Start (First 48 Hours)

Day 1

  1. Flash a blink + UART logging example to verify board.
  2. Connect display and run a minimal LVGL label.
  3. Measure memory usage and confirm PSRAM works.

Day 2

  1. Configure I2S microphone capture at 16 kHz.
  2. Print RMS values or a VU meter to validate audio input.
  3. Play a tone or loopback audio through speaker.
  1. Maker / UI Path: Project 1 -> Project 3 -> Project 4
  2. Firmware Engineer Path: Project 2 -> Project 3 -> Project 5
  3. Full-Stack Path: Project 1 -> Project 2 -> Project 3 -> Project 5

Success Metrics

  • You can run continuous audio capture for 10 minutes with no dropouts.
  • You can stream audio and receive responses under 300 ms end-to-end latency.
  • The UI updates correctly on every state transition.
  • The system recovers from Wi-Fi disconnects without rebooting.
  • You can explain buffer sizing and memory placement decisions.

Appendix: Tooling and Debugging Cheat Sheet

  • ESP-IDF monitor: Use idf.py monitor to capture logs.
  • I2S validation: Dump PCM to file and visualize with Audacity.
  • Network: Use Wireshark to inspect WebSocket frames.
  • Performance: Toggle Wi-Fi power saving and measure latency.

Project Overview Table

# Project Name Main Language Difficulty Time Estimate Core Skills
1 The Eye (Display) C/C++ Beginner 1-2 days LVGL, display drivers
2 The Parrot (Audio) C (ESP-IDF) Advanced 3-5 days I2S, DMA, buffering
3 Dumb Chatbot (API) C/C++ Intermediate 2-4 days WebSocket, streaming
4 HA Satellite YAML (ESPHome) Intermediate 2-4 days ESPHome, voice pipeline
5 The Full Stack XiaoZhi Clone C (ESP-IDF) Expert 2-4 weeks Full-duplex audio, Opus

Project List

Project 1: The “Eye” (Display and State Feedback)

Real World Outcome

You power on the device and see a boot animation on a round display. The screen transitions through states: “offline”, “listening”, “thinking”, and “speaking”. When you say the wake word, the UI pulses. When Wi-Fi drops, an error icon appears. The serial console shows a clean boot and UI state transitions.

Example serial output:

I (1023) ui: screen=boot
I (1450) ui: screen=idle
I (5021) wifi: connected, ip=192.168.1.88
I (5030) ui: screen=listening
I (8123) ui: screen=thinking
I (10210) ui: screen=speaking

The Core Question You’re Answering

How do you build a responsive embedded UI that accurately reflects system state without stealing timing from real-time audio tasks?

Concepts You Must Understand First

  • ESP32-S3 memory hierarchy (PSRAM vs SRAM)
  • LVGL buffer model and flush callbacks
  • Event-driven UI updates

Questions to Guide Your Design

  • What buffer size gives smooth updates without hogging RAM?
  • Should UI updates be timer-driven or event-driven?
  • Which states need visual feedback to reduce confusion?

Thinking Exercise

Draw your state diagram and map each state to a visual cue (icon, animation, color). Ask yourself: can a user tell what is happening without sound?

The Interview Questions They’ll Ask

  1. How do you minimize RAM usage in LVGL?
  2. What is a flush callback and why is it needed?
  3. How do you synchronize UI updates with system state?
  4. Why might UI tasks cause audio glitches?

Hints in Layers

  1. Start with a static label and confirm the display works.
  2. Use LVGL partial buffers (1/10 screen) to reduce memory.
  3. Create a UI task that owns LVGL and accepts state messages.
  4. Map system events to UI messages (e.g., EVT_WIFI_OK).

Books That Will Help

| Book | Chapters | Why | |—|—|—| | Making Embedded Systems | Ch. 1-3, 7 | UI timing and resource constraints | | Code Complete | Ch. 5-6 | State management and clean structure |

Common Pitfalls & Debugging

Problem 1: “Blank white screen”

  • Why: Incorrect SPI wiring or display reset.
  • Fix: Verify pin mapping and reset sequence.
  • Quick test: Draw a single pixel in the corner.

Problem 2: “Random flicker”

  • Why: Buffer size mismatch or flush timing.
  • Fix: Reduce refresh rate and verify buffer sizes.
  • Quick test: Render a static screen and see if it stabilizes.

Definition of Done

  • UI shows all major states (idle, listening, thinking, speaking)
  • No flicker or corruption during 5-minute run
  • UI task runs without blocking audio tasks
  • PSRAM usage logged and within expected bounds

Project 2: The “Parrot” (Low-Level Audio Capture and Playback)

Real World Outcome

You speak into the microphone and hear your voice back through the speaker with minimal delay. A VU meter on the serial console shows real-time audio levels. You can switch between direct loopback and buffered playback to observe latency differences.

Example serial output:

I (2120) audio: i2s_init sr=16000 bits=16
I (2140) audio: dma buffers=8 x 160 samples
I (3001) audio: rms=0.12
I (3011) audio: rms=0.45
I (3021) audio: rms=0.09

The Core Question You’re Answering

How do you move real-time audio through I2S and DMA without glitches or drift?

Concepts You Must Understand First

  • I2S configuration and DMA buffers
  • Ring buffer design
  • FreeRTOS task scheduling

Questions to Guide Your Design

  • What is the minimum buffer size that avoids underruns?
  • How do you detect and log buffer overruns?
  • How do you measure end-to-end latency?

Thinking Exercise

Estimate the time delay introduced by your buffer size and frame size. How would doubling the buffer count change latency?

The Interview Questions They’ll Ask

  1. Why use DMA for audio capture?
  2. How do you size DMA buffers for 16 kHz audio?
  3. What happens if your audio task blocks for 50 ms?
  4. How do you debug an audio underrun?

Hints in Layers

  1. Start with one-way capture and log RMS values.
  2. Add playback with a fixed-size ring buffer.
  3. Use a GPIO toggle to measure latency with an oscilloscope.
  4. Introduce jitter by logging, then remove it.

Books That Will Help

| Book | Chapters | Why | |—|—|—| | Making Embedded Systems | Ch. 8 | Real-time I/O systems | | Computer Organization and Design | Ch. 4 | I/O and DMA fundamentals |

Common Pitfalls & Debugging

Problem 1: “Stuttering audio”

  • Why: Buffer underrun or too small DMA buffers.
  • Fix: Increase dma_buf_count or lower sample rate.
  • Quick test: Add a counter for dropped frames.

Problem 2: “High latency”

  • Why: Buffers too large or multiple copies.
  • Fix: Reduce buffer size and avoid memcpy.
  • Quick test: Measure time between capture and playback.

Definition of Done

  • Continuous audio loopback for 5 minutes
  • No audible stutter or clipping
  • Logged RMS values update at least 10x/sec
  • Latency measured and documented

Project 3: The “Dumb Chatbot” (Streaming Audio to an API)

Real World Outcome

You press a button, the device starts streaming audio to a server, and the server responds with a text transcript and synthesized audio. The device shows “listening” on screen, then “thinking”, then speaks the answer. You can watch WebSocket traffic and see audio packets flowing.

Example serial output:

I (4100) net: ws connected
I (4120) audio: streaming start
I (6840) net: rx transcript="hello there"
I (7021) net: rx tts bytes=18432

The Core Question You’re Answering

How do you stream audio in real time while keeping the device responsive and stable?

Concepts You Must Understand First

  • WebSocket handshake and framing
  • Wi-Fi latency and power-saving effects
  • Audio frame sizing

Questions to Guide Your Design

  • Should you send PCM or compressed audio?
  • How do you detect network stalls?
  • How do you align audio frames to WebSocket packets?

Thinking Exercise

Sketch the latency budget: capture -> encode -> network -> decode -> playback. Which part dominates?

The Interview Questions They’ll Ask

  1. Why choose WebSocket over HTTP polling?
  2. How do you handle network disconnects gracefully?
  3. What happens if packets arrive out of order?
  4. How do you keep audio capture real-time while networking?

Hints in Layers

  1. Start by sending short audio clips, not full streams.
  2. Use binary frames to reduce overhead.
  3. Add a ping/pong timer to detect disconnects.
  4. Add a simple JSON control channel for start/stop.

Books That Will Help

| Book | Chapters | Why | |—|—|—| | TCP/IP Illustrated | Ch. 1-3 | TCP behavior and latency | | Computer Networks | Ch. 1-4 | Protocol layering and reliability |

Common Pitfalls & Debugging

Problem 1: “Audio dropouts”

  • Why: Network stalls or blocking send.
  • Fix: Buffer audio and use non-blocking send.
  • Quick test: Add queue depth logging.

Problem 2: “Slow response”

  • Why: Large frame sizes or power-saving mode.
  • Fix: Reduce frame size; disable Wi-Fi power save.
  • Quick test: Compare RTT with and without power save.

Definition of Done

  • Audio streams for 30 seconds with stable connection
  • Server response is received and played back
  • UI transitions through listening -> thinking -> speaking
  • System recovers from disconnect without reboot

Project 4: Home Assistant Voice Satellite (ESPHome)

Real World Outcome

Your ESP32-S3 appears in Home Assistant as a voice satellite. You can trigger Assist, speak commands, and hear responses. The device updates its LEDs or display based on pipeline state. The Home Assistant Assist Satellite entity defines the pipeline states (IDLE, LISTENING, PROCESSING, RESPONDING) and requires the device to signal when TTS playback finishes so it can return to IDLE. ESPHome’s Voice Assistant component requires Home Assistant 2023.5+ and warns that audio/voice components consume significant RAM/CPU; Bluetooth/BLE components can cause instability.

The Core Question You’re Answering

How do you integrate a voice device into a larger smart-home ecosystem while keeping the firmware simple and maintainable?

Concepts You Must Understand First

  • ESPHome Voice Assistant component and pipeline config
  • Assist Satellite entity state machine (IDLE/LISTENING/PROCESSING/RESPONDING)
  • Audio capture and playback on ESP32-S3
  • Network discovery (mDNS) and Home Assistant integration

Questions to Guide Your Design

  • Which parts should be handled by ESPHome vs custom firmware?
  • How will you expose wake word controls to HA?
  • How do you handle OTA updates reliably?

Thinking Exercise

Compare custom firmware vs ESPHome. What do you gain and lose in each approach?

The Interview Questions They’ll Ask

  1. What is the value of a standardized satellite entity?
  2. How do you keep latency low in an ESPHome pipeline?
  3. How do you design for OTA stability?
  4. What would you log to debug user complaints?

Hints in Layers

  1. Start with ESPHome voice assistant example configs.
  2. Validate mic and speaker before enabling Assist.
  3. Use HA logs to debug handshake issues.
  4. Add LED indicators for pipeline states.

Books That Will Help

| Book | Chapters | Why | |—|—|—| | Making Embedded Systems | Ch. 9 | System integration thinking | | Clean Architecture | Ch. 1-4 | Managing complexity in systems |

Common Pitfalls & Debugging

Problem 1: “Device not discovered”

  • Why: mDNS or network segmentation issues.
  • Fix: Ensure device and HA are on the same subnet.
  • Quick test: Ping and check mDNS entries.

Problem 2: “Audio garbled”

  • Why: Sample rate mismatch or codec mismatch.
  • Fix: Verify ESPHome audio settings.
  • Quick test: Record and playback raw audio locally.

Problem 3: “Random reboots or unstable voice pipeline”

  • Why: Voice components are RAM/CPU heavy, and ESPHome warns that BLE components can cause instability.
  • Fix: Disable BLE components, reduce logging, and confirm free heap during Assist sessions.
  • Quick test: Run a 10-minute Assist session and log minimum heap free.

Definition of Done

  • Device appears in HA with Assist satellite entity
  • Voice command triggers response playback
  • UI/LEDs reflect pipeline state
  • TTS completion signals are sent so the satellite returns to IDLE
  • OTA updates work reliably

Project 5: The Full Stack “XiaoZhi” Clone

Real World Outcome

You speak to the device hands-free. It wakes on a hotword, streams audio in real time, and responds with natural speech. You can interrupt it mid-response, and it immediately stops speaking to listen again. The UI shows state transitions, and the system remains stable for hours.

Example serial output:

I (1200) sys: state=idle
I (8120) wake: detected "Hello XiaoZhi"
I (8135) sys: state=listening
I (10450) net: ws connected
I (12480) sys: state=speaking
I (15550) sys: barge-in detected, stop playback

The Core Question You’re Answering

How do you build a production-grade, full-duplex voice assistant on an MCU with limited resources?

Concepts You Must Understand First

  • All concepts in the Theory Primer
  • State machine design
  • Opus encode/decode pipeline

Questions to Guide Your Design

  • Where do you enforce low-latency constraints?
  • How do you handle barge-in cleanly?
  • How do you prevent audio feedback loops?
  • Where do you store large buffers without starving tasks?

Thinking Exercise

Create a latency budget table. For each stage (capture, encode, network, decode, playback), estimate the worst-case delay.

The Interview Questions They’ll Ask

  1. How do you keep full duplex audio stable?
  2. How do you prioritize tasks to avoid jitter?
  3. What is your reconnect strategy for WebSocket?
  4. How do you measure and reduce latency?

Hints in Layers

  1. Start with half-duplex (listen -> speak) before full duplex.
  2. Add Opus encoding only after PCM streaming is stable.
  3. Use a jitter buffer on playback to smooth network variance.
  4. Implement barge-in by muting playback on VAD.

Books That Will Help

| Book | Chapters | Why | |—|—|—| | Making Embedded Systems | Ch. 10-11 | System integration and reliability | | Computer Systems: A Programmer’s Perspective | Ch. 12 | Concurrency and performance | | Fundamentals of Software Architecture | Ch. 4-6 | Designing robust systems |

Common Pitfalls & Debugging

Problem 1: “Echo feedback”

  • Why: No AEC or poor gain staging.
  • Fix: Lower speaker volume or implement barge-in.
  • Quick test: Mute speaker and verify mic capture stability.

Problem 2: “Random resets”

  • Why: PSRAM or stack overflow under load.
  • Fix: Increase task stack sizes, monitor heap.
  • Quick test: Enable heap tracing and watch free heap.

Definition of Done

  • Wake word triggers reliably with <200 ms response
  • Full duplex works with barge-in
  • Opus streaming is stable for 5 minutes
  • UI shows accurate state at all times
  • System runs for 2 hours without crash