Audio & Voice Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Audio & Voice Pipeline manages audio capture, processing, and playback for Muse gadgets. It provides a synchronized duplex interface for 16 kHz, 16-bit mono audio streams, enabling features like push-to-talk dictation, voice notes, and AI-generated speech playback. Sources: esp32/components/muse/muse_audio.h:23-28, esp32/components/muse/muse_chat_session.cpp:28-40
System Architecture
The pipeline consists of three primary layers: the hardware abstraction layer (HAL), the audio management component, and the chat session manager. The HAL interface interacts directly with I2S peripherals and external codecs like the AIC3204 or XMOS chips. The muse_audio component provides high-level mono buffers and signal processing, such as high-pass filtering. The muse_chat_session coordinates network streaming of voice data to the Muse VM. Sources: esp32/main/voice_board.c:38-48, esp32/components/muse/muse_audio.c:37-45, esp32/components/muse/muse_chat_session.cpp:42-50
Logic and Data Flow
The diagram shows the transition of audio data from physical capture through digital processing to network transmission and eventual playback. Sources: esp32/components/muse/muse_audio.c:166-180, esp32/components/muse/muse_chat_session.cpp:42-50, esp32/main/voice_board.c:267-285
Audio Hardware Interfacing
Hardware initialization occurs in voice_board.c, which configures I2C for codec control and I2S for data transport. The system supports specialized boards like the Seeed SenseCAP Watcher, which utilizes XMOS for advanced processing like Acoustic Echo Cancellation (AEC) and Automatic Gain Control (AGC). Sources: esp32/main/voice_board.c:79-105, esp32/main/voice_board.c:140-165
Codec Configuration
The AIC3204 codec is configured via I2C to handle 48 kHz internal processing, which is then decimated or resampled for the pipeline's 16 kHz requirements. The driver manages digital volume steps from -63.5 dB to +24 dB. Sources: esp32/main/voice_board.c:140-175, esp32/main/voice_board.c:177-188
| Component | I2C Address | Function |
|---|---|---|
| XMOS | 0x42 | AEC, AGC, and Clock Master |
| AIC3204 | 0x18 | DAC and Output Gain Control |
| Sources: esp32/main/voice_board.c:50-51 |
Audio Processing and Management
The muse_audio module manages mono audio frames in 20 ms chunks (320 samples at 16 kHz). It implements a one-pole high-pass filter at 80 Hz to remove DC offset and low-frequency rumble. Sources: esp32/components/muse/muse_audio.h:30-31, esp32/components/muse/muse_audio.c:170-178
Key Management Functions
- muse_audio_read: Performs a blocking read of mixed mono samples, applying high-pass filtering. Sources: esp32/components/muse/muse_audio.c:166-180
- muse_audio_write: Transmits mono samples to the speaker; it substitutes samples with silence if the speaker is disabled in settings. Sources: esp32/components/muse/muse_audio.c:182-198
- muse_audio_power: Toggles power to ADC, DAC, and I2S peripherals to conserve energy during idle states. Sources: esp32/components/muse/muse_audio.c:64-80
Audio Diagnostics
The pipeline includes self-test routines that measure real capture and playback rates and calculate mic correlation and RMS levels in dBFS. Sources: esp32/components/muse/muse_audio.c:98-124
Voice Communication Pipeline
Voice interaction follows a push-to-talk model managed by muse_chat_session.cpp. On devices with PSRAM, audio is processed through a dedicated Noise-encrypted connection. Sources: esp32/components/muse/muse_chat_session.cpp:28-40
Dictation and Voice Notes
When a user speaks, the system can operate in two modes:
- Voice Note: Encodes 16 kHz PCM into Base64 within a JSON envelope for
POST /chat/stream. Sources: esp32/components/muse/muse_chat_session.cpp:446-465 - Streaming Dictation: Resamples audio to 24 kHz for
POST /api/voice/dictation. Sources: esp32/components/muse/muse_chat_session.cpp:395-430
The sequence shows the path of a Voice Note from the microphone to the server. Sources: esp32/components/muse/muse_chat_session.cpp:468-490, esp32/components/muse/muse_chat_link.c:510-525
Audio Playback (TTS)
Assistant replies arrive as text and can be converted to speech. The system includes a minimp3 decoder that converts incoming MP3 fragments into mono PCM, which is then resampled to 16 kHz for the speaker. Sources: esp32/components/muse/muse_chat_session.cpp:665-700
Low-Memory Implementation (Link Mode)
For devices without PSRAM (e.g., ESP32-C6), the pipeline uses muse_chat_link.c. This implementation utilizes Home Link's existing session rather than a dedicated Noise connection. It supports text-only replies and uses bounded buffers (12 KiB subscription lines) to minimize internal RAM usage. Sources: esp32/components/muse/muse_chat_link.c:26-34
Constraints and Behavior
- Buffer Limits: Subscription lines exceeding 12 KiB are skipped to prevent memory overflow. Sources: esp32/components/muse/muse_chat_link.c:45
- Base64 Chunking: Audio is sent in 1536-byte PCM stages, which encode to 2 KB Base64 chunks. Sources: esp32/components/muse/muse_chat_link.c:38-39
- Timeout Handling: A turn fails if no reply is received within 60 seconds of the note upload. Sources: esp32/components/muse/muse_chat_link.c:50
Summary
The Audio & Voice Pipeline provides a robust framework for real-time voice interaction. It abstracts complex hardware configurations and resampling logic, allowing the Muse gadget to communicate efficiently with the cloud VM while managing hardware constraints like memory limits and I/O latency. Sources: esp32/components/muse/muse_audio.h:23-28, esp32/components/muse/muse_chat_session.cpp:28-50