Device Control & Chat Sessions
Relevant source files
The following files were used as context for generating this wiki page:
The Device Control and Chat Session systems manage the lifecycle of communication between a Muse gadget and the Muse Virtual Machine (VM). These systems handle physical hardware interactions, such as button presses and audio sampling, while simultaneously maintaining encrypted network tunnels for real-time voice processing and text-based interactions.
Sources: esp32/README.md:144-150, esp32/components/muse/muse_chat_session.cpp:17-30
Chat Session Architecture
The chat session establishes a persistent HTTP-over-Noise connection to the Muse VM. This transport layer utilizes a WebSocket upgrade on the /v1/noise path, followed by a Noise handshake to secure the multiplexed request streams.
Voice and Data Flow
The system separates processing into two primary tasks: the Voice Task and the Session Task. These tasks communicate via thread-safe queues and stream buffers.
The diagram shows the bidirectional flow of audio data and control commands between hardware and the remote Muse VM.
Sources: esp32/components/muse/muse_chat_session.cpp:32-41, linux/AGENTS.md:104-108
Turn Lifecycle
A "Turn" represents a single interaction cycle, starting from a user initiation and ending when the system settles.
- Initiation: The user presses the talk button, triggering a
CMD_BEGINcommand. - Streaming: The gadget streams 24 kHz PCM16 audio to
/api/voice/dictationor sends a base64-encoded voice note to/chat/stream. - Processing: The VM returns NDJSON transcripts (partials and finals).
- Response: The VM streams assistant messages via the
/chat/subscribestream. - Output: Messages are displayed as captions or converted to speech.
Sources: esp32/components/muse/muse_chat_session.cpp:19-27, esp32/components/muse/muse_chat_session.cpp:520-530
Device Control Interface
Hardware control is abstractly handled through a muse_board_t structure, which defines how specific boards interact with the UI and session logic.
Board Configuration and Capabilities
Different boards support varying levels of interaction, from simple LED status lights to full touchscreen UIs with multiple buttons.
| Capability | Implementation Detail | Source File |
|---|---|---|
| Talk Button | Maps physical GPIOs (e.g., GPIO 0) to session commands. | board_vn_s3_183.c:101 |
| Power Management | Monitors battery millivolts and charging status via ADC channels. | board_vn_s3_183.c:302-320 |
| Settings UI | Provides on-device controls for brightness, volume, and Wi-Fi. | muse_settings_ui.c:980-990 |
| Brightness | Controlled via LEDC PWM duty cycles (0 to 100%). | board_vn_s3_183.c:166-170 |
Physical Interaction Flow
Interaction with the device is often triggered by physical buttons, which are polled every 10ms to detect edge transitions.
The sequence illustrates how a physical button press translates into a remote VM chat turn.
Sources: esp32/devices/AGENTS.md:275-290, esp32/components/muse/muse_chat_session.cpp:665-680
Management and Configuration
The Device Control system includes comprehensive settings accessible via a local UI on boards with screens or via Bluetooth LE for all gadgets.
Settings Module
The settings module manages persistent configuration stored in NVS (Non-Volatile Storage).
- Wi-Fi Management: Scans for APs, saves up to 8 networks, and automatically joins the strongest signal.
- Audio Levels: Manages volume (0-100%) and microphone gain (0-36 dB).
- Session Parameters: Configures the Muse Server host and optional VM ID.
Sources: esp32/components/muse/muse_settings_ui.c:458-470, esp32/components/muse/muse_settings_ui.c:645-660
Connection States
Gadgets report their status through color-coded breathe animations or on-screen icons.
| State | Light / UI Representation | Description |
|---|---|---|
| Orange | Breathing | Ready for initial setup and pairing. |
| Blue | Breathing | Waiting for user confirmation (button press). |
| Green | Solid | Connected to Muse and online. |
| Yellow | Blinking | Attempting to reconnect to Wi-Fi or VM. |
| Purple | Solid | Device is not currently paired with an account. |
Sources: esp32/README.md:120-130, esp32/simulator/src/sim_services.c:36-46
Conclusion
The Device Control and Chat Session systems provide a robust framework for managing the intersection of hardware and AI. By decoupling the low-level audio/button handling from the high-level Noise-encrypted communication, the SDK allows for consistent behavior across diverse ESP32 and Linux hardware platforms. This modularity ensures that gadgets remain responsive to local input while maintaining stable, real-time connections to the Muse processing environment.