High-Level Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Muse Gadget SDK provides the firmware and software to connect off-the-shelf ESP32 boards and Linux computers to the Muse ecosystem. It enables devices to interact with Muse through a secure tunnel, supporting features such as on-screen avatars, push-to-talk voice interactions, and remote command execution. The architecture spans two primary platforms: the ESP32 Device SDK and the Linux Device SDK.
Sources: README.md:21-36, esp32/README.md:16-24, linux/README.md:16-20
Cross-Platform Communication
Both ESP32 and Linux gadgets connect to a Muse Virtual Machine (VM) using a standardized pairing and session protocol. The communication relies on Bluetooth Low Energy (BLE) for initial setup and a secure Noise protocol session for ongoing data exchange.
Connection Workflow
- Pairing: Devices use BLE to pair with the Muse mobile app (iOS/Android).
- Authentication: Every gadget requires an SDK token from
gadgets.muse.aito pair. - Encrypted Session: Once paired, the device establishes an encrypted Noise XX handshake session with the Muse VM.
- Message Exchange: The device registers itself (via
link.register) and listens for commands (link.invoke).
Sources: README.md:38-42, linux/AGENTS.md:22-26, linux/AGENTS.md:95-103
The diagram shows the sequence of establishing a connection between the Muse mobile app, the gadget device, and the Muse VM. Sources: README.md:44-46, linux/AGENTS.md:95-101
ESP32 Firmware Architecture
The ESP32 SDK is built on ESP-IDF v6.0.1. It provides a modular framework for supporting diverse hardware configurations, including different displays, audio codecs, and input methods.
Component Structure
- Main Application: Handles the primary lifecycle and command registration (
main/app.c). - Muse UI: An LVGL-based interface providing an animated avatar, settings menu, and voice interaction feedback.
- Board Support Package (BSP): Abstracted through
muse_board_tstructures to handle hardware-specific initialization (e.g.,board_vn_s3_183.c). - Audio Pipeline: Manages I2S audio for microphones and speakers using
esp_codec_dev.
Sources: esp32/README.md:58-62, esp32/components/muse/boards/board_vn_s3_183.c:380-410, esp32/devices/AGENTS.md:183-205
Hardware Abstraction Layer
Hardware-specific logic is encapsulated in board files. For example, the muse_board_t structure defines the following functional pointers:
| Field | Purpose |
|---|---|
init | Power latches, I2C bus, and PMU setup |
display_start | Panel and LVGL task initialization |
audio_init | Speaker and microphone codec configuration |
poll_buttons | Edge detection for hardware buttons |
read_power | Battery voltage and charging status monitoring |
Sources: esp32/devices/AGENTS.md:183-205, esp32/components/muse/boards/board_vn_s3_183.c:380-410
Linux Device SDK Architecture
The Linux SDK is a Python-based implementation (musegadget) designed for systems like the Raspberry Pi. It operates as a systemd service that manages BLE pairing and the Noise session to the Muse VM.
Core Modules
link_client.py: Manages the Noise XX handshake and the persistent connection to/v1/noise.executor.py: Executes shell commands and file operations received from Muse.ble_server.py: Implements a BlueZ GATT peripheral for pairing via D-Bus.identity.py: Handles persistent device identity and naming (MuseGadgetXXXXXX).
Sources: linux/AGENTS.md:20-33, linux/README.md:57-62
Execution Model
Commands from Muse run as a specific user account (configured via --run-as). If the account has sudo privileges, Muse can perform administrative tasks.
The sequence diagram illustrates the flow of a remote command execution from the Muse VM to the Linux host system. Sources: linux/AGENTS.md:112-117, linux/README.md:57-62
Secure Communication Tunnel
The "Home Link" system creates a tunnel allowing Muse to reach local HTTP APIs or devices on the gadget's local network. This feature is primarily available on devices with sufficient memory (e.g., ESP32-S3 or Linux).
- ESP32: Requires PSRAM to support the tunnel and image processing.
- Linux: Uses the
musegadgetservice to bridge communication between the VM and the local network. - Security: Pairing uses a 6-digit code for out-of-band verification. On ESP32, NVS encryption is recommended to protect Wi-Fi credentials and tokens.
Sources: esp32/README.md:16-18, esp32/README.md:180-186, esp32/devices/AGENTS.md:128-132, linux/README.md:34-40
Conclusion
The High-Level Architecture of the Muse Gadget SDK facilitates a flexible and secure connection between physical hardware and the Muse ecosystem. By abstracting hardware differences through platform-specific SDKs and utilizing a unified Noise-based communication protocol, the system allows developers to extend Muse's capabilities to a wide array of DIY devices, from microcontroller-based smart displays to Linux-powered home automation hubs.
Sources: README.md:21-36, esp32/README.md:170-175, linux/AGENTS.md:95-103