Introduction to Muse Gadgets
Relevant source files
The following files were used as context for generating this wiki page:
Muse Gadgets are open-source devices built using ESP32 boards or Linux-based computers like the Raspberry Pi. The project provides SDKs and firmware to connect these devices to the Muse AI platform, enabling users to interact with physical hardware through voice, text, and automated commands.
The SDKs allow developers to integrate displays, sensors, and actuators. Devices pair with the Muse mobile application on iOS or Android via Bluetooth Low Energy (BLE) and communicate over encrypted sessions to a Muse virtual machine (VM).
Sources: README.md:21-27, esp32/README.md:16-22, linux/README.md:16-21
System Architecture
The Muse Gadget ecosystem consists of three primary components: the hardware gadget (ESP32 or Linux), the Muse mobile application, and the Muse AI backend.
High-Level Data Flow
The following diagram illustrates how a user interacts with a gadget and how data flows to the Muse backend.
The gadget establishes a secure connection to the Muse backend. On ESP32 boards with sufficient memory, a home-network tunnel allows Muse to reach local HTTP APIs.
Sources: README.md:37-43, esp32/README.md:16-22, linux/AGENTS.md:32-34
Device SDKs
The project supports two main platforms, each with distinct capabilities and implementation requirements.
ESP32 Device SDK
The ESP32 firmware is built with ESP-IDF v6.0.1. It supports various hardware configurations, ranging from simple status lights to full UI devices with AMOLED screens and audio input/output.
| Feature Type | Components | Implementation Basis |
|---|---|---|
| Light | Status LED | Kconfig overlay + sdkconfig.defaults |
| Status Screen | Edge bars, animations | Display backend in main/ |
| E-paper | Text-based status | led_status.h implementation |
| Full UI | LVGL Avatar, voice, settings | muse_board_t struct + sdkconfig.muse |
Sources: esp32/README.md:57-61, esp32/devices/AGENTS.md:126-135
Linux Device SDK
The Linux SDK is a Python package (musegadget) that turns a Linux machine into a Muse Home Link. It operates as a background service and executes shell commands or file operations requested by Muse.
Sources: linux/README.md:88-91, linux/AGENTS.md:35-46
Connectivity and Security
Pairing Process
Gadgets use Bluetooth Low Energy for the initial setup. Users must enable "Developer Mode" in the Muse mobile app to discover devices prefixed with MuseGadget.
- Discovery: The device advertises its identity via BLE.
- Authentication: For ESP32, the user presses the BOOT button to confirm pairing. For Linux, the user runs
musegadget pair. - Provisioning: The app sends Wi-Fi credentials to the device.
- Session: The device uses an SDK token to establish an encrypted Noise session with a Muse VM.
Sources: README.md:41-45, esp32/README.md:99-105, linux/README.md:65-71
Security Measures
- Encryption: NVS encryption is recommended for ESP32 to protect Wi-Fi credentials and device tokens stored in flash.
- Noise Protocol: Communication with the backend uses the Noise XX handshake for end-to-end encryption.
- SDK Tokens: Every gadget requires a unique SDK token for identification.
Sources: esp32/README.md:195-200, linux/AGENTS.md:86-90
Functional Capabilities
Gadgets can execute built-in commands and handle custom extensions defined by the developer.
Standard Linux Commands
| Command | Action |
|---|---|
system.run | Executes a shell command and returns output/exit code |
file.read | Reads a file in 64 KB chunks |
file.write | Writes data to a file |
device.health | Reports uptime, load, memory, and temperature |
Sources: linux/README.md:79-84
Avatar Rendering (ESP32)
Devices with screens can display a pixel-art avatar. The system uses a procedural 64x64 renderer with animations for states like listening, thinking, and speaking. Developers can use the avatar.py tool to ask Muse to generate a custom C-based renderer for their device.
Sources: esp32/tools/muse/avatar.py:22-30, esp32/README.md:173-178
Summary
Muse Gadgets bridge the gap between AI agents and physical hardware. By utilizing the ESP32 and Linux SDKs, developers can build custom devices that respond to Muse commands, manage home automation, or provide interactive visual feedback through custom avatars. The project emphasizes a "hacker-first" approach, allowing deep customization of hardware interfaces and backend command execution.
Sources: README.md:25-30, esp32/README.md:21-26