Introduction to Muse Gadget for CYD
Relevant source files
The following files were used as context for generating this wiki page:
The Muse Gadget for CYD is a community board port that adapts Meta's muse-gadget-sdk for the "Cheap Yellow Display" (ESP32-2432S028R). This firmware enables the CYD to connect to the Muse app as a functional gadget, providing a status interface, remote image rendering, and touch-based interaction. Sources: README.md:1-5, board-port/esp32/README.md:154-154
This system operates by flashing open-source firmware onto the ESP32 chip. It handles BLE pairing for initial setup, connects to local Wi-Fi, and establishes an encrypted Noise session with a backend VM. While the CYD hardware lacks PSRAM and extensive flash memory (limiting features like the home-network tunnel or full avatar UI), it fully supports the core control session and visual status updates. Sources: README.md:19-22, board-port/esp32/README.md:17-21
Hardware and Device Features
The CYD board utilizes a 2.8" ILI9341 display and an ESP32-D0WDQ6 chip. The port leverages specific onboard peripherals to mirror the standard Muse Gadget experience. Sources: README.md:1-15, board-port/esp32/main/led_status.c:135-144
Core Capabilities
- Status Screen: Displays connection states via colored bars and pixel-art animations on the ILI9341 screen.
- Onboard RGB LED: Mirrors the status color (e.g., orange for setup, green for connected) using common-anode PWM control.
- Touch Interaction: Uses the XPT2046 touch controller to simulate the physical BOOT button for pairing confirmation.
- Remote Commands: Supports
display.draw_urlfor image rendering anddisplay.textfor title updates sent from the Muse agent.
Sources: README.md:10-18, board-port/esp32/main/led_status.c:266-318
Visual Indicators
The firmware uses specific colors to communicate device state through both the screen bars and the RGB LED.
| Color | Meaning |
|---|---|
| Orange (Breathing) | BLE advertising; ready for setup |
| Blue (Breathing) | Waiting for physical confirmation (touch/button) |
| Blue (Solid) | Joining Wi-Fi or connecting to Muse |
| Green | Connected and session established |
| Yellow (Blinking) | Connection lost or VM switching |
| Purple | Device is not paired |
| Red (Blinking) | System error occurred |
Sources: README.md:73-81, board-port/esp32/README.md:106-115, board-port/esp32/AGENTS.md:329-340
System Architecture and Data Flow
The architecture consists of several modular components within the main and components directories, coordinating networking, security, and peripheral control. Sources: board-port/esp32/AGENTS.md:20-25
The diagram shows the interaction between the core application logic, the networking stack for connectivity, and the peripheral drivers for user interaction. Sources: board-port/esp32/main/app.c:1162-1260, board-port/esp32/main/noise_control.cpp:1152-1240
Connection Lifecycle
- Provisioning: The device starts in BLE mode. The Muse app sends Wi-Fi credentials and an SDK token.
- Wi-Fi Join: The
wifi_mgrconnects to the saved network. - Noise Session:
noise_control.cppinitiates a WebSocket upgrade over TLS tohatch.metaaivm.com. - Command Loop: Once the session is established, the device registers its capabilities and begins listening for
link.invokerequests from the Muse backend.
Sources: board-port/esp32/main/app.c:640-710, board-port/esp32/main/noise_control.cpp:882-930
Implementation Details
Input Handling (Touch-as-Button)
Since the CYD is often used in enclosures, the firmware maps XPT2046 touch events to the system's button handling logic. The cyd_touch_task polls the Z1 pressure value; if it exceeds a threshold (CYD_TOUCH_THRESHOLD), it triggers button_touch_tap(), which replicates a short press of the BOOT button. Sources: board-port/esp32/main/led_status.c:300-318, board-port/esp32/tests/test_link_cyd_extras.py:42-48
// board-port/esp32/main/led_status.c:300
static uint16_t cyd_touch_z1(void) {
uint8_t tx[3] = {CYD_TOUCH_CMD_Z1, 0, 0};
uint8_t rx[3] = {0, 0, 0};
spi_transaction_t t = {
.length = 24, .tx_buffer = tx, .rx_buffer = rx,
};
if (spi_device_transmit(s_touch, &t) != ESP_OK) return 0;
return (uint16_t)(((rx[1] << 8) | rx[2]) >> 4);
}
Sources: board-port/esp32/main/led_status.c:300-307
Display Rendering
The led_status.c module manages the ILI9341 display using esp_lcd. It maintains two primary buffers: s_bar_buf for status indicators and s_anim_buf for animations and images. The display.draw_url command triggers image_fetch_start, which downloads a JPEG or raw RGB565 image, processes it, and renders it to the screen, temporarily replacing the idle animation. Sources: board-port/esp32/main/led_status.c:237-264, board-port/esp32/main/app.c:1162-1180
Technical Configuration
Building the firmware requires ESP-IDF v6.0.1 and a specific SDK token. The CYD port uses a unique Kconfig overlay to define pin maps and peripheral behaviors. Sources: README.md:24-34, board-port/esp32/AGENTS.md:55-65
| Option | Value / Description | File |
|---|---|---|
CONFIG_GADGET_SDK_TOKEN | User's SDK token from gadgets.muse.ai | sdkconfig.token |
CONFIG_HOMEHUB_LED_BACKEND_CYD_ILI9341 | Enables CYD-specific display and LED drivers | devices/sdkconfig.cyd |
LCD_PIN_BL | GPIO 21 (Backlight control) | led_status.c:143 |
CYD_LED_R_GPIO | GPIO 4 (Common Anode Red) | led_status.c:156 |
CYD_TOUCH_SPI_HOST | SPI3_HOST (Dedicated touch SPI) | led_status.c:283 |
Sources: README.md:33-40, board-port/esp32/main/led_status.c:135-165, board-port/esp32/tests/test_link_cyd_extras.py:65-72
Summary
The Muse Gadget for CYD provides a cost-effective platform for interacting with Muse AI agents. By repurposing the CYD's display and touch interface, the firmware delivers a comprehensive subset of the standard SDK's features, including encrypted communication, visual status feedback, and remote content rendering. Developers can further extend the on_ws_command handler in app.c to add custom hardware interactions unique to the CYD. Sources: README.md:1-22, board-port/esp32/AGENTS.md:465-480