Meta muse-gadget-sdk Integration

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• board-port/esp32/AGENTS.md
• board-port/esp32/README.md
• board-port/esp32/main/app.c
• board-port/esp32/main/noise_control.cpp
• board-port/esp32/main/CMakeLists.txt

The Meta muse-gadget-sdk integration provides the firmware framework for connecting ESP32-based hardware to the Muse platform. This integration enables devices to pair with mobile applications via Bluetooth Low Energy (BLE), join Wi-Fi networks, and establish secure, encrypted Noise protocol sessions with a virtual machine (VM) backend. Sources: board-port/esp32/README.md:17-21, board-port/esp32/AGENTS.md:20-25

The system utilizes the ESP-IDF (v6.0.1) framework to manage low-level hardware interactions, including Wi-Fi provisioning, OTA updates, and encrypted storage. It allows developers to extend gadget functionality by registering custom commands and managing hardware-specific status indicators like LEDs or displays. Sources: board-port/esp32/README.md:65-70, board-port/esp32/AGENTS.md:374-380

System Architecture and Components

The integration is structured into several key modules within the main/ directory and external components. The primary logic resides in app.c, while noise_control.cpp handles the secure communication layer.

Core Components

Component	Function
app.c	Manages high-level application logic, state transitions, and provisioning flows.
noise_control.cpp	Implements the Noise protocol session, WebSocket framing, and command dispatch.
ble_server.c	Handles BLE advertising and the community pairing v5 handshake.
wifi_mgr.c	Manages Wi-Fi scanning, connection, and credential storage.
ota.c	Facilitates over-the-air firmware updates via HTTPS.

Sources: board-port/esp32/main/CMakeLists.txt:18-42, board-port/esp32/AGENTS.md:20-25

Communication Flow

The following diagram illustrates the lifecycle of a Muse gadget from boot to a connected state.

The diagram shows the transition from local initialization to secure remote connectivity. Sources: board-port/esp32/main/app.c:1132-1250, board-port/esp32/main/noise_control.cpp:868-885

Provisioning and Pairing

Pairing uses the community pairing v5 protocol. The device remains in an advertising state until a user initiates setup via the Muse mobile app. Sources: board-port/esp32/AGENTS.md:267-275

Pairing Sequence

1. Advertising: The device broadcasts as MuseGadget-XXXXXX.
2. Confirmation: When a pairing request is received, the status LED breathes blue. The user must physically press the BOOT button (or tap the screen on supported boards) to confirm. Sources: board-port/esp32/main/app.c:1010-1035
3. Provisioning: The app sends Wi-Fi credentials (SSID/Password) and SDK tokens to the device over the encrypted BLE link. Sources: board-port/esp32/main/app.c:486-505
4. Validation: The device attempts to join the Wi-Fi network and verify the credentials. Sources: board-port/esp32/main/app.c:508-515

Noise Protocol Integration

The noise_control module manages the secure transport layer to the Muse VM. It uses the ClientSession from musegadgets::noise::core to handle encryption. Sources: board-port/esp32/main/noise_control.cpp:25-30

Session Management

• WebSocket Upgrade: Before the Noise handshake, the device performs an HTTP GET request to /v1/noise with an Upgrade: websocket header. Sources: board-port/esp32/main/noise_control.cpp:329-350
• Handshake: The device completes a three-message Noise handshake to establish session keys. Sources: board-port/esp32/main/noise_control.cpp:445-470
• Keepalives: The system sends WebSocket pings every 60 seconds to prevent middlebox timeout resets. Sources: board-port/esp32/main/noise_control.cpp:945-955

Multi-Stream Multiplexing

The Noise session multiplexes multiple logical streams over a single WebSocket connection:

• Stream 1: Control stream for JSON-based command and control (link.invoke, link.result).
• Stream 2: Data-plane tunnel for raw IP traffic (optional).
• Stream 3: One-shot identity request (GET /identity).
• Streams 16+: Custom application-level requests (Muse-specific).

Sources: board-port/esp32/main/noise_control.cpp:55-65, board-port/esp32/main/noise_control.cpp:665-680

Command and Control

Commands allow the Muse backend to trigger actions on the device. Firmware advertises these commands via link.register. Sources: board-port/esp32/AGENTS.md:374-380

Command Advertisement Example

Commands are registered in noise_control.cpp within build_register_json().

add_command(commands, "device.health",
            "Report basic Link health, including battery level...",
            nullptr, nullptr);

Sources: board-port/esp32/main/noise_control.cpp:715-720

Command Execution Flow

1. Invocation: The VM sends a link.invoke message.
2. Dispatch: on_ws_command in app.c routes the command to the appropriate handler.
3. Async Handling: Slow operations (e.g., network scans, camera captures) are spawned in separate tasks and return an {"_async": true} response immediately.
4. Result: Once finished, the task calls noise_ctrl_send_command_result to push the final data back to the VM.

Sources: board-port/esp32/main/app.c:755-800, board-port/esp32/main/app.c:865-880

Configuration and Board Overlays

The SDK supports various hardware configurations through board-specific sdkconfig overlays located in devices/. Sources: board-port/esp32/README.md:162-167

Important Configuration Keys

Key	Default	Description
CONFIG_GADGET_SDK_TOKEN	(None)	Critical: Required for pairing; generated at gadgets.muse.ai.
CONFIG_HOMEHUB_TUNNEL	y	Enables the L3 home-network tunnel (requires PSRAM).
CONFIG_HOMEHUB_OTA_ENABLED	n	Enables over-the-air updates (default y for UI boards).
CONFIG_HOMEHUB_WIFI_SSID	(Empty)	Developer override for skipping BLE provisioning.

Sources: board-port/esp32/README.md:120-125, board-port/esp32/AGENTS.md:104-108, board-port/esp32/main/app.c:1196-1205

Build and Flash Procedure

To build for a specific board, the tools/board.sh script is used to apply the correct overlays:

tools/board.sh sensecap-indicator build
tools/board.sh cyd flash PORT

Sources: board-port/esp32/README.md:150-155, board-port/esp32/AGENTS.md:120-125

Security Considerations

• NVS Encryption: The firmware recommends enabling NVS encryption to protect Wi-Fi credentials and device tokens stored in flash. Sources: board-port/esp32/README.md:175-180
• Signed Apps: Firmware is signed with a development key. While Secure Boot is not enabled by default for community devices, pairing relies on physical presence (button press) to mitigate remote attacks. Sources: board-port/esp32/README.md:185-189, board-port/esp32/AGENTS.md:273-275
• Token Refresh: Access tokens are refreshed proactively based on uptime (default 30 days) or reactively if the backend returns a 401 Unauthorized response. Sources: board-port/esp32/main/app.c:83-88, board-port/esp32/main/app.c:595-610

The Meta muse-gadget-sdk integration provides a robust framework for creating connected AI gadgets, balancing ease of development with secure communication standards. By leveraging board overlays and a modular command system, it allows for high extensibility across diverse ESP32 hardware.