UI & Asset Generation Tools
Relevant source files
The following files were used as context for generating this wiki page:
The UI & Asset Generation Tools comprise a suite of Python scripts and C utilities used to create, preview, and deploy visual elements for Muse Gadget devices. These tools automate the conversion of standard images into hardware-optimized formats, facilitate the generation of procedural avatars using LLMs, and provide desktop environments for testing UI layouts without physical hardware.
The system ensures that high-level assets are transformed into low-level data structures, such as RGB565 pixel arrays or C-based procedural rendering code, which are compatible with the limited memory and processing power of ESP32-based boards.
Avatar Generation and Management
The avatar system uses a procedural 64x64 pixel-art renderer to display character states such as idling, listening, and speaking. Developers can generate custom avatars by interacting with the Muse LLM through the avatar.py tool, which produces C source code for the firmware.
AI-Driven Avatar Workflow
The avatar.py script manages the lifecycle of custom avatar creation by communicating with a connected Muse device over a serial port. It sends a prompt and the current renderer code to the LLM, extracts the returned C code, validates it through a host-based build check, and finally flashes the updated firmware to the device.
Avatar Validation and Preview
Before deployment, tools validate the procedural code to ensure it meets hardware performance limits (e.g., under 10ms per frame on ESP32-S3).
- Host Sanitizer: Compiles the generated C code with AddressSanitizer and UndefinedBehaviorSanitizer to detect memory corruption.
- GIF Preview: Generates animated previews of all states (boot, idle, thinking, etc.) using
make_gifs.pyto allow visual verification before flashing. - C Previewer:
preview.crenders avatar states into PPM image files for static inspection.
Sources: esp32/tools/muse/avatar.py:144-160, esp32/tools/muse/preview.c:21-25, esp32/tools/muse/AVATAR_RECIPE.md:92-98
Image and Animation Processing
The SDK includes specialized tools to convert standard image assets into the RGB565 format required by the ST7789 and AMOLED displays used in the project.
Image Formatting for Displays
The image_for_display.py tool prepares images for device screens. It performs resizing and color conversion, ensuring the output is optimized for the target resolution. Images are typically sent to the device as binary blobs where pixels are represented in RGB565 format, high-byte first.
Procedural Animation Generation
gen_happy_anim.py generates frame data for specific "happy" animations. It calculates coordinates and sizes for visual elements like hearts and sparkles that overlay the avatar.
| Component | Description | Output Format |
|---|---|---|
| Hearts | Floating heart icons with varying X/Y offsets and scales | C Header Array |
| Sparkles | Orbiting points of light around the character | C Header Array |
| Aura | Mode-specific dithered background glow | Procedural C Code |
Sources: esp32/tools/gen_happy_anim.py:18-40, esp32/tools/muse/preview.c:32-45
UI Components and Layout
The Muse UI is built using the LVGL (Light and Versatile Graphics Library). The implementation focuses on creating a responsive settings menu and state-driven face interface that fits within both round and rectangular panels.
UI Architecture
The UI is organized into tiles. The primary tile displays the avatar face, while swiping left reveals the settings menu.
Settings Interface Elements
The settings UI implements several custom widgets and interaction patterns defined in muse_settings_ui.c:
- Dynamic Page Loading: To save RAM on devices without PSRAM, sub-pages are built when opened and deleted when the user navigates back.
- Compact Mode: UI elements scale automatically for small rectangular screens (under 240px height) or narrow panels.
- Input Methods: Supports both a full keyboard (for screens ≥ 2.0 inches) and a custom numeric/alphanumeric keypad for smaller round displays.
| Widget Type | Function | Source Reference |
|---|---|---|
switch_row | Toggles features like Wi-Fi or Bluetooth | muse_settings_ui.c:248 |
slider | Adjusts volume, brightness, and mic gain | muse_settings_ui.c:275 |
info_row | Displays read-only data like MAC addresses | muse_settings_ui.c:317 |
muse_keypad | Custom input for round screens | muse_settings_ui.c:486 |
Sources: esp32/components/muse/muse_settings_ui.c:51-56, 128-150, 480-495
Asset Deployment Tools
Tools coordinate the transfer of generated assets to the device hardware.
Firmware Build Integration
The CMakeLists.txt for the Muse component dynamically includes custom avatars if they exist in the components/muse/avatar/ directory. If no custom avatar is found, it defaults to the standard muse_pixel.c asset.
Serial Communication
chat.py and avatar.py use serial commands to interact with the device's NVS (Non-Volatile Storage) and memory.
>status: Requests the current board state, including battery levels and connection status.>chat=: Sends message fragments to the device to be forwarded to the Muse LLM.@chat: Device response format for streaming data back to the host tool.
Sources: esp32/components/muse/CMakeLists.txt:49-57, esp32/tools/muse/avatar.py:80-95, esp32/tools/muse/AVATAR_RECIPE.md:113-125
The UI & Asset Generation Tools provide a bridge between high-level design and hardware implementation, enabling rapid iteration of device personalities and interface layouts while maintaining the strict performance requirements of the ESP32 platform.