Troubleshooting & Brick Recovery
Relevant source files
The following files were used as context for generating this wiki page:
Troubleshooting and brick recovery encompasses the procedures and internal logic used to diagnose connectivity issues, handle failed firmware updates, and restore ESP32 devices to a functional state. This system relies on serial log monitoring, automated Over-the-Air (OTA) rollbacks, and manual hardware overrides to ensure device recoverability.
The firmware monitors health through a "heartbeat" log and identifies critical failure states—such as authentication rejections or Wi-Fi timeouts—to trigger corrective actions. Recovery methods range from software-level rollbacks to low-level serial flashing for devices that fail to boot correctly.
Sources: board-port/esp32/README.md:16-18, board-port/esp32/main/app.c:1666-1700
Device Status Diagnostics
The system provides visual and log-based indicators to identify the device's current state and potential failures. The status LED or on-screen bars use specific colors and animations to signal the stage of operation or the nature of an error.
LED and UI Status Indicators
| Color | Pattern | Meaning |
|---|---|---|
| Orange | Breathing | Ready for setup; BLE advertising |
| Blue | Breathing | Physical button press required to confirm pairing |
| Blue | Solid | Joining Wi-Fi or connecting to the Muse VM |
| Green | Solid | Connected; tunnel or control session is active |
| Yellow | Blinking | Reconnecting; connection lost or VM switching |
| Purple | Solid | Device is not paired |
| Red | Blinking | Critical error occurred; check serial log |
Sources: board-port/esp32/README.md:118-128, board-port/esp32/AGENTS.md:436-447
Heartbeat Monitoring
The application maintains a heartbeat loop that logs the full system state every 5 seconds. This log includes uptime, setup stage, connection status for Wi-Fi/WebSocket/Tunnel, and memory availability. Analyzing these logs helps identify if the device is stuck in a specific phase (e.g., "wifi_connecting" or "auth_failed").
Sources: board-port/esp32/main/app.c:1865-1886
Firmware Recovery and OTA Rollback
The device implements an automated rollback mechanism to prevent "bricking" during OTA updates. This ensures that if a new firmware image cannot establish a valid connection to the Muse servers, the device reverts to the previous working version.
OTA Validation Logic
When a new image is installed, the bootloader marks it as PENDING_VERIFY. The device then has a 300-second window (OTA_VERIFY_TIMEOUT_US) to successfully connect to the control WebSocket. If the connection succeeds, the image is validated; otherwise, the device reboots and rolls back to the previous partition.
The diagram shows the decision flow for validating a new OTA image.
Sources: board-port/esp32/main/app.c:79-84, board-port/esp32/main/app.c:1666-1695
Manual Recovery Procedures
Manual recovery is required when the device becomes unresponsive or when credentials must be wiped to resolve persistent authentication errors.
Factory Reset (Hardware)
You can reset the device and forget all Wi-Fi and pairing settings by holding the BOOT button (or the primary device button) for 5 seconds. This triggers setup_wipe_to_clean, which erases the NVS (Non-Volatile Storage) credentials and reboots the device into the initial "Orange" setup state.
Sources: board-port/esp32/README.md:130, board-port/esp32/main/app.c:632-645
Serial Flash Recovery
If the firmware is corrupted and the device cannot boot into the OTA recovery path, you must flash the device via USB. If the chip does not enter the bootloader automatically, follow these steps:
- Hold the BOOT button.
- Tap the RESET button.
- Release the BOOT button.
- Run the flash command:
idf.py -p PORT flash.
Sources: board-port/esp32/README.md:88-89, board-port/esp32/AGENTS.md:329-332
Common Issues and Solutions
| Issue | Potential Cause | Solution |
|---|---|---|
| Red Blinking Light | Firmware Panic or Memory Failure | Connect serial monitor (idf.py monitor) to view the backtrace. |
| Stuck on "Blue" | Invalid Wi-Fi Credentials or SDK Token | Perform a 5-second button hold to reset and re-pair. |
| "Failed to write to target RAM" | USB Bridge Flow Control Issues | Use the paced flash tool for sensitive boards like SenseCAP Watcher. |
| 401 Unauthorized | Revoked SDK Token or Stale Session | Generate a new SDK token at gadgets.muse.ai and rebuild firmware. |
Sources: board-port/esp32/README.md:118-128, board-port/esp32/AGENTS.md:204-210, board-port/esp32/main/noise_control.cpp:520-525
Troubleshooting the SenseCAP Watcher
The SenseCAP Watcher utilizes a CH342 USB bridge that lacks flow control, often causing standard esptool writes to fail with checksum or format errors. This may leave the board in an unbootable state (though not physically bricked).
To recover this board, you must use the paced_esptool.py wrapper, which limits transmission to 64-byte chunks at 115200 baud.
# Example recovery command for SenseCAP Watcher
python ../tools/muse/paced_esptool.py --chip esp32s3 -p PORT -b 115200 \
--before default-reset --after hard-reset write-flash "@flash_args"
Sources: board-port/esp32/AGENTS.md:337-360
Troubleshooting and brick recovery ensure that even in cases of network isolation or firmware failure, the device remains accessible through hardware overrides or automated rollbacks. Maintaining serial access is the most reliable method for diagnosing low-level system panics and memory fragmentation issues.