Choreographed Speech
Relevant source files
The following files were used as context for generating this wiki page:
The Choreographed Speech feature allows the Reachy Mini robot to synchronize physical movements with spoken text. This is achieved through the speak MCP tool, which can parse a string containing both text to be vocalized and special markers that trigger predefined robot movements. This enables more expressive and engaging human-robot interactions by giving the AI a way to control the robot's body language as it speaks.
This system is a core part of the project's goal to provide "embodied consciousness expression." It leverages Text-To-Speech (TTS) services like Grok Voice or Deepgram Aura and integrates with the robot's movement capabilities, including both built-in emotions and a library of recorded moves.
Sources: README.md:3-6, src/server.py:740-743, src/__init__.py:1
Core Architecture
The choreographed speech functionality is primarily handled by the speak MCP tool. When called, it parses the input text for special move markers. The text is split into segments of speech and movement commands. The system then iterates through these segments, executing a move just before speaking the subsequent text chunk.
Sources: src/server.py:768-800
Choreography Parsing
The system uses a regular expression to parse the input string for move markers. The _parse_choreographed_text function is responsible for this logic.
The following diagram illustrates the parsing process:
This diagram shows how the _parse_choreographed_text function breaks down a string into a structured list of text and move segments.
Sources: src/server.py:681-700
Syntax
To embed a movement within speech, a specific marker syntax is used directly in the text string passed to the speak tool.
Syntax: [move:move_name]
Example:
speak("[move:curious1] What's this? [move:surprised1] Oh wow!")
In this example, the robot will first perform the curious1 move, then say "What's this?", then perform the surprised1 move, and finally say "Oh wow!". The move is designed to fire right before its corresponding speech chunk.
Sources: README.md:23-26, src/server.py:744-747
Execution Flow
The execution of a choreographed speech command involves a sequence of steps, from parsing the text to coordinating TTS audio playback with robot motor commands.
The sequence diagram below shows the detailed interaction within the speak tool when it receives a choreographed string.
This diagram illustrates the synchronous nature of the choreography: a move is triggered, the system waits for it to complete, and only then is the associated text chunk spoken.
Sources: src/server.py:774-791
Movement Execution
When a [move:name] marker is parsed, the _do_move function is called to execute the corresponding physical action. This function acts as a dispatcher, first checking if the move name corresponds to one of the built-in emotions. If not, it attempts to play it as a recorded move from the Pollen Robotics library.
This flow shows how the system prioritizes the fast, locally-defined built-in emotions over the more complex, daemon-managed recorded moves.
Sources: src/server.py:321-328
Synchronization
To ensure moves are visually linked to the correct words, the system waits for a move to finish before playing the audio for the subsequent text. This is handled by the _wait_for_moves_complete function, which polls the robot daemon's API until no moves are running.
- Function:
_wait_for_moves_complete(timeout, poll_interval) - Endpoint Polled:
GET {DAEMON_URL}/move/running - Success Condition: The endpoint returns an empty list
[].
This synchronization is critical for the timing of the performance.
Sources: src/server.py:787, src/server.py:953-972
Text-to-Speech (TTS) Providers
The speak tool relies on external TTS services to generate audio. The system supports two providers and selects one based on the presence of environment variables.
| Provider | Status | Primary Use Case | Required Environment Variable |
|---|---|---|---|
| Grok Voice | Supported | Preferred, expressive voice | XAI_API_KEY |
| Deepgram | Supported | Fallback TTS and STT | DEEPGRAM_API_KEY |
Sources: README.md:52-59, README.md:78-83
The selection logic is implemented in the text_to_speech function.
This diagram shows that Grok Voice is prioritized if its API key is available, otherwise the system falls back to using Deepgram.
Sources: src/server.py:440-450
Grok Voice Integration
When using Grok Voice, the system connects to the x.ai API endpoint using the openai SDK. It supports several voices which can be selected via the voice parameter in the speak tool or the GROK_VOICE environment variable.
- Supported Voices:
ara,eve,leo,rex,sal - Default Voice:
eve
Sources: src/server.py:438, src/server.py:488-493, src/server.py:737
Deepgram Integration
If XAI_API_KEY is not set, the system uses Deepgram's Aura-2 model for TTS. This requires the DEEPGRAM_API_KEY to be set. The function sends a POST request to the Deepgram API and saves the resulting MP3 audio to a temporary file.
Sources: src/server.py:453-477
Summary
The Choreographed Speech feature is a powerful component of the reachy-mini-mcp server, transforming the speak tool from a simple text-to-speech function into a performance director. By parsing a simple, embedded syntax, it synchronizes robot movements with vocalizations, allowing an AI to express itself with greater nuance and physicality. The architecture is designed to be flexible, supporting multiple TTS providers and leveraging the full range of the robot's movement capabilities.
Sources: README.md:3-6, src/server.py:740-747