Connecting with AI Models
Relevant source files
The following files were used as context for generating this wiki page:This project, reachy-mini-mcp, serves as a bridge to give AI systems a physical body using the Pollen Robotics Reachy Mini robot. It implements a server compliant with the Model Context Protocol (MCP), allowing AI models such as Claude, GPT, or Grok to perceive and act in the physical world. The connection is facilitated through a set of seven token-efficient tools that enable the AI to speak, listen, see, and express emotions through physical movement.
The core of the system is the MCP server, which exposes the robot's capabilities as simple functions (tools) that the AI can call programmatically. This abstracts away the complexities of robotics, allowing the AI to focus on high-level tasks. The project is licensed under the MIT License.
Sources: README.md, src/init.py:1, LICENSE
System Architecture
The architecture defines a clear, linear flow of information from the AI model to the robot hardware. The MCP server is the central component that translates the AI's tool calls into commands for the Reachy Mini SDK.
Sources: README.md:31-34
This diagram shows the command flow from the AI model to the robot.
Sources: README.md:31-34, src/server.py:4-6
MCP Server Implementation
The server is built using the fastmcp library. It is initialized with a specific name and a set of instructions for the AI model, guiding it on how to use the available tools effectively.
Sources: src/server.py:24-44, pyproject.toml:12
Initialization
The server identifies itself as reachy-mini and provides a detailed prompt to the AI, explaining the purpose of each tool and suggesting when to use them.
# src/server.py:24-44
mcp = FastMCP(
name="reachy-mini",
instructions="""
Reachy Mini robot control for expressive robotics.
Use these tools for robot control:
- show() for 12 built-in emotions (curious, joy, thinking, etc.)
- show(move=...) for 81 recorded emotions from Pollen (fear1, rage1, serenity1, etc.)
- discover() to see available recorded moves
- look() for precise head positioning
- speak() to vocalize with [move:X] markers for choreography
- listen() to hear and transcribe speech
- snap() to capture camera images
- rest() for neutral pose, sleep, or wake
Prefer show() for common emotions, show(move=...) for nuanced expressions.
"""
)
Sources: src/server.py:24-44
Running the Server
The server is started by running the src/server.py script, which calls the mcp.run() method. A cleanup function is registered to ensure the robot connection is properly closed on shutdown.
# src/server.py:1003-1009
def main():
"""Run the MCP server."""
import atexit
atexit.register(cleanup_robot)
mcp.run()
Sources: src/server.py:1003-1009
AI Toolset
The AI interacts with the robot through a set of seven tools, designed to be simple and fit within an AI's working memory, following Miller's Law.
| Tool | Args | Purpose |
|---|---|---|
speak | text, listen_after=0 | Voice + gesture, optionally listen after |
listen | duration=3 | STT via Deepgram Nova-2 |
snap | - | Camera capture (base64 JPEG) |
show | emotion, move="" | Express emotion or play recorded move |
look | roll, pitch, yaw, z, duration | Head positioning (degrees) |
rest | mode="neutral" | neutral / sleep / wake |
discover | library="emotions" | Find available recorded moves |
Sources: README.md:13-23, src/server.py:10-17
Voice and Speech: speak and listen
The speak and listen tools are the primary means of verbal communication. The system uses external voice providers for Text-to-Speech (TTS) and Speech-to-Text (STT).
Voice Providers
The system supports two TTS providers, with Grok Voice being the preferred option if configured.
| Provider | Status | Use Case |
|---|---|---|
| Grok Voice | ✅ Supported | xAI's expressive voice (Eve, Ara, Leo, Rex, Sal) |
| Deepgram | ✅ Supported | TTS (Aura 2) + STT (Nova 2) |
Sources: README.md:36-42
The selection logic is implemented in the text_to_speech function, which checks for the XAI_API_KEY environment variable.
This diagram illustrates the TTS provider selection flow.
Sources: src/server.py:421-426
Choreographed Speech
The speak tool supports embedded move markers ([move:curious1]) to choreograph speech with physical actions. The _parse_choreographed_text function parses these markers, and the speak function executes them in sequence.
This sequence diagram shows how a choreographed speech command is executed.
Sources: src/server.py:734-747, src/server.py:794-835
Movement and Expression: show, look, rest
These tools control the robot's physical presence.
show(emotion, move): The primary tool for expression. It can trigger one of 12 built-in, locally defined emotions (e.g.,curious,joy) or play one of 81 pre-recorded, more nuanced moves from Pollen's libraries (e.g.,fear1,loving1). The built-in emotions are defined in theEXPRESSIONSdictionary.look(...): Provides direct, fine-grained control over the head's orientation (roll, pitch, yaw) and height (z).rest(mode): Manages the robot's power state, allowing it to go into a low-powersleepmode,wakeup, or return to aneutralpose.
Sources: src/server.py:59-160, src/server.py:306-340, src/server.py:343-382, src/server.py:943-962
Perception and Discovery: snap and discover
snap(): Captures an image from the robot's camera and returns it as a base64-encoded JPEG string. This is the AI's primary visual input.discover(library): Queries the robot's daemon to find available moves in Pollen's HuggingFace libraries (emotionsordances). This allows the AI to learn what moves it can perform.
Sources: src/server.py:912-939, src/server.py:981-1000
Configuration for AI Clients
To connect an AI client like Claude, the client's configuration file must be updated to specify how to launch the reachy-mini-mcp server. This involves defining the command, arguments, and necessary environment variables.
Environment Variables
The server's behavior is configured through several environment variables.
| Variable | Required | Default | Purpose |
|---|---|---|---|
XAI_API_KEY | No | - | Grok Voice TTS (preferred) |
GROK_VOICE | No | Eve | Grok voice: Ara, Eve, Leo, Rex, Sal, Mika, Valentin |
DEEPGRAM_API_KEY | Yes* | - | TTS (fallback) + STT |
REACHY_DAEMON_URL | No | http://localhost:8321/api | Daemon API endpoint |
*Required if XAI_API_KEY is not set. |
Sources: README.md:83-91
Example: Claude Desktop Configuration
This JSON snippet shows how to configure the Claude Desktop application to use the reachy-mini-mcp server. It uses poetry to run the server script and passes the DEEPGRAM_API_KEY as an environment variable.
{
"mcpServers": {
"reachy-mini": {
"command": "poetry",
"args": ["-C", "/path/to/reachy-mini-mcp", "run", "python", "src/server.py"],
"env": {
"DEEPGRAM_API_KEY": "your_key_here"
}
}
}
}
Sources: README.md:47-59
Summary
The reachy-mini-mcp project provides a robust and straightforward way to connect powerful AI models to a physical robot. By leveraging the Model Context Protocol and a well-defined set of tools, it abstracts robotic complexities and enables AIs to interact with the world through speech, sight, and movement. The architecture is flexible, supporting multiple voice providers and offering different layers of control, from high-level emotional expressions to precise head movements.