Tutorial: Building a Voice Agent
Relevant source files
The following files were used as context for generating this wiki page:The Voice Agent is an end-to-end, event-driven voice assistant application built on the Loom core runtime. It demonstrates a complete voice interaction loop, from capturing microphone audio to generating and speaking a response from a Large Language Model (LLM). The agent is designed to be modular, with distinct components for each stage of the audio processing and cognitive pipeline.
This tutorial covers the architecture, configuration, and data flow of the Voice Agent. The system is fully event-driven, meaning each component operates independently and communicates through the Loom event bus, ensuring resilience and decoupling.
Sources: apps/voice_agent/README.md:3-5, apps/voice_agent/README.md:202-203
Architecture and Data Flow
The Voice Agent is composed of a series of modules that process audio and text in a sequential pipeline. Each stage is an independent component that subscribes to events from the previous stage and publishes new events for the next.
Sources: apps/voice_agent/README.md:3-5, apps/voice_agent/src/main.rs:38-56
High-Level Pipeline
The following diagram illustrates the primary data flow through the voice agent's components.
This diagram shows the sequential processing pipeline from microphone input to spoken audio output.
Sources: apps/voice_agent/README.md:3-5, loom-audio/README.md:104-110
Event-Driven Interaction
Components communicate asynchronously via the Loom EventBus. This decouples the components and allows them to be managed independently.
The sequence diagram below details the events exchanged between components for a single user query.
This diagram shows the event-based communication between the voice agent's modules via the event bus and tool registry.
Sources: apps/voice_agent/src/main.rs:60-143, apps/voice_agent/README.md:183-191
Core Components
| Component | Crate/Module | Responsibility | Publishes Events |
|---|---|---|---|
| MicSource | loom-audio | Captures audio from the microphone. | audio_chunk |
| VadGate | loom-audio | Detects speech in the audio stream. | vad.speech_start, vad.speech_end, audio_voiced |
| SttEngine | loom-audio | Transcribes voiced audio segments to text using whisper.cpp. | transcript.final |
| WakeWordDetector | loom-audio | Detects wake words/phrases in transcripts. | wake_word_detected, user.query |
| Query Processor | voice_agent | Subscribes to user.query, calls the LLM, and invokes TTS. | (none, invokes tools) |
| TtsSpeakProvider | loom-audio | Synthesizes text to speech and plays it. | tts.start, tts.done, tts.error |
Sources: apps/voice_agent/src/main.rs:38-63, loom-audio/README.md:5-59
Configuration
The Voice Agent is configured through a voice_agent.toml file, environment variables, or a combination of both. The configuration is loaded in a layered approach, allowing for flexible setup.
Sources: apps/voice_agent/src/config.rs:100-128
Configuration Loading Hierarchy
This diagram shows that settings from a TOML file will overlay the default values, which are themselves derived from environment variables or hardcoded fallbacks.
Sources: apps/voice_agent/src/config.rs:100-128
The main configuration is managed by the VoiceAgentConfig struct, which aggregates settings for all sub-modules.
Sources: apps/voice_agent/src/config.rs:6-15
Configuration Sections
LLM Configuration ([llm])
Defines the connection to an OpenAI-compatible LLM endpoint, such as vLLM.
| Parameter | TOML Key | Env Var | Type | Default | Description |
|---|---|---|---|---|---|
| Base URL | base_url | VLLM_BASE_URL | string | http://localhost:8000/v1 | The base URL of the LLM API. |
| Model | model | VLLM_MODEL | string | qwen2.5-0.5b-instruct | The model name to use. |
| API Key | api_key | VLLM_API_KEY | string | (none) | Optional API key for authentication. |
| Timeout | request_timeout_ms | REQUEST_TIMEOUT_MS | u64 | 30000 | Request timeout in milliseconds. |
| Temperature | temperature | VLLM_TEMPERATURE | f32 | 0.7 | LLM generation temperature. |
| System Prompt | system_prompt | VOICE_SYSTEM_PROMPT | string | "You are Loom's..." | The system prompt for the assistant. |
Sources: apps/voice_agent/src/config.rs:18-30, apps/voice_agent/src/config.rs:40-58, apps/voice_agent/README.md:130-137
Text-to-Speech Configuration ([tts])
Configures the TTS engine, with a preference for Piper and a fallback to espeak-ng.
| Parameter | TOML Key | Env Var | Type | Default | Description |
|---|---|---|---|---|---|
| Piper Binary | piper_bin | PIPER_BIN | string | (path search) | Path to the Piper executable. |
| Voice Model | voice | PIPER_VOICE | string | (none) | Path to the .onnx voice model file. |
| Rate | rate | TTS_RATE | f32 | 1.0 | Speech rate multiplier. |
| Volume | volume | TTS_VOLUME | f32 | 1.0 | Playback volume multiplier. |
| Sample Rate | sample_rate | TTS_SAMPLE_RATE | u32 | 16000 | Output audio sample rate. |
| Player | player | TTS_PLAYER | string | aplay | Audio player to use (e.g., aplay, paplay). |
Sources: apps/voice_agent/src/config.rs:33-41, apps/voice_agent/README.md:141-147, loom-audio/src/tts.rs:18-27
Microphone Configuration ([mic])
Settings for audio capture.
| Parameter | TOML Key | Env Var | Type | Default | Description |
|---|---|---|---|---|---|
| Device Name | device_name | MIC_DEVICE | string | (none) | Substring to match for device selection. |
| Chunk Size | chunk_ms | MIC_CHUNK_MS | u32 | 20 | Audio chunk size in milliseconds. |
| Sample Rate | sample_rate_hz | (none) | u32 | 16000 | Desired sample rate in Hz. |
| Channels | channels | (none) | u16 | 1 | Number of audio channels. |
Sources: apps/voice_agent/README.md:151-155, apps/voice_agent/README.md:98-101
VAD Configuration ([vad])
Settings for Voice Activity Detection.
| Parameter | TOML Key | Env Var | Type | Default | Description |
|---|---|---|---|---|---|
| Mode | mode | VAD_MODE | integer | 2 | Aggressiveness from 0 (least) to 3 (most). |
| Frame Size | frame_ms | VAD_FRAME_MS | u32 | 20 | Frame size in ms (10, 20, or 30). |
| Min Start | min_start_ms | VAD_MIN_START_MS | u32 | 60 | Min voiced duration to trigger speech_start. |
| Hangover | hangover_ms | VAD_HANGOVER_MS | u32 | 200 | Silence duration before speech_end. |
Sources: apps/voice_agent/README.md:157-161, apps/voice_agent/README.md:103-106
STT Configuration ([stt])
Settings for Speech-to-Text using whisper.cpp.
| Parameter | TOML Key | Env Var | Type | Default | Description |
|---|---|---|---|---|---|
| Whisper Binary | whisper_bin | WHISPER_BIN | string | whisper | Path to the whisper.cpp executable. |
| Whisper Model | whisper_model | WHISPER_MODEL_PATH | string | ggml-base.en.bin | Path to the GGML model file. |
| Language | language | WHISPER_LANG | string | en | Language for transcription (e.g., en, auto). |
| Extra Args | extra_args | WHISPER_EXTRA_ARGS | array | [] | Extra command-line arguments for whisper. |
Sources: apps/voice_agent/README.md:163-168, apps/voice_agent/README.md:108-112
Wake Word Configuration ([wake])
Settings for the transcript-based wake word detector.
| Parameter | TOML Key | Env Var | Type | Default | Description |
|---|---|---|---|---|---|
| Phrases | phrases | WAKE_PHRASES | array | ["hey loom", "loom"] | List of phrases to detect. |
| Max Distance | max_distance | WAKE_FUZZY_DISTANCE | integer | 1 | Max Levenshtein distance for fuzzy matching. |
| Match Anywhere | match_anywhere | WAKE_MATCH_ANYWHERE | boolean | true | Allow matching anywhere in the transcript. |
| JW Threshold | jaro_winkler_threshold | WAKE_JW_THRESHOLD | float | 0.9 | Jaro-Winkler similarity gate (0.0-1.0). |
Sources: apps/voice_agent/README.md:170-175, apps/voice_agent/README.md:114-118
Extending with Tool Use
The voice agent can be extended to use external tools like web search or weather APIs through the ToolOrchestrator. This component intercepts the user query, discovers available tools, and allows the LLM to decide whether to call a tool to answer the query.
Sources: apps/voice_agent/TOOL_USE_GUIDE.md:7-17
Tool Use Flow
This diagram shows how the ToolOrchestrator enables the LLM to use external tools to fulfill a user's request.
Sources: apps/voice_agent/TOOL_USE_GUIDE.md:9-17
To enable this, you would register CapabilityProvider implementations (e.g., WebSearchProvider, WeatherProvider) with the ToolRegistry and use the ToolOrchestrator to process the query.
Sources: apps/voice_agent/TOOL_USE_GUIDE.md:22-26, core/src/cognitive/llm/tool_orchestrator.rs:118-127
Setup and Execution
Prerequisites
- Rust toolchain
whisper.cppbinary and model- An LLM service (like vLLM)
- A TTS engine (Piper or espeak-ng)
Sources: apps/voice_agent/README.md:15-32
Model Setup
A helper script is provided to download the necessary STT and TTS models.
bash apps/voice_agent/scripts/setup_models.sh
This script downloads a whisper.cpp model and a Piper voice model to default locations.
Sources: apps/voice_agent/scripts/setup_models.sh, apps/voice_agent/README.md:76-85
Running the Agent
After configuring voice_agent.toml and setting up the prerequisites, the agent can be run from the repository root:
cargo run -p voice_agent
The agent will start, initialize all components, and wait for a wake word like "hey loom" to be spoken.
Sources: apps/voice_agent/README.md:40-42, apps/voice_agent/README.md:149-157
Summary
The Voice Agent is a powerful demonstration of the Loom runtime's capabilities, showcasing how to build a complex, multi-stage, event-driven AI application. Its modular architecture allows for easy extension and customization, while the layered configuration system provides flexibility for different deployment environments. By leveraging the loom-audio components and the core ToolRegistry, it provides a solid foundation for building sophisticated voice-enabled assistants.
Sources: apps/voice_agent/README.md