Voice and Audio Processing

Relevant source files

The following files were used as context for generating this wiki page:

• apps/voice_agent/README.md
• apps/voice_agent/src/config.rs
• apps/voice_agent/src/main.rs
• apps/voice_agent/voice_agent.toml
• loom-audio/README.md
• loom-audio/src/stt.rs
• loom-audio/src/tts.rs
• core/src/cognitive/llm/tool_orchestrator.rs

The Voice and Audio Processing system in Loom provides an end-to-end, event-driven pipeline for building voice-enabled AI agents. It captures audio from a microphone, processes it through stages of voice activity detection (VAD) and speech-to-text (STT), detects a wake word, sends the user's query to a Large Language Model (LLM), and finally synthesizes the LLM's response into speech using text-to-speech (TTS).

This entire system is designed to be modular and configurable. Each stage operates independently and communicates through the core Loom EventBus. This architecture allows for robust, real-time interaction, where components can be enabled, disabled, or replaced without affecting the rest of the pipeline. The primary implementation of this system is the voice_agent application.

Sources: apps/voice_agent/README.md, apps/voice_agent/src/main.rs:1-15, loom-audio/README.md

Architecture and Event Flow

The voice processing pipeline is a linear sequence of components that transform raw audio into a spoken response. The entire process is orchestrated by the voice_agent application, which initializes and connects each module to the Loom EventBus.

Sources: apps/voice_agent/README.md, apps/voice_agent/src/main.rs:33-55

High-Level Pipeline

The following diagram illustrates the complete data flow from microphone input to TTS output.

Event-Driven Communication

Components communicate by publishing and subscribing to events on specific topics within the Loom EventBus. This decouples the components and allows for a flexible, resilient system.

Topic	Event Type	Payload/Metadata	Description
audio.mic	audio_chunk	Raw PCM16 audio frames	Captured audio from the microphone.
audio.voiced	audio_voiced	Raw PCM16 audio frames	Audio frames gated by VAD, containing only speech.
vad	vad.speech_start, vad.speech_end	sample_rate	Events indicating the start and end of a speech segment.
transcript	transcript.final	text, duration_ms	The final transcribed text from an utterance.
wake	wake_word_detected	session_id	Fired when a wake phrase is detected in a transcript.
query	user.query	text	The user's query, published after a wake word is detected.
tts	tts.start, tts.done, tts.error	engine, voice, text	Observability events for the TTS process.

Sources: apps/voice_agent/README.md, loom-audio/README.md, loom-audio/src/stt.rs:370-386, loom-audio/src/tts.rs:190-206

Components

The pipeline is composed of several key components, primarily from the loom-audio crate.

Sources: loom-audio/README.md, apps/voice_agent/src/main.rs:40-55

Microphone Capture (MicSource)

This component is responsible for capturing audio from a system microphone using the cpal library. It publishes raw audio chunks to the audio.mic topic.

Sources: apps/voice_agent/src/main.rs:40-41, loom-audio/README.md

Voice Activity Detection (VadGate)

The VadGate listens for audio_chunk events and uses WebRTC VAD to detect speech. It performs two actions:

1. Publishes vad.speech_start and vad.speech_end events to the vad topic.
2. Forwards audio frames containing speech as audio_voiced events to the audio.voiced topic.

Sources: apps/voice_agent/src/main.rs:44-45, loom-audio/README.md

Speech-to-Text (SttEngine)

The SttEngine consumes VAD events and voiced audio to reconstruct a user's utterance. Once vad.speech_end is received, it uses an external whisper.cpp process to transcribe the buffered audio. The final transcription is published as a transcript.final event.

Sources: apps/voice_agent/src/main.rs:48-49, loom-audio/src/stt.rs:101-104

The following diagram shows the internal logic for processing a single utterance.

Wake Word Detection (WakeWordDetector)

This component listens for transcript.final events and performs lightweight, transcript-based wake word detection. It uses fuzzy matching to check for phrases like "hey loom". Upon detection, it publishes a wake_word_detected event and arms the system to treat the next utterance as a query, which is then published to the query topic.

Sources: apps/voice_agent/src/main.rs:52-53, loom-audio/README.md

Text-to-Speech (TtsSpeakProvider)

The TtsSpeakProvider is a Tool that exposes a tts.speak capability. It synthesizes text into speech using a local engine. It gracefully degrades by preferring Piper for high-quality synthesis and falling back to the more widely available espeak-ng. If no engine is found, it logs the text.

Sources: apps/voice_agent/src/main.rs:58-62, loom-audio/src/tts.rs:1-15

The engine selection logic is as follows:

Voice Agent Application

The voice_agent binary is the main application that orchestrates the entire pipeline. Its main function in main.rs is responsible for:

1. Initializing the Loom runtime and EventBus.
2. Loading configuration from voice_agent.toml and environment variables.
3. Instantiating and starting each audio component (MicSource, VadGate, SttEngine, WakeWordDetector).
4. Registering the TtsSpeakProvider as a tool.
5. Subscribing to the query topic to process user queries.

Sources: apps/voice_agent/src/main.rs:24-62

Query Handling Flow

When a user.query event is received, the voice_agent's main loop triggers a sequence to get a response from the LLM and speak it.

Configuration

The voice_agent is highly configurable through a combination of a TOML file and environment variables. The configuration is managed by the VoiceAgentConfig struct. The system loads settings in the following order of precedence:

1. voice_agent.toml file.
2. Environment variables.
3. Default values coded in the application.

Sources: apps/voice_agent/src/config.rs:100-126, apps/voice_agent/README.md

An example voice_agent.toml demonstrates how to configure all components:

query_topic = "query"

[llm]
base_url = "http://localhost:8000/v1"
model = "Qwen/Qwen2.5-0.5B-Instruct"
temperature = 0.6
system_prompt = "You are Loom's helpful and concise voice assistant."

[tts]
piper_bin = "/opt/piper/piper"
voice = "./models/piper/en_US-amy-medium/en_US-amy-medium.onnx"
rate = 1.0

[mic]
chunk_ms = 20
sample_rate_hz = 16000

[vad]
mode = 2
hangover_ms = 200

[stt]
whisper_bin = "loom-audio/whisper.cpp/build/bin/whisper-cli"
whisper_model = "loom-audio/whisper.cpp/models/ggml-base.en.bin"
language = "en"

[wake]
phrases = ["hey loom", "loom"]
max_distance = 1

Sources: apps/voice_agent/voice_agent.toml, apps/voice_agent/src/config.rs:170-375

Component Configuration Tables

The following tables summarize the key configuration options for each module, which can be set via voice_agent.toml or corresponding environment variables.

STT (stt)

TOML Key	Environment Variable	Description	Default
whisper_bin	WHISPER_BIN	Path to the whisper.cpp executable.	whisper
whisper_model	WHISPER_MODEL_PATH	Path to the whisper model file.	ggml-base.en.bin
language	WHISPER_LANG	Language for transcription (e.g., "en", "auto").	en
extra_args	WHISPER_EXTRA_ARGS	Comma-separated extra arguments for whisper.	[]
temp_dir	STT_TEMP_DIR	Directory for temporary WAV files.	System temp dir
Sources: apps/voice_agent/src/config.rs:271-297, loom-audio/src/stt.rs:29-65

TTS (tts)

TOML Key	Environment Variable	Description	Default
piper_bin	PIPER_BIN	Path to the Piper executable.	Auto-detected from PATH
voice	PIPER_VOICE	Path to the Piper voice model (.onnx).	None
rate	TTS_RATE	Speech rate multiplier.	1.0
volume	TTS_VOLUME	Speech volume multiplier.	1.0
player	TTS_PLAYER	Preferred audio player (e.g., aplay).	Auto-detected
Sources: apps/voice_agent/src/config.rs:356-375, loom-audio/src/tts.rs:40-75

LLM (llm)

TOML Key	Environment Variable	Description	Default
base_url	VLLM_BASE_URL	Base URL of the OpenAI-compatible API.	http://localhost:8000/v1
model	VLLM_MODEL	The model name to use.	qwen2.5-0.5b-instruct
api_key	VLLM_API_KEY	API key for the LLM service.	None
temperature	VLLM_TEMPERATURE	LLM sampling temperature.	0.7
system_prompt	VOICE_SYSTEM_PROMPT	System prompt for the voice assistant.	"You are Loom's helpful..."
Sources: apps/voice_agent/src/config.rs:331-354, apps/voice_agent/src/config.rs:43-63

Conclusion

The voice and audio processing system is a cornerstone of Loom's capability to create interactive, human-like agents. Its modular, event-driven design provides a robust and extensible foundation for voice interaction. By leveraging external tools for STT and TTS, it achieves high-quality processing while remaining simple to configure and run. The voice_agent demo serves as a powerful example of how these components are integrated to create a fully functional voice assistant.

Sources: apps/voice_agent/README.md, ROADMAP.md