Voice & Speech Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Voice & Speech Pipeline in Clawdbot is a multi-layered system responsible for audio input processing (Speech-to-Text), wake-word detection, and expressive audio synthesis (Text-to-Speech). It integrates local on-device processing via macOS Speech.framework with cloud-based services like ElevenLabs and OpenAI to provide a low-latency, conversational interface.
The pipeline supports various interaction modes, including proactive telephony calls, manual audio generation via CLI tools, and automatic speech generation for messaging platforms like Telegram.
System Architecture
The pipeline is divided into an ingestion layer for processing microphone or telephony streams and a synthesis layer for generating assistant responses.
The diagram above illustrates the end-to-end flow from audio capture to speech synthesis.
Sources: Swabble/README.md, src/tts/tts.ts:740-850
Synthesis Providers & Configuration
Clawdbot utilizes several Text-to-Speech (TTS) providers with specific feature sets and latency profiles. These are managed via messages.tts configuration.
Supported Providers
- OpenAI: Provides high-quality voices like
alloy,ash, andnova. It is often used for telephony due to low latency. - ElevenLabs: Offers highly expressive voices and supports specialized audio tags for emotions.
- Edge TTS: A local-friendly option that supports a wide range of languages and neural voices.
- sag: A CLI tool specifically for ElevenLabs that supports local playback and "mac-style say" UX.
Sources: src/tts/tts.ts:31-45, skills/sag/SKILL.md:1-15, extensions/voice-call/CHANGELOG.md:4-8
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
auto | TtsAutoMode | off | Determines when TTS triggers (always, inbound, tagged, off). |
maxTextLength | number | 4000 | The hard limit for text input before truncation or failure. |
maxLength | number | 1500 | Threshold for triggering auto-summarization. |
summarize | boolean | true | Whether to use an LLM to shorten long text before synthesis. |
mode | TtsMode | final | Controls if TTS happens on every tool turn or just the final reply. |
Sources: src/tts/tts.ts:25-30, src/tts/tts.ts:88-144
Speech-to-Text & Wake Word Detection
For local interactions, Clawdbot utilizes the Swabble daemon. This component targets macOS 26 and uses Speech.framework for on-device, zero-network transcription.
Swabble Pipeline
- Audio Capture: Uses
AVAudioEngineto tap the microphone. - Buffer Conversion: The
BufferConvertertransforms raw audio intoAnalyzerInput. - Transcription:
SpeechAnalyzerandSpeechTranscribergenerate text segments. - Wake Gating: The system looks for the wake word (default:
clawdorclaude). If detected, the subsequent transcript is passed to a configured hook.
Sources: Swabble/README.md, appcast.xml:102-104
Expressive Audio & Directives
The pipeline supports "Model-driven TTS tags," allowing agents to insert cues into text that influence the synthesized output. This is particularly prominent when using ElevenLabs via the sag tool or the core TTS engine.
TTS Directive Parsing
The system parses text for [[tts:...]] directives to override settings per-message. Supported overrides include:
provider: Change the synthesis engine (e.g.,[[tts: provider=elevenlabs]]).voice: Select a specific voice ID or name.stability / speed / style: Fine-tune ElevenLabs voice settings.
Sources: src/tts/tts.ts:446-590, appcast.xml:10-12
Audio Tags (ElevenLabs v3)
When using expressive models, specific bracketed tags can be used at the start of lines to modify tone:
[whispers],[shouts],[sings][laughs],[sighs],[excited]
Sources: skills/sag/SKILL.md:25-33
Telephony & Voice Calls
The @clawdbot/voice-call extension leverages the pipeline for real-time telephony. It deep-merges plugin configuration with the core messages.tts settings.
Telephony requires specific output formats (PCM) and sample rates (e.g., 22050Hz for ElevenLabs, 24000Hz for OpenAI) to ensure compatibility with media streams.
Sources: src/tts/tts.ts:55-60, src/tts/tts.ts:948-1020, extensions/voice-call/CHANGELOG.md:44-55
Summarization Logic
To manage token costs and audio duration, the pipeline includes a summarization step for long responses.
- Detection: If the response text exceeds
maxLength(default 1500 chars), the summarizer is triggered. - LLM Invocation: The system uses a
summaryModel(resolved via agent config) to create a concise version of the text. - Tone Preservation: The summarizer is instructed to maintain the original tone and style while hitting the target character count.
- Synthesis: The summarized text is then sent to the chosen TTS provider.
Sources: src/tts/tts.ts:646-708, src/tts/tts.ts:1078-1110
Conclusion
The Voice & Speech Pipeline provides a flexible architecture that balances local privacy (via Swabble) with high-fidelity synthesis (via ElevenLabs/OpenAI). By supporting both real-time telephony and asynchronous messaging (Telegram voice notes), it enables Clawdbot to function as a multimodal assistant capable of expressive and context-aware vocal interaction.