Voice Capabilities
Relevant source files
The following files were used as context for generating this wiki page:
The Voice system in Cogmo provides high-quality speech-to-text (STT) for inbound messages and text-to-speech (TTS) for outbound replies. It is designed as a modular, provider-agnostic subsystem that allows operators to mix and match different services—such as using OpenAI for cost-effective transcription and ElevenLabs for natural voice character—independently. The primary interface for these capabilities is currently Telegram, which supports native OGG/Opus voice clips.
Sources: design/voice.md:3-12, changelog.d/2026-05-06-voice-slice-1.md:1-5
Architecture Overview
The voice subsystem is integrated into the agent loop at four critical points: capability detection at the adapter level, cost ceiling enforcement via channel settings, user preference resolution (profile vs. conversation overrides), and per-turn modality resolution. The architecture ensures that the agent remains modality-agnostic while receiving hints to optimize its response style when voice output is active.
Sources: design/voice.md:16-24, changelog.d/2026-05-06-voice-slice-1.md:11-14
Component Relationship
The diagram below illustrates how the Voice Resolver coordinates between the database configuration and the external provider implementations.
A brief explanation: The Orchestrator calls the Resolver at the start of every turn. The Resolver decrypts secrets and checks a content hash to determine if it should return a cached provider bundle or build a new one based on updated database configurations.
Sources: design/voice.md:58-69, src/voice/resolver.ts:18-45
Core Interfaces and Providers
Cogmo separates voice functionality into two distinct interfaces: TtsProvider and SttProvider. This split allows for hybrid configurations where different vendors handle different directions of the audio flow.
Sources: design/voice.md:46-52, changelog.d/2026-05-06-voice-slice-1.md:3-5
Provider Implementation Details
| Provider | Type | Implementation Notes |
|---|---|---|
OpenAIVoiceProvider | TTS + STT | Uses gpt-4o-mini-tts and gpt-4o-mini-transcribe. Emits OGG/Opus directly via response_format: "opus". |
ElevenLabsTtsProvider | TTS | Used for high-quality voice character. Uses fetch directly to minimize dependencies. Supports OGG via opus_48000_128. |
DeepgramSttProvider | STT | Deferred/Proposed for higher accuracy on noisy inputs. |
Sources: design/voice.md:53-57, src/voice/openai.ts:10-25, src/voice/elevenlabs.ts:12-30
Data Flow and Modality Resolution
Voice mode resolution occurs at the beginning of every turn. It evaluates the adapter's capabilities, presence of voice configuration, and the effective voice_mode (which can be auto, always, or never).
Sources: design/voice.md:104-118, changelog.d/2026-05-06-voice-slice-1.md:14-17
Turn Resolution Sequence
This sequence diagram shows how an inbound voice message is processed into text and how a corresponding voice reply is generated.
A brief explanation: Inbound voice is transcribed before the agent loop begins. If voice mode is active, the agent's text response is sent to the TTS provider, and the resulting audio is delivered to the user as a follow-up to the text message.
Sources: design/voice.md:131-144, changelog.d/2026-05-06-voice-slice-1.md:6-11
Configuration and Setup
Voice settings are managed via a singleton voice_config table that references the secrets table for credentials. This ensures that sensitive API keys are encrypted at rest using the master key.
Sources: design/voice.md:72-88, src/setup/wizard.ts:684-715
Database Schema: voice_config
| Field | Type | Description |
|---|---|---|
tts_provider | Enum | openai, openai_compatible, or elevenlabs. |
stt_provider | Enum | openai or openai_compatible. |
tts_model | Text | Model ID (e.g., gpt-4o-mini-tts). |
tts_voice | Text | Voice ID or ElevenLabs UUID. |
tts_base_url | Text | Custom endpoint for OpenAI-compatible providers. |
Sources: design/voice.md:76-88, src/setup/wizard.ts:646-670
Setup Wizard Integration
The cogmo setup wizard guides users through configuring voice providers. It performs a live 1-word "probe" to validate the TTS configuration before saving.
// Example of the TTS probe in src/setup/wizard.ts
const probe = buildTtsProbe(tts);
await probe.tts({ text: "hi", voice: tts.voice, model: tts.model, format: "ogg" });
Sources: src/setup/wizard.ts:803-815, design/voice.md:90-95
User Interaction and UI
Users control voice behavior through Telegram commands and profile settings.
- /voice auto: Mirror the inbound modality (voice in, voice out).
- /voice always: Always reply with voice.
- /voice off / never: Disable voice replies.
- /voice clear: Revert to profile defaults.
When voice mode is active, the system prompt is injected with a "Voice mode" hint. This instructs the LLM to keep responses short (1-2 sentences), avoid markdown/tables, and skip narration of background tasks to ensure a natural listening experience.
Sources: design/voice.md:155-163, design/voice.md:167-171, changelog.d/2026-05-06-voice-slice-1.md:18-24
Summary
Cogmo's Voice Capabilities provide a robust, asynchronous audio interface that respects user preferences and channel constraints. By leveraging a provider-agnostic architecture and a per-turn resolution strategy, the system maintains high flexibility while ensuring that expensive or high-latency audio operations are handled durably through orchestrated steps. The integration of OGG/Opus allows for seamless interaction within the Telegram ecosystem without requiring complex external transcoding.
Sources: design/voice.md:185-195, changelog.d/2026-05-06-voice-slice-1.md:25-30