Voice Input (Experimental)
Relevant source files
The following files were used as context for generating this wiki page:The Voice Input feature adds offline speech-to-text capabilities to caro, allowing users to speak their natural language command descriptions instead of typing them. This functionality aligns with the project's local-first, privacy-focused approach by ensuring all transcription happens on-device with no network calls. The feature is designed to improve accessibility and user experience.
This is complemented by a planned, but distinct, Caro Voice Synthesis feature, which aims to create a unique human-like voice for the caro assistant for text-to-speech output.
Sources: speech-to-text-integration-plan.md:3-6, caro-voice/RESEARCH.md:3-9, presentation/roadmap-slides.md:115-129
Speech-to-Text (Input)
This section details the system for converting user speech into text for command generation.
User Workflow
The new workflow is initiated with the --voice flag. The tool listens for audio input, transcribes it locally, and then proceeds with the standard command generation and confirmation flow.
# New workflow with speech input
$ cmdai --voice
🎤 Listening... (press Ctrl+C to stop)
[User speaks: "find all PDF files larger than 10MB"]
✓ Transcribed: "find all PDF files larger than 10MB"
Generated command:
find . -name "*.pdf" -size +10M
Execute this command? (y/N)
Sources: speech-to-text-integration-plan.md:8-19
The following diagram illustrates the sequence of events when a user invokes the --voice command.
Architecture
The recommended architecture for speech-to-text is whisper.cpp with whisper-rs Rust bindings, chosen for its maturity, performance, and Metal support on Apple Silicon. The system is designed with a modular, trait-based approach to separate concerns.
Sources: speech-to-text-integration-plan.md:23-40, speech-to-text-integration-plan.md:64-74
Component Structure
The architecture introduces two new modules: audio for input capture and transcription for speech-to-text processing.
This diagram shows the new audio and transcription modules within the caro source tree.
Sources: speech-to-text-integration-plan.md:77-87
Trait-Based Design
The core logic is abstracted through traits to allow for different implementations.
/// Platform-agnostic audio input trait
#[async_trait]
pub trait AudioCapture {
async fn start_recording(&mut self) -> Result<()>;
async fn stop_recording(&mut self) -> Result<Vec<f32>>;
fn is_recording(&self) -> bool;
}
/// Speech-to-text transcription trait
#[async_trait]
pub trait Transcriber {
async fn transcribe(&self, audio: &[f32]) -> Result<String>;
fn model_info(&self) -> ModelInfo;
async fn is_available(&self) -> bool;
}
/// Whisper backend implementation
pub struct WhisperTranscriber {
ctx: WhisperContext,
model_path: PathBuf,
}
Sources: speech-to-text-integration-plan.md:90-109
Implementation Plan
The feature is planned to be developed in five phases.
Configuration
Voice input features can be configured in ~/.config/cmdai/config.toml.
| Option | Type | Default | Description |
|---|---|---|---|
enabled | boolean | true | Enables or disables the voice feature. |
model | string | "tiny" | Model size to use (tiny, base, small, etc.). |
auto_stop | boolean | true | Enables auto-stop on silence detection. |
language | string | "auto" | Language for transcription (auto, en, es, etc.). |
vad.enabled | boolean | true | Enables or disables Voice Activity Detection. |
vad.threshold | float | 0.5 | VAD sensitivity threshold. |
vad.silence_duration_ms | integer | 1500 | Silence duration to trigger auto-stop. |
Sources: speech-to-text-integration-plan.md:193-204
Dependencies
| Type | Name | Version | Purpose |
|---|---|---|---|
| Rust Crate | whisper-rs | 0.12 | Rust bindings for whisper.cpp. |
| Rust Crate | cpal | 0.15 | Cross-platform audio I/O. |
| Rust Crate | vad-rs | (optional) | Voice Activity Detection. |
| System | Xcode Tools | - | Required for Metal framework on macOS. |
| System | ALSA dev | - | Required for audio on Linux (e.g., libasound2-dev). |
Sources: speech-to-text-integration-plan.md:126, speech-to-text-integration-plan.md:149, speech-to-text-integration-plan.md:258-267
Caro Voice Synthesis (Output)
This is a planned feature to give caro a unique, recognizable voice for text-to-speech output. The goal is to create a human-sounding voice that is "seeded" from the real-life vocalizations of a dog named Kyaro. This feature is part of the v2.0.0 roadmap.
Sources: caro-voice/RESEARCH.md:3-9, presentation/roadmap-slides.md:115-129
Technical Approach
The recommended approach is a multi-stage timbre transfer pipeline using Differentiable Digital Signal Processing (DDSP).
This diagram shows the proposed pipeline for creating Caro's voice.
Implementation Plan
Phase 1: Audio Collection & Preparation
A dataset of 8-12 minutes of Kyaro's vocalizations will be recorded and preprocessed.
Required Audio Samples:
| Category | Target Duration | Purpose |
|---|---|---|
| Standard barks | 2-3 min | Core timbre extraction |
| Alert barks | 1-2 min | Sharp, attention-getting |
| Happy sounds | 2 min | Playful, excited |
| Whines/cries | 1-2 min | Emotional range |
| Grumbles | 1-2 min | Low, sustained |
Sources: caro-voice/PLAN.md:17-31, caro-voice/RESEARCH.md:104-115
Phase 2: Feature Extraction
Acoustic features like pitch (F0), formants, and MFCCs will be extracted from the processed audio using Python libraries.
# scripts/preprocess.py
import librosa
import soundfile as sf
from pathlib import Path
def preprocess_audio(input_dir, output_dir, sr=44100):
"""Normalize and clean Kyaro audio samples."""
input_path = Path(input_dir)
output_path = Path(output_dir)
output_path.mkdir(exist_ok=True)
for audio_file in input_path.glob("*.wav"):
# Load audio
y, _ = librosa.load(audio_file, sr=sr, mono=True)
# Normalize
y = librosa.util.normalize(y)
# Trim silence
y, _ = librosa.effects.trim(y, top_db=30)
# Save
output_file = output_path / audio_file.name
sf.write(output_file, y, sr)
print(f"Processed: {audio_file.name}")
Sources: caro-voice/PLAN.md:43-68
Phase 3 & 4: Model Training and Synthesis
A DDSP autoencoder will be trained on the Kyaro audio to learn its timbre. This timbre model will then be applied to a neutral human base voice (generated by a TTS model like F5-TTS) to produce the final "Caro" voice.
Sources: caro-voice/PLAN.md:95-114, caro-voice/PLAN.md:116-120
Dependencies
Key Python libraries for the voice synthesis project include:
| Library | Purpose |
|---|---|
torch / torchaudio | Core machine learning framework. |
librosa | Audio analysis and feature extraction. |
ddsp | Differentiable Digital Signal Processing for timbre transfer. |
f5-tts | High-quality base TTS model. |
parselmouth | Praat-like formant extraction. |
Sources: caro-voice/RESEARCH.md:120-127, caro-voice/PLAN.md:195-219
Summary
The voice capabilities in caro are twofold: an implemented plan for speech-to-text input and a researched plan for text-to-speech output. The input feature uses whisper.cpp for robust, private, offline transcription. The planned output feature is an innovative approach to create a unique brand voice for the assistant by blending human TTS with acoustic features from a real dog. Both features adhere to the project's local-first philosophy.
Sources: speech-to-text-integration-plan.md:3-6, caro-voice/RESEARCH.md:3-9