Audio Processing (STT & TTS)
Relevant source files
The following files were used as context for generating this wiki page:
The Audio Processing module in tldw_server provides a robust infrastructure for Speech-to-Text (STT) and Text-to-Speech (TTS) operations. It supports a wide array of media formats including video and audio, enabling features such as real-time streaming transcription, long-form media synthesis, and multi-speaker diarization. This system is integrated into the broader media ingestion and RAG (Retrieval-Augmented Generation) pipelines, allowing users to search, analyze, and interact with transcribed content.
The module abstracts various backends, supporting both commercial providers (OpenAI, Anthropic, Google) and local inference engines (faster_whisper, NVIDIA NeMo, Kokoro ONNX). It employs advanced techniques like VAD (Voice Activity Detection), silence removal via FFmpeg, and chunked processing to ensure high accuracy and performance across different hardware profiles, including CUDA-accelerated environments.
Sources: AGENTS.md:52-58, Docs/Architecture.md:209-222, CLAUDE.md:52-58
Speech-to-Text (STT) Architecture
The STT system handles both file-based and real-time streaming transcription. It uses a multi-layered approach to process audio, involving preprocessing for silence removal and diarization to distinguish between different speakers.
STT Ingestion Flow
When media is ingested, the audio pipeline extracts raw text and metadata. This content is then normalized into chunks and stored in the Media DB v2 for retrieval and RAG.
The flow starts with raw audio or video, which is preprocessed using FFmpeg to remove silence and normalize formats before being passed to specific transcription services.
Sources: Docs/Design/Audio_Pipeline.md:10-15, Docs/Architecture.md:189-207, Docs/Product/STT_Module_vNext_PRD.md:112-120
Streaming Protocol (WebSocket v2)
The vNext STT module implements an enhanced WebSocket protocol (protocol_version=2) to provide better control over streaming sessions. This includes explicit state management for pausing, resuming, and flushing finals.
| Action | Description |
|---|---|
pause | Stop processing new audio and buffer it up to a bounded queue (default 2.0s). |
resume | Continue processing queued audio in FIFO order. |
stop | Flush finals for processed audio and close the session. |
commit | Force a final transcript for processed audio in the current utterance. |
Sources: Docs/Product/STT_Module_vNext_PRD.md:46-65
Text-to-Speech (TTS) System
The TTS system is designed for high reliability and low latency, utilizing an LRU (Least Recently Used) model cache to avoid repeated loading of local inference engines.
TTS Processing Pipeline
For long-form content, the system employs sentence-based chunking and crossfade merging to prevent artifacts between audio segments.
This diagram illustrates the internal logic for handling long-form TTS requests, emphasizing chunking, cache utilization, and quality checks.
Sources: Docs/Product/PRD_TTS_Speech_UX_Upgrades.md:101-118
Key TTS Features
- Model Caching: LRU cache for local instances (e.g., Kokoro ONNX) to reduce latency on MPS/CUDA hardware.
- Reliability Upgrades: Implementation of min-token patching and dynamic max tokens to prevent silent or truncated outputs.
- Progress Reporting: Real-time status updates via SSE/WebSocket, providing ETA and per-segment progress.
- History Management: Per-user persistence of TTS artifacts with metadata including duration, provider, and parameters.
Sources: Docs/Product/PRD_TTS_Speech_UX_Upgrades.md:101-131
Diarization and Audio Enhancement
The project utilizes advanced diarization techniques to improve transcript readability by grouping segments by speaker embeddings.
Diarization Strategy
The system moves beyond basic segmenting by generating speaker embeddings for each Whisper-generated segment and matching them against known speakers in the current recording. This significantly reduces the "bonus speaker" effect often found in standard pyannote implementations.
# Conceptual logic for improved diarization
for segment in segments:
embedding = generate_speaker_embedding(segment)
if match_found(embedding, known_speakers):
add_to_speaker_array(segment, matched_speaker)
else:
create_new_speaker(segment, embedding)
Sources: Docs/Design/Audio_Pipeline.md:28-44
Silence Removal
To optimize transcription accuracy and speed, FFmpeg is used to strip silence before VAD or STT processing.
ffmpeg -i video-audio.m4a \
-af "silenceremove=start_periods=1:start_duration=0:start_threshold=-50dB:\
stop_periods=-1:stop_duration=0.02:stop_threshold=-50dB,\
apad=pad_dur=0.02" \
-c:a aac -b:a 128k output_minpause.m4a -y
Sources: Docs/Design/Audio_Pipeline.md:10-15
Data Models and Persistence
Audio processing results are stored with rich metadata to support versioning and administrative oversight.
Transcription Run History
The Media DB v2 tracks transcription runs to allow for versioned transcripts and idempotent reruns.
| Field | Type | Description |
|---|---|---|
transcription_run_id | Integer | Logical run key for append-only history. |
idempotency_key | String | Scoped to media_id to prevent duplicate runs. |
latest_run_id | Integer | Globally current default run for a media item. |
auto_commit | Boolean | True if emitted by server-side VAD. |
Sources: Docs/Product/STT_Module_vNext_PRD.md:82-108
Summary
The Audio Processing module provides a scalable, provider-agnostic framework for media analysis. By combining advanced local models with commercial APIs and robust preprocessing (FFmpeg, Diarization), tldw_server ensures that audio and video content are accurately transformed into actionable text for research and interaction. Key architectural features like the WebSocket v2 protocol and TTS model caching prioritize performance and user visibility into long-running tasks.