Voice Interaction & VAD

Relevant source files

The following files were used as context for generating this wiki page:

• instructions/design-backend-prompt.txt
• src/main.ts
• src/renderer.tsx
• backend/src/handler.ts
• SOUL.md

Voice Interaction and Voice Activity Detection (VAD) constitute the hands-free primary interface for the JARVIS desktop environment. The system implements a privacy-first "hard-mute" architecture by default, requiring explicit user authorization before audio capture begins. Once active, the system utilizes a pipeline consisting of local Web Audio API-based silence detection, high-quality Speech-to-Text (STT) via xAI Whisper, and Text-to-Speech (TTS) response synthesis.

Sources: instructions/design-backend-prompt.txt:64-70, src/renderer.tsx:185-190

Architecture Overview

The voice system is distributed across the Electron Main process, the Renderer process, and the local Backend service. The Renderer handles the immediate audio stream and VAD logic, the Main process provides secure IPC bridges to external APIs (like xAI), and the Backend manages the voice engine status and model orchestration.

Voice Pipeline Flow

The following diagram illustrates the lifecycle of a voice command from capture to spoken response.

Voice Activity Detection (VAD)

The system utilizes the Web Audio API to analyze microphone input in real-time. This local analysis ensures that audio is only recorded and transmitted when human speech is detected, minimizing bandwidth and protecting privacy.

• VAD Threshold: Configured conservatively (target 0.8 in protocol) to process only clear, direct speech.
• Silence Frames: The system waits for approximately 16 frames (~500ms) of silence after speech detection before stopping the MediaRecorder.
• Safety Timer: A maximum recording duration of 7 seconds is enforced to prevent runaway recording.

Sources: SOUL.md:120-130, src/renderer.tsx:325-350

VAD Logic Summary

Parameter	Value / Description
SPEECH_THRESHOLD	22 (RMS level for detection)
SILENCE_FRAMES_MAX	16 (~500ms at typical frame rates)
MAX_RECORD_TIME	7000ms (Auto-stop safety)
FFT_SIZE	64 (Analysis window size)

Sources: src/renderer.tsx:325-340

Speech-to-Text (STT) and Command Parsing

Once the VAD detects a complete utterance, the audio blob is sent to the Electron Main process. JARVIS primarily uses xAI Whisper for transcription. The resulting text is then filtered and parsed for deterministic commands before being sent to the AI Agent.

Noise & Hallucination Filtering

The system includes heuristics to ignore common STT artifacts such as "Vielen Dank" or "Untertitelung" (hallucinations common in Whisper when listening to silence). It also uses a "Barge-in" logic: if the user starts speaking while JARVIS is responding, the TTS playback and current model generation are immediately aborted to favor the new user input.

Sources: src/renderer.tsx:110-125, src/renderer.tsx:220-245

Deterministic Command Handling

Specific voice patterns trigger immediate actions without requiring LLM inference:

• Stop Commands: "Stop", "Abbrechen", "Ruhe".
• Hardware Control: "Aktiviere Iron-Man-Modus" (Unmutes microphone).
• Navigation: "Öffne Spotify", "Lade Wikipedia auf der Hauptbühne".
• Vision Queries: "Was siehst du?", "Analysiere mein Outfit" (Triggers camera/screenshot snapshot).

Sources: src/renderer.tsx:255-280, src/renderer.tsx:880-920

Text-to-Speech (TTS)

JARVIS supports multiple TTS providers, with xAI (Grok Zenith voice) as the primary high-quality engine and Fish Audio as a zero-shot neural fallback.

• Synthesis Process: Markdown and special characters are stripped from responses before synthesis to ensure natural phrasing.
• Playback: Audio is returned as an ArrayBuffer via IPC and played back using the Web Audio API in the renderer.
• Visual State: The Orb identity core enters the responding state (bright cyan/green expansion) during playback.

Sources: instructions/design-backend-prompt.txt:75-80, src/renderer.tsx:365-400

TTS Configuration Options

Option	Type	Default	Description
ttsProvider	enum	xai	Primary engine (xai or fishaudio)
ttsVoice	string	zenith	Voice ID (e.g., zenith for authoritative JARVIS tone)
sttLanguage	string	de	ISO language code for transcription

Sources: src/main.ts:25-35, backend/src/handler.ts:510-520

Privacy and Security Principles

The voice interaction system adheres to strict security constraints:

1. Hard Mute: No background audio capture occurs until the user toggles the microphone state.
2. Local Loopback: Voice engine status and control routes are restricted to the local loopback address (127.0.0.1).
3. Intent Confirmation: Critical actions derived from voice commands require "Human-in-the-loop" approval via an Action Proposal card.

Sources: instructions/design-backend-prompt.txt:64-70, instructions/design-backend-prompt.txt:125-135

Conclusion

The Voice Interaction & VAD system transforms JARVIS from a simple chat interface into a reactive assistant. By combining local VAD for efficiency with cloud-based STT/TTS for quality, the system achieves low-latency performance while maintaining user privacy through a hard-mute architecture.