Voice Assistant Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Voice Assistant Integration provides real-time, bidirectional vocal communication between users and the AI agent within the Happy ecosystem. Primarily powered by ElevenLabs Conversational AI, this system enables users to interact with their coding sessions using natural language, receiving spoken responses while maintaining hands-on-keyboard productivity.
The integration spans multiple platforms, including macOS, iOS, Android, and Web, utilizing platform-specific audio engines (AVFoundation for Apple platforms, Android system TTS fallback for Android) while sharing a common WebSocket-based communication protocol. It handles complex tasks such as Voice Activity Detection (VAD), audio streaming, and contextual updates to ensure the AI remains aware of the user's current environment.
Sources: apps/macos/CLAUDE.md:214-230, apps/macos/Happy/Services/VoiceService.swift:522-535, apps/web/react/CLAUDE.md:31-33
System Architecture
The voice integration is built on a client-server model where the client application manages local hardware (microphone/speakers) and communicates with ElevenLabs via WebSockets.
High-Level Components
| Component | Description |
|---|---|
| ElevenLabs API | Backend service providing Conversational AI, TTS (Text-to-Speech), and VAD. |
| VoiceService | The core business logic layer handling WebSocket lifecycle, audio capture, and playback. |
| Audio Capture Manager | platform-specific component (e.g., AVAudioEngine) for processing microphone input. |
| Audio Playback Manager | Component for streaming and playing back PCM audio chunks received from the AI. |
| Voice UI | Floating indicators and controls (mute, language) for managing the voice state. |
Sources: apps/macos/Happy/Services/VoiceService.swift:30-70, apps/macos/CLAUDE.md:220-240
Communication Flow
The following diagram illustrates the interaction between the Happy Client and the ElevenLabs Conversational AI service.
The client maintains a persistent WebSocket connection to wss://api.elevenlabs.io/v1/convai/conversation, transmitting audio in 16kHz mono 16-bit PCM format.
Sources: apps/macos/Happy/Services/VoiceService.swift:85-135, apps/macos/CLAUDE.md:231-239
Technical Implementation (macOS/iOS)
The macOS implementation, detailed in VoiceService.swift, serves as the reference architecture for Apple platforms.
State and Mode Management
The system tracks the "mode" of the conversation to provide visual feedback and manage audio priorities:
- Idle: The assistant is connected but waiting.
- Listening: The assistant is actively processing microphone input.
- Speaking: The assistant is playing back synthesized audio.
Sources: apps/macos/Happy/Services/VoiceService.swift:76-83, apps/macos/CLAUDE.md:244-245
Audio Handling Logic
The AudioCaptureManager uses AVAudioEngine to install a tap on the input bus. It converts native input to the required 16kHz PCM format before Base64 encoding for WebSocket transmission.
Audio playback utilizes an AVAudioPlayerNode and a DispatchQueue to manage pending buffers, ensuring smooth streaming of audio chunks as they arrive from the server.
Sources: apps/macos/Happy/Services/VoiceService.swift:233-280, apps/macos/Happy/Services/VoiceService.swift:188-210
Multi-Platform Availability
| Platform | Core Implementation | Notes |
|---|---|---|
| macOS | VoiceService.swift (AVFoundation) | Referenc integration with ElevenLabs; supports VAD and low latency. |
| iOS | Swift/SwiftUI | ElevenLabs TTS + system fallback support. |
| Android | VoiceService.kt (Kotlin) | ElevenLabs TTS with Android System TTS fallback. |
| Web | RealtimeVoiceSession.tsx | Real-time session management via React. |
Sources: apps/mobile/android/CLAUDE.md:200-205, apps/mobile/ios/CLAUDE.md:31-40, apps/web/react/CLAUDE.md:31-33
Configuration and Protocol
WebSocket Event Protocol
The system utilizes specific JSON payloads for client-to-server and server-to-client events.
Client Events:
user_audio_chunk: Raw audio data.user_message: Text fallback/injection.contextual_update: Updates agent on environment changes (e.g., "User opened file.ts").conversation_initiation_client_data: Initial handshake.
Server Events:
audio: Streaming chunks for playback.user_transcript: Real-time text of user speech.agent_response: The AI's generated text.interruption: Signal to stop playback when user speaks over agent.
Sources: apps/macos/Happy/Services/VoiceService.swift:105-180, apps/macos/CLAUDE.md:241-247
Voice Settings
Users can customize the integration through the application settings:
- Language Selection: Supports codes such as
en,es,ru,pl,pt,ca, andzh. - Mute Toggle: Disables microphone tap processing.
- Voice Assistant Enable/Disable: Global toggle to prevent unintended background activity.
Sources: apps/macos/Happy/Services/VoiceService.swift:574-585, apps/macos/CLAUDE.md:265-270
Conclusion
The Voice Assistant Integration is a critical component of the "Happy" ecosystem, transforming the AI agent from a standard chatbot into a conversational companion. By leveraging high-quality models from ElevenLabs and native platform audio APIs, it provides a responsive and context-aware interface that enhances remote development workflows.
Sources: apps/macos/CLAUDE.md:214-220, apps/web/react/CLAUDE.md:31-33