Voice Interaction (STT & TTS)

Relevant source files

The following files were used as context for generating this wiki page:

• backend/modules/voice/routes.py
• backend/modules/voice/tts.py
• Ai-Assistant-v2/auth-service/server.js
• ai-assistant-react/src/components/Chat/ChatInterface.jsx
• PROJECT_STRUCTURE_GIT.md

The Voice Interaction system provides the AI Assistant with auditory capabilities, allowing it to transcribe spoken language into text (Speech-to-Text) and synthesize text into audible speech (Text-to-Speech). This module is integrated into the Python backend service to leverage local machine learning libraries while being exposed via a RESTful API for frontend consumption.

This system enables hands-free interaction, enhancing accessibility and providing a more natural conversational experience. It operates alongside the Dual AI Models to process user intents and deliver spoken responses.
Sources: README.md, backend/modules/voice/routes.py:1-10

System Architecture

The voice interaction architecture is split between the Python AI Service (which handles the heavy processing) and the React Frontend (which provides the user controls).

Architecture Components

• SpeechToText (STT): Handles microphone input or audio file processing to extract text.
• TextToSpeech (TTS): Uses the pyttsx3 library to convert AI-generated text into speech.
• Voice Routes: Flask-based endpoints that expose voice functionalities.
• Frontend Controls: UI buttons and state management for triggering listening or speaking modes.

Text-to-Speech (TTS) Implementation

The TTS system is built around the pyttsx3 library, which provides a cross-platform interface for text-to-speech synthesis.

The TextToSpeech Class

The TextToSpeech class encapsulates the engine initialization, configuration, and execution logic. It supports direct speech output and saving synthesized audio to files.

• Initialization: The engine is configured with a default rate of 150 words per minute and a volume level of 0.9.
• Voice Selection: It retrieves available system voices and defaults to the first index.
• Execution: The speak method uses engine.say() for immediate playback or engine.save_to_file() for persistence, followed by engine.runAndWait() to process the command queue.

Sources: backend/modules/voice/tts.py:4-60

TTS Configuration Options

Parameter	Type	Default	Description
rate	Integer	150	Speed of the synthesized speech (words per minute).
volume	Float	0.9	Volume level of the speech (0.0 to 1.0).
voice_index	Integer	0	The index of the system voice to be used.
save_to_file	String	None	Optional path to save the audio output as a file.

Sources: backend/modules/voice/tts.py:16-25, backend/modules/voice/routes.py:127-140

Speech-to-Text (STT) and Listen Endpoints

The STT system allows the assistant to capture user input through a microphone or via uploaded audio files.

Workflow Sequence

The following diagram illustrates the sequence of events when a user initiates a voice command.

API Reference (Voice Module)

The voice features are exposed via the voice_bp Blueprint in the Python backend.

Endpoints Summary

Method	Endpoint	Description
POST	/voice/listen	Activates microphone to capture and transcribe speech.
POST	/voice/speak	Synthesizes provided text into audible speech.
GET	/voice/voices	Returns a list of available TTS voices on the host system.
POST	/voice/configure	Updates TTS settings like rate, volume, and voice index.
POST	/voice/recognize_file	Accepts a multipart audio file and returns transcribed text.

Sources: backend/modules/voice/routes.py:12-178

Implementation Detail: Voice Configuration

The configuration endpoint allows dynamic updates to the TTS engine without restarting the service.

@voice_bp.route('/configure', methods=['POST'])
def configure_voice():
    data = request.get_json() or {}
    results = {}
    if 'voice_index' in data:
        results['voice_set'] = tts.set_voice(data['voice_index'])
    if 'rate' in data:
        results['rate_set'] = tts.set_rate(data['rate'])
    if 'volume' in data:
        results['volume_set'] = tts.set_volume(data['volume'])
    return jsonify({'status': 'success', 'results': results}), 200

Sources: backend/modules/voice/routes.py:127-148

Frontend Integration

In the ChatInterface.jsx component, voice interaction is represented as a feature within the sidebar. Although marked as "coming soon" in some UI alerts, the backend infrastructure is fully mapped to handle the requests.

User Settings

The Node.js authentication service stores user preferences regarding voice interaction in the UserSchema, specifically under settings.voiceEnabled.

• Path: Ai-Assistant-v2/auth-service/server.js:33-33
• Default: false

UI Component Structure

The frontend provides a dedicated "Voice" section in the sidebar with a button to trigger the listening process.
Sources: ai-assistant-react/src/components/Chat/ChatInterface.jsx:265-270, Ai-Assistant-v2/auth-service/server.js:33

Conclusion

The Voice Interaction system provides a robust foundation for multi-modal communication within the AI Assistant platform. By utilizing local TTS via pyttsx3 and a dedicated STT module, the application ensures low-latency voice responses and accurate command recognition, all while maintaining user-specific configuration via the central authentication service.