Google Gemini Provider Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Google Gemini Provider Integration allows the SDK to interface with Google's Gemini models for Text-to-Speech (TTS) capabilities. This integration supports single-utterance generation, multi-speaker native dialogues, and pseudo-streaming by wrapping standard content generation responses. It leverages the Google Generative Language API (v1beta) to produce high-quality audio outputs from text inputs using a variety of prebuilt voices.
Sources: src/providers/google/index.ts:1-20, README.md:120-130
Architecture and Components
The integration is centered around the GoogleSpeechProvider class, which implements the SpeechProvider interface. It handles request construction, authentication via API keys, and post-processing of raw audio data into standard formats.
Core Class: GoogleSpeechProvider
The GoogleSpeechProvider manages the configuration for interacting with Google's API, including the base URL, API key resolution, and fetch implementation.
The provider defaults to gemini-2.5-flash-preview-tts and uses a default base URL of https://generativelanguage.googleapis.com/v1beta.
Sources: src/providers/google/index.ts:31-110, skills/speech-sdk/references/providers/google.md:1-10
Supported Models and Features
Google Gemini integration supports several models, each with distinct language support and feature sets.
| Model ID | Release Date | Key Features | Languages |
|---|---|---|---|
gemini-3.1-flash-tts-preview | 2026-04-15 | Streaming, Native Audio Tags | 80+ (including am, bn, cmn, fil, etc.) |
gemini-2.5-flash-preview-tts | 2025-05-01 | Streaming (Pseudo) | 24 (en, fr, de, es, zh, ja, etc.) |
gemini-2.5-pro-preview-tts | 2025-05-01 | Streaming (Pseudo) | 24 (en, fr, de, es, zh, ja, etc.) |
Sources: src/providers/google/index.ts:40-100, skills/speech-sdk/references/providers/google.md:12-18
Generation Data Flow
The generate method follows a specific flow to convert text into a playable WAV file. Gemini returns raw 16-bit mono PCM, which the SDK wraps in a WAV container to ensure direct playability for clients.
Request Payload Structure
Requests to the :generateContent endpoint include a generationConfig that specifies audio as the response modality and provides a voice_config.
{
"contents": [{ "role": "user", "parts": [{ "text": "..." }] }],
"generationConfig": {
"responseModalities": ["audio"],
"speech_config": {
"voice_config": {
"prebuilt_voice_config": { "voice_name": "Kore" }
}
}
}
}
Sources: src/providers/google/index.ts:135-180, src/tests/google-speech-model.test.ts:50-70
Audio Tag Processing
The SDK provides a standardized way to handle expressive audio tags (e.g., [whispers], [laugh]).
- Native Support:
gemini-3.1-flash-tts-previewsupports tags natively; they are passed through verbatim. - Fallback: Older models (2.5 series) do not support native tags. The SDK strips these tags and issues a warning to prevent them from being spoken literally.
Sources: src/providers/google/index.ts:114-125, src/tests/google-speech-model.test.ts:180-210
Native Multi-Speaker Dialogue
Google Gemini supports a "native" path for conversations with exactly two speakers. This is handled by the generateDialogue method.
Dialogue Logic
- Labeling: The SDK maps unique voices to internal labels like
Speaker1andSpeaker2. - Concatenation: Text from turns is joined with newlines, prefixed by the speaker label (e.g.,
Speaker1: Hello). - Multi-Speaker Config: The request body uses
multi_speaker_voice_configinstead of the standardvoice_config.
Pseudo-Streaming Implementation
While Gemini provides a streamGenerateContent endpoint, it buffers the full synthesis server-side. The SDK's stream method delegates to generate() and wraps the resulting bytes in a ReadableStream that emits the entire file as a single chunk. True progressive streaming for Gemini (Live API via WebSockets) is not implemented in this provider.
Sources: src/providers/google/index.ts:208-228, src/tests/google-stream.test.ts:35-50
Configuration and Environment
Users can configure the provider using the createGoogle factory or by setting environment variables.
- Environment Variable:
GOOGLE_API_KEY - Default Sample Rate: 24,000 Hz
- Media Type:
audio/wav(after SDK wrapping)
Sources: src/providers/google/index.ts:311-320, skills/speech-sdk/references/providers/google.md:4-8
The Google Gemini integration provides a robust implementation for high-quality speech generation and two-person dialogues, ensuring audio compatibility across platforms by automatically handling PCM-to-WAV conversion.