Other TTS Providers (Deepgram, Fish, fal, etc.)
Relevant source files
The following files were used as context for generating this wiki page:
The Speech SDK provides a unified abstraction layer for 13 different Text-to-Speech (TTS) providers, allowing developers to switch between high-performance models with zero lock-in. While major providers like OpenAI and ElevenLabs are central to the SDK, specialized support for "Other" providers—including Deepgram, Fish Audio, Mistral, Google Gemini, and Hume—enables advanced features like ultra-low latency, native multi-speaker dialogue, and voice cloning.
Sources: README.md:12-14, skills/speech-sdk/SKILL.md:3-8
Provider Implementation Matrix
Each provider is implemented as a class following the SpeechProvider interface. They handle authorization, request body mapping, and response parsing specific to their respective APIs.
| Provider | Prefix | Default Model | Auth Header / Key | Notable Features |
|---|---|---|---|---|
| Deepgram | deepgram | aura-2 | Token | Ultra-low latency aura models |
| Fish Audio | fish-audio | s2-pro | Bearer | Native dialogue, model in headers |
google | gemini-2.5-flash-preview-tts | key (Query Param) | Multi-speaker dialogue, PCM-to-WAV wrapping | |
| Mistral | mistral | voxtral-mini-tts-2603 | Bearer | Native voice cloning (ref_audio) |
| Hume | hume | octave-2 | X-Hume-Api-Key | Utterance-based logic, native timestamps |
| Resemble | resemble | default | Authorization | Base64 audio content mapping |
Sources: README.md:63-78, src/tests/deepgram-speech-model.test.ts:46-57, src/providers/google/index.ts:153-156
Architecture and Data Flow
The architecture follows a standard flow where a request is resolved to a specific provider implementation. The provider then transforms the generic generateSpeech options into the specific format required by the external API.
This diagram illustrates how different providers handle request parameters differently (e.g., Deepgram using query strings vs Fish Audio using custom headers).
Sources: AGENTS.md:21-25, src/tests/deepgram-speech-model.test.ts:98-115
Provider-Specific Logic
Deepgram
Deepgram implementation is unique in its use of query parameters for model configuration. While most providers pass model and voice IDs in the JSON body, Deepgram appends them to the URL string.
- Endpoint:
https://api.deepgram.com/v1/speak - Parameters:
modelis constructed asmodelId-voiceId(e.g.,aura-2-thalia-en). - Auth: Uses the
Tokenprefix in theAuthorizationheader.
Sources: src/tests/deepgram-speech-model.test.ts:20-25, src/tests/deepgram-speech-model.test.ts:46-57
Fish Audio
Fish Audio utilizes the s2-pro model and requires the model ID to be passed as a custom model header rather than within the request body.
- Voice Mapping: Maps the
voiceparameter toreference_idin the JSON body. - Dialogue: Supports native multi-speaker capabilities (1-4 voices).
Sources: src/tests/fish-audio-speech-model.test.ts:20-25, README.md:105
Mistral and Voice Cloning
Mistral provides native support for voice cloning. The SDK allows developers to pass either a voice_id (string) or raw audio data for cloning.
- Cloning Logic: If the
voiceparameter is an object containingaudio, it is converted to base64 and passed asref_audioin the request body. - Duration Metadata: Mistral returns
audio_duration_secondswithin ausageobject, which the SDK converts toaudioDurationMs.
Sources: src/tests/mistral-speech-model.test.ts:46-65, src/tests/mistral-speech-model.test.ts:145-160
Google Gemini (TTS)
The Google provider integrates Gemini-based TTS models. It is one of the few providers where the API key is passed as a query parameter (key=...) instead of a header.
- PCM Wrapping: Gemini returns raw 16-bit mono PCM. The SDK automatically wraps this in a
RIFF/WAVEcontainer to ensure the output is a standard playableaudio/wavfile. - Multi-Speaker Constraints: Supports exactly 2 voices for native dialogue.
Sources: src/providers/google/index.ts:182-190, src/providers/google/index.ts:207-215, src/providers/google/index.ts:245-246
Hume
Hume implementations focus on the octave-1 and octave-2 models. They use a unique utterances array structure in the request body.
- Native Timestamps:
octave-2supports native word-level alignment via the/v0/ttsendpoint. The SDK flattens the nested snippet response into a standardWordTimestamp[]. - Endpoint Routing: Requests for timestamps are routed to
/v0/tts, while standard requests use/v0/tts/file.
Sources: src/tests/hume-speech-model.test.ts:46-55, src/tests/hume-timestamps.test.ts:10-25, src/tests/hume-timestamps.test.ts:60-75
Advanced Feature Support
The "Other" providers often fill specific niches in the feature set of the SDK, particularly regarding audio tags and streaming.
| Feature | Provider Support | Implementation Detail |
|---|---|---|
| Audio Tags | Cartesia, Google (3.1) | [laugh] or emotion tags are mapped to SSML or passed natively. |
| Streaming | Inworld, Deepgram, Google | Models marked with the streaming feature return ReadableStream<Uint8Array>. |
| Voice Cloning | Mistral, fal-ai | Accepts { audio } or { url } in the voice parameter. |
Sources: README.md:180-195, src/providers/google/index.ts:133-145, src/tests/mistral-speech-model.test.ts:46-57
Conclusion
The inclusion of specialized providers like Deepgram, Fish, and Mistral ensures that the Speech SDK provides a complete toolkit for modern audio applications. By abstracting the significant differences in authentication (Headers vs Query Params), payload structure (Flat vs Utterances), and audio encoding (MP3 vs Raw PCM), the SDK allows for a consistent developer experience across the entire spectrum of AI speech technology.