Supported Providers Overview
Relevant source files
The following files were used as context for generating this wiki page:
The Speech SDK is a universal, provider-agnostic TypeScript toolkit designed to interface with multiple Text-to-Speech (TTS) services through a unified API. It abstracts the complexities of individual provider implementations—such as authentication, endpoint structures, and data formats—into a single public function, generateSpeech(), allowing for zero lock-in and cross-platform compatibility across Node.js, Edge runtimes, and browsers.
The core architecture relies on a provider abstraction layer where each supported service implements a common SpeechProvider interface. This enables features like multi-speaker Conversations and Word-Level Timestamps to function consistently regardless of the underlying backend.
Sources: README.md:10-15, AGENTS.md:20-25, skills/speech-sdk/SKILL.md:10-12
Provider Architecture and Resolution
The SDK employs a factory pattern to instantiate providers. Users can specify a provider and model using a "provider/model" string (e.g., "openai/tts-1") or by using specific factory functions like createOpenAI().
Resolution Flow
When a request is made, the SDK resolves the provider and model ID to determine which implementation to use. Built-in providers are mapped in src/resolve-provider.ts, which handles the instantiation of the corresponding class.
The SDK supports a "provider-first" resolution where passing just the provider name (e.g., "deepgram") defaults to that provider's pre-configured default model.
Sources: AGENTS.md:26-34, src/resolve-provider.ts, skills/speech-sdk/SKILL.md:26-30
Capability Matrix
The SDK currently supports 13 major providers. Each provider is associated with a specific prefix used for model identification and a corresponding environment variable for authentication.
| Provider | Prefix | Default Model | Env Var | Key Features |
|---|---|---|---|---|
| OpenAI | openai | gpt-4o-mini-tts | OPENAI_API_KEY | Audio Tags (Instructions) |
| ElevenLabs | elevenlabs | eleven_multilingual_v2 | ELEVENLABS_API_KEY | Native Timestamps, Native Dialogue |
| Deepgram | deepgram | aura-2 | DEEPGRAM_API_KEY | Streaming |
| Cartesia | cartesia | sonic-3 | CARTESIA_API_KEY | Native Timestamps (SSE) |
| Hume | hume | octave-2 | HUME_API_KEY | Native Timestamps, Native Dialogue |
google | gemini-2.5-flash-preview-tts | GOOGLE_API_KEY | Native Dialogue (2 voices) | |
| Fish Audio | fish-audio | s2-pro | FISH_AUDIO_API_KEY | Native Dialogue |
| Murf | murf | GEN2 | MURF_API_KEY | Native Timestamps |
| Resemble | resemble | default | RESEMBLE_API_KEY | Native Timestamps |
| Inworld | inworld | inworld-tts-1.5-max | INWORLD_API_KEY | Native Timestamps |
| Mistral | mistral | voxtral-mini-tts-2603 | MISTRAL_API_KEY | Voice Cloning |
| xAI | xai | grok-tts | XAI_API_KEY | TTS Support |
| fal | fal-ai | (User specified) | FAL_API_KEY | Native Dialogue (Dia) |
Sources: README.md:43-57, skills/speech-sdk/SKILL.md:14-16, src/providers/google/index.ts:117-133
Provider Features and Specializations
Providers vary in their support for advanced SDK features. The SDK uses a features array within the model definition to signal capabilities.
Word-Level Timestamps
The SDK implements a cascade for timestamps:
- Native: If the provider returns alignment data (e.g., ElevenLabs, Murf, Hume Octave-2), it is used directly.
- STT Fallback: If
timestamps: "on"is requested for a provider without native support (e.g., OpenAI, Deepgram), the SDK transcribes the generated audio via an STT model (defaulting to Whisper).
Sources: README.md:104-124, skills/speech-sdk/SKILL.md:65-75
Multi-Speaker Conversations
The generateConversation function routes turns based on provider capabilities:
- Native Path: Used for providers with multi-speaker endpoints (ElevenLabs v3, Google Gemini, Hume Octave, Fish Audio S2-Pro).
- Stitch Path: Used for mixed-provider conversations or providers without native support. It executes turns in parallel and concatenates PCM data.
Audio Tag Support
The SDK supports a standardized bracket syntax [tag] for expressive cues. Handling varies by provider:
- Passthrough: ElevenLabs and Google (Gemini 3.1) handle certain tags natively.
- Transformation: Cartesia transforms emotion tags to SSML. OpenAI maps tags to the
instructionsfield for thegpt-4o-mini-ttsmodel. - Stripping: Unsupported tags are removed, and a warning is added to the
SpeechResult.
Sources: README.md:173-185, src/providers/google/index.ts:153-163, src/generate-speech.ts:110-130
Implementation Details
Every provider implementation must satisfy the SpeechProvider interface, which requires:
id: Unique provider identifier string.defaultModel: The fallback model ID.generate(): Method to produce audio from text.stream(): (Optional) Method for incremental audio delivery.processAudioTags(): (Optional) Logic for handling expressive cues.
Example: Google Provider Data Flow
The Google Gemini implementation demonstrates how the SDK handles raw PCM data by wrapping it in a WAV container before returning it to the user.