ElevenLabs Provider Integration
Relevant source files
The following files were used as context for generating this wiki page:
The ElevenLabs provider integration in the Speech SDK enables high-fidelity text-to-speech (TTS) capabilities, including streaming audio, native word-level timestamps, and multi-speaker dialogue. It implements the SpeechProvider interface, allowing it to be used interchangeably with other providers through the core generateSpeech and streamSpeech APIs.
The integration handles API communication with ElevenLabs, mapping project-specific voice and model IDs to the appropriate REST endpoints. It supports advanced features such as character-to-word alignment aggregation and specialized multi-speaker "dialogue" endpoints for efficient conversational audio generation.
Sources: src/providers/elevenlabs/index.ts:25-27, README.md:93-100, AGENTS.md:21-25
Architecture and Data Flow
The ElevenLabsSpeechProvider class centralizes the logic for request construction, response parsing, and feature detection. It follows the factory pattern, where the createElevenLabs function returns a ResolvedModel that wraps the provider instance and a specific model ID.
Request Lifecycle
The provider transforms generic SDK options into ElevenLabs-specific parameters. Key transformations include extracting certain fields from providerOptions (like output_format) into URL query parameters while leaving others in the JSON request body.
Component Interaction
The integration interacts with internal utilities for API key resolution and error handling. It also utilizes a specialized alignment module to convert raw character-level timing data into the WordTimestamp format used by the SDK.
Key Features and Capabilities
The ElevenLabs integration supports several distinct models, each with varying language support and feature sets.
Supported Models
| Model ID | Release Date | Default | Features |
|---|---|---|---|
eleven_v3 | 2025-06-08 | No | Streaming, Audio Tags, Native Timestamps, Dialogue |
eleven_multilingual_v2 | 2023-08-22 | Yes | Streaming, Native Timestamps |
eleven_flash_v2_5 | 2024-12-01 | No | Streaming, Native Timestamps |
eleven_flash_v2 | 2024-12-01 | No | Streaming, Native Timestamps (English only) |
| Sources: src/providers/elevenlabs/index.ts:122-152, README.md:73 |
Native Dialogue support
The eleven_v3 model supports a specialized /v1/text-to-dialogue endpoint. This allows for multi-speaker audio generation in a single request, which is more efficient than local stitching.
- Constraints: 1–10 voices, maximum 2,000 total characters.
- Logic: The provider maps an array of
turnsto aninputsarray containingtextandvoice_idpairs.
Sources: src/providers/elevenlabs/index.ts:327-340, README.md:118
Implementation Details
Request Construction
The buildRequest private method handles the extraction of specific options that ElevenLabs expects as query parameters.
// src/providers/elevenlabs/index.ts:162-177
private buildRequest(
text: string,
modelId: string,
providerOptions: Record<string, unknown> | undefined
) {
const opts = providerOptions ?? {};
const {
output_format,
enable_logging,
optimize_streaming_latency,
...bodyOptions
} = opts as Record<string, unknown>;
const body: Record<string, unknown> = {
...bodyOptions,
text,
model_id: modelId,
};
// ... query string logic
}
Sources: src/providers/elevenlabs/index.ts:162-177
Audio Format Mapping
When using the /with-timestamps endpoint, ElevenLabs returns audio as a base64 string inside a JSON payload. Since the Content-Type header is for application/json, the provider must manually derive the audio's media type based on the requested output_format.
| Format Prefix | Derived Media Type |
|---|---|
mp3_ | audio/mpeg |
pcm_ | audio/pcm;rate={rate} |
ulaw_ | audio/basic;rate={rate} |
opus_ | audio/opus |
| Sources: src/providers/elevenlabs/index.ts:405-428 |
Alignment Logic
ElevenLabs provides character-level alignment (arrays of characters, start times, and end times). The alignmentToWordTimestamps utility iterates through these characters, stripping whitespace and aggregating them into words to provide the word-level granularity required by the SDK.
Sources: src/providers/elevenlabs/alignment.ts:18-35
Error Handling and Metadata
The integration captures the ElevenLabs request-id and audio-duration-seconds from response headers and exposes them via providerMetadata and metadata.audioDurationMs.
- API Keys: Resolves
ELEVENLABS_API_KEYfrom environment variables by default. - Retries: Retries 5xx and network errors but fails fast on 4xx validation errors.
- Validation: Throws a
SpeechSDKErrorif avoiceID is not provided, as ElevenLabs requires a specific voice for all TTS operations.
Sources: src/providers/elevenlabs/index.ts:223-230, src/tests/elevenlabs-speech-model.test.ts:250-260, README.md:209-215
Conclusion
The ElevenLabs Provider Integration provides a robust implementation of text-to-speech features within the Speech SDK. By supporting native streaming, dialogue, and word-level timestamps, it allows developers to leverage ElevenLabs' advanced capabilities through a simplified, unified interface.