Multi-Speaker Conversations
Relevant source files
The following files were used as context for generating this wiki page:
The Multi-Speaker Conversations feature in the Speech SDK allows developers to generate complex audio outputs involving multiple voices, such as podcasts, interviews, and dialogues. The primary entry point for this functionality is the generateConversation function, which produces a single, high-quality audio file from an array of individual dialogue "turns."
This system abstracts the complexity of coordinating multiple Text-to-Speech (TTS) requests, handling everything from provider-specific native dialogue endpoints to local PCM (Pulse Code Modulation) stitching and volume normalization. It supports cross-provider mixing, allowing different speakers in the same conversation to be powered by different AI models (e.g., combining OpenAI and ElevenLabs).
Sources: skills/speech-sdk/references/conversation.md:1-12, README.md:118-124
Core Architecture and Dispatch Logic
The SDK employs an automated dispatch mechanism to determine the most efficient way to generate multi-speaker audio. It evaluates the provided turns and the capabilities of the requested models to choose between two primary paths: Native Dialogue and Stitch Fallback.
Dispatch Paths
- Native Path: Triggered when all turns belong to a single provider that supports a native multi-speaker endpoint (e.g., ElevenLabs v3 or Google Gemini). This path is preferred as it usually results in more natural prosody and a single API round-trip.
- Stitch Path: Triggered if turns use different providers or if the selected provider lacks native dialogue support. In this mode, the SDK executes
generateSpeechcalls in parallel and concatenates the results locally.
The following diagram illustrates the decision logic used by the SDK:
Provider Native Capabilities
Different providers have specific constraints and "caps" when using their native dialogue endpoints:
| Provider | Models | Voice Constraints |
|---|---|---|
| ElevenLabs | eleven_v3 | 1–10 voices, ≤ 2,000 chars |
gemini-2.5-flash-preview-tts, gemini-3.1-flash-tts-preview | Exactly 2 voices | |
| Hume | octave-1, octave-2 | 1–4 voices |
| Fish Audio | s2-pro | 1–4 voices |
Sources: README.md:139-144, src/providers/google/index.ts:285-290
Implementation Details
The Stitching Process
On the stitch path, the SDK performs several technical operations to ensure a seamless output:
- Parallel Execution: Individual turns are requested concurrently, governed by
maxConcurrency(default: 6). - PCM Conversion: Providers are forced into a PCM/WAV mode using
getStitchOptionsto ensure the audio can be concatenated without decoding compressed artifacts. - Gap Insertion: A configurable silence (defined by
gapMs, default: 300) is inserted between turns. - Concatenation: The raw audio buffers are joined into a single
audio/wavfile.
Sources: skills/speech-sdk/references/conversation.md:46-55, src/tests/e2e/conversation-stitch-four-providers.e2e.test.ts:32-40
Volume Normalization
To prevent jarring volume shifts between different voices or providers, the SDK performs RMS (Root Mean Square) normalization by default.
- Target Loudness: The default target is
-20dBFS, which is the industry standard for podcasts. - Behavior: On the stitch path, normalization is always applied. On the native path, it is applied if the provider supports a decodable output mode.
- Configuration: Can be disabled by setting
normalizeVolume: falseor customized viavolumeDbfs.
Sources: skills/speech-sdk/references/conversation.md:57-64, README.md:189-195
Data Structures
GenerateConversationOptions
The configuration interface for creating a multi-speaker dialogue.
interface GenerateConversationOptions {
turns: ConversationTurn[] // Array of dialogue pieces
model?: string | ResolvedModel // Global default model
gapMs?: number // Default: 300
maxConcurrency?: number // Default: 6
normalizeVolume?: boolean // Default: true
volumeDbfs?: number // Default: -20
timestamps?: "on" | "auto" | "off" // Alignment settings
}
Sources: skills/speech-sdk/references/conversation.md:27-44
Word-Level Timestamps
When timestamps: "on" is used, the SDK generates a unified array of word alignments across all speakers. If the stitch path is used, the SDK automatically offsets the timing of each turn based on the cumulative duration of previous audio and the inter-turn gaps.
Error Handling
| Error | Context |
|---|---|
ConversationInputError | Triggered by invalid input such as empty turns or missing text. |
DialogueConstraintError | Thrown when the native path is selected but speaker requirements (e.g., voice count) are not met. |
StitchUnsupportedError | Thrown when a provider/model cannot produce the PCM/WAV format required for local mixing. |
Sources: skills/speech-sdk/references/conversation.md:81-89, README.md:255-265
Conclusion
Multi-speaker conversations represent a high-level orchestration layer within the Speech SDK. By combining parallel request handling, intelligent dispatching, and post-processing techniques like PCM stitching and RMS normalization, the system provides a unified interface for generating complex audio content regardless of the underlying TTS providers used. Individual turns are seamlessly integrated into a single audio result with consistent loudness and accurate, cumulative timestamps.