Execution Flow & Dispatch

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• skills/speech-sdk/SKILL.md
• README.md
• src/providers/google/index.ts
• src/conversation/pcm-concat.ts

The Speech SDK is designed as a universal, provider-agnostic interface for Text-to-Speech (TTS) services. The core architectural principle revolves around a unified execution flow that abstracts away specific provider implementations, allowing for seamless switching between 13 different AI speech providers including OpenAI, ElevenLabs, Google Gemini, and Cartesia. Sources: AGENTS.md, README.md

Execution is categorized into three primary entry points: single utterance generation (generateSpeech), incremental audio streaming (streamSpeech), and multi-speaker dialogue management (generateConversation). The dispatch logic automatically determines whether to utilize native provider features—such as multi-speaker endpoints—or to fallback to local processing like PCM audio stitching and volume normalization. Sources: skills/speech-sdk/SKILL.md, README.md

Core Generation Flow

The standard execution path for a single TTS request follows a linear sequence from the public API to the specific provider implementation. This flow includes model resolution, provider instantiation, and audio processing.

Workflow Stages

1. Entry Point: The user calls generateSpeech() with a model (e.g., "openai/tts-1") and text.
2. Model Resolution: The SDK uses resolveModel() to parse the provider/model string and instantiate the correct built-in provider.
3. Provider Dispatch: The request is dispatched to the provider.generate() method of the specific implementation (e.g., GoogleSpeechProvider).
4. Result Formatting: The raw provider response is wrapped into a SpeechResult object containing audio bytes and metadata.

Sources: AGENTS.md, src/providers/google/index.ts:135-140

The diagram shows the synchronous path from a user request to the final speech result. Sources: AGENTS.md, src/providers/google/index.ts

Conversation Dispatch Logic

generateConversation handles complex multi-turn dialogues. It employs a "Smart Dispatch" system that selects between two distinct execution paths based on the capabilities of the requested model and the composition of the turns. Sources: skills/speech-sdk/SKILL.md, README.md

1. Native Dialogue Path

If the turns utilize a single provider that supports native multi-speaker endpoints, the SDK dispatches the entire conversation in a single API request. This path provides a natural mix and lower latency. Sources: README.md, src/providers/google/index.ts:246-260

Provider	Supported Models	Constraint
Google	gemini-3.1-flash-tts-preview	Exactly 2 voices
ElevenLabs	eleven_v3	1–10 voices, ≤ 2,000 chars
Hume	octave-2	1–4 voices
Fish Audio	s2-pro	1–4 voices

Sources: README.md, src/providers/google/index.ts:240-245

2. Local Stitching Path

When turns involve multiple providers or providers lacking native dialogue support, the SDK executes turns in parallel and concatenates the results locally.

• Concurrency: Turns are executed up to a maxConcurrency (default 6).
• Stitching: Audio is decoded to raw PCM, resampled to a target rate, and combined with a gapMs silence.
• Normalization: By default, every turn is RMS-leveled to approximately -20 dBFS to ensure consistent loudness across different providers.

Sources: skills/speech-sdk/SKILL.md, src/conversation/pcm-concat.ts:182-195, src/conversation/pcm-concat.ts:245-255

This diagram illustrates the parallel execution and local assembly required for cross-provider conversations. Sources: skills/speech-sdk/SKILL.md, src/conversation/pcm-concat.ts

Audio Processing & Normalization

The SDK ensures audio consistency through internal utilities that handle PCM data manipulation and volume leveling.

Normalization Logic

Normalization uses Root-Mean-Square (RMS) amplitude calculation to target a specific loudness. The default target is -20 dBFS, which is the standard for broadcast/podcast voice levels. Sources: src/conversation/pcm-concat.ts:210-220, src/conversation/pcm-concat.ts:245-250

// From src/conversation/pcm-concat.ts
export function normalizeRms(
  segments: readonly Pcm16Segment[],
  targetRmsAmplitude = DEFAULT_TARGET_RMS_INT16
): Pcm16Segment[] {
  return segments.map((s) => {
    const segRms = rmsPcm16(s.pcm);
    if (segRms === 0) return { ...s };
    return { ...s, pcm: scaleClamp(s.pcm, targetRmsAmplitude / segRms) };
  });
}

Sources: src/conversation/pcm-concat.ts:257-268

Streaming Dispatch

Streaming is handled via streamSpeech(), which returns a standard ReadableStream<Uint8Array>. The SDK dispatches this to the provider's stream() method.

• Native Streaming: Providers like Cartesia or OpenAI stream chunks as they are generated.
• WAV Wrapping: Some providers (like Google Gemini) synthesize the full clip before flushing. The SDK wraps these responses in a single-chunk ReadableStream to maintain a consistent API interface. Sources: src/providers/google/index.ts:215-225

Error Handling & Retries

The execution flow includes built-in resiliency using p-retry. The SDK retries network errors and 5xx status codes automatically with exponential backoff. Sources: AGENTS.md, README.md

Configuration	Default Value	Description
maxRetries	2	Number of attempts for 5xx/network errors
abortSignal	undefined	Standard AbortSignal to cancel requests
timeout	N/A	Provider-specific or determined by fetch implementation

Sources: README.md, skills/speech-sdk/SKILL.md

Summary

The Execution Flow & Dispatch system in the Speech SDK provides a robust abstraction layer that balances provider-native optimizations with consistent local fallback processing. By automating model resolution, parallelizing multi-speaker turns, and enforcing volume normalization, the SDK ensures that developers can generate professional-grade audio without managing the idiosyncratic complexities of individual TTS APIs. Sources: AGENTS.md, README.md, src/conversation/pcm-concat.ts