SDK Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Speech SDK is a universal, provider-agnostic TypeScript library designed to facilitate Text-to-Speech (TTS) operations across multiple AI providers including OpenAI, ElevenLabs, Google Gemini, and Cartesia. It is engineered to run seamlessly across Node.js, Edge runtimes, and browser environments, providing a unified interface for speech generation, streaming, and multi-speaker conversation management.
The architecture centers around a provider abstraction layer that standardizes request and response formats. This allows developers to switch between 13+ supported providers with minimal code changes, leveraging a single public function, generateSpeech(), to handle core logic such as model resolution and retry orchestration.
Sources: AGENTS.md, README.md:1-20, skills/speech-sdk/SKILL.md:1-15
Core Component Structure
The SDK is built on a modular architecture that separates the public API surface from individual provider implementations.
Main Entry Points
The SDK exposes three primary functions for interacting with TTS services:
generateSpeech: For single utterance generation.streamSpeech: For chunked audio delivery viaReadableStream.generateConversation: For complex, multi-speaker dialogues that utilize native provider endpoints or local audio stitching.
Sources: skills/speech-sdk/SKILL.md:14-18, src/index.ts:15-30
Component Relationship Diagram
The following diagram illustrates how the public API interacts with the internal resolution and provider layers.
This diagram shows the core flow from the public API entry point to the specific provider implementation.
Sources: AGENTS.md:20-25, src/speech-provider.ts
Provider Abstraction Layer
All supported services must implement the SpeechProvider interface, ensuring a consistent internal API.
The SpeechProvider Interface
The interface enforces a structure that includes model metadata, supported features, and methods for audio generation and streaming. Key features tracked per provider include streaming, audio-tags, and timestamps.
| Component | Description |
|---|---|
id | Unique identifier for the provider (e.g., "google", "cartesia"). |
defaultModel | The fallback model ID used if none is specified. |
models | An array of ModelInfo objects defining capabilities and languages. |
generate() | Async method to produce a complete SpeechResult. |
stream() | Async method to return a ReadableStream<Uint8Array>. |
Sources: src/speech-provider.ts:5-40, src/providers/google/index.ts:114-125, src/providers/cartesia/index.ts:35-45
Provider Implementation: Google Gemini Example
The GoogleSpeechProvider demonstrates how specific API logic is encapsulated. It handles Gemini-specific requirements such as wrapping raw 16-bit mono PCM into a playable WAV container.
// Example of provider-specific audio wrapping
const sampleRate = parseMediaTypeParam(part.inlineData.mimeType ?? "", "rate") ?? DEFAULT_GEMINI_SAMPLE_RATE;
const pcm = base64ToBytes(part.inlineData.data);
const wav = await wrapPcm16Mono(pcm, sampleRate);
return {
audio: wav,
mediaType: "audio/wav",
};
Sources: src/providers/google/index.ts:193-206
Data Flow: Speech Generation
The SDK follows a strict sequence to transform text into audio across different providers.
Sequence Diagram: generateSpeech
The following sequence diagram tracks a request from the user through the resolution and provider execution phases.
This sequence shows the standard lifecycle of a TTS request.
Sources: AGENTS.md:20-35, src/providers/google/index.ts:160-200
Multi-Speaker Conversation Logic
The generateConversation system uses a dual-path logic to handle dialogues.
- Native Path: If a provider supports native multi-speaker endpoints (e.g., ElevenLabs v3, Gemini TTS, Hume Octave), the SDK sends the entire conversation in a single request for natural mixing.
- Stitch Path: If multiple providers are mixed or the provider lacks native support, the SDK executes turns in parallel, normalizes volume (RMS-leveling to ~-20 dBFS), and concatenates PCM data with configurable gaps (
gapMs).
Sources: skills/speech-sdk/SKILL.md:46-65, README.md:110-130
Multi-Speaker Capabilities Table
| Provider | Supported Models | Voice Constraints |
|---|---|---|
| ElevenLabs | eleven_v3 | 1–10 voices, ≤ 2,000 chars |
| Gemini 2.5/3.1 Flash/Pro | Exactly 2 voices | |
| Hume | octave-1, octave-2 | 1–4 voices |
| Fish Audio | s2-pro | 1–4 voices |
Sources: README.md:128-135, src/providers/google/index.ts:250-260
Metadata and Result Handling
The SDK returns a standardized SpeechResult or StreamSpeechResult to the consumer, regardless of the underlying provider.
SpeechResult Structure
interface SpeechResult {
audio: {
uint8Array: Uint8Array;
base64: string; // lazy-computed
mediaType: string;
};
metadata: {
latencyMs: number;
inputChars: number;
provider: string;
model: string;
audioDurationMs?: number;
ttfbMs?: number;
};
timestamps?: WordTimestamp[];
providerMetadata?: Record<string, unknown>;
warnings?: string[];
}
Sources: README.md:210-225, skills/speech-sdk/SKILL.md:25-30
Architecture Summary
The Speech SDK architecture is defined by its Provider Pattern, allowing for zero lock-in and high extensibility. By abstracting the complexities of 13 different API structures—including differing authentication methods (Bearer vs. API Key), audio formats (PCM, MP3, WAV), and feature sets (Timestamps, Streaming)—the SDK provides a consistent, high-level interface for modern TTS applications. Its reliance on standard TypeScript interfaces and functional factory patterns (createOpenAI, createGoogle) ensures that it remains lightweight and compatible across various JavaScript environments.
Sources: AGENTS.md:20-40, README.md:1-15, src/index.ts