Generating Speech
Relevant source files
The following files were used as context for generating this wiki page:
Generating speech is the core functionality of the Speech SDK, primarily exposed through the generateSpeech() function. This system provides a universal, provider-agnostic interface for converting text to audio across 13 different AI providers, including OpenAI, ElevenLabs, Google Gemini, and Cartesia. The SDK is designed to run in Node.js, Edge runtimes, and browsers, abstracting the complexities of individual provider APIs into a single, consistent workflow.
Sources: AGENTS.md:15-17, README.md:10-14, skills/speech-sdk/SKILL.md:9-11
Core Architecture and Data Flow
The speech generation process follows a structured pipeline that involves model resolution, text preprocessing, provider execution, and result post-processing. The entry point for this logic is src/generate-speech.ts, which coordinates the interaction between the SDK's core logic and specific provider implementations.
Generation Pipeline
The following diagram illustrates the high-level flow of a generateSpeech request:
The SDK utilizes p-retry to handle 5xx errors and network failures automatically, while immediately failing on 4xx errors.
Sources: src/generate-speech.ts:71-167, AGENTS.md:19-21
Function Signature and Configuration
The generateSpeech function accepts a comprehensive options object that configures how the audio is generated and processed.
| Parameter | Type | Description |
|---|---|---|
model | string | ResolvedModel | The provider/model string (e.g., "openai/tts-1") or a factory-resolved model. |
text | string | The input text to be synthesized into speech. |
voice | Voice | A string ID or a voice object for reference cloning (URL or base64). |
volumeDbfs | number | Target loudness in dBFS (≤ 0). Forces audio/wav output. |
timestamps | "on" | "auto" | "off" | Controls word-level alignment generation. Default is "auto". |
providerOptions | object | Pass-through parameters for the specific provider API. |
maxRetries | number | Number of retry attempts for transient errors. Default is 2. |
Sources: src/generate-speech.ts:25-59, skills/speech-sdk/SKILL.md:112-124
Audio Tags and Preprocessing
Before sending text to a provider, the SDK performs "Progressive Disclosure" and preprocessing. This includes detecting and handling bracketed audio tags (e.g., [laugh], [sigh]).
- Native Support: Providers like ElevenLabs (
eleven_v3) or Google Gemini (gemini-3.1-flash) receive these tags natively. - SSML Transformation: Some providers, like Cartesia, transform emotion tags into SSML (e.g.,
<emotion value="angry"/>). - Stripping: If a provider does not support a specific tag, it is stripped from the text, and a warning is added to the final
SpeechResult.
Sources: src/generate-speech.ts:182-192, src/providers/google/index.ts:114-124, README.md:173-188
Word-Level Timestamps
The SDK provides a sophisticated "cascade" system for word-level alignment timing, measured in seconds from the start of the audio.
- Native Support: ElevenLabs, Murf, Hume, Inworld, Cartesia, and Resemble provide native timestamps.
- STT Fallback: When
timestamps: "on"is set but the provider has no native alignment, the SDK transcribes the generated audio via OpenAI Whisper (openai/whisper-1) to derive timings.
Sources: src/generate-speech.ts:133-149, README.md:104-138, skills/speech-sdk/SKILL.md:65-79
Result and Metadata
The output of generateSpeech is a SpeechResult object containing the audio data and comprehensive execution metadata.
interface SpeechResult {
readonly audio: GeneratedAudioFile;
readonly metadata: SpeechMetadata;
readonly timestamps?: readonly WordTimestamp[];
readonly providerMetadata?: Record<string, unknown>;
readonly warnings?: string[];
}
Sources: skills/speech-sdk/references/result.md:3-13
Audio File Management
The GeneratedAudioFile allows for lazy base64 computation. The uint8Array contains the raw bytes, while the base64 string is only calculated if accessed by the user. Common media types include audio/mpeg and audio/wav.
Sources: skills/speech-sdk/references/result.md:15-19, src/providers/google/index.ts:168-180
Execution Metadata
The SDK tracks several performance and diagnostic metrics during the generation process:
- latencyMs: Total time from function call to result.
- inputChars: Length of the text after preprocessing.
- audioDurationMs: The actual length of the resulting audio in milliseconds.
- provider: The ID of the provider used (e.g., "google").
- model: The specific model ID used.
Sources: src/generate-speech.ts:151-157, README.md:204-209
Conclusion
Generating speech in the Speech SDK is a highly resilient process that abstracts multiple provider APIs into a single developer-friendly interface. It handles complex features like volume normalization, cross-provider audio tag support, and a multi-tiered timestamp system, ensuring consistent behavior regardless of the underlying text-to-speech engine.