Introduction & Quick Start
Relevant source files
The following files were used as context for generating this wiki page:
@speech-sdk/core is a universal, cross-platform TypeScript SDK designed for text-to-speech (TTS) integration. It provides a single, unified API to interact with 13 different speech providers, including OpenAI, ElevenLabs, Google Gemini, and Cartesia, ensuring zero vendor lock-in. The SDK is designed to run seamlessly in Node.js, Edge runtimes, and browser environments.
Sources: README.md:1-12, skills/speech-sdk/SKILL.md:1-6
The system architecture focuses on provider abstraction, allowing developers to switch between different high-quality speech models by simply changing a string identifier. The core flow involves resolving a model, calling the specific provider implementation, and returning a standardized SpeechResult.
Sources: AGENTS.md:21-25
Core Architecture & Flow
The SDK operates through a centralized entry point that abstracts the complexity of different provider APIs. The primary execution path flows from the public API through a resolution layer to specific provider logic.
The diagram above illustrates how the SDK routes requests to different providers while maintaining a consistent output format.
Sources: AGENTS.md:21-30, skills/speech-sdk/SKILL.md:65-79
Key Components
| Component | Description | File Path |
|---|---|---|
generateSpeech | Public entry point for single-utterance TTS generation. | src/generate-speech.ts |
streamSpeech | Handles chunked audio output via ReadableStream. | src/stream-speech.ts |
generateConversation | Manages multi-speaker dialogues and audio stitching. | src/generate-conversation.ts |
SpeechProvider | The interface all 13 built-in providers implement. | src/speech-provider.ts |
SpeechResult | Standardized object containing audio bytes, base64, and metadata. | src/speech-result.ts |
Sources: AGENTS.md:21-30, skills/speech-sdk/SKILL.md:9-12
Installation & Environment Setup
The SDK is distributed as an ESM-only package and can be installed via standard package managers.
npm install @speech-sdk/core
Sources: README.md:31-33, AGENTS.md:46
Authentication
The SDK automatically resolves API keys from environment variables based on the provider prefix.
| Provider | Env Var |
|---|---|
| OpenAI | OPENAI_API_KEY |
| ElevenLabs | ELEVENLABS_API_KEY |
GOOGLE_API_KEY | |
| Cartesia | CARTESIA_API_KEY |
| Inworld | INWORLD_API_KEY |
Sources: README.md:80-92, src/providers/google/index.ts:140
Basic Usage
The SDK provides three primary functions to handle different audio generation requirements.
Single Utterance
The generateSpeech function is used for standard TTS. It requires a model, text, and voice.
import { generateSpeech } from "@speech-sdk/core"
const result = await generateSpeech({
model: "openai/gpt-4o-mini-tts",
text: "Hello from SpeechSDK!",
voice: "alloy",
})
Sources: skills/speech-sdk/SKILL.md:14-25
Streaming Audio
For real-time applications, streamSpeech returns a ReadableStream<Uint8Array>.
import { streamSpeech } from "@speech-sdk/core"
const result = await streamSpeech({
model: "elevenlabs/eleven_v3",
text: "Streaming content...",
voice: "voice-id",
})
for await (const chunk of result.audio) {
process.stdout.write(chunk)
}
Sources: skills/speech-sdk/SKILL.md:30-40
Multi-Speaker Conversations
generateConversation handles complex scenarios with multiple voices. It uses a "native" path for providers supporting multi-speaker endpoints (e.g., ElevenLabs, Google Gemini) and a "stitch" path for others.
import { generateConversation } from "@speech-sdk/core/conversation"
const result = await generateConversation({
turns: [
{ voice: "alloy", text: "Welcome to the show." },
{ voice: "echo", text: "Thanks for having me!" },
],
})
Sources: skills/speech-sdk/SKILL.md:42-55, README.md:115-125
Data Structures
SpeechResult
Every generation call returns a SpeechResult object, ensuring consistency regardless of the underlying provider.
| Field | Type | Description |
|---|---|---|
audio.uint8Array | Uint8Array | Raw audio bytes. |
audio.base64 | string | Lazy-computed base64 string. |
audio.mediaType | string | MIME type (e.g., audio/mpeg, audio/wav). |
metadata | SpeechMetadata | Latency, input chars, provider, and model info. |
timestamps | WordTimestamp[] | Word-level alignment (optional). |
Sources: skills/speech-sdk/references/result.md:3-23, skills/speech-sdk/SKILL.md:20-23
Timing and Alignment
Word-level timestamps can be requested by passing timestamps: "on". If the provider does not support native alignment, the SDK falls back to an STT round-trip (typically OpenAI Whisper).
Error Handling & Retries
The SDK includes built-in retry logic for 5xx status codes and network errors using exponential backoff. It does not retry 4xx errors (client errors).
| Error Class | Trigger Condition |
|---|---|
ApiError | Provider returns a non-2xx response. |
NoSpeechGeneratedError | Empty provider response or empty input. |
StreamingNotSupportedError | Calling streamSpeech on a model without streaming features. |
SpeechSDKError | Base class for all SDK-specific errors. |
Sources: AGENTS.md:27, README.md:248-270
Summary
The Speech SDK provides a robust, provider-agnostic framework for integrating AI-driven speech synthesis into TypeScript applications. By abstracting 13 different providers into a single interface, it simplifies the implementation of complex features like multi-speaker dialogue, real-time streaming, and word-level alignment while maintaining high technical standards for error handling and performance across Node.js and browser environments.