Streaming Audio

Relevant source files

The following files were used as context for generating this wiki page:

• skills/speech-sdk/references/streaming.md
• src/stream-speech.ts
• src/tests/stream-speech.test.ts
• src/providers/google/index.ts
• src/providers/inworld/index.ts
• src/speech-provider.ts
• README.md

Streaming Audio in the Speech SDK allows developers to receive audio data incrementally as it is generated by a provider. This is accomplished via the streamSpeech function, which returns a standard Web ReadableStream<Uint8Array>. This capability is essential for real-time applications such as voice agents, low-latency browser playback, and processing long-form text without waiting for the entire synthesis to complete.

Sources: skills/speech-sdk/references/streaming.md:1-5, README.md:95-98

Core Architecture and Flow

The streaming system is built around a provider-agnostic interface. When streamSpeech is called, the SDK resolves the appropriate provider and model, checks for streaming support, and initiates the request. If the provider supports native streaming, the SDK pipes the provider's response directly to the consumer.

Execution Flow

1. Resolution: The SDK parses the model string (e.g., openai/tts-1) and retrieves the corresponding provider instance.
2. Feature Validation: The SDK verifies if the selected model has the streaming feature enabled in its metadata.
3. Provider Execution: The SDK calls the provider's internal stream() method.
4. Error Handling & Retries: The system applies retry logic for network errors or 5xx responses until the initial response headers are received.

The diagram shows the high-level logic within the streamSpeech entry point, including feature verification and retry mechanisms.
Sources: src/stream-speech.ts, src/tests/stream-speech.test.ts:47-60, README.md:104-106

API Reference

streamSpeech Function

The primary entry point for streaming audio.

Parameter	Type	Description
model	string | ResolvedModel	The provider/model identifier or a resolved model object.
text	string	The text to convert to speech.
voice	Voice	The voice identifier, URL, or audio reference.
maxRetries	number	Number of retries for 5xx/network errors (default: 2).
abortSignal	AbortSignal	Allows cancelling the stream request.

Sources: README.md:213-225, src/stream-speech.ts

StreamSpeechResult

The object returned by a successful streamSpeech call.

Field	Type	Description
audio	ReadableStream<Uint8Array>	The raw audio bytes delivered in chunks.
mediaType	string	The MIME type of the audio (e.g., audio/mpeg, audio/wav).
metadata	SpeechMetadata	Includes latencyMs, ttfbMs (Time to First Byte), and inputChars.
warnings	string[]	List of non-fatal issues, such as unsupported audio tags.

Sources: skills/speech-sdk/references/streaming.md:14-20, src/tests/metadata.test.ts:121-135

Provider Implementation Details

Providers implement streaming differently based on their underlying APIs.

Native NDJSON Streaming (e.g., Inworld)

Providers like Inworld use newline-delimited JSON (NDJSON) to stream audio chunks. The SDK includes a parseInworldNdjsonStream utility to decode these lines and extract the audioContent.

The diagram illustrates the process of decoding NDJSON lines into raw bytes for the end user.
Sources: src/providers/inworld/index.ts:162-185, src/providers/inworld/index.ts:237-260

Simulated Streaming (e.g., Google Gemini)

Some providers, such as Google Gemini, may buffer the full synthesis server-side before delivering it. In these cases, the SDK's stream() method delegates to the generate() method and wraps the resulting full buffer in a single-chunk ReadableStream to maintain a consistent API.

Sources: src/providers/google/index.ts:161-171, src/tests/google-stream.test.ts:33-40

Error Handling

Streaming introduces specific error scenarios:

• StreamingNotSupportedError: Thrown if streamSpeech is called on a model that lacks the streaming feature or if the provider has not implemented the stream() method.
• Retry Behavior: Retries (configured via maxRetries) apply only until the response headers are received. If an error occurs mid-stream after the headers have arrived, the error propagates directly to the stream consumer.
• AbortSignal: Passing an AbortSignal allows the caller to terminate the request and close the stream connection immediately.

Sources: src/tests/stream-speech.test.ts:47-60, README.md:104-106, skills/speech-sdk/references/streaming.md:46-54

Usage Example: Browser Playback

To play a stream in the browser, chunks must be collected and converted into a Blob or handled via Media Source Extensions.

const result = await streamSpeech({
  model: "cartesia/sonic-2",
  text: "Streaming in the browser.",
  voice: "voice-id",
});

const chunks: Uint8Array[] = [];
for await (const chunk of result.audio) {
  chunks.push(chunk);
}

const blob = new Blob(chunks, { type: result.mediaType });
const url = URL.createObjectURL(blob);
new Audio(url).play();

Sources: skills/speech-sdk/references/streaming.md:33-44

Streaming Audio provides a high-performance path for audio delivery by leveraging ReadableStream, ensuring that the Speech SDK remains efficient across Node.js, Edge, and Browser environments.