SpeechResult & Data Structures
Relevant source files
The following files were used as context for generating this wiki page:
The SpeechResult interface is the primary return type for the generateSpeech function in the Speech SDK. It serves as a unified container for synthesized audio data, execution metadata, word-level alignment, and provider-specific diagnostics. This structure abstracts away the varied response formats of the 13+ supported providers, offering a consistent developer experience across Node.js, Edge, and Browser environments.
The data structures within the SDK are designed for efficiency, particularly through the use of lazy-computed properties for heavy operations like base64 encoding and structured metadata for tracking performance metrics like latency and character counts.
Core Data Model
The high-level architecture of a speech generation result is centered around the SpeechResult interface, which aggregates several sub-structures.
The diagram shows the relationship between the primary SpeechResult object and its constituent data structures.
Sources: src/speech-result.ts:1-25, skills/speech-sdk/references/result.md:1-25
The SpeechResult Interface
The SpeechResult interface provides a comprehensive overview of the generation attempt.
| Field | Type | Description |
|---|---|---|
audio | GeneratedAudioFile | The raw audio data and its encoding information. |
metadata | SpeechMetadata | Performance and usage statistics for the request. |
timestamps | WordTimestamp[] | Optional word-level alignment data (start/end times in seconds). |
providerMetadata | Record<string, unknown> | Raw, untransformed extras from the provider (e.g., requestId). |
warnings | string[] | Non-fatal issues, such as unsupported audio tags being stripped. |
Sources: skills/speech-sdk/references/result.md:3-11, src/generate-speech.ts:133-139
Audio Data Handling
Audio is encapsulated in the GeneratedAudioFile structure. The SDK utilizes the DefaultGeneratedAudioFile class to implement lazy evaluation for base64 strings to minimize memory overhead when only raw bytes are needed.
GeneratedAudioFile Structure
- uint8Array: Contains the raw audio bytes. This is the primary format used for file writing or streaming to responses.
- base64: A string representation of the audio. It is lazy-computed from the
uint8Arrayon first access. - mediaType: The MIME type of the audio (e.g.,
audio/mpeg,audio/wav).
Sources: src/speech-result.ts:5-15, skills/speech-sdk/references/result.md:13-17
Data Flow for Audio Retrieval
This flow illustrates how audio data is processed and stored within the result structures, including the lazy computation of base64 strings.
Sources: src/generate-speech.ts:101-122, src/speech-result.ts:15-25
Metadata and Performance Tracking
The SpeechMetadata structure tracks the lifecycle and cost of the speech generation request.
| Field | Type | Description |
|---|---|---|
latencyMs | number | Total time from request start to completion. |
inputChars | number | The length of the text processed (after tag stripping). |
provider | string | The ID of the provider used (e.g., "openai"). |
model | string | The specific model identifier used. |
audioDurationMs | number | Optional duration of the generated audio in milliseconds. |
ttfbMs | number | Optional Time To First Byte (used in streaming). |
Sources: src/metadata.ts:1-15, src/generate-speech.ts:125-131
Word-Level Alignment (Timestamps)
Timestamps provide word-level granularity for the generated audio, essential for captions or highlighting text during playback.
WordTimestamp Structure
- text: The specific word or token.
- start: Offset from the beginning of the audio in seconds.
- end: End time in seconds.
Sources: skills/speech-sdk/references/result.md:19-23
Timestamp Generation Logic
Timestamps are populated based on the TimestampMode and provider capabilities:
- Native: If the model supports native timestamps (e.g., ElevenLabs, Cartesia) and mode is
"auto"or"on", they are returned directly. - Derived: If mode is
"on"and the provider lacks native support, the SDK performs an STT round-trip (fallback toopenai/whisper-1by default). - Off: Timestamps are suppressed even if natively available.
Sources: src/generate-speech.ts:40-47, src/generate-speech.ts:143-157
Error Structures
The SDK defines several specialized error classes that extend SpeechSDKError.
| Error Class | Trigger Condition |
|---|---|
ApiError | Provider returns a non-2xx status code. |
NoSpeechGeneratedError | Empty input or empty provider response. |
VolumeAdjustmentUnsupportedError | volumeDbfs requested but provider has no decodable mode. |
StreamingNotSupportedError | streamSpeech() called on a model without streaming features. |
Sources: src/index.ts:7-13, src/generate-speech.ts:73-77, README.md:188-204
Summary
SpeechResult and its associated data structures provide a robust, standardized interface for interacting with diverse Text-to-Speech providers. By centralizing audio data in the GeneratedAudioFile with lazy base64 conversion and providing detailed SpeechMetadata, the SDK ensures that developers can easily handle audio output, track performance, and implement advanced features like synchronized highlighting via WordTimestamp.