STT Fallback & Whisper
Relevant source files
The following files were used as context for generating this wiki page:
STT Fallback is a mechanism within the Speech SDK designed to provide word-level timestamps when a Text-to-Speech (TTS) provider does not support native alignment. By default, the SDK utilizes OpenAI Whisper as the primary Speech-to-Text (STT) provider to transcribe synthesized audio and derive the necessary timing data.
This system ensures that developers can obtain consistent WordTimestamp metadata regardless of whether the chosen TTS model (such as OpenAI TTS-1 or Google Gemini) provides this information out-of-the-box. The fallback is triggered when the timestamps option is set to "on".
Architecture and Dispatch Logic
The SDK follows a specific cascade to resolve word-level timestamps. It first attempts to fetch native alignment from the provider. If the provider lacks native support and the user has explicitly requested timestamps, the SDK initiates an STT round-trip.
The Timestamp Cascade
- Native Support: The SDK checks if the model declares native timestamp capabilities (e.g., ElevenLabs or Cartesia).
- User Override: If no native support exists, the SDK checks for a custom
timestampProviderpassed in thegenerateSpeechoptions. - Default Fallback: If no override is provided, it defaults to OpenAI Whisper (
openai/whisper-1), requiring anOPENAI_API_KEY.
Sources: skills/speech-sdk/references/timestamps.md:27-33, src/generate-speech.ts:167-185
Component Interaction Diagram
The following diagram illustrates how the generateSpeech function interacts with both TTS and STT providers when native timestamps are unavailable.
The flow ensures that expensive STT round-trips are only performed when strictly necessary based on the TimestampMode.
Sources: src/generate-speech.ts:145-188, skills/speech-sdk/references/timestamps.md:19-25
Timestamp Modes and Configuration
The behavior of the fallback system is governed by the TimestampMode type.
| Mode | Behavior |
|---|---|
"auto" (Default) | Returns timestamps only if the provider supplies them natively. No extra API calls or latency. |
"on" | Guarantees timestamps. Uses native if available; otherwise triggers the STT round-trip (Whisper). |
"off" | Suppresses all timestamps, even if the provider supports them for free. |
Sources: README.md:143-149, src/generate-speech.ts:46-52
Configuration Parameters
The generateSpeech function accepts specific options to manage the STT fallback:
timestamps:TimestampMode("on" | "auto" | "off").timestampProvider: AResolvedSTTModelused to override the default Whisper fallback. This can be constructed usingcreateOpenAISTT({ apiKey })("whisper-1").
Sources: src/generate-speech.ts:46-60, skills/speech-sdk/references/timestamps.md:46-55
STT Fallback in Conversations
When using generateConversation, the STT fallback handles multi-turn audio stitching. If the conversation follows the "Stitch Path" (mixing providers or using providers without native dialogue support), the SDK manages offsets for derived timestamps.
Monotonic Timing
For stitched audio, the SDK ensures that word timings are monotonic across turn boundaries. Derived timestamps for each turn are offset by the cumulative duration of previous turns plus the configured gapMs.
Derived timings for the second turn would be adjusted to start at 2.3s in the final concatenated file.
Sources: skills/speech-sdk/references/timestamps.md:65-72, src/tests/e2e/conversation-timestamps.e2e.test.ts:14-41
Implementation Details
The core logic for the fallback is located in src/generate-speech.ts within the generateSpeech function.
Key Internal Functions
deriveTimestampsViaSTT: Called whentimestampMode === "on"and no native timestamps are returned. It pipes the synthesizedUint8Arrayaudio to the STT provider.logTimestampDecision: A debug utility that logs whether the SDK is requesting native alignment or falling back to STT.describeSTTTarget: Identifies the STT model being used for the fallback in debug logs.
Sources: src/generate-speech.ts:175-185, src/generate-speech.ts:223-255
Default Provider (OpenAI Whisper)
The SDK includes a built-in provider for OpenAI Whisper. It targets the v1/audio/transcriptions endpoint and expects a verbose_json response format to extract word-level segments.
Sources: src/tests/e2e/openai.e2e.test.ts:133-145, skills/speech-sdk/references/timestamps.md:32-33
Error Handling
| Error | Scenario |
|---|---|
TimestampKeyMissingError | Thrown when timestamps: "on" is used but no OPENAI_API_KEY is found for the default Whisper fallback. |
ApiError | Propagated from the underlying STT provider (e.g., 401 Unauthorized or 429 Rate Limit) during the transcription round-trip. |
Sources: skills/speech-sdk/references/timestamps.md:92-96, README.md:275-276
Summary
The STT Fallback system provides a critical bridge for feature parity across the 13+ supported TTS providers. By leveraging OpenAI Whisper or custom STT models, the Speech SDK ensures that word-level alignment is always available for applications requiring captions, highlights, or synchronized animations, even when the primary speech engine is audio-only.