Word-Level Timestamps

Relevant source files

The following files were used as context for generating this wiki page:

• skills/speech-sdk/references/timestamps.md
• src/generate-speech.ts
• src/providers/resemble/alignment.ts
• src/providers/murf/alignment.ts
• src/providers/inworld/alignment.ts
• src/providers/hume/alignment.ts
• src/captions.ts
• README.md

Word-level timestamps provide precise alignment between synthesized audio and the corresponding text. This feature allows developers to identify exactly when each word begins and ends within the generated audio file, enabling use cases such as word highlighting, closed captioning, and synchronized animations. Timings are provided at word granularity, with start and end values represented in seconds from the beginning of the audio. Sources: skills/speech-sdk/references/timestamps.md:3-5, README.md:158-160

The SDK implements a "cascade" logic to resolve these timestamps, prioritizing native provider data when available and falling back to Speech-to-Text (STT) processing when necessary. This functionality is supported in both single-utterance generation via generateSpeech and multi-speaker dialogues via generateConversation. Sources: skills/speech-sdk/references/timestamps.md:19-25, src/generate-speech.ts:167-175

Timestamp Resolution Logic

The SDK uses a tiered approach to determine how to obtain word-level alignment based on the requested TimestampMode.

Modes of Operation

Mode	Behavior
"auto"	Returns timestamps only if the TTS provider supplies them natively. No extra API calls or STT fallbacks are triggered.
"on"	Guarantees timestamps. It uses native alignment if available; otherwise, it transcribes the synthesized audio using an STT provider.
"off"	Suppresses all timestamp generation, even if the provider would have returned them natively.

Sources: skills/speech-sdk/references/timestamps.md:11-17, src/generate-speech.ts:36-42

The Resolution Cascade

When timestamps are requested, the generateSpeech function follows a specific sequence to resolve the data:

1. Native Support: If the model declares native timestamp support, the SDK includes a request flag (e.g., includeTimestamps: true) in the provider's synthesis call. Sources: src/generate-speech.ts:77-80
2. User Override: If native alignment is missing and mode is "on", the SDK checks for a user-provided timestampProvider (a ResolvedSTTModel). Sources: src/generate-speech.ts:46-51
3. Default Fallback: If no override exists, it defaults to OpenAI Whisper (openai/whisper-1). This requires an OPENAI_API_KEY. Sources: skills/speech-sdk/references/timestamps.md:25-27, src/generate-speech.ts:210-213

The following diagram illustrates the decision flow for timestamp acquisition:

Provider-Specific Implementations

Different providers return alignment data in various formats, which the SDK normalizes into a standard WordTimestamp[] structure.

Supported Native Providers

Provider	Mechanism	Normalization Logic
ElevenLabs	/with-timestamps	Splitting character-level alignment into words based on spaces.
Murf	wordDurations	Converts startMs and endMs to seconds.
Inworld	wordAlignment	Zips parallel words, wordStartTimeSeconds, and wordEndTimeSeconds arrays.
Hume	snippets	Flattens nested snippets and filters for type: "word".
Resemble	audio_timestamps	Aggregates grapheme-level characters and times into word boundaries.
Cartesia	/tts/sse	Merges interleaved chunk and timestamp events from the SSE stream.

Sources: README.md:179-191, src/providers/murf/alignment.ts:24-34, src/providers/inworld/alignment.ts:24-43, src/providers/hume/alignment.ts:34-58

Resemble Grapheme Aggregation

Resemble returns alignment at the grapheme (character) level. The SDK walks the graph_chars array, accumulating non-whitespace characters into words while attaching punctuation to the adjacent word to match the behavior of other providers. Sources: src/providers/resemble/alignment.ts:33-40

Caption Generation

The SDK provides a utility, timestampsToCaptions, to convert the raw WordTimestamp[] into standard subtitle formats. It supports SubRip (SRT) and WebVTT (VTT). Sources: README.md:198-204

Formatting and Constraints

• SRT: Uses comma-decimal separators (00:00:00,000). Sources: src/captions.ts:16-18
• VTT: Uses period-decimal separators (00:00:00.000) and includes the required WEBVTT header. Sources: src/captions.ts:20-22, README.md:223-228
• Segmentation: Cues break primarily on sentence boundaries (., !, ?). Sources: README.md:238-239
• Constraints: Long sentences are subdivided based on maxCharsPerCue, maxCueDurationMs, and longPhraseCommaBreakChars. Sources: README.md:240-241

Data Structures

interface WordTimestamp {
  readonly text: string;
  readonly start: number; // seconds
  readonly end: number;   // seconds
}

Sources: skills/speech-sdk/references/timestamps.md:83-87, skills/speech-sdk/references/result.md:17-21

Error Handling

When using the derived timestamp path (timestamps: "on"), specific errors may occur:

Error	Cause
TimestampKeyMissingError	Triggered when an STT fallback is required but no OPENAI_API_KEY (or custom provider key) is configured.
ApiError	Propagated from the underlying STT service if the transcription call fails.

Sources: skills/speech-sdk/references/timestamps.md:94-98, README.md:307

Summary

Word-level timestamps in the Speech SDK provide a unified interface for accessing alignment data across multiple TTS providers. By combining native provider capabilities with an intelligent STT fallback mechanism, the SDK ensures that developers can obtain precise word timings regardless of the underlying engine's native features, facilitating the creation of synchronized audio-visual experiences. Sources: skills/speech-sdk/references/timestamps.md:19-32, README.md:158-165