Audio Tags & Expression

Relevant source files

The following files were used as context for generating this wiki page:

• src/audio-tags.ts
• skills/speech-sdk/references/audio-tags.md
• src/generate-speech.ts
• src/providers/google/index.ts
• src/tests/audio-tags.test.ts
• README.md
• src/tests/cartesia-speech-model.test.ts
• src/tests/elevenlabs-speech-model.test.ts

Audio Tags provide a standardized, bracket-based syntax [tag] to inject expressive cues, non-verbal sounds, or emotional directives into text-to-speech synthesis. This system allows developers to control vocal characteristics like laughter, whispering, or specific emotional tones across different AI voice providers using a unified interface.

The SDK manages these tags through a "progressive enhancement" strategy: it passes tags through to providers that support them natively, transforms them into specific formats (like SSML) for others, or strips them safely to prevent the text-to-speech engine from literally speaking the bracketed instructions.

Sources: README.md:32-33, README.md:154-156, src/generate-speech.ts:161-169

Core Logic & Data Flow

The processing of audio tags occurs during the preprocessing stage of the generateSpeech function. Before the text is sent to a provider's API, the SDK determines how to handle detected tags based on the capabilities of the specific model being used.

Processing Pipeline

1. Detection: The SDK scans the input text for patterns matching [tag].
2. Delegation: If a provider implements a custom processAudioTags method, the SDK delegates handling to the provider.
3. Fallback: If no custom handler exists, the SDK detects tags and strips them if they are not natively supported by the model, issuing warnings to the user.

The diagram above illustrates the decision logic within the generateSpeech pipeline for handling expressive tags.
Sources: src/generate-speech.ts:79-83, src/generate-speech.ts:161-171

Provider Implementation Strategies

Different providers handle tags based on their underlying API capabilities. The SDK categorizes these into four main behaviors:

Provider	Model Example	SDK Behavior	Implementation Detail
OpenAI	gpt-4o-mini-tts	Mapping	Mapped to the native instructions field.
ElevenLabs	eleven_v3	Passthrough	Tags are passed verbatim in the text string.
Google	gemini-3.1-flash	Passthrough	Supports native tags like [whispers] or [laugh].
Cartesia	sonic-3	Transformation	Emotion tags are converted to <emotion> SSML tags.
Generic	Various	Stripping	Tags are removed and added to result.warnings.

Sources: README.md:162-167, src/providers/google/index.ts:122-127, src/tests/cartesia-speech-model.test.ts:123-130

Native Passthrough (ElevenLabs & Google)

For models like eleven_v3 or gemini-3.1-flash-tts-preview, the SDK returns the text unchanged. These models are trained to interpret bracketed text as non-verbal performance cues.
Sources: src/providers/google/index.ts:122-127, src/tests/elevenlabs-speech-model.test.ts:181-186

SSML Transformation (Cartesia)

The Cartesia provider identifies specific emotion tags (e.g., [angry], [sad], [happy]) and converts them into SSML elements. For example, [angry] Hello becomes <emotion value="angry"/> Hello. Unsupported tags like [French accent] are stripped with a warning.
Sources: src/tests/cartesia-speech-model.test.ts:132-140, src/tests/cartesia-speech-model.test.ts:159-165

Stripping and Warnings

For providers that do not support tags, the stripAudioTags utility removes the tags and generates a standardized warning message to ensure the user is aware that the expressive cues will be missing from the generated audio.

// Example Warning Format
"Audio tag [laugh] is not supported by openai/tts-1 and was removed."

Sources: src/audio-tags.ts:28-30, src/tests/audio-tags.test.ts:63-68

Technical Utilities

The SDK provides internal utilities for tag manipulation located in src/audio-tags.ts.

Tag Detection

The detectAudioTags function uses a regular expression to find all instances of text enclosed in square brackets. It handles adjacent tags and tags containing spaces or numbers.
Sources: src/tests/audio-tags.test.ts:5-25

Tag Stripping

The stripAudioTags function performs the following actions:

1. Identifies all [tag] patterns.
2. Removes them from the string.
3. Collapses resulting extra whitespace.
4. Trims leading/trailing whitespace.
5. Returns an object containing the cleaned text and an array of warnings.

Sources: src/tests/audio-tags.test.ts:71-99

Error Handling & Constraints

• Empty Text: If stripping all tags results in an empty string, the SDK throws a NoSpeechGeneratedError.
• Case Sensitivity: Some providers (like Cartesia) handle emotion matching case-insensitively.
• Brackets: The SDK ignores unclosed brackets (e.g., [unclosed text) to avoid stripping legitimate content.

Sources: src/generate-speech.ts:85-90%20to%20avoid%20stripping%20legitimate%20content.%0A%0ASources%3A%20%5Bsrc/generate-speech.ts%3A85-90), src/tests/cartesia-speech-model.test.ts:149-152, src/tests/audio-tags.test.ts:40-42

Summary

The Audio Tags system provides a high-level abstraction for vocal expression. By centralizing the logic for detection, passthrough, and transformation, the SDK ensures that developers can write expressive text once and have it handled appropriately across a diverse range of 13+ text-to-speech providers.