Cartesia Provider Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Cartesia Provider Integration enables the Speech SDK to interface with Cartesia's high-performance text-to-speech (TTS) models, specifically the sonic series. This integration supports standard audio generation, progressive streaming, and advanced features like word-level native timestamps and complex audio tags.
The integration is encapsulated within the CartesiaSpeechProvider class, which implements the SpeechProvider interface. It manages authentication via API keys, handles different API endpoints for standard and timestamped requests, and processes audio tags to leverage Cartesia's SSML capabilities.
Sources: src/providers/cartesia/index.ts:25-30, skills/speech-sdk/references/providers/cartesia.md:1-12
Architecture and Data Flow
The Cartesia integration bifurcates its logic based on whether the user requires word-level timestamps. Standard requests use the /tts/bytes endpoint for simplicity and efficiency, while timestamped requests use the Server-Sent Events (SSE) /tts/sse endpoint to receive interleaved audio and timing data.
Request Routing Logic
The diagram illustrates the internal branching logic within the provider. When includeTimestamps is true, the SDK switches from a simple byte response to a stream processing mode that manually assembles raw PCM chunks.
Sources: src/providers/cartesia/index.ts:182-192, src/providers/cartesia/index.ts:223-231
Key Components and Functions
CartesiaSpeechProvider Class
The central class responsible for the integration. It maintains configurations such as apiKey, baseURL, and the fetch implementation.
| Component | Description |
|---|---|
generate() | Main entry point for generating audio. Routes to /tts/bytes or generateWithTimestamps(). |
stream() | Provides progressive audio streaming via the /tts/bytes endpoint. |
processAudioTags() | Translates bracketed tags (e.g., [happy]) into Cartesia-specific SSML or removes unsupported tags. |
getStitchOptions() | Provides configuration for multi-speaker conversation stitching, defaulting to 24kHz PCM. |
Sources: src/providers/cartesia/index.ts:182, src/providers/cartesia/index.ts:275, src/providers/cartesia/index.ts:133, src/providers/cartesia/index.ts:314
Native Timestamps Processing
Cartesia provides native alignment by interleaving chunk and timestamps events over SSE. The SDK uses collectCartesiaSse to drain this stream, base64-decoding audio data and merging timestamp messages.
The sequence shows how the SDK handles the asynchronous arrival of audio and metadata, ensuring they are correctly reassembled before being returned to the caller.
Sources: src/providers/cartesia/index.ts:223-273, src/providers/cartesia/index.ts:335-408
Models and Features
The provider supports two primary models with varying feature sets.
| Model ID | Streaming | Audio Tags | Native Timestamps | Default Sample Rate |
|---|---|---|---|---|
sonic-3 | Supported | Yes (SSML) | Supported | 44,100 Hz |
sonic-2 | Supported | No | Supported | 44,100 Hz |
Sources: src/providers/cartesia/index.ts:34-80, skills/speech-sdk/references/providers/cartesia.md:14-21
Audio Tag Support
sonic-3 supports emotion tags and specific sound effects. The processAudioTags method handles the following:
- Emotions: Over 50 emotions (e.g.,
neutral,angry,excited) are converted to<emotion value="..."/>SSML tags. - Passthrough: The
[laughter]tag is passed through natively to the model. - Unsupported: Unknown tags are stripped from the transcript, and a warning is added to the result metadata.
Sources: src/providers/cartesia/index.ts:82-179, src/tests/cartesia-speech-model.test.ts:137-185
Configuration and Defaults
The provider uses a standard set of default values for audio quality while allowing overrides via providerOptions.
// Default Output Format for /tts/bytes
{
container: "wav",
encoding: "pcm_f32le",
sample_rate: 44_100
}
// Internal Format for /tts/sse (Timestamps)
{
container: "raw",
encoding: "pcm_s16le",
sample_rate: 24_000
}
Sources: src/providers/cartesia/index.ts:194-198, src/providers/cartesia/index.ts:241-245
Authentication
Authentication is handled via the X-API-Key header. The SDK also sends a Cartesia-Version header (fixed to 2025-04-16) and an X-User-Agent for telemetry.
Sources: src/providers/cartesia/index.ts:208-210, src/tests/cartesia-speech-model.test.ts:46-52
Error Handling
The provider leverages handleErrorResponse to process non-2xx API responses. It throws SpeechSDKError for specific Cartesia failures, such as malformed SSE payloads or error events received within the stream.
Sources: src/providers/cartesia/index.ts:311, src/providers/cartesia/index.ts:373-375