Adding Custom Providers

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• src/speech-provider.ts
• src/providers/openai/index.ts
• src/providers/elevenlabs/index.ts
• src/providers/google/index.ts
• src/providers/cartesia/index.ts
• package.json

The Speech SDK is designed with a provider abstraction layer that enables universal support for multiple Text-to-Speech (TTS) services. Developers can add new providers by implementing the SpeechProvider interface, which standardizes how the SDK interacts with various external APIs. This architecture ensures that the public API remains consistent regardless of the underlying service used.

Sources: AGENTS.md:15-18, src/speech-provider.ts

Core Architecture and Data Flow

The SDK follows a structured flow where the public generateSpeech() function resolves a model string to a specific provider instance. The provider then handles the transformation of text into audio data.

This diagram shows the high-level request flow from the public entry point to the provider-specific implementation.
Sources: AGENTS.md:18-20, src/providers/openai/index.ts

Implementation Steps

To add a new provider to the SDK, follow these four primary steps:

1. Implement the SpeechProvider Interface

Create a new directory in src/providers/<name>/ and implement the SpeechProvider interface. This class must handle authentication, request formatting, and response parsing for the specific provider.

Key members of the interface include:

• id: A unique string identifier for the provider.
• defaultModel: The fallback model ID used if none is specified.
• generate(): The core method for producing a single audio utterance.
• stream(): (Optional) A method for returning a ReadableStream of audio bytes.
• processAudioTags(): (Optional) Logic to handle or strip bracketed expressive cues like [laugh].

Sources: AGENTS.md:28-31, src/speech-provider.ts, src/providers/cartesia/index.ts:153-176

2. Create a Factory Function

Providers are typically instantiated via a factory function (e.g., createOpenAI, createElevenLabs). This function accepts a configuration object and returns a function that produces a ResolvedModel.

export function createCustomProvider(config: Config = {}) {
  const provider = new CustomProvider(providerConfig);

  return function custom(modelId?: string): ResolvedModel<string> {
    return {
      provider,
      modelId: modelId ?? provider.defaultModel,
    };
  };
}

Sources: AGENTS.md:33-35, src/providers/google/index.ts:333-341

3. Register with resolve-provider.ts

The internal resolution logic must be updated to recognize the new provider. Add a case for the provider in the createBuiltinProvider() function within src/resolve-provider.ts. This allows the SDK to automatically instantiate the provider when a string like "my-provider/model-id" is passed to generateSpeech().

Sources: AGENTS.md:31, src/resolve-provider.ts

4. Export the Provider

Update package.json to include the new provider under the exports field. This ensures that users can import the specific provider factory directly if needed.

Sources: AGENTS.md:32, package.json:20-75

Provider Capability Matrix

Providers can optionally support advanced features by defining them in their models array and implementing specialized methods.

Feature	Method / Property	Description
Streaming	stream()	Returns a ReadableStream<Uint8Array>.
Audio Tags	processAudioTags()	Processes [tag] syntax (e.g., mapping to SSML or API fields).
Native Dialogue	generateDialogue()	Uses a single provider endpoint for multi-speaker output.
Timestamps	includeTimestamps	Returns word-level alignment data natively.

Sources: src/speech-provider.ts, src/providers/elevenlabs/index.ts:161-175, src/providers/google/index.ts:145-155, src/providers/cartesia/index.ts:25-50

Communication and Security

Providers are responsible for resolving their own API keys, typically using the resolveApiKey utility. This utility looks for a provided key in the configuration or falls back to an environment variable (e.g., OPENAI_API_KEY).

This sequence diagram illustrates how authentication and communication are handled during the generation process.
Sources: src/provider-utils.ts, src/providers/openai/index.ts:54-60, src/providers/google/index.ts:182-185

Testing Requirements

Every new provider must include a suite of unit tests in src/__tests__/<name>-speech-model.test.ts. These tests should verify:

• Correct URL construction.
• Proper header inclusion (Authorization, X-User-Agent, Content-Type).
• Correct body mapping, including providerOptions passthrough.
• Accurate response parsing for both audio bytes and metadata.
• Error handling for non-2xx responses using handleErrorResponse.

Sources: AGENTS.md:14-16, src/providers/openai/index.ts:104, src/providers/google/index.ts:221

Conclusion

Adding a custom provider involves implementing a standardized interface and registering it within the SDK's resolution logic. By adhering to the SpeechProvider pattern, new services can immediately benefit from the SDK's built-in retry logic, volume normalization, and multi-speaker stitching capabilities.