OpenAI Provider Integration
Relevant source files
The following files were used as context for generating this wiki page:
The OpenAI Provider Integration enables the Speech SDK to interface with OpenAI's Text-to-Speech (TTS) and Speech-to-Text (STT) services. It allows developers to generate high-quality audio from text using models like gpt-4o-mini-tts and tts-1, supporting both standard generation and incremental streaming.
This integration is a core component of the SDK's multi-provider architecture, providing a unified interface for speech synthesis while handling OpenAI-specific logic such as API key resolution, base URL configuration, and specialized audio tag processing for advanced models.
Sources: AGENTS.md:12-14, src/providers/openai/index.ts:1-15
Architecture and Core Components
The OpenAI integration is centered around the OpenAISpeechProvider class, which implements the universal SpeechProvider interface. This ensures consistency across the SDK's 13 supported providers.
Key Classes and Functions
| Component | Description |
|---|---|
OpenAISpeechProvider | The main class implementing the SpeechProvider interface. It manages request construction, authentication, and communication with OpenAI's /audio/speech endpoint. |
createOpenAI() | A factory function used to instantiate the provider with custom configurations like API keys and base URLs. |
buildRequestInput() | An internal method that prepares the payload for OpenAI, specifically handling the mapping of SDK audio tags to OpenAI instructions. |
processAudioTags() | Determines if a model supports native SDK tags or if they should be stripped before processing. |
Sources: src/providers/openai/index.ts:16-160, AGENTS.md:23-28
Model Configuration
The provider supports multiple models, each with distinct features and release dates.
| Model ID | Release Date | Default | Features |
|---|---|---|---|
gpt-4o-mini-tts | 2025-03-20 | Yes | Streaming, Audio Tags, Derived Timestamps |
tts-1 | 2023-11-06 | No | Streaming, Derived Timestamps |
tts-1-hd | 2023-11-06 | No | Streaming, Derived Timestamps |
Sources: src/providers/openai/index.ts:18-112, README.md:55-70
Data Flow and Logic
The integration follows the standard SDK core flow: the user calls a public function, which resolves to the OpenAI provider, which then executes the API request.
Speech Generation Flow
The diagram shows the sequence from the initial SDK call to the final result, highlighting the provider-specific steps.
Sources: AGENTS.md:16-20, src/providers/openai/index.ts:133-162
Audio Tag Transformation
For the gpt-4o-mini-tts model, the SDK supports a bracket syntax (e.g., [cheerfully]) for expressive cues. The provider extracts these tags and converts them into an instructions string passed to the OpenAI API.
This sequence illustrates how inline tags are decoupled from the text and moved into the specific instruction field for compatible OpenAI models.
Sources: src/providers/openai/index.ts:114-131, src/tests/openai-speech-model.test.ts:166-180
Configuration and Options
The OpenAI provider can be configured via a factory function or environmental variables.
Provider Configuration
| Option | Type | Description | Default |
|---|---|---|---|
apiKey | string | OpenAI API Key. | process.env.OPENAI_API_KEY |
baseURL | string | Base URL for API requests. | https://api.openai.com/v1 |
fetch | function | Custom fetch implementation. | globalThis.fetch |
Sources: src/providers/openai/index.ts:10-15, README.md:215-225
Provider Options (Pass-through)
Parameters passed in providerOptions are sent directly to the OpenAI request body without transformation.
await generateSpeech({
model: "openai/tts-1",
text: "Testing speed and format.",
voice: "alloy",
providerOptions: {
speed: 1.2,
response_format: "opus"
},
});
Sources: src/tests/e2e/openai.e2e.test.ts:66-74, src/tests/openai-speech-model.test.ts:60-75
Timestamps (Whisper Fallback)
OpenAI TTS models do not currently provide native word-level alignment. When timestamps: "on" is requested, the SDK performs a "derived" timestamp operation by piping the synthesized audio through OpenAI Whisper.
- Mode
"auto": Returnsundefinedfor OpenAI as there is no native alignment. - Mode
"on": Transcribes synthesized audio viaopenai/whisper-1(derived). - Custom Provider: Users can override the fallback using
timestampProvider.
Sources: README.md:120-135, src/tests/e2e/openai.e2e.test.ts:119-165
Error Handling
The integration uses shared utility functions to handle API failures.
- Retries: The SDK retries 5xx and network errors with exponential backoff.
- 4xx Errors: These are not retried. An
ApiErroris thrown containing the status code and response body. - Abortion: The provider respects
AbortSignalfor cancelling pending requests.
Sources: README.md:255-275, src/providers/openai/index.ts:153-156, src/tests/e2e/openai.e2e.test.ts:51-64
The OpenAI Provider Integration provides a robust and flexible bridge to OpenAI's audio capabilities, maintaining the SDK's promise of a unified API while leveraging model-specific features like gpt-4o-mini-tts instructions and Whisper-based timestamp derivation.