Voice Cloning
Relevant source files
The following files were used as context for generating this wiki page:
Voice cloning in the Speech SDK allows developers to generate speech that mimics a specific speaker's characteristics using reference audio data or URLs. Instead of providing a static voice ID string, the system accepts a voice object containing the reference material. This feature is ephemeral; the reference audio is used for the specific generation request and is not typically saved as a permanent voice on the provider's platform.
Sources: skills/speech-sdk/references/voice-cloning.md:1-5, README.md:167-170
Architecture and Data Flow
The SDK abstracts the complexity of different provider implementations by providing a unified Voice type. When generateSpeech is called, the SDK resolves the model and passes the voice object—whether a string, audio bytes, or a URL—to the specific provider implementation.
Voice Data Types
The Voice type is defined as a union that accommodates different cloning methods:
| Type | Description | Structure |
|---|---|---|
| Voice ID | A standard string representing a pre-defined voice. | string |
| Inline Audio | Base64 string or Uint8Array of raw audio bytes. | { audio: string | Uint8Array } |
| Remote URL | A public URL pointing to a reference audio file. | { url: string } |
Sources: skills/speech-sdk/references/voice-cloning.md:31-35, src/speech-provider.ts:10-14
Data Flow Diagram
The following diagram illustrates how the SDK processes different voice inputs during the generation flow.
The SDK handles internal conversions, such as transforming Uint8Array buffers into Base64 strings required by certain APIs like Mistral.
Sources: src/generate-speech.ts:50-55, src/providers/mistral/index.ts:70-85
Provider Implementations
Different providers support cloning through varied mechanisms. The SDK maps the generic Voice object to the provider-specific API fields.
Mistral Cloning
Mistral uses the ref_audio field for cloning. If a voice object with audio is provided, the SDK omits the voice_id and sends the Base64 encoded audio instead.
// Mapping logic in Mistral provider
if (typeof voice === "object" && "audio" in voice) {
const audio = voice.audio;
body.ref_audio = typeof audio === "string" ? audio : btoa(String.fromCharCode(...audio));
} else if (typeof voice === "string") {
body.voice_id = voice;
}
Sources: src/providers/mistral/index.ts:72-82, src/tests/mistral-speech-model.test.ts:48-75
Fal.ai Cloning
Fal support involves sending the model ID in the URL and mapping the voice object to either a voice string field or an audio_url field in the request body.
Sources: src/tests/fal-speech-model.test.ts:85-115, src/providers/fal/index.ts:45-60
Cartesia Cloning
Cartesia supports cloning specifically for the sonic-3 model using audio data. It requires specific output formats (e.g., pcm_f32le) and maps voices to a nested object structure.
Sources: skills/speech-sdk/references/voice-cloning.md:39, src/tests/cartesia-speech-model.test.ts:60-75
Feature Support Matrix
Cloning support varies by provider and specifically by the model used within that provider.
| Provider | Cloning Support | Primary Method | Relevant Models |
|---|---|---|---|
| Cartesia | Yes | Audio Data | sonic-3 |
| Mistral | Yes | Audio Data | All models |
| fal-ai | Yes | URL / Data | Variable (e.g., f5-tts) |
| Fish Audio | Yes | Reference ID / Inline | s2-pro |
| Hume | Yes | Named Voice | octave-2 |
| Resemble | Yes | Voice UUID | default |
Sources: skills/speech-sdk/references/voice-cloning.md:37-45, README.md:175-185
Implementation Example
Cloning via a URL using the fal-ai provider:
import { createFal } from "@speech-sdk/core/fal-ai"
import { generateSpeech } from "@speech-sdk/core"
const fal = createFal()
await generateSpeech({
model: fal("fal-ai/f5-tts"),
text: "Hello from a cloned voice!",
voice: { url: "https://example.com/reference.wav" },
})
Sources: skills/speech-sdk/references/voice-cloning.md:20-29
Conclusion
Voice cloning provides a flexible way to personalize speech output without the overhead of permanent voice management. By supporting both local audio buffers and remote URLs, the SDK ensures compatibility with high-performance edge environments and browser-based applications alike.
Sources: README.md:10-15, skills/speech-sdk/references/voice-cloning.md:1-5