Voice Patching
Relevant source files
The following files were used as context for generating this wiki page:
Voice Patching is a specialized utility within the Piper text-to-speech engine designed to modify ONNX voice models to support phoneme alignment output. By default, many Piper models only output raw audio; patching enables the extraction of duration data for each phoneme id, allowing applications to synchronize visual elements or highlighting with the generated speech.
This process involves identifying internal model tensors that calculate phoneme widths and explicitly marking them as graph outputs. This structural modification allows the inference engine to return a second tensor containing sample counts per phoneme alongside the primary audio data.
Sources: src/piper/patch_voice_with_alignment.py:1-5, CHANGELOG.md:32-32
Alignment Extraction Logic
The patching process focuses on the Ceil operator within the ONNX computational graph. These operators typically represent the duration calculation where phoneme widths are scaled and rounded. By exposing the output of these nodes, Piper can retrieve the exact number of audio samples attributed to each phoneme id.
Automated Tensor Detection
The patching tool attempts to autodetect the correct alignment tensor by scanning for Ceil nodes. If multiple candidates exist or none are found, manual overrides via the --tensor-name parameter are required.
The flow above illustrates the logic used to identify and expose internal duration tensors.
Sources: src/piper/patch_voice_with_alignment.py:35-77
Integration with Inference Engines
Once a model is patched, both the Python and C++ inference implementations can utilize the additional output tensor to provide detailed PhonemeAlignment data.
Python Implementation
In the Python API, PiperVoice.synthesize checks if the model returns more than one tensor. If it does, the second tensor is treated as phoneme id samples. These samples are then mapped back to the original phonemes (including BOS, PAD, and EOS tokens) to create a list of PhonemeAlignment objects.
| Component | Description |
|---|---|
PhonemeAlignment | Data class containing a phoneme string, its associated IDs, and total sample count. |
AudioChunk.phoneme_id_samples | The raw numpy array of sample counts directly from the patched ONNX output. |
AudioChunk.phoneme_alignments | The high-level mapping of text phonemes to audio durations. |
Sources: src/piper/voice.py:38-43, src/piper/voice.py:440-480
C++ (libpiper) Implementation
The C++ library similarly detects the presence of the second output tensor during the piper_synthesize_next call. It scales the raw width values by the model's hop_length (default 256) to convert them into absolute sample counts.
The sequence shows how the C++ API handles the dual-output nature of patched models.
Sources: libpiper/src/piper.cpp:337-353, libpiper/include/piper_impl.hpp:32-32
Utility Configuration
The patch_voice_with_alignment.py script provides the following CLI options for model modification:
| Argument | Description | Default |
|---|---|---|
model | Path to the input ONNX voice model. | Required |
--output | Path to write the patched model. | Overwrites input |
--tensor-name | Manually specify the tensor to mark as output. | Autodetect (Ceil) |
Sources: src/piper/patch_voice_with_alignment.py:20-29
Verification and Testing
Patched models are verified by checking that the number of alignment samples matches the number of phoneme IDs generated for a given sentence. Testing involves mocking the model output to ensure the alignment logic correctly groups IDs and their corresponding durations into logical phoneme boundaries.
# From test_synthesize_alignment
audio_chunks = list(voice.synthesize("Test 1."))
assert audio_chunks[0].phoneme_id_samples is not None
assert len(expected_alignments) == len(audio_chunks[0].phoneme_id_samples)
Sources: tests/test_piper.py:277-285
Summary
Voice Patching is a critical utility for advanced TTS applications requiring word or phoneme-level timing. By manipulating the ONNX graph to expose internal duration calculations, Piper transforms a standard audio-only model into a feature-rich synthesis engine capable of providing precise speech-to-text alignments. This enables features such as synchronized lip-syncing or karaoke-style text highlighting.