VITS Model Structure
Relevant source files
The following files were used as context for generating this wiki page:
The VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) model structure in Piper is a high-performance, end-to-end architecture designed for local neural text-to-speech synthesis. It integrates a conditional variational autoencoder with adversarial training to generate high-quality audio directly from phoneme sequences.
In the Piper project, the VITS model is utilized both for training (via PyTorch Lightning) and for efficient inference (via ONNX Runtime). The architecture supports multi-speaker synthesis and provides fine-grained control over speech characteristics such as length and noise scales.
Sources: src/piper/train/vits/config.py, src/piper/voice.py:126-140
Core Architecture Overview
The VITS model consists of several interconnected modules that handle the transition from symbolic text representation to raw waveforms. The primary components include a text encoder, a posterior encoder (used during training), a stochastic duration predictor, and a decoder (generator).
The diagram above illustrates the high-level data flow where phonemes and speaker identities are processed to produce audio.
Sources: src/piper/train/vits/config.py:72-87, src/piper/train/export_onnx.py:53-65
Key Components and Modules
Text Encoder and Prior
The text encoder processes phoneme IDs into a hidden representation. It uses an attention-based architecture to capture long-range dependencies in the input text. The encoder's output serves as the mean and variance for the prior distribution in the variational autoencoder.
- Input: Phoneme sequences (sequences of integers).
- Parameters:
num_symbols,inter_channels,hidden_channels,filter_channels,n_heads,n_layers. - Functionality: Maps discrete phonemes to a continuous latent space.
Sources: src/piper/train/vits/config.py:73-81, src/piper/train/vits/dataset.py:480-500
Decoder (Generator)
The decoder is responsible for converting the latent representation $z$ into the final audio waveform. It employs upsampling layers and residual blocks to generate high-fidelity audio. Piper supports different quality configurations for the decoder.
| Configuration | Resblock Type | Upsample Rates | Initial Channels |
|---|---|---|---|
| Low Quality | 2 | (8, 8, 4) | 256 |
| High Quality | 1 | (8, 8, 2, 2) | 512 |
Sources: src/piper/train/vits/config.py:27-58
Multi-Speaker Support
The model supports multiple speakers by using a speaker embedding. When n_speakers > 1, a Speaker ID (sid) is passed to the encoder, flow-based prior, and decoder to condition the synthesis on a specific voice.
- Speaker ID Map: Maps speaker names to integer IDs.
- Global Information (gin): When multi-speaker is enabled,
gin_channelsis typically set to 512 to handle speaker conditioning.
Sources: src/piper/train/vits/config.py:84-87, src/piper/train/vits/dataset.py:129-145
Synthesis Logic and Inference
During inference, the posterior encoder is bypassed. The model samples from the prior distribution and uses the predicted durations to align the text features before decoding.
The sequence above shows the transition from raw text to the generation of an AudioChunk using the ONNX model.
Sources: src/piper/voice.py:214-250, src/piper/train/export_onnx.py:46-65
Inference Parameters
Inference can be tuned using several scales defined in the model configuration:
- Noise Scale (
noise_scale): Controls the variance of the generated audio (default: 0.667). - Length Scale (
length_scale): Controls the speed of speech; >1.0 is slower, <1.0 is faster (default: 1.0). - Noise Scale W (
noise_w_scale): Controls the stochasticity of the duration predictor (default: 0.8).
Sources: src/piper/voice.py:382-415, src/piper/train/vits/config.py:148-151
Training Data Module
The VitsDataModule handles the preparation of training data, including audio normalization, silence trimming via VAD (Voice Activity Detection), and spectrogram generation.
| Data Type | Description | Source File Reference |
|---|---|---|
| Phoneme IDs | Padded LongTensors of phoneme indices | dataset.py:480 |
| Spectrograms | Mel-spectrograms for posterior conditioning | dataset.py:504 |
| Speaker IDs | Optional integer IDs for multi-speaker | dataset.py:508 |
| Audio | Normalized 22050Hz (default) audio waveforms | dataset.py:101 |
Sources: src/piper/train/vits/dataset.py:45-120, src/piper/train/vits/dataset.py:475-520
The VITS model structure in Piper provides a robust framework for high-quality text-to-speech. By separating the architecture into discrete modules for text encoding, duration prediction, and audio decoding, the system achieves a balance between synthesis flexibility and execution efficiency.