Phonemization Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Phonemization Pipeline in Piper is a critical subsystem responsible for transforming raw input text into a sequence of phoneme identifiers (IDs) that the neural synthesis engine can process. This process involves multiple stages, including text normalization, language-specific phoneme generation (using tools like espeak-ng or specialized Chinese G2P models), and mapping those phonemes to numerical IDs based on a voice-specific configuration.
The pipeline supports different phoneme types: espeak (standard), pinyin (for Chinese), and text (direct IPA). It also handles complex text structures such as raw phoneme blocks wrapped in [[ ]], which allow users to bypass the standard G2P logic for specific words or phrases.
Sources: src/piper/config.py:15-19, src/piper/voice.py:169-204, libpiper/src/piper.cpp:145-185
Pipeline Architecture and Data Flow
The pipeline operates in a sequential manner, moving from high-level text to low-level IDs. The data flow differs slightly between the Python implementation and the C++ library implementation, but the core logic remains consistent.
High-Level Flow
The following diagram illustrates the standard path from text input to phoneme ID generation.
The pipeline begins by determining the PhonemeType from the PiperConfig. For most languages, it routes through the EspeakPhonemizer, which utilizes a C extension (espeakbridge) to interface with the espeak-ng library. For Chinese, it uses a specialized ChinesePhonemizer based on the G2PW model.
Sources: src/piper/voice.py:170-193, src/piper/phonemize_espeak.py:15-20, src/piper/phonemize_chinese.py:180-210
Supported Phonemization Methods
Piper provides three distinct methods for converting text to phonemes, defined by the PhonemeType enum.
| Phoneme Type | Implementation Class | Primary Tool/Model | Description |
|---|---|---|---|
espeak | EspeakPhonemizer | espeak-ng | Standard multi-language IPA-based phonemization. |
pinyin | ChinesePhonemizer | g2pW | Specialized Chinese G2P using neural models for Pinyin. |
text | N/A (Direct) | unicodedata | Direct use of provided text as phonemes (NFD normalization). |
Sources: src/piper/config.py:15-19, src/piper/voice.py:170-176, src/piper/phonemize_chinese.py:173-178
Espeak-ng Integration
The EspeakPhonemizer initializes espeak-ng using a data directory and processes text through the espeakbridge module. It decomposes phonemes into UTF-8 codepoints to ensure that accent characters are treated as separate units.
Key characteristics:
- Language Switch Removal: It filters out language switch flags (e.g.,
(en)) thatespeak-ngmight insert around foreign words. - Sentence Splitting: Logic in
espeakbridge.cidentifies sentence terminators (periods, questions, exclamations) to group phonemes into discrete sentences.
Sources: src/piper/phonemize_espeak.py:22-51, src/piper/espeakbridge.c:64-90
Chinese (Pinyin) Pipeline
The Chinese pipeline performs advanced pre-processing before G2P conversion, specifically handling numerical expansions and temperature formatting.
- Normalization: Converts temperatures (e.g., "-7°C" to "零下七度") and percentages (e.g., "77%" to "百分之七十七").
- G2P Conversion: Uses the
G2PWConverterto generate Pinyin syllables with tones. - Syllable Splitting: Pinyin syllables (like
hang2) are split into initials (h), finals (ang), and tones (2).
Sources: src/piper/phonemize_chinese.py:212-250, src/piper/phonemize_chinese.py:318-333
Phoneme to ID Mapping
Once phonemes are generated as strings or codepoints, they must be converted to integers based on the phoneme_id_map provided in the voice configuration.
Mapping Logic
The system uses special identifiers to mark sentence boundaries and maintain structural consistency:
- PAD (0): Interleaved between every phoneme ID to assist the neural model's attention mechanism.
- BOS (1): Beginning of Sentence.
- EOS (2): End of Sentence.
In the C++ implementation, the mapping process iterates through UTF-8 codepoints and looks up vectors of IDs in the phoneme_id_map. If a codepoint is found, its corresponding IDs are added to the sequence, each followed by a PAD ID.
Sources: libpiper/include/piper_impl.hpp:22-30, libpiper/src/piper.cpp:185-230, src/piper/phoneme_ids.py:84-110
Internal Data Structures
PiperConfig (Python)
The PiperConfig class holds the mapping and phoneme type configuration required for the pipeline.
@dataclass
class PiperConfig:
num_symbols: int
num_speakers: int
sample_rate: int
espeak_voice: str
phoneme_id_map: Mapping[str, Sequence[int]]
phoneme_type: PhonemeType
# ...
Sources: src/piper/config.py:22-45
piper_synthesizer (C++)
The C++ struct maintains the state of the phoneme queue during synthesis.
struct piper_synthesizer {
std::string espeak_voice;
PhonemeIdMap phoneme_id_map;
// ...
std::queue<std::pair<std::vector<Phoneme>, std::vector<PhonemeId>>>
phoneme_id_queue;
};
Sources: libpiper/include/piper_impl.hpp:67-90
Synthesis and Alignment
In the synthesize method of PiperVoice, phonemization is the first step. If requested, the pipeline can also generate alignments between phonemes and audio samples. This is achieved by correlating the output of the ONNX model (which provides sample counts per phoneme ID) back to the original phoneme strings.
The alignment logic must account for the interleaved PAD IDs. It sums the sample counts for a phoneme ID and its subsequent PAD to attribute the total duration to a single phoneme character.
Sources: src/piper/voice.py:218-285, libpiper/src/piper.cpp:385-410
Summary
The Phonemization Pipeline serves as the bridge between human-readable text and the machine-readable numerical sequences required for neural TTS. By supporting multiple engines (espeak-ng, g2pW) and providing a robust mapping system with interleaved padding and boundary markers, it ensures that the synthesis engine receives highly structured and accurate phonetic data.