Tokenization Logic
Relevant source files
The following files were used as context for generating this wiki page:
Tokenization Logic in the Wikispeech server is the foundational process of breaking down raw text or structured SSML input into a hierarchical data structure suitable for speech synthesis. This process transforms input strings into paragraphs, sentences, phrases, and individual tokens, often enriching them with phonetic or linguistic metadata.
The system supports multiple entry points for tokenization, including a "basic" Python-based regex tokenizer and external adapters like MaryTTS. It is a critical step in the Textprocessing pipeline, ensuring that subsequent components like lexicon lookups or G2P (Grapheme-to-Phoneme) converters can operate on discrete, well-defined linguistic units.
Sources: wikispeech_server/tokeniser.py, wikispeech_server/wikispeech.py:348-355
Hierarchical Data Structure
The primary output of the tokenization logic is a "Utterance" (utt) object. This nested dictionary structure represents the linguistic hierarchy of the input text.
| Level | Description | Key in Parent |
|---|---|---|
| Utterance | The root object containing all processed data. | N/A |
| Paragraph | Large blocks of text, often separated by double newlines or <p> tags. | paragraphs |
| Sentence | Discrete grammatical units within a paragraph. | sentences |
| Phrase | Sub-sentence units often delimited by punctuation like commas or semicolons. | phrases |
| Token | The smallest unit, typically a word or punctuation mark, including its orthography. | tokens |
Sources: wikispeech_server/tokeniser.py:47-57, wikispeech_server/adapters/marytts_adapter.py:273-285
Internal Tokenization Process
The internal tokenise function uses regular expressions to segment text. It identifies boundaries for paragraphs, sentences, and phrases based on standard punctuation and specific language rules (e.g., Basque Spanish characters).
Segmentation Rules
The tokenizer uses the following regex patterns to identify boundaries:
- Paragraphs: Split by
</p>tags or double newlines (\n\n). - Sentences: Split by
</s>tags or specific terminal punctuation (.?!:) followed by a space and an uppercase letter or number. - Phrases: Split by
</phrase>tags or punctuation like),;-. - Tokens: Split by
</token>tags or whitespace.
Sources: wikispeech_server/tokeniser.py:37-43, wikispeech_server/tokeniser.py:84-90
Punctuation Handling
Tokens are further processed to strip and categorize punctuation. The system distinguishes between:
- prepunct: Punctuation at the start of a token (e.g., the
(in(hello). - orth: The core word or symbol.
- punct: Punctuation at the end of a token (e.g., the
!inworld!).
The diagram shows the top-down flow from raw text to the final structured JSON format.
Sources: wikispeech_server/tokeniser.py:126-140
External Adapter Tokenization
In addition to the internal logic, tokenization can be performed by external services through adapters.
MaryTTS Tokenization
The marytts_adapter provides a marytts_preproc function. This sends the input text to a MaryTTS server with the OUTPUT_TYPE set to INTONATION or ALLOPHONES. The resulting MaryXML is then parsed back into the standard Wikispeech hierarchy.
- MTU Handling: The adapter specifically handles Multi-Token Units (
<mtu>), where a single original orthography maps to multiple expanded tokens. - SSML Mapping: If the input is SSML, the adapter can map existing phoneme transcriptions into a format MaryTTS understands before tokenizing the rest of the text.
Sources: wikispeech_server/adapters/marytts_adapter.py:65-96, wikispeech_server/adapters/marytts_adapter.py:302-330
JavaScript Implementation (Client-Side)
In the workflow demo, tokenization results are handled by getSentenceAndTokens. This logic flattens the nested paragraphs and sentences into a list of words for the Lexicon tab, while tracking frequency and g2p methods.
This sequence illustrates how client-side scripts interact with the server's tokenization endpoints.
Sources: workflow_demo/test_workflow.js:296-339
API Configuration
Tokenization behavior is controlled through parameters passed to the /textprocessing/ endpoint.
| Parameter | Type | Description |
|---|---|---|
input | String | The text or SSML to be tokenized. |
lang | String | ISO 639-1 code (e.g., "en", "sv"). |
textprocessor | String | The name of the textprocessor config (includes the tokenizer module). |
process | String | Set to tokenise for partial processing (returns structure before lexicon lookup). |
Sources: wikispeech_server/options.py:73-100, wikispeech_server/wikispeech.py:461-464
Data Enrichment and Cleanup
Post-tokenization, tokens are often cleaned or prepared for lexicon lookup:
- Case Normalization: Tokens are usually converted to lowercase for dictionary matching.
- Orthography Cleanup: Soft hyphens (
\xad) and Arabic diacritics (Tashkeel) are removed to ensure consistent lookup keys. - G2P Tagging: If a token originates from an SSML
<phoneme>tag, it is marked asinput_ssml_transcriptionto prevent it from being overwritten by general G2P rules.
Sources: wikispeech_server/adapters/lexicon_client.py:17-43, wikispeech_server/adapters/lexicon_client.py:73-81
Conclusion
Tokenization Logic within Wikispeech is a multi-layered system that converts unstructured text into a highly granular hierarchy. By supporting both standard regex-based segmentation and advanced XML parsing from engines like MaryTTS, the system provides a flexible framework for multi-lingual speech synthesis preparation. This structure allows subsequent modules to focus on specific linguistic tasks like phonetic transcription and timing estimation.