Tokenizers
Relevant source files
The following files were used as context for generating this wiki page:
Tokenizers convert raw text into numerical sequences (tokens) that Large Language Models can process and decode model outputs back into human-readable text. Within the oxidize project, the tokenization system supports multiple encoding algorithms, special token handling, and streaming detokenization across Rust, C, C++, Go, and Python implementations.
The system handles Byte-Pair Encoding (BPE), SentencePiece, WordPiece, and Tiktoken formats. It interfaces directly with GGUF model metadata to automatically configure the correct tokenizer for a given model architecture.
Sources: oxidize-core/src/format/tokenizer.rs:1-20, oxidize-c/include/oxidize/tokenizer.h:1-15, PRD.md:255-265
Architecture and Core Components
The tokenization logic is centralized in the LoadedTokenizer enum, which dispatches requests to specific format implementations. The architecture prioritizes round-trip parity between different language ports to ensure consistent model behavior across platforms.
The diagram shows the flow from model metadata to specific tokenizer implementations and their primary operations.
Sources: oxidize-core/src/format/tokenizer.rs:25-50, oxidize-c/include/oxidize/tokenizer.h:74-95
Supported Formats
The system implements four primary tokenization algorithms, typically mapped to specific model families:
| Format | Model Examples | Description |
|---|---|---|
| BPE | GPT-2, Qwen, LFM2 | Byte-level encoding with byte-to-unicode mapping. |
| SentencePiece | Llama, Gemma | Unigram-based segmentation using Viterbi best-path search. |
| WordPiece | BERT | Greedy longest-match with ## continuation markers. |
| Tiktoken | GPT-4, Claude | Raw byte-level BPE without unicode mapping. |
Sources: oxidize-core/src/format/tokenizer.rs:218-228, oxidize-c/include/oxidize/tokenizer.h:10-15
Tokenizer Lifecycle and Configuration
Tokenizers are initialized by reading the tokenizer.ggml.model key from GGUF metadata. This metadata also provides the vocabulary, scores (for unigram models), and merge ranks (for BPE models).
Metadata Configuration Table
| Metadata Key | Type | Purpose |
|---|---|---|
tokenizer.ggml.model | String | Identifies the tokenizer algorithm (e.g., "llama", "gpt2"). |
tokenizer.ggml.tokens | String Array | The full vocabulary of the model. |
tokenizer.ggml.merges | String Array | Pairwise merge rules for BPE/Tiktoken. |
tokenizer.ggml.scores | Float Array | Log-probabilities for SentencePiece unigram. |
tokenizer.ggml.add_bos_token | Boolean | Determines if a Beginning-Of-String token is prepended. |
Sources: oxidize-core/src/format/tokenizer.rs:215-230, oxidize-core/src/format/tokenizer.rs:430-500, oxidize-c/include/oxidize/tokenizer.h:100-110
Special Tokens
The SpecialTokens structure manages reserved IDs for model control. It tracks IDs for unknown, bos (Beginning of String), eos (End of String), pad (Padding), separator, cls, and mask. These IDs are used to apply EncodeOptions during the tokenization process.
Sources: oxidize-core/src/format/tokenizer.rs:168-185, oxidize-c/include/oxidize/tokenizer.h:111-125
Encoding and Decoding Logic
Encoding Process
Encoding converts text into a Vec<u32> of token IDs. For BPE models, the system handles "Special Pieces" (e.g., <|im_start|>) by matching them verbatim before processing surrounding text through byte-level BPE. This prevents the fragmentation of control markers.
Streaming Detokenization
The StreamingDetokenizer handles incremental decoding of model output. It maintains a buffer for pending_bytes to correctly handle UTF-8 characters that might be split across multiple tokens. Text is only emitted once a complete UTF-8 sequence is formed.
Sources: oxidize-core/src/format/tokenizer.rs:115-140, oxidize-c/include/oxidize/tokenizer.h:160-185
Chat Templates
Tokenizers facilitate structured dialogue via chat templates. The system provides a fast-path for ChatML-family templates, converting ChatMessage objects into the canonical <|im_start|>{role}\n{content}<|im_end|>\n format.
// oxidize-core/src/format/tokenizer.rs:248-255
if template.contains("<|im_start|>") {
let mut out = String::new();
for message in messages {
out.push_str("<|im_start|>");
out.push_str(message.role);
out.push('\n');
out.push_str(message.content);
out.push_str("<|im_end|>\n");
}
}
Sources: oxidize-core/src/format/tokenizer.rs:245-265, oxidize-c/include/oxidize/tokenizer.h:145-155, oxidize-cpp/src/train/train_data.cpp:115-130
Advanced Features
- Token Healing: Finds better token boundaries at the end of a prompt to reduce artifacts when the input ends on a partial word.
- Viterbi Segmentation: Used in
SentencePieceUnigramTokenizerto find the most probable sequence of tokens by maximizing total log-probability scores. - GPT-2 Byte Mapping: Maps raw bytes to printable unicode characters (e.g., space to 'Ä ') to ensure control characters are represented in the BPE vocabulary.
Sources: oxidize-core/src/format/tokenizer.rs:85-110, oxidize-core/src/format/tokenizer.rs:670-715, oxidize-c/include/oxidize/tokenizer.h:186-200
Conclusion
The Tokenizer system in oxidize provides a robust, multi-format interface for LLM text processing. By integrating deeply with GGUF metadata and maintaining high-fidelity ports in C, Go, and Python, it ensures that model inference remains consistent regardless of the deployment environment or underlying hardware.