fix(rag): tokenize special-token spellings as document text
PR Checklist
- Conventional commit message
- Regression tests added
- Changeset added
- Docs updated (no new API; restores ordinary document ingestion)
What is the current behavior?
A document containing the literal string <|endoftext|> or <|fim_prefix|> makes createTikTokenizer().countTokens(), .tokenize() and new TokenChunker().chunk() throw The text contains a special token that is not allowed. This prevents RAG ingestion of otherwise ordinary text, for example documentation about tokenizers.
What is the new behavior?
Use js-tiktoken's ordinary-text encoding mode (no allowed or disallowed special tokens). These spellings stay literal document content rather than becoming special token IDs. The public regression verifies that <|endoftext|> counts as seven ordinary tokens with both cl100k_base and o200k_base.
new TokenChunker().chunk('Text about <|endoftext|>.', { maxTokens: 4 });
The chunks now reconstruct the source text instead of throwing. Ordinary documents retain the same tokenization.
Notes for reviewers
- Against main
e9fc599, the public-export regression has 6 failures / 2 controls before, all 8 pass after. Full RAG suite: 55 passed. - CJS, ESM and DTS builds pass. Changed-file Biome and
git diff --checkpass. - Full package
tsc --noEmitreports the same three pre-existing diagnostics on the exact baseline and this change:code-chunker.spec.ts:60andlate-chunker.spec.ts:16,27. No new diagnostic; this check is not claimed as passing. - The public tokenizer/chunker and real js-tiktoken run locally; no model or provider service is involved.
- #1510 addresses multibyte character offsets in the decoder loop. This is independent: its actual patch leaves the encoding call unchanged. This PR does not claim to fix those existing Unicode offset cases.
AI assistance (Codex) was used for investigation, implementation, review and tests. No human code review is claimed.
Summary by cubic
Stops RAG token counting and chunking from throwing on documents containing literal special-token spellings such as <|endoftext|> or <|fim_prefix|>.
- Switches encoding to js-tiktoken's ordinary-text mode so these spellings remain literal document content, while ordinary documents keep the same tokenization.
- Adds regression tests verifying
<|endoftext|>counts as seven tokens under bothcl100k_baseando200k_base, and that chunked output reconstructs the source text.
Summary by CodeRabbit
- Bug Fixes
- Token spellings such as
<|endoftext|>and<|fim_prefix|>in document text are now treated as ordinary text during token counting and chunking. - Chunking preserves the original text and respects token limits when these spellings appear.
- Token spellings such as