Tokenization System

Relevant source files

The following files were used as context for generating this wiki page:

• src/tokenize.ts
• src/tokenize.test.ts
• skill-packages/cxs/references/advanced-queries.md
• src/query/snippet.ts
• AGENTS.md
• README.md

The Tokenization System in cxs is responsible for transforming raw text from Codex session logs and user queries into searchable tokens. It ensures consistency between the indexing pipeline (where sessions are stored in SQLite FTS) and the query pipeline (where user input is processed to find matches). The system employs a hybrid strategy that handles Western languages via word segmentation and CJK (Chinese, Japanese, Korean) characters through overlapping bigrams to ensure stable recall.

Sources: src/tokenize.ts:12-21, README.md:120-130

Core Tokenization Strategy

The system differentiates between CJK and Non-CJK characters to optimize retrieval performance. Non-CJK text is processed using standard word segmentation, while CJK text utilizes a bigram approach because technical terms are often fragmented too aggressively by standard dictionary-based segmenters.

Segmentation Logic

1. Non-CJK Runs: Uses Intl.Segmenter with granularity: "word" to extract word-like tokens. This preserves identifiers and handles hyphenated terms (e.g., sing-box becomes [sing, box]).
2. CJK Runs: Splits continuous CJK strings into overlapping bigrams. For example, "测试测试" is processed into "测试" and "试测".
3. Noise Reduction: Single CJK characters are dropped as they typically carry no retrieval value and act as noise.

Sources: src/tokenize.ts:1-21, src/tokenize.test.ts:24-26

Data Flow Diagram

The following diagram illustrates how text is processed into a final token list:

This flow ensures that mixed-language strings are handled correctly by toggling between specialized buffers.
Sources: src/tokenize.ts:25-54

Key Functions and Components

The system provides several utility functions to support both indexing and query-time operations.

Function	Purpose	Logic Summary
tokenize(text)	Primary entry point for splitting text.	Uses buffers to separate and process CJK vs. Non-CJK runs.
tokenizedText(text)	Whitespace-joined representation for storage.	Joins tokenize() results with spaces for FTS virtual columns.
queryTerms(query)	Extracts distinct tokens for a query.	Calls tokenize() and deduplicates the resulting array.
hasCjk(text)	Checks for presence of CJK characters.	Uses regex CJK_CHAR to detect Han, Hiragana, Katakana, or Hangul.
isCjkToken(token)	Validates if a token is purely CJK.	Uses regex CJK_ONLY for character validation.

Sources: src/tokenize.ts:25-88

Query Processing Behavior

When a user submits a query, cxs does not pass the raw string directly to SQLite FTS. Instead, it applies the tokenization logic to the query to ensure alignment with the index.

Transformation Rules

• Every term extracted by the tokenizer is enclosed in double quotes.
• Multiple terms are combined using the AND operator.
• Native FTS operators like OR, NEAR, or wildcards (*) are not supported as they are treated as part of the text or ignored if they aren't "word-like".

Sources: skill-packages/cxs/references/advanced-queries.md:3-12

CJK Fallback Mechanism

Because single CJK characters are dropped during tokenization, a query consisting of a single Kanji/Hanzi would result in an empty token list. To handle this, the system provides a fallback to an anchored LIKE scan when hasCjk() is true but the token count is zero.

Sources: AGENTS.md:25-30, skill-packages/cxs/references/advanced-queries.md:16-24

Snippet Highlighting

The tokenization system also informs how search results are highlighted in the UI. Snippets are generated by locating tokens within the source text and wrapping them in <mark> tags.

The system prioritizes phrase matches first, then falls back to scoring windows based on the number of distinct terms and total matched characters within a given slice of text.
Sources: src/query/snippet.ts:18-45, skill-packages/cxs/references/advanced-queries.md:40-45

Technical Implementation Details

The tokenization relies on specific Unicode script patterns to identify CJK characters:

const CJK_CHAR = /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}]/u;
const CJK_ONLY = /^[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}]+$/u;

Non-CJK words are lowercased and punctuation is ignored through the isWordLike property of the Intl.Segmenter segments.

Sources: src/tokenize.ts:2-4, src/tokenize.test.ts:32-34

Conclusion

The Tokenization System is a critical component for ensuring that cxs provides reliable search across diverse Codex session logs. By using a hybrid bigram/word-segmentation approach, it balances the need for technical term precision in Western languages with stable recall for CJK scripts. This consistency between the indexing and query layers allows for predictable retrieval and accurate snippet generation.