Advanced Queries & CJK Handling
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Advanced Queries and CJK (Chinese, Japanese, Korean) handling in cxs are designed to provide precise retrieval of local Codex session logs while navigating the limitations of standard SQLite Full-Text Search (FTS). The system employs a progressive retrieval strategy that moves from broad session discovery to local context anchoring.
The scope of this module includes query tokenization, multi-lingual text processing, and heuristic reranking. By implementing custom tokenization for CJK characters and specific field weighting for session metadata, cxs ensures that even brief or non-English queries can locate relevant historical discussions. Sources: README.md:143-156, skill-packages/cxs/references/advanced-queries.md:3-8
Query Processing & Tokenization Logic
The cxs engine does not pass user input directly to SQLite FTS. Instead, it processes queries through a tokenizer that enforces specific logical constraints to improve accuracy.
Tokenization Workflow
- Term Processing: Every term in the query is wrapped in double quotes.
- Logical Conjunction: Terms are joined using the
ANDoperator, meaning all provided keywords must exist in the candidate messages or session metadata. - Heuristic Reranking: While FTS handles initial recall, the original full sentence is used during the reranking phase to maintain the relevance of natural phrases.
The diagram shows the transition from raw user input to processed tokens and finally to reranked results. Sources: skill-packages/cxs/references/advanced-queries.md:5-12, AGENTS.md:15-20
CJK Character Handling
Standard tokenization often fails with CJK characters due to the lack of whitespace. cxs implements a specific strategy for these languages:
- Bigram Segmentation: Continuous CJK strings are split into overlapping bigrams.
- Single Character Limitation: A single CJK character does not form a valid FTS term.
- LIKE Fallback: If tokenization results in an empty set for a CJK query, the system falls back to a bounded
LIKEscan to ensure recall.
Sources: skill-packages/cxs/references/advanced-queries.md:18-24, AGENTS.md:27-30
Retrieval Architecture & Ranking
cxs utilizes a multi-layered retrieval chain to balance performance and accuracy. Recall is performed across both individual messages and session-level summaries.
Field Weighting in Session FTS
When performing session-level recall, different metadata fields are assigned weights to prioritize titles and summaries over raw reasoning logs.
| Field | Weight | Description |
|---|---|---|
title | 8.0 | The primary session title. |
compact_text | 4.0 | Compacted session representation. |
summary_text | 3.0 | Derived summary of the conversation. |
reasoning_summary_text | 1.2 | Summary of the AI's internal reasoning. |
Sources: AGENTS.md:31-34, README.md:149-152
Progression Flow
Retrieval follows a strict "progressive" workflow to avoid overwhelming the agent with irrelevant data:
The sequence illustrates the escalation from a low-cost database search to a higher-cost full-page read from the raw session logs. Sources: README.md:144-148, skill-packages/cxs/references/progressive-workflow.md:3-10
Advanced Command Parameters
The find and list commands provide specific flags to handle complex retrieval scenarios, such as finding the "latest" discussion rather than the "most relevant."
Finding "Latest + Keyword"
By default, find sorts by relevance. To find the most recent occurrence of a topic, the --sort ended flag must be used. Additionally, the --exclude-session flag is used to prevent the current active session from appearing in the search results (self-hit).
# Search for 'xsearch', sort by latest, and exclude current session
cxs find "xsearch" --sort ended --exclude-session <current_uuid> -n 5 --json
Sources: src/cli.ts:101-106, skill-packages/cxs/references/cli-surface.md:47-55
Constraining Scope with Selectors
Queries can be scoped using structured JSON selectors to improve performance and precision.
| Selector Kind | Purpose |
|---|---|
cwd | Constrains search to a specific project directory. |
date_range | Constrains search to a specific timeframe. |
cwd_date_range | Combines project and time constraints. |
Sources: src/cli.ts:98-100, README.md:96-103
Summary
Advanced Queries and CJK Handling in cxs provide a robust framework for excavating historical context from Codex sessions. By combining CJK-aware tokenization, weighted field ranking, and a progressive read workflow, the system ensures that developers and agents can precisely locate relevant configurations or discussions without the "cold start" problem of reading raw JSONL files. Sources: skill-packages/cxs/SKILL.md:5-15, AGENTS.md:15-20