Smart Term Expansion
Relevant source files
The following files were used as context for generating this wiki page:
Smart Term Expansion is a specialized NLP preprocessing system within KamehaDB designed to bridge the gap between free-text user queries and canonical data stored in databases. It functions by detecting specific entities in user input—such as country names, states, currencies, or boolean concepts—and expanding them into a list of canonical variants (codes, abbreviations, or alternate names). This ensures that the AI assistant can generate accurate WHERE filters, matching stored values that might differ from the user's literal phrasing (e.g., matching "Germany" against "DE" or "DEU").
The system operates as a client-side or sidecar utility that enriches the prompt sent to the LLM. By providing the model with concrete data variants verbatim, it reduces the model's reliance on internal training data for specific codes (like ISO 3166-1) which can often be unreliable.
Sources: packages/shared/src/term-expansion.ts:1-12, CHANGELOG.md:37-41
Architecture and Data Flow
The Smart Term Expansion logic is centralized in the @kamehadb/shared package, allowing it to be used by both the desktop frontend and the sidecar backend. The process follows a pipeline of tokenization, lookup, and prompt rendering.
Expansion Pipeline
- Tokenization: The raw user text is split into individual tokens and bigrams (pairs of adjacent tokens).
- Stop Word Filtering: Common English words are filtered out to reduce noise, unless they match a known canonical expansion (e.g., "us" as United States).
- Alias Lookup: Tokens are matched against various internal dictionaries (Countries, US States, Currencies, etc.).
- Prompt Injection: The resulting variants are formatted into a human-readable instruction block for the AI model.
The following diagram illustrates the transformation of a user query into an enriched prompt context:
Core Components
Logic and Functions
The system relies on several key exported functions and types to manage the expansion lifecycle.
| Function / Type | Description |
|---|---|
expandTerms(userText: string) | Scans user message and returns an array of TermExpansion objects. |
renderExpansionsForPrompt(expansions[]) | Formats expansions into a string block: - "term" -> "var1", "var2". |
lookup(term: string) | Internal helper that routes a term to the correct alias dictionary based on length and shape. |
TermExpansion (Type) | Interface containing the original term and its expanded variants[]. |
Sources: packages/shared/src/term-expansion.ts:342-378, packages/shared/src/index.ts:3-4
Alias Dictionaries
Smart Term Expansion uses pre-baked maps for various domains to ensure high performance and accuracy without external API calls.
- Country Aliases: Maps names like "germany" to
['germany', 'german', 'de', 'deu', 'deutschland']. - US State Aliases: Supports two-way lookups between abbreviations (e.g., "ca") and full names ("california").
- Currency/Language Aliases: Maps common names to ISO codes and symbols (e.g., "dollar" ->
$,USD). - Boolean Aliases: Maps concepts like "active" or "yes" to
['true', '1', 't', 'active']. - Street Abbreviations: Expands standard address components like "st" to "street" or "ave" to "avenue".
Sources: packages/shared/src/term-expansion.ts:14-232
Implementation Details
Tokenization Strategy
The tokenizer handles nuances like punctuation and multi-word entities. It splits on non-alphanumeric characters but preserves bigrams to capture terms like "United Kingdom" or "New York" that would lose meaning if treated as separate tokens.
function tokenize(text: string): string[] {
// Split on whitespace + punctuation, keep tokens of length >= 2.
return text
.toLowerCase()
.split(/[^a-z0-9À-ɏ]+/i)
.filter((t) => t.length >= 2);
}
Sources: packages/shared/src/term-expansion.ts:314-320
Multi-Domain Conflict Handling
Since abbreviations can be ambiguous (e.g., "DE" could mean Germany or Delaware), the lookup function is designed to return combined results from all matching dictionaries, allowing the SQL generator to use OR logic in filters.
AI Integration
The system is integrated into the ai-chat-panel.tsx workflow indirectly through the sidecar's /ai/chat endpoint. When a user sends a message, the sidecar invokes expandTerms on the input. The resulting expansions are appended to the system prompt, instructing the model to use the provided variants verbatim in its generated WHERE clauses.
This integration provides:
- Case-Insensitivity: Assisting the AI in handling mixed-case data.
- Fuzzy Matching: Helping resolve pluralization or slight phrasal differences.
- Verbatim Accuracy: The model is explicitly told:
use these variants verbatim in WHERE filters.
Sources: apps/desktop/src/components/ai-chat-panel.tsx:327-340, CHANGELOG.md:42-51, packages/shared/src/term-expansion.ts:376-381
Conclusion
Smart Term Expansion serves as a critical translation layer in KamehaDB's AI-driven database exploration. By codifying domain-specific knowledge about countries, states, and technical codes into the expansion dictionaries, the system ensures that user intent is accurately reflected in generated SQL queries, regardless of how the underlying data is keyed. This robust pre-processing significantly improves the reliability of natural language to SQL conversion for real-world datasets.