Search & Full Text Search (FTS)
Relevant source files
The following files were used as context for generating this wiki page:
Search provides a way to find stored memories, including observations, session summaries, and user prompts. The system combines lexical Full-Text Search (FTS) using SQLite FTS5 with semantic search using vector embeddings when available.
The system indexes text by character trigrams to support multiple languages and CJK text. It ranks results using the BM25 algorithm for lexical matches and cosine similarity for semantic matches. Users interact with search through the web viewer, Model Context Protocol (MCP) tools, or CLI commands.
Sources: src/search.rs:1-12, src/mcp.rs:136-137
Lexical Search Architecture
The lexical search system uses an SQLite FTS5 virtual table with a trigram tokenizer. This approach ensures that CJK characters are indexed individually and that the system finds documents even with partial matches or different word forms.
Trigram Tokenization
Oboete splits queries into character trigrams. Each run of text between whitespace and punctuation is processed into overlapping three-character pieces.
- Hiragana filtering: All-hiragana trigrams are skipped because they represent common particles and verb endings that match too many documents.
- Case Folding: The index uses Unicode case folding so that matches are case-insensitive.
- Limit: A query uses at most 64 deduplicated trigrams to maintain performance.
Sources: src/search.rs:1-8, src/search.rs:37-65
Query Processing Flow
If a query produces trigrams, they are joined with OR operators and matched against the FTS table. If a query is too short for trigrams, the system falls back to literal LIKE terms for every word, requiring all terms to match.
The diagram shows the logic path taken based on query length and character composition.
Sources: src/search.rs:138-164, src/search.rs:168-185
Semantic Search and Hybrid Fusion
When enabled, semantic search uses the bge-m3 model to generate 1024-dimensional vectors. The system performs a hybrid search by running lexical and semantic searches in parallel and fusing the results.
Vector Retrieval
- Providers: Currently uses Cloudflare Workers AI.
- Optimization: Sign bits of vectors are indexed for Hamming distance search, and then rescored using full fp32 cosine similarity.
- Caching: Normalized fp32 vectors are stored in an
embeddingstable to avoid redundant API calls.
Sources: src/embed.rs:19-32, src/embed.rs:81-125
Reciprocal Rank Fusion (RRF)
The search system fuses lexical and semantic results using Reciprocal Rank Fusion with a constant $k = 60$. It prioritizes "knowledge" documents (observations and summaries) over user prompts.
Sources: src/search.rs:104-135
Data Structures and API
Search Hit Structure
A search result is represented by the Hit struct, which contains metadata and the document content.
| Field | Type | Description |
|---|---|---|
doc | String | Document ID (e.g., o123, s5, p7) |
kind | String | Kind of document (e.g., decision, summary, prompt) |
repo | String | Repository key where the event occurred |
when | String | Formatted local time of the event |
title | String | Document title (empty for summaries/prompts) |
body | String | Full text body of the document |
Sources: src/search.rs:15-22
Search Interface Endpoints
The search system is exposed via a JSON API for the web viewer and an MCP server for agents.
| Endpoint/Tool | Parameters | Source |
|---|---|---|
/api/search | q (query), repo, limit | src/view.rs:207-214 |
mcp:search | query, all, repo, limit | src/mcp.rs:136-159 |
Sources: src/view.rs:207-214, src/mcp.rs:136-159
Result Presentation
Snippet Generation
The system generates a snippet for search hits by finding the passage with the most query terms.
- It flattens the document body and locates all query term occurrences.
- It uses a sliding window of a specified width to find the passage containing the highest count of unique query terms.
- It returns the text within that window, adding ellipses if the passage is truncated.
Sources: src/search.rs:271-309
Web Viewer Integration
The web viewer (app.js) handles search results by rendering them as entries. It initially shows the snippet and provides a "Full text" button that fetches the complete document via the API.
The sequence diagram illustrates the lifecycle of a search request from the user interface.
Sources: assets/viewer/app.js:239-257, assets/viewer/app.js:313-322
Search and FTS in Oboete provide a robust retrieval mechanism that handles technical language and diverse character sets. By combining character-level trigrams with modern vector embeddings, the system ensures that coding agents can accurately recall previous decisions and context.
Sources: src/search.rs:1-12, src/mcp.rs:1-10