Semantic & Hybrid Search Engine
Relevant source files
The following files were used as context for generating this wiki page:
- crates/mcb-providers/src/hybrid_search/engine.rs
- crates/mcb-providers/src/hybrid_search/mod.rs
- crates/mcb-providers/tests/unit/hybrid_search/hybrid_search_tests.rs
- crates/mcb-utils/src/constants/search.rs
- crates/mcb-domain/tests/integration/semantic_search_workflow.rs
- crates/mcb-server/src/formatter/search.rs
- README.md
The Semantic & Hybrid Search Engine is a core component of the MCB (Memory Context Browser) project, designed to provide high-relevance retrieval of code snippets and memory entries. It bridges the gap between traditional keyword-based retrieval and modern vector-based semantic understanding. By combining BM25 (Best Matching 25) lexical scores with semantic similarity scores from vector embeddings, the engine ensures that searches remain accurate even when specific keywords are missing or when natural language queries are used.
This system is implemented within the mcb-providers crate and integrates with the mcb-domain entities to facilitate repository-level context indexing. It supports multiple embedding providers (such as OpenAI, Ollama, and FastEmbed) and vector stores (Milvus, Qdrant, etc.) to generate and store the semantic component of the search.
Sources: crates/mcb-providers/src/hybrid_search/mod.rs:1-15, README.md:12-25
Architecture and Score Fusion
The engine utilizes a parallel processing and fusion architecture. When a query is received, it is processed through two distinct paths: the Lexical Matcher (BM25) and the Semantic Matcher (Vector Search). The resulting scores are then combined using a weighted fusion formula to produce the final ranking.
Search Data Flow
The following diagram illustrates how the engine processes a query to produce ranked results:
The BM25 score is normalized using a sigmoid function to bring it into a 0-1 range, making it compatible with semantic similarity scores. The final hybrid score is calculated as:
Hybrid Score = (bm25_weight * normalized_bm25) + (semantic_weight * semantic_score)
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:14-35, crates/mcb-providers/src/hybrid_search/engine.rs:100-107
Core Components
1. HybridSearchEngine
The primary orchestrator that implements the HybridSearchProvider trait. It maintains an internal RwLock<HashMap<String, CollectionIndex>> to manage separate search indexes for different projects or collections.
| Feature | Description | Default Value |
|---|---|---|
| BM25 Weight | Influence of keyword matching in the final score | 0.4 (40%) |
| Semantic Weight | Influence of vector similarity in the final score | 0.6 (60%) |
| Collection Support | Supports multiple isolated indexes by name | N/A |
| Normalization | Sigmoid-based scaling for BM25 scores | N/A |
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:43-52, crates/mcb-utils/src/constants/search.rs:14-17
2. BM25 Scorer
The lexical component of the engine. It calculates scores based on term frequency (TF) and inverse document frequency (IDF).
- Term Frequency Saturation (
k1): Controls how quickly the importance of a term plateaus as its frequency increases. Default: 1.2. - Length Normalization (
b): Adjusts scores based on the length of the document relative to the average document length. Default: 0.75.
Sources: crates/mcb-providers/src/hybrid_search/mod.rs:42-51, crates/mcb-utils/src/constants/search.rs:8-11
Indexing Process
Before searching, code chunks must be indexed. The engine performs deduplication using a unique key derived from the file path and line numbers.
Result Formatting and User Experience
The search results are formatted into a human-readable structure by the mcb-server crate before being presented via the MCP protocol.
- Code Previews: The system includes snippets of the matched code, limited by
CODE_PREVIEW_MAX_LINES. - Language Highlighting: Syntax hints are applied based on file extensions (e.g.,
.rs->rust,.py->python). - Performance Warnings: If a search exceeds the
SEARCH_SLOW_THRESHOLD_MS(1000ms), a performance note is appended to the response.
Sources: crates/mcb-server/src/formatter/search.rs:15-45, crates/mcb-server/src/formatter/search.rs:105-125, crates/mcb-utils/src/constants/search.rs:38
Algorithmic Constants
Key constants governing the behavior of the search engine are centralized in mcb-utils.
| Constant | Value | Purpose |
|---|---|---|
BM25_TOKEN_MIN_LENGTH | 2 | Minimum characters for a word to be indexed |
HYBRID_SEARCH_MAX_CANDIDATES | 100 | Limit for candidate retrieval before fusion |
SEARCH_OVERFETCH_MULTIPLIER | 2 | Multiplier for initial retrieval to ensure enough results after filtering |
RRF_K | 60.0 | Parameter for Reciprocal Rank Fusion |
Sources: crates/mcb-utils/src/constants/search.rs:12-32
Implementation Example: Hybrid Score Calculation
The following snippet demonstrates how the engine performs the fusion between lexical and semantic scores:
// crates/mcb-providers/src/hybrid_search/engine.rs:100-111
fn hybrid_score_for(
&self,
index: &CollectionIndex,
result: &SearchResult,
query_terms: &[String],
) -> f64 {
let doc_key = format!("{}:{}", result.file_path, result.start_line);
let Some(&doc_idx) = index.document_index.get(&doc_key) else {
return self.semantic_weight * result.score;
};
let document = &index.documents[doc_idx];
let bm25_score = index.scorer.score_with_tokens(document, query_terms);
let normalized_bm25 = Self::normalize_bm25_score(bm25_score);
self.bm25_weight * normalized_bm25 + self.semantic_weight * result.score
}
This logic ensures that if a document is not present in the local lexical index, it still contributes to results based on its semantic score alone.
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:100-111
Conclusion
The Semantic & Hybrid Search Engine provides a robust retrieval layer for the MCB project. By balancing lexical precision (BM25) with semantic breadth (vector embeddings), it allows AI agents to navigate complex codebases efficiently. Its modular design allows it to scale across multiple collections while maintaining high performance through sigmoid normalization and weighted fusion.