Hybrid Search Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Hybrid Search Engine is a core component of the MCB (Memory Context Browser) project, designed to provide high-relevance code search by combining lexical and semantic methodologies. It integrates BM25 text-based ranking with vector-based semantic similarity scores to produce a final ranked list of results. This approach ensures that searches are sensitive both to exact keyword matches (lexical) and to the conceptual meaning of natural language queries (semantic).
The engine operates within the mcb-providers crate, implementing the HybridSearchProvider trait defined in the domain layer. It manages separate indexes for different collections, re-ranking semantic results fetched from vector stores using a weighted score fusion algorithm.
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:1-30, crates/mcb-providers/src/hybrid_search/mod.rs:1-15, README.md:10-15
Core Architecture and Data Flow
The Hybrid Search Engine follows a parallel processing and fusion pattern. It takes a natural language query and a set of initial semantic search results, then enriches these results with lexical scores derived from a local BM25 index.
Search Execution Flow
The following diagram illustrates how the engine processes a search request:
The search process involves extracting lexical tokens from the query while simultaneously retrieving semantically similar chunks from a vector store. The two scores are combined using a weighted formula.
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:12-28, crates/mcb-providers/src/hybrid_search/mod.rs:7-22
Score Fusion Logic
The engine calculates a Hybrid Score for each result based on specific weights assigned to BM25 and Semantic components. The default weights are typically 0.4 for BM25 and 0.6 for Semantic search.
| Component | Weight Variable | Default Value | Description |
|---|---|---|---|
| BM25 | bm25_weight | 0.4 | Weight for lexical keyword matching. |
| Semantic | semantic_weight | 0.6 | Weight for vector similarity matching. |
Formula: Hybrid Score = (bm25_weight * Normalized_BM25) + (semantic_weight * Semantic_Score)
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:43-45, crates/mcb-utils/src/constants/search.rs:14-17
Components and Implementation Details
HybridSearchEngine Struct
The primary structure responsible for managing collections and performing search operations. It uses a RwLock to handle concurrent access to various CollectionIndex instances.
pub struct HybridSearchEngine {
bm25_weight: f64,
semantic_weight: f64,
collections: RwLock<HashMap<String, CollectionIndex>>,
}
Sources: crates/mcb-providers/src/hybrid_search/engine.rs:40-46
BM25 Scorer
The BM25 (Best Matching 25) implementation scores documents based on term frequency (TF) and inverse document frequency (IDF). It employs two main parameters:
- k1: Term frequency saturation (default 1.2).
- b: Document length normalization (default 0.75).
The engine normalizes raw BM25 scores using a sigmoid function to ensure they fall within a 0.0 to 1.0 range before fusion.
Sources: crates/mcb-providers/src/hybrid_search/mod.rs:39-50, crates/mcb-providers/src/hybrid_search/engine.rs:90-96
Collection Management
Indexes are organized by "collections," allowing the engine to isolate search data for different projects or repositories.
Algorithmic Constants
The following constants define the behavior of the search algorithms and are maintained as the single source of truth in the utilities crate.
| Constant | Value | File Path |
|---|---|---|
HYBRID_SEARCH_BM25_K1 | 1.2 | mcb-utils/src/constants/search.rs |
HYBRID_SEARCH_BM25_B | 0.75 | mcb-utils/src/constants/search.rs |
BM25_TOKEN_MIN_LENGTH | 2 | mcb-utils/src/constants/search.rs |
HYBRID_SEARCH_MAX_CANDIDATES | 100 | mcb-utils/src/constants/search.rs |
SEARCH_SLOW_THRESHOLD_MS | 1000 | mcb-utils/src/constants/search.rs |
Sources: crates/mcb-utils/src/constants/search.rs:7-40
User Feedback and Formatting
The search results are formatted for display via the mcb-server formatter. It includes information about query duration, relevance scores, and code previews. If a search takes longer than the SEARCH_SLOW_THRESHOLD_MS (1 second), a performance warning is appended to the response.
Search Response Elements
- Query: The original search string.
- Duration: Time taken in seconds.
- Results Found: Total count of matches.
- Code Preview: Snippets of the found code with syntax highlighting hints.
- Relevance Score: The final fused hybrid score.
Sources: crates/mcb-server/src/formatter/search.rs:16-50, crates/mcb-server/src/formatter/search.rs:95-105
Summary
The Hybrid Search Engine provides a balanced retrieval mechanism by combining the precision of keyword matching (BM25) with the breadth of semantic understanding (Vector Search). Through its collection-based architecture and weighted fusion logic, it delivers ranked results that are both lexically accurate and conceptually relevant to user queries.
Sources: crates/mcb-providers/src/hybrid_search/engine.rs, crates/mcb-providers/src/hybrid_search/mod.rs