Hybrid Search Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Hybrid Search Engine in CodeScholar is a multi-modal retrieval system designed to provide context-aware academic paper discovery for AI agents. It combines semantic vector-based similarity with traditional text-matching algorithms to ensure both conceptual relevance and keyword precision. The system operates primarily through a Model Context Protocol (MCP) server, allowing external agents to query a structured library of research papers, techniques, and problem types.
The engine facilitates "emergent discovery" by mapping cross-domain connections, allowing users to find research foundations for specific code concepts. It supports a 3-layer knowledge architecture: Problems, Techniques, and Papers, which are synthesized using a multi-pass pipeline.
Sources: server.js:1-15, scripts/generate_knowledge_base.py:10-20
Architecture and Core Components
The hybrid search architecture utilizes a dual-path retrieval strategy. The first path leverages Vectra, a local vector index, for semantic search based on embeddings. The second path uses a weighted text search across paper titles, abstracts, and authors.
Search Components
| Component | Technology | Description |
|---|---|---|
| Vector Index | Vectra (LocalIndex) | Stores paper embeddings for cosine similarity-based retrieval. |
| Text Index | In-memory Map/Array | Stores lower-case metadata for keyword and phrase matching. |
| Hybrid Resolver | JavaScript (Node.js) | Merges, deduplicates, and re-ranks results from both paths. |
| Concept Extractor | Python (AST/Regex) | Extracts searchable terms from raw source code context. |
Sources: server.js:42-55, server.js:115-130, server.js:487-520
System Data Flow
The following diagram illustrates how a user query is processed through the hybrid pipeline to produce ranked results.
The system first determines if vector search is available (requiring a local Vectra index). If so, it generates an embedding for the query and retrieves semantic candidates, which are then combined with keyword-matched results.
Sources: server.js:134-170, server.js:527-550
Search Methodologies
Semantic Vector Search
The engine uses the vectorSearch function to query the LocalIndex. It supports metadata filtering for publication year and minimum citation counts. Results are returned with a relevance_score derived from cosine similarity.
Sources: server.js:93-113
Keyword Text Search
The textSearch function implements a weighted scoring system for multi-term matching:
- Title Match: +3 points per term.
- Abstract Match: +2 points per term.
- Author Match: +1 point per term.
- Exact Phrase Match: +5 points bonus.
- Citation Bonus: Normalized score based on citation count (up to +1).
Sources: server.js:176-210
Hybrid Re-ranking Logic
The hybridSearch function merges results from both sources. Vector scores are scaled (multiplied by 10) to align with text scores. Papers found by both methods receive a "boost" by summing their respective scores.
Sources: server.js:134-164
Sources: server.js:134-145, server.js:540-560
Implementation Details
The engine is exposed via several MCP tools that allow agents to interact with the underlying research data.
Primary Search Tools
| Tool Name | Input Parameters | Purpose |
|---|---|---|
search_papers | query, limit, year, min_citations | General semantic and keyword search. |
link_papers_to_code | file_path, directory, min_score | Analyzes code to find relevant research foundations. |
find_related_papers | concepts, code_context | Finds papers based on specific algorithms or techniques. |
Sources: server.js:315-350, server.js:460-480
Code Analysis and Extraction
The system utilizes a Python-based concept extractor (concept_extractor.py) to parse source code. This identifies algorithms, data structures, and research concepts to generate optimized search queries.
Sources: server.js:630-650, scripts/generate_knowledge_base.py:25-45
Knowledge Base Layers
The engine searches across a structured hierarchy defined in the generation pipeline:
- Problems: Entry points defining high-level challenges (e.g., "code quality improvement").
- Techniques: Specific solutions and algorithms (e.g., "Quality Aware Code Fixing").
- Papers: The underlying evidence and benchmarks.
Sources: scripts/generate_knowledge_base.py:14-19, knowledge/techniques/hybrid-data-storage-architecture.md:5-15
Conclusion
The Hybrid Search Engine provides a robust mechanism for linking software implementation to academic research. By combining the strengths of semantic embeddings and weighted keyword matching, it ensures that AI agents can accurately navigate complex knowledge spaces. This dual approach is critical for the "CodeScholar" mission of making research implementable for software engineers.
Sources: server.js:780-795, scripts/generate_knowledge_base.py:650-670