Query Generation Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Query Generation Strategy in CodeScholar is a multi-faceted approach designed to bridge the gap between source code analysis and academic research retrieval. It transforms raw code concepts, user search terms, and interaction data into optimized search queries capable of surfacing highly relevant academic papers, techniques, and problem definitions. This strategy is critical for the system's "link_papers_to_code" tool, which provides the research foundation for specific implementation details.
Sources: server.js:476-486, scripts/generate_knowledge_base.py:16-24
Core Query Components
The system utilizes several distinct methodologies for generating queries, ranging from direct text extraction to complex semantic expansion and feedback-driven refinement.
Concept-Based Generation
Queries are primarily derived from "concepts" extracted during code analysis. This process identifies algorithms, techniques, and data structures (e.g., "transformer", "binary tree", "gradient descent") and uses them as seeds for query expansion.
Sources: scripts/tests/test_query_generator.py:44-78, server.js:568-575
Pseudo Relevance Feedback (PRF)
For exploratory search scenarios, the system employs a Pseudo Relevance Feedback mechanism. This approach analyzes user interaction logs and search result patterns to generate potential feedback queries that refine the search based on observed user behavior.
| Strategy | Description | Key Mechanism |
|---|---|---|
| Direct | Uses the exact name of the extracted concept. | Literal text match |
| Concept Expansion | Broadens terms into academic synonyms. | Synonyms and category mapping |
| Compound | Combines multiple concepts into complex queries. | Multi-term logical grouping |
| Pseudo Relevance | Refines queries based on search results. | User interaction patterns |
Sources: scripts/tests/test_query_generator.py:12-104, knowledge/papers/end-to-end-vertical-web-search-pseudo-relevance-feedback-queries-recommendation-.json:21-28
Query Transformation Pipeline
The following diagram illustrates how raw inputs are transformed into the final ranked queries used by the MCP server tools.
The pipeline ensures that a single concept can result in multiple high-confidence search terms to maximize library coverage.
Sources: server.js:488-515, scripts/tests/test_query_generator.py:145-160
Implementation Logic
Query Weighting and Ranking
The system does not treat all generated queries equally. A QueryGenerator class manages GeneratedQuery objects, each assigned a confidence score and a QueryStrategy type. Queries are ranked by confidence, and duplicates are merged by retaining the highest confidence score.
Sources: scripts/tests/test_query_generator.py:12-32, scripts/tests/test_query_generator.py:101-131
Semantic and Hybrid Search
The server.js implementation provides a hybrid search mode. It attempts to generate a query embedding for semantic search (using OpenAI) while falling back to a weighted text search if embeddings are unavailable.
- Title Match: Highest weight (+3)
- Abstract Match: Medium weight (+2)
- Exact Phrase Match: Bonus (+5)
- Citation Bonus: Normalized weight based on citation count
Sources: server.js:156-180, server.js:636-658
Complex Question Decomposition
For complex knowledge base queries, the strategy involves a "Hereditary Attentive Template-based" approach. This method decomposes a complex question into logical sub-questions and identifies relevant templates to guide the retrieval process.
This multi-step decomposition is used for queries that require logical reasoning across large datasets.
Sources: knowledge/papers/a-hereditary-attentive-template-based-approach-for-complex-knowledge-base-questi.json:16-24
Technical Constraints and Parameters
The query generation and execution behavior is governed by specific parameters to ensure performance and relevance.
| Parameter | Default Value | Description |
|---|---|---|
max_queries | 15 | Maximum number of queries to generate from concepts. |
min_confidence | 0.3 | Threshold to filter out low-confidence generated queries. |
limit (Search) | 10 | Max results returned per query in search_papers. |
feedback_threshold | 0.5 | Threshold for PRF query inclusion. |
template_size | 5 | Size of the template used for question decomposition. |
Sources: server.js:336-350, scripts/tests/test_query_generator.py:175-185, knowledge/papers/end-to-end-vertical-web-search-pseudo-relevance-feedback-queries-recommendation-.json:31-35, knowledge/papers/a-hereditary-attentive-template-based-approach-for-complex-knowledge-base-questi.json:28-31
The Query Generation Strategy enables CodeScholar to function as a bridge between active development and academic research. By combining structural code analysis with semantic expansion and iterative feedback, the system ensures that developers find the research foundations relevant to their specific implementation contexts.