Paper Search Tools
Relevant source files
The following files were used as context for generating this wiki page:
Paper Search Tools represent the core functionality of the CodeScholar system, designed to bridge the gap between source code and academic research. These tools facilitate the discovery, retrieval, and analysis of academic papers by leveraging hybrid search mechanisms that combine traditional text matching with modern vector embeddings.
The system operates as a Model Context Protocol (MCP) server, providing a suite of tools that allow AI agents and developers to search a local library, discover external papers from sources like arXiv and Semantic Scholar, and link specific code concepts to published methodologies.
Sources: server.js:1-15, README.md:1-10
Search Architecture and Mechanisms
CodeScholar employs a multi-tiered search strategy to ensure both keyword precision and semantic depth. The architecture supports local library indexing and external API integration.
Hybrid Search Logic
The system's hybridSearch function merges results from two distinct paths:
- Vector Search: Utilizes
Vectra(a local vector database) to rank papers by cosine similarity to a query embedding. This requires anOPENAI_API_KEYto generate embeddings for queries. - Text Search: A fallback and enrichment layer that performs multi-term matching across titles, abstracts, and authors. It applies weighted scoring (e.g., title matches are weighted higher than abstract matches).
Sources: server.js:108-145, server.js:147-194, README.md:55-60
Local Knowledge Base Structure
The knowledge base is organized into a 3-layer architecture for efficient consumption:
- Problems: High-level entry points (e.g., "Code Quality Assessment") featuring decision matrices.
- Techniques: Implementable algorithm reference cards with pseudocode and complexity analysis.
- Papers: The evidence layer containing DOIs, benchmarks, and methodology details.
Sources: scripts/generate_knowledge_base.py:16-21, knowledge/README.md:17-27
The following diagram illustrates the search flow within the MCP server:
The diagram shows how the system branches between local library queries (using hybrid logic) and external API calls for paper discovery.
Sources: server.js:350-410, server.js:525-595
Key Search Tools and API Endpoints
The CodeScholar server exposes several tools through the MCP ListToolsRequestSchema.
Core Search Tools
| Tool Name | Description | Key Parameters |
|---|---|---|
search_papers | Primary search tool for local library. Supports hybrid vector+text search. | query, limit, year, min_citations |
discover_papers | Searches external sources (arXiv, Semantic Scholar) for papers not in local library. | query, sources, max_results |
link_papers_to_code | Flagship tool. Analyzes code, generates queries, and scores paper applicability. | file_path, directory, min_score |
find_related_papers | Finds papers based on a list of concepts or code context. | concepts, code_context |
Sources: server.js:283-345, README.md:12-18
Applicability Scoring
For tools like link_papers_to_code, the system uses an ApplicabilityScorer to rank papers based on four weighted components:
- Concept Overlap: Measures the intersection of code-extracted concepts and paper text.
- Text Relevance: Keyword similarity in title and abstract.
- Recency: Favors newer publications using a decay function.
- Citation Impact: Normalizes citation counts to influence the final score.
Sources: scripts/tests/test_applicability_scorer.py:126-170, README.md:20-22
Data Models and Schemas
Paper Metadata Schema
The system expects papers in a structured JSON format (typically unified_library.json).
| Field | Type | Description |
|---|---|---|
id | String (UUID) | Unique identifier for the paper. |
title | String | Full title of the academic work. |
abstract | String | Summary of the research. |
identifiers | Object | Contains doi, doi_url, and semantic_scholar_id. |
metadata | Object | Contains year and citation_count. |
Sources: server.js:203-220, knowledge/papers/a-data-driven-methodology-for-quality-aware-code-fixing.json:1-15
Problem Entry Point Schema
Used in the problems/ directory to guide users toward specific techniques.
{
"problem_name": "Name of the problem",
"decision_matrix": [
{
"technique": "Algorithm Name",
"speed": "fast/medium/slow",
"accuracy": "low/medium/high",
"best_when": "Specific scenario"
}
],
"start_here": "Recommended first step"
}
Sources: knowledge/problems/code-quality-assessment.json:1-40, scripts/generate_knowledge_base.py:534-566
Implementation Details
The following sequence diagram represents the interaction when linking code to research:
The sequence demonstrates the orchestration between the JavaScript MCP server and the underlying Python scripts for specialized analysis.
Sources: server.js:640-750, README.md:40-52
Summary
Paper Search Tools provide a robust framework for discovering academic context within software development workflows. By integrating local vector search with real-time external discovery and automated code analysis, the system enables developers to ground their implementation choices in proven research. The modular design, combining Node.js for server operations and Python for intensive scoring and extraction, allows for high extensibility and precise relevance ranking.