arXiv API Integration
Relevant source files
The following files were used as context for generating this wiki page:
The arXiv API Integration is a core component of the CodeScholar system, designed to facilitate the discovery of cutting-edge academic research that is not currently present in the local library. It enables the system to bridge the gap between static local data and the evolving landscape of academic literature, specifically focusing on computer science and related technical domains.
This module allows AI agents to perform external searches via the Model Context Protocol (MCP), retrieving paper metadata including titles, abstracts, and identifiers (DOIs/arXiv IDs). The integration serves as a primary source for the discover_papers tool, ensuring that users can find recent research foundations for their codebases.
Sources: README.md:12, server.js:14-15, server.js:621-625
System Architecture and Integration Flow
The integration operates as a bridge between the Node.js MCP server and a specialized Python-based source module. When a user or agent requests a discovery search, the server invokes a sub-process to communicate with the arXiv API.
High-Level Data Flow
The following diagram illustrates the flow of a discovery request from the user interface through the MCP server to the arXiv API.
The flow ensures that results are retrieved, parsed, and converted into the unified CodeScholar format before being presented to the user.
Sources: server.js:632-646, README.md:38-40
The Discovery Tool (discover_papers)
The discover_papers tool is the primary interface for the arXiv integration. It is registered within the MCP server and accepts parameters to control the search scope and source selection.
Tool Parameters
| Parameter | Type | Description |
|---|---|---|
query | string | The search terms or natural language question. |
sources | array | List of sources to search (e.g., ["arxiv", "semantic_scholar"]). |
max_results | number | Maximum results to return per source (default: 5). |
Sources: server.js:464-478
Result Filtering Logic
The server implements a deduplication layer to ensure that papers already residing in the local unified_library.json are filtered out of the discovery results. This is achieved by checking:
- DOI Matching: Compares the DOI of the discovered paper against the local library index.
- Fuzzy Title Matching: Converts titles to lowercase and checks for exact matches in the
titleIndex.
Sources: server.js:665-675
Backend Implementation: arxiv_source.py
The logic for querying the arXiv API is encapsulated in scripts/sources/arxiv_source.py. This module handles the construction of API queries, the execution of asynchronous HTTP requests, and the transformation of XML responses into a structured format compatible with the rest of the project.
Core Functions
search_arxiv(query, max_results): The entry point for the module. It constructs the API URL and handles the request lifecycle.- Response Parsing: Transforms the arXiv XML entry fields (like
<title>,<summary>, and<id>) into the project's standard metadata schema.
Sources: server.js:640-642, scripts/sources/arxiv_source.py
Unified Data Mapping
Discovered papers are mapped to the project's unified format, which is also used during the initial import of legacy papers. This ensures consistency across the knowledge base.
Deployment and Configuration
The integration requires a Python environment with dependencies defined in requirements.txt. The MCP server must be configured with the correct paths to access the scripts.
Server-Side Execution
The Node.js server uses child_process.execSync to run a Python snippet that imports the search_arxiv function. This allows for a clean separation of concerns between the TypeScript/JavaScript MCP logic and the Python-based data retrieval logic.
// Example of the bridging logic used in server.js
const pythonCmd = `python -c "import sys, json, asyncio; sys.path.insert(0, '${scriptsPath}'); from sources.arxiv_source import search_arxiv; results = asyncio.run(search_arxiv('${args.query}', ${maxResults})); print(json.dumps(results))"`;
Sources: server.js:633-644, README.md:21-23
Summary
The arXiv API Integration allows CodeScholar to function as a dynamic research tool rather than a static repository. By combining asynchronous Python-based retrieval with Node.js sub-process management and a robust deduplication layer, the system provides users with immediate access to the latest research papers tailored to their specific code context. This integration is essential for the project's goal of linking software engineering tasks to their academic foundations.
Sources: README.md:3-8, server.js:520-530, scripts/generate_knowledge_base.py:1010-1030