Semantic Scholar Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Semantic Scholar integration in CodeScholar serves as a primary source for enriching academic paper metadata and discovering new research. It provides the project with critical data points such as citation counts, author lists, and stable identifiers (DOIs and Semantic Scholar IDs), which are used to score the applicability of research to specific codebases.
This integration is implemented through two main avenues: a data enhancement pipeline that updates the local unified_library.json and a real-time discovery tool exposed via the Model Context Protocol (MCP) server. By leveraging the Semantic Scholar Graph API, the system ensures that the knowledge base remains accurate and reflects the current impact of indexed research.
Sources: scripts/enhance_existing_papers.py:12-15, server.js:1010-1014, README.md:10-15
Metadata Enhancement Pipeline
The enhancement pipeline is a specialized script designed to augment existing paper records with high-fidelity data from Semantic Scholar. It primarily uses DOIs as keys to query the Semantic Scholar Graph API.
Enrichment Logic
The script iterates through the unified_library.json, identifies papers with DOIs, and fetches updated metadata. The fetched data includes citationCount, authors, year, and publicationDate. This data is then merged back into the unified paper format, adding a specific source entry for Semantic Scholar.
The diagram above illustrates the sequential process of enriching local paper metadata using the Semantic Scholar API.
Sources: scripts/enhance_existing_papers.py:83-118, scripts/enhance_existing_papers.py:141-160
Technical Configuration
The integration enforces strict rate limits and timeouts to comply with Semantic Scholar's public API policies.
| Parameter | Value | Description |
|---|---|---|
| API Endpoint | https://api.semanticscholar.org/graph/v1 | Base URL for Graph API requests |
| Rate Limit Delay | 0.01 seconds | Enforces ~100 requests/second limit |
| Fields Requested | paperId, title, abstract, citationCount, authors, year, publicationDate | Specific metadata fields retrieved |
| Timeout | 30 seconds | Network request timeout |
Sources: scripts/enhance_existing_papers.py:17-25, scripts/enhance_existing_papers.py:53-56
Real-time Discovery via MCP
The server.js component integrates Semantic Scholar as an external discovery source. This allows AI agents to find research that is not yet present in the local knowledge base.
Discovery Tool (discover_papers)
The discover_papers tool executes a Python sub-process that calls the semantic_scholar_source.py module. It supports keyword searches and allows filtering by year. The results are compared against the local library by DOI or title to ensure only "new" papers are presented to the user.
This sequence shows how the MCP server orchestrates a real-time search on Semantic Scholar and filters results based on existing data.
Sources: server.js:847-862, server.js:988-1033
Search Execution
The discovery logic uses a max_results parameter to limit payload size and can target specific publication years (e.g., "2024" or ranges like "2023-2024").
Sources: server.js:859-862, server.js:994-1000
Data Model Integration
Papers enriched by Semantic Scholar follow a specific schema within the knowledge/papers/ directory. This schema identifies the paper's origin and provides direct links for human researchers.
Identifier Schema
Papers are assigned a paperId from Semantic Scholar, which is stored in the identifiers block alongside the doi.
| Field | Location | Example / Format |
|---|---|---|
| Semantic Scholar ID | identifiers.semantic_scholar_id | UUID or alphanumeric string |
| Citation Count | metadata.citation_count | Integer (e.g., 45) |
| Author List | authors | Array of strings |
| Source URL | sources[].url | https://www.semanticscholar.org/paper/{id} |
Sources: scripts/enhance_existing_papers.py:101-112, knowledge/papers/artificial-intelligence-assisted-academic-writing-recommendations-for-ethical-us.md:10-15
Applicability Scoring Impact
Semantic Scholar data is vital for the ApplicabilityScorer. The citation count and publication year retrieved from the API are used as weights (2 of the 4 scoring components) to determine how relevant a paper is to a given piece of source code.
Sources: README.md:12-14, server.js:1162-1180
Summary
The Semantic Scholar integration is a cornerstone of the CodeScholar architecture, providing the objective "impact" data (citations) and "recency" data (publication dates) necessary for ranking research. It functions as both a background enrichment service for the static knowledge base and a real-time bridge to the broader academic landscape through the MCP server's discovery tools.