Semantic Search Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Semantic Search Engine is a core component of the Chain-of-Ideas (CoI) Agent, designed to facilitate deep literature discovery and research paper analysis. It leverages the Semantic Scholar API to query academic databases and employs Large Language Model (LLM) embeddings to perform semantic re-ranking, ensuring that retrieved documents are highly relevant to the researcher's specific topic and idea.
The system is primarily encapsulated in the SementicSearcher class, which handles asynchronous API calls, PDF downloading, and document parsing using tools like scipdf. This engine supports both the DeepResearchAgent and ReviewAgent by providing the data necessary for trend analysis, novelty checking, and experimental design validation.
Sources: searcher/sementic_search.py:73-77, agents.py:75-76, README.md:39-44
Core Architecture and Data Flow
The Semantic Search Engine operates through a multi-stage process involving query generation, API interaction, and content extraction. It uses a vector-based similarity approach to refine search results.
The diagram above illustrates the flow from an initial research query to the generation of a structured Result object containing parsed paper content.
Sources: searcher/sementic_search.py:176-218, agents.py:108-129
Key Components
| Component | Description |
|---|---|
SementicSearcher | The primary coordinator class for API interactions and local file management. |
Result | A data structure storing a paper's title, abstract, parsed article content, citation count, and publication year. |
| Semantic Scholar API | External provider used for metadata and Open Access PDF discovery. |
scipdf | Library used to parse PDF binaries into structured JSON dictionaries containing sections and references. |
| Cosine Similarity | Mathematical method used to rank paper embeddings against the user's research topic. |
| Sources: searcher/sementic_search.py:64-70, searcher/sementic_search.py:73-77, searcher/sementic_search.py:157-169 |
Search Logic and Re-ranking
The engine goes beyond keyword matching by utilizing LLM embeddings. When a search is initiated, the engine can optionally re-rank candidates by calculating the cosine similarity between the research topic embedding and the candidate paper's metadata (title and abstract).
Asynchronous Search Process
The search_async method handles the lifecycle of a search request:
- Filtering: Excludes "banned" papers or those already read by the agent to maintain search novelty.
- Reranking: If an LLM and topic are provided, it generates embeddings for candidate papers and sorts them by similarity score.
- Acquisition: It checks for local PDFs first; if missing, it attempts to download via
openAccessPdfURLs provided by the API.
Sources: searcher/sementic_search.py:176-218, searcher/sementic_search.py:143-155
Similarity Calculation
The engine uses NumPy to perform vectorized cosine similarity calculations across paper embeddings.
def cal_cosine_similarity_matric(self, matric1, matric2):
dot_product = np.dot(matric1, matric2.T)
norm1 = np.linalg.norm(matric1, axis=1)
norm2 = np.linalg.norm(matric2, axis=1)
cos_sim = dot_product / np.outer(norm1, norm2)
return cos_sim.flatten().tolist()
Sources: searcher/sementic_search.py:129-141
Related Paper Discovery
The engine supports recursive research through "Forward" and "Backward" searches. The search_related_paper_async method retrieves both citations (papers that cited the current work) and references (papers cited by the current work).
This recursive discovery allows the DeepResearchAgent to build "idea chains" that track the evolution of research topics over time.
Sources: searcher/sementic_search.py:220-272, agents.py:207-250
Configuration and API Integration
The engine requires specific environment variables and API keys to function, primarily the SEMENTIC_SEARCH_API_KEY. It also utilizes a specific set of fields from the Semantic Scholar Graph API to ensure it receives enough data for full parsing.
| Parameter | Default Fields Used |
|---|---|
| Paper Metadata | title, abstract, year, citationCount |
| Access Info | isOpenAccess, openAccessPdf |
| Deep Data | citations.title, references.abstract, publicationDate |
| Sources: searcher/sementic_search.py:82-84, main.py:12-18, README.md:46-48 |
Document Parsing and Content Extraction
Once a PDF is acquired, the engine transforms it into a machine-readable format. The read_paper_content_with_ref method extracts:
- Sections: Headings and text blocks for semantic analysis.
- References: A mapped list of
ref_id,title, andyearused for bibliography tracing. - Ref IDs: Links within the text to specific citations, enabling the LLM to understand context-specific references.
Sources: searcher/sementic_search.py:343-353, prompts/deep_research_agent_prompts.py:85-115
Conclusion
The Semantic Search Engine acts as the primary sensory organ for the CoI Agent, bridging the gap between raw academic databases and structured research synthesis. By combining traditional API filtering with modern vector-based re-ranking and deep PDF parsing, it enables the autonomous discovery of novel research directions and the rigorous validation of scientific ideas.