PDF Parsing Integration

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• searcher/sementic_search.py
• agents.py
• main.py
• requirements.txt
• prompts/deep_research_agent_prompts.py

The PDF Parsing Integration module is a critical component of the Chain-of-Ideas (CoI) Agent, responsible for converting unstructured scientific literature into structured data for LLM analysis. This system enables the autonomous agent to "read" research papers, extract specific sections (Abstract, Introduction, Methodology), and identify internal references to build conceptual "idea chains."

The integration primarily relies on external tools like SciPDF Parser and Grobid to handle the heavy lifting of PDF layout analysis, while the internal SementicSearcher class manages the lifecycle of downloading, caching, and extracting text from these documents.
Sources: README.md:38-44, searcher/sementic_search.py:68-81

System Architecture

The PDF parsing workflow is split into two primary layers: the infrastructure layer (handling the PDF-to-XML conversion) and the application layer (handling the data extraction for research agents).

External Dependencies

The system requires a specific environment to process PDFs:

• SciPDF Parser: A Python library used to interface with Grobid.
• Grobid: A machine learning library for extracting and restructuring PDF documents into TEI/XML.
• Java JDK 11: Required to run the Grobid server.
  Sources: README.md:46-70, requirements.txt:13

Logical Data Flow

The following diagram illustrates how a research topic leads to a parsed paper structure.

The diagram shows the transition from raw search results to structured data used by the LLM.
Sources: searcher/sementic_search.py:195-236, agents.py:165-175

Key Components and Logic

SementicSearcher Class

The SementicSearcher class in searcher/sementic_search.py acts as the primary orchestrator for PDF interactions. It handles the local caching of PDFs to avoid redundant downloads and manages the parsing through the scipdf library.

Key Methods for PDF Processing:

Method	Description
download_pdf_async	Retrieves a PDF from a URL and saves it to a local directory.
read_arxiv_from_path	Uses scipdf.parse_pdf_to_dict to convert a local PDF into a Python dictionary.
read_paper_content	Iterates through sections in the parsed dictionary to extract headings, text, and reference IDs.
read_paper_content_with_ref	Appends a structured <References> block containing ref_id, title, and year.

Sources: searcher/sementic_search.py:238-274, searcher/sementic_search.py:292-311

Data Structures

The parser converts a PDF into a structured "Article" dictionary. The DeepResearchAgent specifically extracts the following fields for its reasoning chains:

• Title & Abstract: Used for relevance judging.
• Sections: A list of objects containing heading, text, and publication_ref.
• References: A list of bibliography entries including ref_id, title, and year.

Sources: searcher/sementic_search.py:284-311, agents.py:175-185

Sequence of PDF Integration

When the DeepResearchAgent performs a "deep research" task, it follows a sequence to integrate PDF data into its memory.

This sequence demonstrates how the agent waits for structured text before proceeding with literature synthesis.
Sources: agents.py:165-180, searcher/sementic_search.py:180-210

Implementation Details

Extraction Regex and Cleanup

The system uses utility functions to clean up the text extracted from the parsed PDFs. The extract function identifies content between custom XML-style tags used in the LLM prompts.

def extract(text, type):
    if text:
        target_str = get_content_between_a_b(f"<{type}>", f"</{type}>", text)
        return target_str if target_str else text
    return ""

Sources: searcher/sementic_search.py:27-38

Integration in Agents

The DeepResearchAgent utilizes the parsed content to build "Idea Chains." It specifically looks for references within the parsed PDF to perform backward search, allowing the agent to navigate from a core paper to its predecessors.
Sources: agents.py:246-260, prompts/deep_research_agent_prompts.py:100-115

Summary

The PDF Parsing Integration provides the necessary bridge between raw PDF files and the structured knowledge required by the CoI Agent. By leveraging Grobid and SciPDF, the system extracts high-fidelity sections and bibliography data, which are then formatted into prompts for the DeepResearchAgent to generate novel research directions.
Sources: agents.py:210-220, README.md:72-80