Grobid & SciPDF Setup

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• searcher/sementic_search.py
• agents.py
• requirements.txt
• main.py

Introduction

The Grobid and SciPDF setup provides the foundational infrastructure for the Chain-of-Ideas (CoI) Agent to process and understand scientific literature. This system enables the conversion of raw PDF documents into structured machine-readable formats, allowing the agent to extract specific sections like abstracts, introductions, and references for further analysis.

Within the project, these tools act as the bridge between raw data acquisition from sources like Semantic Scholar and the high-level reasoning performed by the DeepResearchAgent and ReviewAgent. By utilizing Grobid as a background service and scipdf_parser as the Python interface, the project automates the extraction of complex academic metadata.

Sources: README.md, agents.py:73-81

System Architecture and Integration

The PDF processing pipeline relies on a client-server architecture. Grobid runs as a Java-based service that handles the heavy lifting of PDF parsing, while the scipdf library acts as a client within the Python environment.

Components Table

Component	Responsibility	Technology
Grobid	Parsing PDF structures, identifying sections, and extracting citations.	Java/JDK 11
SciPDF Parser	Python wrapper for communicating with Grobid and converting XML/TEI to dictionaries.	Python, Spacy
SementicSearcher	Manages local storage and invokes parsing during the search/research phase.	Python

Sources: README.md:33-65, searcher/sementic_search.py:245-251

Data Flow Diagram

The following diagram illustrates how a PDF file is processed from its raw state to structured data used by the research agents.

The diagram shows the lifecycle of a document within the system, emphasizing the external dependency on the Grobid service.
Sources: searcher/sementic_search.py:228-285

Installation and Configuration

Environment Requirements

The project requires specific versions of Java and Python libraries to ensure compatibility with the Grobid service.

1. Java Environment: Grobid requires OpenJDK 11.
2. Python Dependencies: Core libraries include scipdf, spacy, and the en_core_web_sm model.
3. Service Configuration: Grobid typically serves on local port 8080.

Sources: README.md:33-50, requirements.txt

Deployment Steps

The setup is divided into library installation and service initialization.

# 1. Install SciPDF Parser and Spacy model
pip install git+https://github.com/titipata/scipdf_parser
python -m spacy download en_core_web_sm

# 2. Setup Java for Grobid
wget https://download.oracle.com/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz
export JAVA_HOME=Your_path/jdk-11.0.2

# 3. Start Grobid (Standard method)
cd scipdf_parser
bash serve_grobid.sh

Sources: README.md:33-63

Technical Implementation in Source Code

PDF Parsing Logic

The SementicSearcher class in searcher/sementic_search.py is the primary consumer of the SciPDF/Grobid integration. It uses scipdf.parse_pdf_to_dict to transform files into a structured article_dict.

Key methods involved in structured extraction include:

• read_arxiv_from_path(pdf_path): Uses scipdf to parse a local PDF.
• read_paper_content(article): Iterates through the sections dictionary generated by Grobid to aggregate text and reference IDs.
• read_paper_content_with_ref(article): Specifically extracts the <References> section by mapping Grobid's ref_id to paper titles and years.

Sources: searcher/sementic_search.py:245-251, searcher/sementic_search.py:274-285

Class Relationship Diagram

The following class diagram shows how the searcher interacts with the parsing utility and how the data is stored.

This diagram represents the internal class structure where SementicSearcher orchestrates the parsing flow.
Sources: searcher/sementic_search.py:72-78, searcher/sementic_search.py:92-95

Conclusion

The Grobid & SciPDF setup is a critical preprocessing layer for the CoI Agent. By containerizing or running Grobid as a service and interfacing through scipdf_parser, the system converts unstructured scientific PDF data into a rich, section-aware format. This allows the agents to perform deep research tasks, such as tracing idea chains and analyzing experimental designs, which would be impossible with plain text extraction.

Sources: README.md, agents.py:175-185