PDF Processing Suite
Relevant source files
The following files were used as context for generating this wiki page:
The PDF Processing Suite is a specialized utility layer within the DeepCode repository designed to handle the acquisition, conversion, and structural analysis of research papers and technical documentation. Its primary purpose is to facilitate the "Paper-to-Code" workflow by transforming unstructured PDF data into machine-readable Markdown formats, which can then be ingested by LLM agents for algorithmic extraction and implementation planning.
This suite operates at the boundary between external information sources and the project's internal implementation agents. It handles various input types, including direct URLs, local PDF files, and text inputs, ensuring that the core research content is preserved and formatted correctly for downstream analysis.
Sources: prompts/code_prompts.py:34-40, tools/pdf_converter.py:10-15
Core Components and Architecture
The suite is structured around distinct functional modules that handle different stages of the document pipeline, from initial path detection to final format conversion.
Input Analysis and Acquisition
The entry point for PDF processing is the PaperInputAnalyzerAgent. This component classifies incoming data into one of four categories: URL, file, directory, or raw text. Once classified, the PAPER_DOWNLOADER_PROMPT logic dictates the storage path, typically assigning a numeric ID based on the count of existing files in the ./deepcode_lab/papers/ directory.
Sources: prompts/code_prompts.py:42-78
Document Conversion System
The PDFConverter class provides the heavy lifting for file transformations. It supports two primary conversion paths:
- Office to PDF: Utilizes LibreOffice (via
headlessmode) to convert.doc,.docx,.ppt,.pptx,.xls, and.xlsxfiles. - Text to PDF: Utilizes the
ReportLablibrary to convert.txtand.mdfiles into formatted PDFs, including support for inline Markdown elements like bolding, italics, and links.
Sources: tools/pdf_converter.py:30-40, tools/pdf_converter.py:343-350
Workflow Integration
The suite is integrated into the broader CodeImplementationWorkflow. During the implementation loop, the system uses segmented reading strategies to bypass token limits when processing large PDF-converted Markdown files.
The diagram shows the data flow from initial user input through the PDF processing tools to the final analysis agents.
Sources: prompts/code_prompts.py:42-55, workflows/code_implementation_workflow.py:110-120
PDF Conversion Logic
The PDFConverter implements a robust multi-platform strategy for handling document conversions, specifically targeting headless environments used by agents.
Office Document Conversion
For Office formats, the system searches for libreoffice or soffice binaries. It uses a temporary directory for the conversion process to prevent workspace clutter before moving the final PDF to the designated pdf_output folder.
| Parameter | Description | Source |
|---|---|---|
doc_path | Path to the source Office document | tools/pdf_converter.py:47 |
output_dir | Optional target directory; defaults to pdf_output | tools/pdf_converter.py:48 |
--headless | LibreOffice flag to run without a GUI | tools/pdf_converter.py:126 |
--convert-to | Specifies the target format (pdf) | tools/pdf_converter.py:127 |
Sources: tools/pdf_converter.py:42-160
Markdown and Text Processing
When converting text or Markdown to PDF, the suite performs inline styling. The _process_inline_markdown method uses regular expressions to map Markdown syntax to ReportLab's internal XML-like tags.
# Example of inline markdown processing
text = re.sub(r"\*\*(.*?)\*\*", r"<b>\1</b>", text) # Bold
text = re.sub(r"`([^`]+?)`", r'<font name="Courier">...</font>', text) # Code
Sources: tools/pdf_converter.py:343-380
Intelligent Reading and Segmentation
To handle the high token count of research papers converted from PDF, the suite employs a "Segmented Reading" strategy defined in the PAPER_ALGORITHM_ANALYSIS_PROMPT. This ensures that implementation details are not lost due to LLM context window limits.
Segmentation Parameters
| Query Type | Keywords | Max Chars | Purpose |
|---|---|---|---|
algorithm_extraction | algorithm, method, formula | 6000 | Extracting core logic |
concept_analysis | framework, architecture, system | 6000 | Mapping system design |
code_planning | setup, implementation, configuration | 8000 | Generating file structures |
Sources: prompts/code_prompts.py:165-175, prompts/code_prompts.py:270-280, prompts/code_prompts.py:375-385
Sequence of Document Processing
The sequence diagram illustrates how the agent interacts with the processed PDF content using the segmented reader tool to avoid token overflow.
Sources: prompts/code_prompts.py:170-185, workflows/agents/memory_agent_concise.py:530-545
Implementation Details
The PDFConverter utility is designed with platform awareness, particularly for Windows systems where console windows are hidden during subprocess execution.
# Subprocess configuration for background conversion
subprocess_kwargs = {
"capture_output": True,
"check": True,
"timeout": 10,
"encoding": "utf-8"
}
if platform.system() == "Windows":
subprocess_kwargs["creationflags"] = 0x08000000 # CREATE_NO_WINDOW
Sources: tools/pdf_converter.py:82-95
Conclusion
The PDF Processing Suite serves as the foundation for the DeepCode "Paper-to-Code" pipeline. By providing robust conversion utilities in pdf_converter.py and intelligent acquisition logic in the agent prompts, it transforms complex academic PDFs into structured Markdown files. This allows the implementation agents to perform targeted algorithm extraction through segmented reading, ultimately leading to the generation of reproducible codebases.