Document Segmentation Agent
Relevant source files
The following files were used as context for generating this wiki page:
The Document Segmentation Agent (and its underlying server infrastructure) is a specialized system within the DeepCode project designed to handle large-scale research papers and technical documentation. Its primary purpose is to overcome LLM token limits by intelligently breaking down complex documents into semantic segments that preserve the integrity of algorithms, mathematical formulas, and technical concepts.
The system uses content-aware analysis rather than simple structural chunking to ensure that related information—such as a theorem and its proof or an algorithm and its pseudocode—remains grouped together. This facilitates high-quality information retrieval for downstream tasks like Algorithm Extraction and Code Planning.
Sources: tools/document_segmentation_server.py:1-40, prompts/code_prompts.py:168-180
System Architecture
The segmentation system operates as a Model Context Protocol (MCP) server, providing tools that other agents can call to process and query document content. It consists of three primary internal components: the DocumentAnalyzer, the DocumentSegmenter, and the DocumentIndex.
The diagram above illustrates the flow from raw document input to the retrieval of scored semantic segments.
Sources: tools/document_segmentation_server.py:65-75, workflows/code_implementation_workflow_index.py:270-285
Core Components
DocumentAnalyzer
The DocumentAnalyzer performs semantic content analysis to detect the document type and select the optimal segmentation strategy. It uses weighted indicators to score the density of algorithms, technical concepts, and implementation details.
- Algorithm Indicators: Keywords like "procedure", "method", "framework", and "step".
- Technical Concept Indicators: Keywords like "formula", "theorem", "lemma", and "proof".
- Implementation Indicators: Keywords like "code", "programming", "module", and "interface".
Sources: tools/document_segmentation_server.py:108-160
DocumentSegmenter
This component executes the selected strategy to create intelligent segments. Unlike standard chunkers, it identifies "formula chains" and "algorithm blocks" to prevent breaking critical technical logic across segments.
Sources: tools/document_segmentation_server.py:228-245
Segmentation Strategies
The system dynamically chooses a strategy based on the document's semantic density and detected type.
| Strategy | Usage Context | Key Feature |
|---|---|---|
semantic_research_focused | Academic papers with complex content | Preserves standard paper sections (Intro, Method, etc.) |
algorithm_preserve_integrity | High algorithm density documents | Groups input/output specs with pseudocode blocks |
concept_implementation_hybrid | Mixed conceptual and code documents | Maps implementation details to their defining concepts |
semantic_chunking_enhanced | Extremely long documents (>15k chars) | Uses advanced boundary detection for chunking |
content_aware_segmentation | General technical documents | Adaptive chunking based on content density |
Sources: tools/document_segmentation_server.py:42-55, tools/document_segmentation_server.py:185-200
Data Models
The system maintains a structured index of segments to allow for multi-level relevance scoring.
DocumentSegment Metadata
Each segment is stored with the following attributes:
- content_type: Classified as "introduction", "methodology", "algorithm", "results", etc.
- relevance_scores: Pre-calculated scores for different query types (
concept_analysis,algorithm_extraction,code_planning). - keywords: Extracted using frequency and position weighting (e.g., words in the first 25% of content carry more weight).
Sources: tools/document_segmentation_server.py:82-96, tools/document_segmentation_server.py:730-755
Intelligent Retrieval Process
When an agent requests document segments via read_document_segments, the system performs a scoring and ranking process to return the most relevant context.
This process ensures that "algorithm_extraction" queries prioritize segments containing pseudocode and formulas, while "code_planning" queries prioritize implementation details.
Sources: tools/document_segmentation_server.py:614-635, tools/document_segmentation_server.py:760-785
API Tools
The Document Segmentation Agent provides the following MCP tools:
| Tool Name | Parameters | Description |
|---|---|---|
analyze_and_segment_document | paper_dir, force_refresh | Analyzes structure and creates semantic segments stored in a JSON index. |
read_document_segments | query_type, keywords, max_segments | Retrieves the most relevant segments based on a specific analysis goal. |
get_document_overview | paper_dir | Returns metadata about all available segments and the used strategy. |
Sources: tools/document_segmentation_server.py:25-40, tools/document_segmentation_server.py:539-555
Implementation Detail: Algorithm Integrity
To preserve algorithms, the system identifies patterns such as input:, output:, for each, and while. It expands the segment boundary backward and forward from these matches to include the surrounding natural language description, ensuring the LLM receives the full context of the procedure.
Sources: tools/document_segmentation_server.py:210-225, tools/document_segmentation_server.py:388-410
Conclusion
The Document Segmentation Agent is a critical utility for technical document processing in DeepCode. By moving beyond simple text splitting and employing semantic, content-aware segmentation, it allows implementation agents to access high-density technical information without losing the logical connections between mathematical formulations and their algorithmic implementations.
Sources: tools/document_segmentation_server.py:1-60, prompts/code_prompts.py:180-200