File Processing Utilities
Relevant source files
The following files were used as context for generating this wiki page:
File Processing Utilities in the DeepCode project encompass a suite of tools, prompts, and agentic workflows designed to automate the lifecycle of technical documentation and source code management. The system is primary oriented toward "paper-to-code" tasks, where research papers are analyzed, downloaded, converted, and subsequently used to generate executable codebases. These utilities manage the transition from raw input (URLs, PDFs, or text) to structured file trees and production-grade implementations.
The scope of these utilities includes input classification, metadata extraction, directory structure generation via shell commands, and concise memory management for file-by-file code generation. By leveraging Model Context Protocol (MCP) tools, the system maintains a workspace where files are created, read, and tested iteratively.
Sources: prompts/code_prompts.py:34-118, workflows/code_implementation_workflow.py:68-112
Paper Input Analysis and Downloading
The initial phase of the file processing pipeline involves identifying and normalizing the source material. The system uses a specialized agent to analyze strings and determine if they represent local file paths, remote URLs, or raw text.
Input Classification Rules
The PAPER_INPUT_ANALYZER_PROMPT defines the logic for classifying inputs into four distinct types:
- URL: Inputs starting with
http://orhttps://. - File: Local paths pointing to PDF files.
- Directory: Paths pointing to folders.
- Text: Inputs with no detected path or URL.
Sources: prompts/code_prompts.py:41-60
Processing and Format Conversion
Once classified, the PAPER_DOWNLOADER_PROMPT governs the movement and conversion of files. Papers are saved using a numeric ID system in the ./deepcode_lab/papers/ directory. If a file is provided in PDF format, the utility triggers a file-downloader tool to convert the content into Markdown (.md) for easier LLM consumption.
| Input Type | Action Taken | Tool Used |
|---|---|---|
| URL | Download and extract metadata | file-downloader |
| File | Move to labs and convert to .md | file-downloader |
| Directory | Verify existence; set failure status | N/A |
| Text | Use provided info; no file ops | N/A |
Sources: prompts/code_prompts.py:73-95
File Tree and Project Structure Generation
After analysis, the system generates the physical project structure. This is handled by a StructureGeneratorAgent which interprets an implementation plan to create a directory hierarchy.
Shell Command Generation
The utility translates the "File Structure" section of an implementation plan into a series of shell commands. It strictly uses mkdir -p for directories and touch for files to ensure the environment is ready for code implementation.
Sources: prompts/code_prompts.py:532-550, workflows/code_implementation_workflow.py:157-190
This diagram illustrates the flow from plan analysis to physical directory creation:
Iterative Code Implementation and Writing
The core utility for file creation is the code implementation workflow, which operates on a file-by-file basis using the write_file tool.
The Write-File Protocol
Implementation is governed by the CODE_IMPLEMENTATION_PROMPT. Developers (agents) are instructed to use heredoc syntax (cat > [path] << 'EOF') to write complete, executable files. This ensures that the environment remains consistent and that files are created in a specific logical order:
- Configuration/Environment files.
- Core utilities and base classes.
- Main implementation modules.
- Tests and documentation.
Sources: prompts/code_prompts.py:575-603, workflows/code_implementation_workflow.py:204-245
Implementation Components Tracking
The ConciseMemoryAgent tracks the status of these files to avoid redundant work. It parses the initial plan to build an all_files_list and monitors the write_file tool results to move files from "unimplemented" to "implemented" status.
Sources: workflows/agents/memory_agent_concise.py:108-142, workflows/agents/memory_agent_concise_index.py:84-106
Memory-Efficient File Summarization
To maintain context without exceeding token limits, the system implements a "Code Knowledge Base." This utility summarizes every file as it is written.
The Summary Loop
When a file is implemented via write_file, the create_code_implementation_summary method is triggered. It uses an LLM to analyze the code and extract:
- Implementation Progress: Core ideas and completion status.
- Dependencies: Function signatures, parameters, and return types required by other files.
- Required Packages: Dependencies for
requirements.txt.
Sources: workflows/agents/memory_agent_concise.py:263-315, workflows/agents/memory_agent_concise_index.py:263-315
Storage and Persistence
Summaries are appended to a persistent Markdown file named implement_code_summary.md. This file serves as the long-term memory for the project, allowing subsequent agent rounds to use read_code_mem to understand previously written code without reading the entire source file.
Workspace Management Tools
The system relies on Model Context Protocol (MCP) tools to interact with the file system. These tools are coordinated by the CodeImplementationWorkflow class.
| Tool Name | Purpose | Implementation Reference |
|---|---|---|
set_workspace | Sets the base directory for all file operations | code_implementation_workflow.py:465 |
write_file | Creates or updates a file with provided content | memory_agent_concise.py:530 |
read_file | Reads the raw content of a specific file | memory_agent_concise.py:531 |
read_code_mem | Retrieves summaries from implement_code_summary.md | memory_agent_concise.py:529 |
get_file_structure | Returns a visual tree of the current workspace | memory_agent_concise.py:536 |
Sources: workflows/code_implementation_workflow.py:460-475, workflows/agents/memory_agent_concise.py:528-540
File Processing Utilities facilitate a structured, automated approach to software reproduction by bridging the gap between academic paper analysis and physical code implementation. By combining robust shell command generation for project scaffolding with an intelligent, summary-based memory system, the utilities ensure that complex codebases can be built iteratively and accurately.