Code Indexing Tools
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Code Indexing Tools within the DeepCode project serve as an intelligent bridge between existing codebases and target implementation requirements. These tools analyze, categorize, and map source code relationships to facilitate high-accuracy paper reproduction and automated software development. By leveraging Large Language Models (LLMs) and structured metadata extraction, the system builds searchable indexes that guide implementation agents through complex architectural patterns and functional dependencies.
The scope of these tools includes automated file structure extraction from implementation plans, repository-wide relationship mapping, and the generation of concise code summaries for memory management. These components work together to ensure that implementation agents have a context-aware understanding of both the "Working Directory" and "Reference Indexes" during the development lifecycle.
Core Components and Architecture
Codebase Index Workflow
The indexing process is orchestrated by a central workflow that integrates file analysis, relationship building, and report generation. It utilizes an initial implementation plan to define a target structure and then maps existing codebase components to that structure.
The workflow manages the transition from raw project plans to structured intelligence, supporting specific directory patterns like src/, core/, and lib/ while filtering out non-essential directories such as __pycache__ or node_modules.
Sources: workflows/codebase_index_workflow.py:27-42, workflows/codebase_index_workflow.py:385-400
Code Reference Indexer
This specialized Model Context Protocol (MCP) tool provides unified searching capabilities for implementation agents. It handles the loading of index files and provides structured implementation suggestions based on relevance scores and direct relationships.
| Function | Purpose | Key Data Structures |
|---|---|---|
search_code_references | Unified tool for searching reference code patterns. | CodeReference, RelationshipInfo |
load_index_files | Parses JSON index files from the directory. | Dict[str, Dict] cache |
calculate_relevance | Computes similarity based on filename, type, and keywords. | Relevance Score (0.0 - 1.0) |
Sources: tools/code_reference_indexer.py:38-65, tools/code_reference_indexer.py:126-155
Indexing Logic and Data Flow
Relationship Mapping and Scoring
The indexing tools calculate relevance between a target file (the file being implemented) and reference files (existing code) using a multi-weighted scoring algorithm.
- Filename Similarity (0.3 weight): Matches stems of target and reference names.
- File Type Matching (0.2 weight): Matches file extensions.
- Keyword Matching (0.5 weight): Scans main functions, key concepts, and summaries.
Sources: tools/code_reference_indexer.py:126-155
Memory and Summary Indexing
During active implementation, the system maintains a "Knowledge Base" of the latest code changes. This is handled by the ConciseMemoryAgent, which summarizes implementations to prevent token overflow while preserving dependency information.
Configuration and Management
The indexing behavior is governed by a detailed configuration schema that defines path locations, LLM parameters, and file filtering rules.
| Config Key | Description | Default / Example Value |
|---|---|---|
max_file_size | Maximum file size for analysis. | 1,048,576 (1MB) |
supported_extensions | List of parsable code files. | .py, .js, .ts, .java, .cpp |
min_confidence | Minimum score to include a relationship. | 0.3 |
request_delay | Delay between LLM calls to avoid rate limits. | 0.5s |
Sources: workflows/codebase_index_workflow.py:255-300
Prompt Engineering for Indexing
The system uses specific prompts to drive the indexing and analysis phases:
- PAPER_REFERENCE_ANALYZER_PROMPT: Identifies GitHub repositories with high implementation value for referencing.
- PAPER_ALGORITHM_ANALYSIS_PROMPT: Uses segmented reading to extract formulas and implementation hints into structured YAML.
Sources: prompts/code_prompts.py:100-140, prompts/code_prompts.py:175-200
Conclusion
Code Indexing Tools represent the intelligence layer of the DeepCode framework. By combining automated codebase scanning in CodebaseIndexWorkflow with the retrieval capabilities of CodeReferenceIndexer, the system enables agents to perform informed, dependency-aware code generation. The use of concise memory summaries further ensures that the indexing process remains scalable for large-scale paper reproductions and multi-file projects.