Tree-sitter Parsing Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Tree-sitter Parsing Pipeline is the core structural engine of the code-review-graph project. It transforms raw source code into a multi-language Abstract Syntax Tree (AST), which is then processed into a persistent knowledge graph stored in SQLite. This pipeline enables token-efficient code reviews by providing AI tools with precise structural context—such as callers, callees, and inheritance—rather than requiring the ingestion of entire files.
The pipeline is designed for high performance, supporting incremental updates that re-parse only changed files. It handles a wide array of programming languages, ranging from systems languages like C++ and Rust to web technologies like TypeScript and Vue SFCs, as well as Jupyter Notebooks.
Sources: CLAUDE.md:5-10, README.zh-CN.md:工作原理
Pipeline Architecture
The parsing process follows a structured data flow where source code is converted into nodes and edges within a SQLite-backed graph store.
The diagram shows how files enter the pipeline, undergo change detection, are parsed via Tree-sitter, and eventually result in a structural graph.
Sources: CLAUDE.md:16-25, README.zh-CN.md:工作原理
Core Components
1. Multi-language Parser
The pipeline utilizes Tree-sitter grammars to extract nodes (functions, classes, imports) and edges (calls, inheritance, test coverage). For complex scenarios where Tree-sitter may be insufficient, the system uses targeted fallbacks to ensure broad source-language support.
Key extracted elements include:
- Nodes: Files, Classes, Functions, Types, and Tests.
- Edges:
CALLS,INHERITS,IMPORTS_FROM,CONTAINS,TESTED_BY, andIMPLEMENTS.
Sources: CLAUDE.md:16-18, visualization.py:461-470, CONTRIBUTING.md:46-52
2. Incremental Update Logic
To maintain performance in large repositories, the pipeline tracks changes via SHA-256 hashing. It identifies which files have changed since the last build and only re-indexes those specific files and their immediate dependencies.
| Feature | Description |
|---|---|
| Full Build | Re-parses every file in the repository to rebuild the graph from scratch. |
| Incremental Update | Uses Git-based detection and file hashes to update only the modified portions. |
| Watch Mode | A background daemon that triggers updates automatically upon file system changes. |
Sources: CLAUDE.md:21, main.py:84-106, README.zh-CN.md:增量更新
3. Post-processing and Enrichment
Once the initial AST data is stored, the pipeline runs post-processing steps to generate higher-level architectural insights.
- Flow Detection: Traces execution paths from entry points (e.g., HTTP handlers or CLI commands) to calculate "criticality" scores for specific code paths.
- Community Detection: Uses the Leiden algorithm to cluster related code entities into architectural "communities."
- Full-Text Search (FTS): Rebuilds a SQLite FTS5 index for hybrid search (keyword + vector).
Sources: main.py:121-140, CLAUDE.md:27-32, tests/test_integration_v2.py:165-185
Data Model and Schema
The resulting graph is stored in a SQLite database (typically located at .code-review-graph/graph.db).
Entity Relationship Diagram
The schema represents nodes as code entities and edges as the relationships between them.
Sources: tests/test_integration_v2.py:50-130, visualization.py:41-60
Implementation Details
The pipeline is primarily invoked through the build_or_update_graph_tool, which manages the transition between parallel file parsing and sequential database commits. On Windows, the pipeline uses WindowsSelectorEventLoopPolicy and asyncio.to_thread to avoid deadlocks during high-intensity parsing and process-pool interactions.
# code_review_graph/main.py:84-110
@mcp.tool()
async def build_or_update_graph_tool(
full_rebuild: bool = False,
repo_root: Optional[str] = None,
base: str = "HEAD~1",
postprocess: str = "full",
recurse_submodules: Optional[bool] = None,
) -> dict:
return await asyncio.to_thread(
build_or_update_graph,
full_rebuild=full_rebuild,
repo_root=_resolve_repo_root(repo_root),
base=base,
postprocess=postprocess,
recurse_submodules=recurse_submodules,
)
Sources: main.py:84-110, main.py:750-765
Summary
The Tree-sitter Parsing Pipeline provides the foundational intelligence for the code-review-graph system. By combining high-fidelity AST parsing with incremental updates and architectural post-processing, it creates a structural map that significantly reduces token consumption for AI agents. This pipeline ensures that instead of reading entire files, AI tools can perform "impact radius" analysis to focus only on the code that truly matters for a given change or task.
Sources: CLAUDE.md:5-10, README.zh-CN.md:功能一览