Code Splitters & AST Analysis
Relevant source files
The following files were used as context for generating this wiki page:
Code splitting is a fundamental component of the Claude Context architecture, responsible for breaking down raw source code into manageable, semantically meaningful "chunks" for vector embedding and retrieval. By utilizing Abstract Syntax Tree (AST) analysis, the system identifies logical code units such as functions, classes, and interfaces to ensure that context is preserved within each chunk rather than being sliced arbitrarily by character counts.
The project implements two primary strategies: a sophisticated AST-based splitter and a character-based fallback splitter (LangChain). This multi-layered approach ensures that even when complex language parsers are unavailable or fail, the system maintains high-quality indexing capabilities across a wide range of programming environments.
Sources: packages/core/README.md, packages/core/src/context.ts:114-118
Architecture and Core Logic
The splitting system is designed around a unified Splitter interface, which mandates a split method returning CodeChunk objects. The Context class acts as the orchestrator, delegating file content to the configured splitter based on the environment and language.
The diagram shows the decision logic for selecting a splitting strategy and the subsequent refinement process.
Sources: packages/core/src/splitter/ast-splitter.ts:44-63, packages/core/src/context.ts:544-555
Data Structures
The system uses standardized structures to pass data between the splitters and the indexing engine.
| Interface | Field | Type | Description |
|---|---|---|---|
| CodeChunk | content | string | The actual text of the code segment. |
metadata | object | Contains startLine, endLine, language, and filePath. | |
| Splitter | split() | Method | The core function to transform code into chunks. |
setChunkSize() | Method | Configures maximum character limit for chunks. |
Sources: packages/core/src/splitter/index.ts:1-20
AST Code Splitter
The AstCodeSplitter leverages tree-sitter parsers to understand the syntactic structure of the code. It identifies "splittable node types"—specific AST nodes that represent self-contained logical units.
Splittable Node Types by Language
The splitter identifies specific nodes to create clean boundaries for chunks.
| Language | Node Types Tracked for Splitting |
|---|---|
| TypeScript / JS | function_declaration, class_declaration, method_definition, interface_declaration, type_alias_declaration |
| Python | function_definition, class_definition, decorated_definition, async_function_definition |
| Java / Scala | method_declaration, class_declaration, interface_declaration, constructor_declaration |
| Go | function_declaration, method_declaration, type_declaration, var_declaration |
| Rust | function_item, impl_item, struct_item, enum_item, mod_item |
Sources: packages/core/src/splitter/ast-splitter.ts:14-25, packages/vscode-extension/src/stubs/ast-splitter-stub.js:30-40
Refinement and Overlap Logic
Even when split by AST nodes, some logical units (like very large classes) may exceed the chunkSize. In these cases, the AstCodeSplitter performs a secondary "refine" step.
- AST Extraction: Traverse the tree and extract nodes matching the language's splittable types.
- Size Check: If a node's content exceeds the
chunkSize(default 2500), it is passed to a character-based sub-splitter. - Overlap Addition: To prevent loss of context at boundaries, the splitter adds a
chunkOverlap(default 300) by prepending text from the end of the previous chunk to the current one.
Sources: packages/core/src/splitter/ast-splitter.ts:101-135, packages/core/src/splitter/ast-splitter.ts:175-200
VSCode Web-Tree-Sitter Implementation
In the VSCode extension environment, the system utilizes web-tree-sitter via a specialized stub. This allows AST analysis to run within the extension's restricted Node.js context by loading .wasm language parsers dynamically.
The sequence shows the initialization and loading of WASM-based parsers for AST analysis in the browser-like VSCode environment.
Sources: packages/vscode-extension/src/stubs/ast-splitter-stub.js:68-100, packages/vscode-extension/src/stubs/ast-splitter-stub.js:145-165
Fallback Strategy (LangChain Splitter)
When AST parsing is not possible—due to unsupported languages or parser initialization failures—the system falls back to the LangChainCodeSplitter. This splitter uses character-based logic while attempting to respect line boundaries.
Configuration Options
These settings are configurable via the ContextConfig or VSCode settings.
| Parameter | Type | Default | Description |
|---|---|---|---|
chunkSize | number | 2500 (AST) / 1000 (LC) | Maximum characters per chunk. |
chunkOverlap | number | 300 (AST) / 200 (LC) | Overlap between consecutive chunks. |
splitterType | enum | 'ast' | Choice between ast or langchain. |
Sources: packages/core/src/context.ts:52-55, packages/core/README.md:120-130
Conclusion
The "Code Splitters & AST Analysis" system provides the semantic intelligence necessary for effective code retrieval. By prioritizing structural analysis via AST trees and providing robust character-based fallbacks, the project ensures that code is indexed in logically coherent blocks. This architecture directly supports the high-quality context provided to Claude, enabling more accurate AI-assisted development across diverse programming languages.
Sources: packages/core/src/context.ts:120-135, packages/core/README.md:200-210