Language Chunking
Relevant source files
The following files were used as context for generating this wiki page:
Language Chunking is a core subsystem within the MCB project designed to split source code into semantically meaningful segments for indexing and search. By utilizing Tree-sitter for Abstract Syntax Tree (AST) analysis, the system identifies structural boundaries (such as functions, classes, and modules) rather than relying on arbitrary line breaks. This ensures that the context provided to AI agents is structurally coherent and relevant to the codebase.
The chunking engine supports 14 major programming languages and implements a fallback mechanism for unsupported file types. It serves as the primary data preparation step for semantic code search, allowing the Memory Context Browser to index repositories with high precision.
Sources: README.md:16-17, crates/mcb-domain/src/entities/code_chunk.rs:6-9, crates/mcb-providers/src/language/mod.rs:1-8
System Architecture
The chunking architecture is divided into three layers following Clean Architecture principles: the Domain entity layer, the Provider implementation layer, and the Utility constant layer.
Core Components
| Component | Responsibility |
|---|---|
IntelligentChunker | Orchestrates the chunking process, selecting between AST-based and generic strategies. |
CodeChunk | The primary domain entity representing a segment of code with line numbers and metadata. |
LanguageProcessor | Language-specific implementations (e.g., RustProcessor) that define extraction rules. |
LanguageConfig | Holds configuration like default chunk sizes and priority thresholds for specific languages. |
Sources: crates/mcb-providers/src/language/common/engine/chunker.rs:15-18, crates/mcb-domain/src/entities/code_chunk.rs:36-52, crates/mcb-providers/src/language/mod.rs:10-48
Processing Flow
The system first attempts to detect the programming language based on the file extension. If a supported processor exists, it uses Tree-sitter to parse the content; otherwise, it defaults to a line-based generic chunking method.
The diagram shows the decision path between intelligent AST-based chunking and fallback generic chunking.
Sources: crates/mcb-providers/src/language/common/engine/chunker.rs:30-50
Intelligent Chunker Implementation
The IntelligentChunker is the primary engine for processing code. It implements the CodeChunker port from the domain layer and provides both synchronous and asynchronous (via tokio::task::spawn_blocking) interfaces.
AST-Based Extraction
For supported languages, the chunker utilizes the LANGUAGE_PROCESSORS registry. It invokes language-specific logic to navigate the AST and extract nodes based on NodeExtractionRule definitions, which include parameters for min_length, max_depth, and priority.
Generic Fallback
If AST parsing fails or the language is unsupported, the system employs chunk_generic. This method splits content into fixed-size windows defined by CHUNK_SIZE_GENERIC (defaulting to 15 lines).
// crates/mcb-providers/src/language/common/engine/chunker.rs:77-109
fn chunk_generic(content: &str, file_name: &str, language: &Language) -> Vec<CodeChunk> {
let lines: Vec<&str> = content.lines().collect();
let mut chunks = Vec::new();
let chunk_size = CHUNK_SIZE_GENERIC;
for (chunk_idx, chunk_lines) in lines.chunks(chunk_size).enumerate() {
let start_line = chunk_idx * chunk_size;
let end_line = start_line + chunk_lines.len() - 1;
// ... creation of CodeChunk entity
}
chunks
}
Sources: crates/mcb-providers/src/language/common/engine/chunker.rs:52-75, crates/mcb-utils/src/constants/lang.rs:69-106
Language Configuration and Constants
The system maintains a centralized set of constants in mcb-utils to ensure consistency across the indexing pipeline.
Language Mappings
The system maps file extensions to internal Language identifiers and defines specific chunk sizes for optimized processing.
| Language | Extension | Chunk Size (Lines) |
|---|---|---|
| Rust | .rs | 20 |
| Python | .py, .pyi | 15 |
| JS / TS | .js, .ts | 15 |
| Go | .go | 15 |
| Java | .java | 15 |
Sources: crates/mcb-utils/src/constants/lang.rs:114-135, crates/mcb-utils/src/constants/lang.rs:137-152
Extraction Parameters
Default constraints are applied to node extraction to prevent the creation of noisy or trivial chunks:
- MIN_LENGTH: 20 characters.
- MIN_LINES: 1 line.
- MAX_DEPTH: 3 levels deep in the AST.
- MAX_CHUNKS_PER_FILE: 75 chunks.
Sources: crates/mcb-utils/src/constants/lang.rs:56-58, crates/mcb-utils/src/constants/lang.rs:98-106
Data Model: CodeChunk
The CodeChunk entity represents the output of the chunking process. It contains the raw content alongside rich metadata used by vector stores and search handlers.
The diagram represents the structure of the CodeChunk entity and its relationship with the Language value object.
Sources: crates/mcb-domain/src/entities/code_chunk.rs:36-52
Supported Languages
MCB currently provides complete AST processors for the following languages via Tree-sitter integration:
- Systems: Rust, C, C++, C#
- Web/Backend: JavaScript, TypeScript, Go, Java, PHP, Ruby
- Mobile/Modern: Swift, Kotlin
- Data/Scripting: Python
Sources: crates/mcb-providers/src/language/mod.rs:25-40, crates/mcb-infrastructure/Cargo.toml:104-118
Conclusion
Language Chunking in MCB provides the structural intelligence necessary for semantic code understanding. By combining AST-based extraction for high-fidelity languages with a robust generic fallback, the system ensures that every file in a repository can be indexed and retrieved with contextually appropriate boundaries. This architecture facilitates precise search results by maintaining the semantic integrity of code blocks throughout the vectorization process.