AST-Aware Parsing
Relevant source files
The following files were used as context for generating this wiki page:
AST-Aware Parsing in the MCB project provides deep codebase understanding by leveraging the Abstract Syntax Tree (AST) of various programming languages. Unlike simple text-based scanning, this system uses tree-sitter and specialized analysis tools to recognize logical code structures such as functions, classes, and modules across 14 supported languages including Rust, Python, and JavaScript.
This capability is central to the project's semantic search, code chunking, and architecture validation features. It allows the system to enforce complex structural rules, calculate code metrics (like cognitive complexity), and provide context-aware insights to AI agents.
Sources: README.md:16-17, crates/mcb-providers/Cargo.toml:110-123, crates/mcb-validate/Cargo.toml:73-82
Core Components and Architecture
The AST-aware parsing infrastructure is distributed across specialized crates, primarily mcb-providers for data extraction and mcb-validate for structural analysis.
Parsing Infrastructure
The project utilizes the tree-sitter ecosystem for robust, incremental parsing. The mcb-providers crate includes parsers for a wide array of languages to support diverse codebases.
| Component | Description |
|---|---|
| tree-sitter | The core incremental parsing library used for AST generation. |
| Language Parsers | Specific crates (e.g., tree-sitter-rust, tree-sitter-python) that define the grammar for individual languages. |
| rust-code-analysis | A specialized library used for advanced metrics calculation like cyclomatic and cognitive complexity. |
Sources: crates/mcb-providers/Cargo.toml:107-123, crates/mcb-validate/Cargo.toml:73-82, crates/mcb-validate/tests/integration/integration_metrics.rs:11-12
Data Flow Diagram
The following diagram illustrates how source code is transformed into actionable AST data and metrics within the system.
The diagram shows the dual path for code analysis: structural parsing via tree-sitter and metric analysis via rust-code-analysis.
Sources: crates/mcb-validate/src/utils/source.rs:163-176, crates/mcb-validate/tests/unit/validators/declarative_validator_tests.rs:110-125, crates/mcb-validate/tests/integration/integration_metrics.rs:24-35
Structural Extraction
MCB provides utilities to extract meaningful units from code, such as functions and blocks, while filtering out noise like comments or test code.
Function Analysis
The FunctionInfo structure captures metadata about parsed functions, including their body lines and control-flow characteristics.
- Meaningful Body: Body lines excluding braces and signatures.
- Control Flow Detection: Identifies if the function contains logic branching.
- Start Line: Tracks the 1-based position for reporting and navigation.
Sources: crates/mcb-validate/src/utils/source.rs:136-150
Rule-Based Selectors
The validation engine uses YAML-defined rules that leverage AST selectors to target specific code constructs.
selectors:
- language: "rust"
node_type: "function_item"
ast_query: "(function_item name: (identifier) @name)"
Sources: crates/mcb-validate/tests/unit/validators/declarative_validator_tests.rs:118-125, crates/mcb-validate/rules/schema.json:96-135
Metrics and Complexity Analysis
Beyond structural identity, the system analyzes the "quality" of code using metrics derived from the AST. This is handled primarily by the RcaAnalyzer.
Supported Metrics
The system tracks several key complexity indicators:
- Cognitive Complexity: Measures how difficult a function is to understand (e.g., nested
ifstatements). - Cyclomatic Complexity: Measures the number of linearly independent paths through the code.
- Function Length: Tracks the number of lines in a single function to prevent "god functions."
- Nesting Depth: Monitors the depth of nested blocks.
Sources: crates/mcb-validate/tests/integration/integration_metrics.rs:41-94, crates/mcb-validate/rules/schema.json:148-228
Metrics Validation Sequence
This sequence shows how the RcaAnalyzer evaluates a file against configured thresholds.
Sequence of events during a metrics-based validation check.
Sources: crates/mcb-validate/tests/integration/integration_metrics.rs:112-133, crates/mcb-validate/tests/integration/integration_metrics.rs:271-290
Language Support
The parsing system is designed to be multi-language, supporting various syntax grammars through the Model Context Protocol (MCP) and tree-sitter.
| Language | Primary Node Types | Parser Crate |
|---|---|---|
| Rust | source_file, function_item | tree-sitter-rust |
| Python | module, function_definition | tree-sitter-python |
| JavaScript | program, function_declaration | tree-sitter-javascript |
Sources: crates/mcb-providers/Cargo.toml:110-123, crates/mcb-validate/rules/schema.json:101-125, crates/mcb-validate/tests/unit/validators/declarative_validator_tests.rs:56-61
Conclusion
AST-Aware Parsing moves the MCB project beyond simple string matching into true structural understanding. By combining tree-sitter's precise parsing with the metric analysis capabilities of rust-code-analysis, the system provides a robust foundation for architecture validation and intelligent code navigation. This enables AI agents to interact with codebases at the level of logical abstractions rather than just raw text.
Sources: README.md:16-17, crates/mcb-validate/src/utils/source.rs:163-176, crates/mcb-validate/tests/integration/integration_metrics.rs:341-356