AST-Aware Analysis & Parsing
Relevant source files
The following files were used as context for generating this wiki page:
AST-Aware Analysis & Parsing is a core capability of the Memory Context Browser (MCB) that enables deep codebase understanding through structural representation of source code. Unlike standard text-based search, this system utilizes the Tree-sitter parsing library to generate Abstract Syntax Trees (ASTs) for 14 supported languages, including Rust, Python, and JavaScript/TypeScript.
This structural awareness is leveraged across the project for semantic code search, language-specific chunking, and complex architecture validation. By understanding the syntax and hierarchy of code elements such as functions, classes, and modules, the system can perform sophisticated metric analysis and rule enforcement that remains robust against purely cosmetic code changes.
Sources: README.md:14-16, crates/mcb-providers/Cargo.toml:84-106
Architecture and Core Components
The AST analysis system is integrated into several layers of the MCB workspace. The mcb-providers crate handles the low-level parsing and language-specific chunking, while mcb-validate utilizes AST data for architectural rule enforcement and code metrics calculation.
Analysis Flow
The following diagram illustrates how source code is transformed from raw text into structured insights:
Analysis flow derived from architectural descriptions. Sources: README.md:65-78, crates/mcb-validate/tests/integration/integration_metrics.rs:7-10
Multi-Language Support
The system supports a wide array of languages through specific Tree-sitter parsers integrated as dependencies.
| Category | Supported Languages |
|---|---|
| Mainstream | Rust, Python, Go, Java |
| Web/Scripting | JavaScript, TypeScript, PHP, Ruby |
| Systems/Mobile | C, C++, C#, Swift, Kotlin |
| Metadata | YAML, TOML, JSON (via Schema) |
Sources: README.md:14-16, crates/mcb-providers/Cargo.toml:84-106, crates/mcb-validate/rules/schema.json:101-125
Complexity Metrics Analysis
The RcaAnalyzer (Rust Code Analysis Analyzer) is the primary component for calculating code quality metrics using AST data. It identifies structural patterns to determine the complexity of functions and methods.
Supported Metric Types
- Cognitive Complexity: Measures how difficult code is to understand based on control flow nesting.
- Cyclomatic Complexity: Measures the number of linearly independent paths through a program's source code.
- Function Length: Tracks the physical line count of function bodies.
- Nesting Depth: Measures the maximum depth of nested control structures (e.g., if statements, loops).
Sources: crates/mcb-validate/tests/integration/integration_metrics.rs:207-227, crates/mcb-validate/rules/schema.json:157-220
Metric Analysis Logic
The analyzer extracts functions and methods (including those inside impl blocks in Rust) and evaluates them against defined thresholds.
Declarative Rule Matching
The DeclarativeValidator allows users to define validation rules using YAML schemas that target specific AST nodes. This enables the creation of custom linting rules without modifying the core Rust codebase.
AST Selectors and Queries
Rules can use two methods for structural matching:
- Selectors: Match specific node types (e.g.,
function_itemin Rust orfunction_definitionin Python). - AST Queries: Use Tree-sitter S-expression query syntax for complex pattern matching (e.g., matching a function with a specific name or signature).
Sources: crates/mcb-validate/tests/unit/validators/declarative_validator_tests.rs:56-78, crates/mcb-validate/rules/schema.json:96-135
Rule Schema Definition
The rule schema defines how AST patterns are mapped to severity levels and messages.
{
"selectors": [
{
"language": "rust",
"node_type": "function_item"
}
],
"ast_query": "(function_item name: (identifier) @name)",
"severity": "warning",
"message": "Function node detected"
}
Sources: crates/mcb-validate/rules/schema.json:96-140, crates/mcb-validate/tests/unit/validators/declarative_validator_tests.rs:136-150
Parsing Utilities
MCB provides utility functions in mcb-validate for scanning and filtering source lines before analysis, ensuring that comments and test modules do not skew metrics.
Key Utility Functions
source_lines: Iterates through source lines, skipping comments and#[cfg(test)]modules.extract_functions: Parses non-test source lines to extract structuredFunctionInfoincluding start lines and meaningful bodies.is_fn_signature_or_brace: Helper to identify structural boundaries in source code.
Sources: crates/mcb-validate/src/utils/source.rs:66-90, crates/mcb-validate/src/utils/source.rs:127-141
Conclusion
AST-Aware Analysis & Parsing provides the structural foundation for MCB's advanced features. By moving beyond simple text analysis, the system achieves a higher level of accuracy in code search and enforces strict architectural standards through multi-language AST inspection and complexity metric tracking. This ensures that the context provided to AI agents is structurally sound and adheres to defined project conventions.