Language Parser Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Language Parser Engine serves as the foundational multi-language analysis component of the Arbor ecosystem. It leverages Tree-sitter to transform raw source code into structured Abstract Syntax Trees (ASTs), specifically designed to extract semantic entities such as functions, classes, and their inter-relationships.
This engine provides high-performance code analysis through two primary mechanisms: a legacy extension-based extraction system and a modern, query-driven engine (ArborParser). It is architected to support incremental updates, ensuring that re-parsing files updates existing nodes within the semantic dependency graph rather than creating duplicates.
Sources: crates/arbor-core/README.md, crates/arbor-core/src/parser_v2.rs:7-14, crates/arbor-core/src/lib.rs:1-10
Engine Architecture
The parser engine is divided into a core orchestration layer and language-specific implementations. It handles file reading, automatic language detection, AST generation, and entity extraction.
Data Flow for File Parsing
The following diagram illustrates the lifecycle of a source file as it is processed by the engine into semantic nodes and relations.
The engine first identifies the programming language via the file extension before invoking the Tree-sitter parser to generate a concrete tree.
Sources: crates/arbor-core/src/parser.rs:32-51, crates/arbor-core/src/parser_v2.rs:118-144
Core Components
| Component | Description |
|---|---|
ArborParser | The high-performance parser that manages compiled Tree-sitter queries for multiple languages. |
LanguageParser | A trait defined for language-specific extraction logic. |
CodeNode | The primary data structure representing a code entity (e.g., Function, Class). |
SymbolRelation | Represents an edge between two symbols, such as a call or an import. |
CompiledQueries | A cache of pre-compiled Tree-sitter queries (symbols, imports, calls) to optimize indexing. |
Sources: crates/arbor-core/src/parser_v2.rs:25-85, crates/arbor-core/src/node.rs, crates/arbor-core/src/languages/mod.rs
Symbol and Relation Extraction
The ArborParser utilizes pre-compiled queries to identify symbols and their interactions. This approach allows for efficient scanning of large codebases by searching for specific patterns in the AST.
Supported Node Entities
The engine extracts a wide variety of symbols depending on the language's syntax:
- Functions & Methods: Declarations and definitions.
- Classes & Structs: Object-oriented and data structures.
- Interfaces & Traits: Contract definitions.
- Modules & Namespaces: Scoping entities.
- Imports & Includes: Dependency markers.
Sources: crates/arbor-core/src/parser_v2.rs:184-230, crates/arbor-core/src/languages/cpp.rs:35-130
Relation Types
The engine tracks how symbols interact through RelationType enums, which are used to build the global dependency graph.
Enclosing Symbol Resolution
When a function call is detected, the engine must determine its "parent" or caller. It does this by searching for the smallest symbol that encompasses the line number where the call occurs.
fn find_enclosing_symbol<'a>(
&self,
line: u32,
symbols: &'a [CodeNode],
) -> Option<&'a CodeNode> {
symbols
.iter()
.filter(|s| s.line_start <= line && s.line_end >= line)
.min_by_key(|s| s.line_end - s.line_start) // Smallest enclosing
}
Sources: crates/arbor-core/src/parser_v2.rs:341-351
Language-Specific Implementations
The engine supports 10 languages, with C++, Rust, and TypeScript having detailed extraction logic.
C++ Parser Logic
The C++ implementation (CppParser) recursively traverses the AST to handle complex structures like namespaces and nested class members.
- Namespaces: Extracted as
NodeKind::Module. - Classes/Structs: Members are extracted by recursing into
field_declaration_listordeclaration_list. - Visibility: Detects
public,protected, andprivatekeywords to set theVisibilityofCodeNode. - Signatures: Reconstructs function signatures by combining return types, names, and parameters.
Sources: crates/arbor-core/src/languages/cpp.rs:16-110, crates/arbor-core/src/languages/cpp.rs:315-330
Multi-Language Query Support
The modern ArborParser uses language-specific query strings to find nodes.
| Language | Symbol Query Focus | Call Query Focus |
|---|---|---|
| Rust | function_item, struct_item, trait_item | call_expression |
| Python | function_definition, class_definition | call |
| TypeScript | class_declaration, method_definition | call_expression with member support |
| Java | method_declaration, constructor_declaration | method_invocation |
Sources: crates/arbor-core/src/parser_v2.rs:360-490
Error Handling
The parsing process is designed to be resilient. While the engine handles I/O errors and unsupported languages via the ParseError enum, it is capable of graceful degradation during syntax errors, extracting as much information as possible from a partially valid tree.
| Error Variant | Cause |
|---|---|
IoError | Failure to read the source file from disk. |
EmptyFile | File contains no content (except valid Python __init__.py). |
UnsupportedLanguage | Extension does not map to a supported parser. |
QueryError | Failure to compile a Tree-sitter query string. |
ParserError | Tree-sitter failed to generate a tree from source. |
Sources: crates/arbor-core/src/error.rs, crates/arbor-core/src/parser.rs:38-48, crates/arbor-core/src/parser_v2.rs:136-140
Summary
The Language Parser Engine is the critical first step in Arbor's code analysis pipeline. By combining the precision of Tree-sitter ASTs with high-performance cached queries, it extracts the essential nodes and relationships required to build a semantic dependency graph. This engine supports incremental updates and handles multiple paradigms—from C++ namespaces to Python decorators—enabling deterministic, execution-aware analysis across diverse codebases.