Generating Grammars with Enums
Relevant source files
The following files were used as context for generating this wiki page:
The rust-code-analysis project utilizes a specialized mechanism for generating language grammars through an internal crate named enums. This system bridges the gap between external Tree-sitter parsing libraries and the Rust-based metrics evaluation engine. By converting Tree-sitter grammars into structured Rust enums, the project ensures type safety and high performance when traversing Abstract Syntax Trees (ASTs) to compute various Supported Metrics.
The workflow involves integrating a Tree-sitter crate, registering it within the enums crate metadata, and executing generation scripts that output specialized Rust source files. These generated files contain the enum definitions used by the primary library to identify and process different syntactic constructs across all supported languages.
Sources: rust-code-analysis-book/src/developers/new-language.md:14-19, rust-code-analysis-book/src/developers/update-grammars.md:5-9, src/lib.rs:13-25
Grammar Generation Architecture
The architecture relies on the enums crate acting as a code generator. It consumes Tree-sitter language definitions and produces Rust code using templates.
Core Components
| Component | Responsibility |
|---|---|
| enums Crate | The primary generator that iterates through supported languages to create Rust enum files. |
| Tree-sitter Crates | External or internal dependencies (e.g., tree-sitter-rust) providing the base grammar data. |
| RustTemplate | An Askama template used to format the generated Rust enum code. |
| recreate-grammars.sh | The main automation script that triggers the enums crate and cleans up previous builds. |
Sources: enums/src/rust.rs:11-18, rust-code-analysis-book/src/developers/new-language.md:16-19, recreate-grammars.sh:3-12
Data Flow for New Language Support
The following diagram illustrates the sequence of operations required to generate a new grammar and integrate it into the core library.
The flow starts with dependency management and registration, ending with the generation of physical Rust source files that the main library imports.
Sources: rust-code-analysis-book/src/developers/new-language.md:21-41, enums/src/rust.rs:21-40
The Generation Process
The generation logic is primarily contained within enums/src/rust.rs. It utilizes the generate_rust function to iterate over all registered languages.
Language Registration and Code Generation
- Iteration: The system iterates through
Lang::into_enum_iter()to process every supported language. - Metadata Extraction: For each language, it retrieves the Tree-sitter language object, the language name, and token names.
- Template Rendering: It populates a
RustTemplatestruct with the camel-cased language name and a vector of token names. - File Creation: A new
.rsfile is created in the target directory (usuallysrc/languages/) using a naming convention based on the language (e.g.,language_rust.rs).
pub fn generate_rust(output: &Path, file_template: &str) -> std::io::Result<()> {
for lang in Lang::into_enum_iter() {
let language = get_language(&lang);
let name = get_language_name(&lang);
let c_name = camel_case(name.to_string());
// ... file creation and rendering logic
let args = RustTemplate { c_name, names };
file.write_all(args.render().unwrap().as_bytes())?;
}
Ok(())
}
Sources: enums/src/rust.rs:21-38, rust-code-analysis-book/src/developers/new-language.md:36-39
Macro Integration
The generated enums are consumed by the main library via several internal macros defined in src/macros.rs. These macros automate the implementation of traits and the creation of parser instances.
mk_langs!: The top-level macro that defines supported languages, their extensions, and associated parsers.get_language!: Maps internal identifiers to specific Tree-sitter language functions (e.g.,tree_sitter_rust::language()).mk_code!: Generates the struct and parser type for a specific language.
Sources: src/macros.rs:1-12, src/macros.rs:242-251, rust-code-analysis-book/src/developers/new-language.md:58-85
Internal and Third-Party Grammars
The project distinguishes between standard third-party grammars (available on crates.io) and internal Mozilla-defined grammars.
Third-Party Updates
Third-party grammars are updated by modifying the version in the main Cargo.toml and enums/Cargo.toml, then running ./recreate-grammars.sh.
Sources: rust-code-analysis-book/src/developers/update-grammars.md:19-28
Mozilla-Specific Grammars
Internal grammars like tree-sitter-mozcpp and tree-sitter-mozjs require custom scripts because they often inherit from or modify standard grammars.
The sequence above shows how generate-mozcpp.sh handles symbol conflicts by renaming Tree-sitter functions before generating the final grammar.
Sources: generate-grammars/generate-mozcpp.sh:22-63, generate-grammars/generate-grammar.sh:9-16, rust-code-analysis-book/src/developers/update-grammars.md:46-60
Automation Scripts
The project provides several bash scripts to maintain grammar consistency.
| Script | Purpose |
|---|---|
recreate-grammars.sh | Cleans enums build, runs the generator for all languages and C macros, and formats output. |
generate-grammar.sh | Generates a specific Mozilla-defined grammar using npm and the Tree-sitter CLI. |
generate-mozjs.sh | Specialized script for updating Mozilla JavaScript grammars with proper symbol prefixes. |
Sources: recreate-grammars.sh:1-12, generate-grammars/generate-grammar.sh:1-15, generate-grammars/generate-mozjs.sh:45-56
C Macro Generation
In addition to language grammars, the enums crate generates Rust representations of C macros and special tokens. This is handled by create_macros_file which reads definitions from .txt files in a data directory and renders them using the CMacrosTemplate.
Sources: enums/src/rust.rs:41-76
Summary
The "Generating Grammars with Enums" system is a critical build-time feature that enables rust-code-analysis to support a wide array of languages. By automating the conversion of Tree-sitter grammars into Rust enums via the enums crate and associated shell scripts, the project maintains a type-safe interface for AST analysis while allowing for rapid updates to both third-party and custom internal language definitions.