Managing and Updating Grammars
Relevant source files
The following files were used as context for generating this wiki page:
In rust-code-analysis, grammars serve as the foundational logic for parsing various programming languages to extract syntax and semantic data. The project utilizes tree-sitter as the parsing engine. Grammars are categorized into third-party grammars (external crates) and internal grammars (Mozilla-defined variants for Firefox-specific needs).
Because grammars are not static and are subject to bug fixes or language evolution, they must be updated periodically. These updates are managed through a combination of manual configuration changes in Cargo.toml files and automated Bash and Python scripts.
Sources: rust-code-analysis-book/src/developers/update-grammars.md:3-15, README.md:16-20
Grammar Categorization
The project maintains two distinct types of grammars, each requiring different update procedures.
| Grammar Type | Description | Maintenance |
|---|---|---|
| Third-party | Maintained by external developers; published on crates.io. | Update version in Cargo.toml. |
| Internal | Mozilla-defined grammars (e.g., tree-sitter-mozcpp, tree-sitter-mozjs). | Manual script execution and local generation. |
Sources: rust-code-analysis-book/src/developers/update-grammars.md:17-23
Updating Third-party Grammars
Third-party grammars are updated by modifying the version numbers in the project's root Cargo.toml and the enums/Cargo.toml file.
- Version Update: Replace the version string for the specific grammar (e.g.,
tree-sitter-java = "x.xx.x"). - Recreation: Execute the
./recreate-grammars.shscript to refresh all grammar structures and data. - Validation: Fix any failed tests or regressions introduced by the new grammar version.
Sources: rust-code-analysis-book/src/developers/update-grammars.md:25-38
Updating Internal Grammars
Internal grammars require a more complex workflow involving the tree-sitter-cli and specialized generation scripts.
General Internal Update Workflow
For standard internal grammars like tree-sitter-ccomment or tree-sitter-preproc:
- Update
tree-sitter-cliversion in the grammar'spackage.json. - Align version fields in the main
Cargo.tomlandenums/Cargo.toml. - Run the generation script:
./generate-grammars/generate-grammar.sh <grammar_name>.
Sources: rust-code-analysis-book/src/developers/update-grammars.md:40-60
Specialized Mozilla Grammars
Mozilla-specific grammars (mozcpp and mozjs) have dedicated scripts that fetch specific revisions of upstream grammars, apply patches to avoid symbol collisions, and regenerate the final grammar.
The diagram shows the automated flow for updating Mozilla-defined grammars like mozjs and mozcpp.
Sources: generate-grammars/generate-mozjs.sh:10-75, generate-grammars/generate-mozcpp.sh:10-70
Validation and Testing Infrastructure
Updating grammars can introduce breaking changes in metric calculations. The project uses a Python-based utility, check-grammar-crate.py, to compare metrics before and after an update.
Metric Comparison Logic
The tool computes metrics on a target repository (usually a local clone of gecko-dev) using both the "old" (master branch) and "new" (current branch) grammar versions.
Sequence of events during metric validation following a grammar update.
Sources: check-grammar-crate.py:46-150, check-grammars-crates.sh:10-45
Automation Scripts
Several scripts automate the CI process for grammar updates:
check-grammars-crates.sh: Detects if a grammar crate version has changed inCargo.tomlcompared to the master branch. If changed, it triggers the metric computation and comparison.generate-grammar.sh: Handles the Node.js dependencies required fortree-sitter generate, runs the generation, and triggers./recreate-grammars.sh.
Sources: check-grammars-crates.sh:5-30, generate-grammars/generate-grammar.sh:10-25
Supporting a New Language
Adding a completely new language involves integrating a new tree-sitter grammar into the enums crate and the main library.
- Enum Integration: Add the language crate to
enums/Cargo.tomland updateenums/src/languages.rsandenums/src/macros.rs. - Code Generation: Run
./recreate-grammars.shto generate the Rust source file for the new grammar insrc/languages/. - Library Integration: Register the new grammar in
src/languages/mod.rsand define language properties (extensions, description) insrc/langs.rsusing themk_langs!macro.
Sources: rust-code-analysis-book/src/developers/new-language.md:12-65
Summary
Grammar management in rust-code-analysis is a structured process that ensures parsing accuracy across supported languages. By distinguishing between third-party and internal grammars, the project maintains flexibility for custom requirements (Firefox) while leveraging community improvements. The integration of automated metric comparison tools ensures that grammar updates do not silently regress the quality or accuracy of the static analysis metrics provided by the library.