Testing Strategies
Relevant source files
The following files were used as context for generating this wiki page:
The rust-code-analysis project employs a multi-layered testing strategy designed to ensure the accuracy of code metric extraction across multiple programming languages. This strategy includes standard unit and integration tests, specialized snapshot testing for regression detection, and automated grammar validation scripts to maintain compatibility with upstream parser changes.
Sources: README.md:47-66, rust-code-analysis-book/src/developers/README.md:42-45
Core Testing Framework
The primary mechanism for verifying code changes is the standard Rust test runner. Developers are instructed to always verify changes using cargo test across the entire workspace with all features enabled.
Testing Commands
| Command | Description |
|---|---|
cargo test --workspace --all-features --verbose | Executes all unit and integration tests in the project. |
cargo insta test --review | Runs snapshot tests and opens a review interface for changes. |
cargo clippy --workspace --all-targets | Performs static analysis to catch common mistakes and style issues. |
Sources: rust-code-analysis-book/src/developers/README.md:42-45, README.md:58-62, rust-code-analysis-book/src/developers/README.md:65-68
Snapshot Testing with Insta
The project heavily utilizes insta for snapshot testing. This approach allows the system to compare current metric outputs against a "gold standard" reference file. This is particularly useful for verifying the complex FuncSpace structures generated during analysis.
The following diagram illustrates the snapshot testing flow implemented in the test suite:
The testing logic involves reading a source file from a test repository, guessing its language, and asserting that the resulting YAML representation of the metrics matches the stored snapshot.
Sources: README.md:56-62, tests/common/mod.rs:24-60
Implementation Details
In tests/common/mod.rs, the act_on_file function handles the core logic of a snapshot test:
- Source Loading: Uses
read_file_with_eolto prepare the buffer. - Language Identification: Either uses a configured language or
guess_language. - Metric Extraction: Calls
get_function_spacesto generate analysis results. - Redaction: Floating point values are rounded to three decimal places, and file paths are redacted to ensure tests are deterministic across different environments.
Sources: tests/common/mod.rs:24-52, src/tools.rs:260-272
Integration Testing: Repository Analysis
Large-scale integration tests are performed by running analysis over entire projects, such as pdf.js. These tests use a ConcurrentRunner to process multiple files in parallel and compare the output against snapshots.
Integration tests often include exclusion lists to skip files that currently cause parser errors or are irrelevant to the specific test case.
Sources: tests/pdf_js_test.rs:6-126, tests/common/mod.rs:63-88
Grammar and Crate Validation
Because the library relies on Tree-Sitter grammars, it includes specialized scripts to detect breaking changes when these external dependencies are updated.
Automated Comparison Logic
The check-grammar-crate.py script and check-grammars-crates.sh perform the following:
- Metric Baseline: Computes metrics on a clean master branch for a target repository (e.g.,
gecko-dev). - Update Simulation: Computes metrics again using the updated grammar.
- Diffing: Uses
json-minimal-teststo identify exactly which metrics changed and identifies the "minimal" code snippet that triggered the change.
Sources: check-grammar-crate.py:12-25, check-grammars-crates.sh:65-80
Supported Grammar Mappings
The validation scripts maintain specific mappings for grammar testing:
| Grammar | Extensions Tested |
|---|---|
tree-sitter-rust | *.rs |
tree-sitter-java | *.java |
tree-sitter-python | *.py |
tree-sitter-mozcpp | *.cpp, *.h, *.mm, *.c, etc. |
Sources: check-grammar-crate.py:46-73
Developer Guidelines for Testing
The project maintains strict requirements for contributions to ensure continued reliability:
- Mandatory Tests: Every new feature must be accompanied by at least one unit or integration test.
- CI Integrity: Snapshot tests must be reviewed and updated if valid changes occur.
- Verification: All tests must pass via
cargo testbefore submitting a pull request. - Code Quality: Tools like
clippyandrustfmtare integrated into the workflow to catch logical and stylistic errors before they reach the testing phase.
Sources: rust-code-analysis-book/src/developers/README.md:42-63, rust-code-analysis-book/src/developers/README.md:88-92
Summary
Testing in rust-code-analysis is a comprehensive process that balances developer-led unit tests with automated regression detection. By combining snapshot testing for complex data structures and automated grammar validation for external dependencies, the project ensures high technical accuracy across its various supported programming languages.