Testing Strategy & Fixtures
Relevant source files
The following files were used as context for generating this wiki page:
The code-review-graph project utilizes a multi-layered testing strategy designed to ensure the integrity of its knowledge graph, the accuracy of its multi-language parsers, and the reliability of its MCP (Model Context Protocol) tools. This strategy encompasses unit tests for core components, integration tests for full-pipeline validation, and specialized evaluation benchmarks to measure performance and token efficiency.
The project relies heavily on a "local-first" data model, meaning many tests involve the creation of temporary SQLite databases and mock repository structures to simulate real-world coding environments. This approach allows for isolated verification of graph operations, such as community detection and impact analysis, without dependencies on external services.
Core Testing Architecture
The testing infrastructure is primarily built using pytest for the Python core and Mocha for the VS Code extension components. The architecture is categorized into specialized test suites that target different layers of the application.
The diagram shows the relationship between different test modules and the shared SQLite data layer used for verification.
Sources: CONTRIBUTING.md, CLAUDE.md, tests/test_integration_v2.py
Key Testing Components
| Component | Responsibility | Relevant Files |
|---|---|---|
| Unit Tests | Validating individual functions in parser.py, graph.py, and tools/. | test_parser.py, test_tools.py |
| Integration Tests | End-to-end validation of flows, communities, and FTS search. | test_integration_v2.py |
| Evaluation Framework | Measuring MRR (Mean Reciprocal Rank), Precision/Recall, and Token Efficiency. | test_eval.py |
| Fixture Management | Providing sample code in various languages (Python, TS, Dart, Vue). | tests/fixtures/ |
Sources: CONTRIBUTING.md:16-32, tests/test_integration_v2.py:24-43
Language Parser Verification
A critical part of the testing strategy is the verification of the Tree-sitter multi-language parser. Tests ensure that source files are correctly transformed into NodeInfo and EdgeInfo objects.
Parser Test Logic
- Language Detection: Verifies detection via file extensions and shebang lines for extension-less scripts.
- Node Extraction: Ensures Classes, Functions, and Tests are correctly identified with proper line ranges.
- Edge Resolution: Validates the creation of
CALLS,CONTAINS,INHERITS, andIMPORTS_FROMrelationships. - Cross-File Resolution: Confirms that calls to imported functions resolve to the correct qualified names.
This flowchart illustrates the data flow within the parser tests from raw source to graph nodes.
Sources: tests/test_parser.py:17-100, tests/test_parser.py:140-180
Graph and Tool Integration
Tests for MCP tools (found in tests/test_tools.py) validate the high-level API exposed to AI coding tools. These tests often use a setup_method to seed a temporary GraphStore with a "realistic" graph structure.
Seeding Realistic Graphs
To test complex features like "Impact Radius" or "Community Detection," the test suite seeds a graph containing:
- Layered Architectures: Mocking API, Auth, and DB layers.
- Inter-file Dependencies: Establishing calls across file boundaries.
- Test-to-Code Mapping: Using
TESTED_BYedges to link production code to test nodes.
Sources: tests/test_integration_v2.py:46-130, tests/test_tools.py:33-70
Refactoring and Impact Analysis
The refactoring tests (tests/test_refactor.py) verify the safety and accuracy of graph-powered operations:
- Rename Preview: Generates an edit list without modifying files, returning a
refactor_id. - Dead Code Detection: Identifies unreferenced functions while excluding entry points and framework-specific lifecycle hooks (e.g., Angular's
ngOnInit). - Dry Run Execution: Uses a
dry_runflag to return a unified diff before applying changes to disk.
Sources: tests/test_refactor.py:64-90, tests/test_refactor.py:150-180, tests/test_refactor.py:418-445
Evaluation Framework Benchmarks
The project includes an evaluation framework (code_review_graph.eval) to quantitatively measure the effectiveness of the graph.
Metric Definitions
| Metric | Calculation / Purpose | Source |
|---|---|---|
| Token Efficiency | Compares raw file tokens vs. graph-provided context tokens to calculate reduction percentage. | test_eval.py:27-32 |
| MRR (Mean Reciprocal Rank) | Measures the quality of search results for expected nodes. | test_eval.py:41-53 |
| Precision & Recall | Evaluates the accuracy of impact analysis against a "ground truth" of co-changed files. | test_eval.py:56-78 |
Benchmark Execution Flow
The sequence diagram details how the evaluation runner orchestrates benchmarks against mock repositories.
Sources: tests/test_eval.py:236-285, tests/test_eval.py:126-150
Testing Constraints and Security Invariants
The test suite also enforces security and performance invariants:
- SQL Parameterization: All tests verify that queries use
?placeholders to prevent injection. - Path Traversal: The
_validate_repo_roothelper is tested to ensure it only accepts directories containing.git,.svn, or.code-review-graph. - Sanitization: The
_sanitize_namefunction is tested for its ability to strip control characters and truncate long names (256 char cap) to defend against prompt injection.
Sources: CONTRIBUTING.md:46, tests/test_tools.py:317-325, tests/test_tools.py:534-570
Conclusion
The testing strategy of code-review-graph is geared toward maintaining high precision in code relationship mapping. By combining traditional unit testing of parsers with a specialized evaluation framework for AI-context efficiency, the project ensures that the resulting knowledge graph is both technically accurate and practically useful for reducing LLM token overhead.