Evaluation Framework & Benchmarks
Relevant source files
The following files were used as context for generating this wiki page:
The Evaluation Framework in code-review-graph is a specialized suite designed to measure the efficiency, accuracy, and performance of the code knowledge graph. It provides automated benchmarking against real-world open-source repositories to validate how effectively the graph reduces token usage for AI agents while maintaining high retrieval precision and recall.
Sources: code_review_graph/eval/reporter.py:78-85, CLAUDE.md:65-65, tests/test_eval.py:202-206
The framework operates by running a series of predefined benchmarks, such as token efficiency and impact accuracy, across multiple test commits. Results are aggregated into CSV files and transformed into comprehensive Markdown reports for technical analysis.
Sources: code_review_graph/eval/runner.py, code_review_graph/eval/reporter.py:108-115
Framework Architecture
The framework is structured into a runner that coordinates execution, a registry of specialized benchmark modules, and a reporter for data visualization.
Component Overview
| Component | Responsibility |
|---|---|
| Runner | Coordinates repository cloning, graph building, and benchmark execution across test commits. |
| Benchmark Registry | A central mapping of benchmark names to their respective implementation modules. |
| Scorer | Logic for calculating metrics like MRR (Mean Reciprocal Rank), Precision, Recall, and F1. |
| Reporter | Aggregates CSV data into human-readable Markdown tables and full technical reports. |
Sources: code_review_graph/eval/runner.py, code_review_graph/eval/reporter.py, tests/test_eval.py:15-38
Data Flow Logic
The evaluation process follows a strict pipeline: repository setup, graph initialization, benchmark execution, and finally, result reporting.
The diagram illustrates the flow from initial configuration to the final generation of evaluation reports.
Sources: code_review_graph/eval/runner.py, code_review_graph/eval/reporter.py:78-105
Benchmark Modules
The framework includes several key benchmarks to evaluate different aspects of the system.
Token Efficiency
Measures the reduction in tokens sent to an AI agent when using the graph versus a "naive" full-file approach. It calculates ratios for both "naive-to-graph" and "standard-to-graph" contexts.
Sources: tests/test_eval.py:27-33, code_review_graph/eval/reporter.py:164-175
Impact Accuracy
Evaluates the graph's ability to identify the "blast radius" of a change. It uses two ground-truth modes:
- Graph-derived: A circular upper bound based on graph traversal.
- Co-change: Files actually changed in the same commit, providing a real-world accuracy metric.
Sources: code_review_graph/eval/reporter.py:84-88, tests/test_eval.py:461-480
Agent Baseline
Compares the graph's retrieval capabilities against a standard grep top-k baseline. It derives search terms from natural language questions and measures the token ratio between a standard grep search and the graph's structured query response.
Sources: code_review_graph/eval/benchmarks/agent_baseline.py, tests/test_eval.py:530-545
Multi-Hop Retrieval
Tests the graph's capacity to resolve complex, multi-step dependencies that require traversing several edges (e.g., following a chain of function calls across multiple files).
Sources: code_review_graph/eval/benchmarks/multi_hop_retrieval.py
Configuration and Implementation
Benchmarks are configured using YAML files that define repository metadata, test commits, and expected search queries.
Configuration Structure
name: code-review-graph
url: https://github.com/marlon-costa-dc/code-review-graph
language: python
test_commits:
- sha: "b72413c"
description: "Initial eval framework"
search_queries:
- query: "hybrid_search"
expected: "code_review_graph/search.py::hybrid_search"
Sources: code_review_graph/eval/configs/code-review-graph.yaml, tests/test_eval.py:108-132
Execution and Reporting
Benchmarks are triggered via the CLI command code-review-graph eval. The runner ensures that each benchmark execution is isolated, often rebuilding the graph or performing incremental updates to match the state of specific git commits.
This sequence shows the interaction between the runner and specific benchmark implementations during a test run.
Sources: code_review_graph/eval/runner.py, tests/test_eval.py:202-248
Performance and Metrics
The framework tracks build performance, including file and node counts, and the time taken for community and flow detection.
| Metric | Calculation / Description |
|---|---|
| MRR | Mean Reciprocal Rank of the expected result in search outputs. |
| Token Ratio | naive_tokens / graph_tokens; higher is better efficiency. |
| F1 Score | Harmonic mean of precision and recall for impact analysis. |
| Nodes/Sec | Throughput during the graph build process. |
Sources: tests/test_eval.py:36-65, code_review_graph/eval/reporter.py:228-235
The Evaluation Framework provides the empirical foundation for code-review-graph, ensuring that architectural changes improve agent efficiency and retrieval accuracy across diverse codebases. By quantifying token savings and impact precision, it allows developers to maintain a "local-first" knowledge graph that is both fast and token-efficient.
Sources: CLAUDE.md:8-13, code_review_graph/eval/reporter.py:78-95