Manual Evaluation & Comparison
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The "Manual Evaluation & Comparison" system in cxs provides a structured framework for assessing the retrieval accuracy and ranking performance of the CLI tool. It allows developers to run predefined queries against the indexed session data, validate the results against expected outcomes (predicates), and compare performance across different code versions or batches.
This system serves as a critical acceptance gate to ensure that changes to the indexing or ranking logic do not introduce regressions. It categorizes evaluation into general manual queries and "Dogfood" goldens—real-world historical search cases derived from actual developer usage.
Sources: AGENTS.md:126-128, CONTRIBUTING.md:18-24, eval/run-manual-eval.ts:24-34
Manual Evaluation Core Logic
The core evaluation logic centers on the evaluateManualQuery function, which tests whether search results meet specific assertions defined in a manual query object.
Predicate Matching
Evaluation is based on "predicates"—labels associated with specific needles (substrings) that must appear in the search results. A query is considered a "pass" only if all defined predicates find a match within the top-K results provided.
The system allows different predicates to be satisfied by different results within the same top-K window.
Sources: eval/manual-eval-core.ts:25-68, eval/manual-eval-core.test.ts:29-47
Data Structures
| Interface | Description | Key Fields |
|---|---|---|
ManualQuery | Defines the test case and expectations. | id, query, intent, expectedTitleOrSummaryContains, expectedCwdContains |
ManualQueryEvaluation | The result of an evaluation run. | mark (pass/fail/skip), predicateResults |
ManualPredicateResult | Status of a specific assertion. | label, needle, matched (boolean) |
Sources: eval/manual-eval-core.ts:3-23
Execution Flow (run-manual-eval.ts)
The manual evaluation runner automates the process of executing CLI commands and capturing outputs for a batch of queries.
Process Architecture
- Setup: Creates a timestamped directory in
data/cxs-eval/to store the batch artifacts. - Execution: Iterates through
manual-queries.json, executingcxs findfor each query with a limit of 5. - Context Retrieval: For the top result of each query, it automatically triggers a
read-rangeorread-pagecommand to capture the context used by the agent. - Reporting: Generates a
README.mdsummary and ascorecard.jsoncontaining detailed metrics.
Dogfood Evaluation
Dogfood evaluation is a specialized subset of manual evaluation that uses real-world "golden" cases.
Candidate Lifecycle
Developers can capture "dogfood candidates" when they encounter retrieval issues during normal usage. These candidates are promoted to "hard" goldens only after explicit user confirmation.
- Good/Query-Refine: Standard successful or self-corrected usage.
- Coverage-Issue: Problems stemming from missing sync.
- Dogfood-Candidate: Reproducible ranking or recall failures worthy of inclusion in the test suite.
Sources: AGENTS.md:104-118, eval/run-dogfood-eval.ts:18-29
Evaluation Modes
Dogfood items can specify multiple query attempts to find the correct result. The evaluator tracks pass@1 and pass@K metrics.
| Feature | Description |
|---|---|
| Multi-Attempt | Tries multiple query variations defined in the golden item until one passes. |
| Stale Handling | Can skip or include items marked as "stale" using the --include-stale flag. |
| Blocking | Items marked as "hard" status will cause a non-zero exit code on failure. |
Sources: eval/run-dogfood-eval.ts:68-80, eval/run-dogfood-eval.ts:117-124, eval/run-dogfood-eval.ts:233-237
Batch Comparison
The comparison utility allows for regression testing by analyzing the differences between two evaluation batches (e.g., "before" and "after" a code change).
Comparison Metrics
The comparison focuses on changes in the Top 1 result for every query. It identifies:
- Total queries executed in both batches.
- Number of queries where the
Top 1session changed. - Specific shifts in titles and session UUIDs.
Sources: eval/compare.test.ts:21-43, CONTRIBUTING.md:27-31
Summary
The evaluation suite in cxs provides a robust, evidence-based approach to software quality. By combining automated predicate matching, multi-attempt dogfood testing, and batch comparison tools, the project ensures that retrieval performance remains consistent as the system evolves. Developers are encouraged to run npm run eval:manual and npm run eval:compare before submitting significant changes to the ranking or indexing logic.
Sources: CONTRIBUTING.md:37-45, AGENTS.md:126-130