Dogfood Evaluation Framework
Relevant source files
The following files were used as context for generating this wiki page:
The Dogfood Evaluation Framework is a developer-centric testing system designed to validate the retrieval performance of the cxs CLI using real-world historical session data. It allows developers to maintain a "Golden Set" of search queries and expected results (sessions, contexts, and rankings) to serve as an acceptance gate for changes to indexing, ranking, or retrieval logic.
The framework operates by executing the cxs CLI against defined "goldens," evaluating the output against predicates, and generating detailed scorecards and README reports. Unlike standard unit tests, dogfooding focuses on personal history archaeology and the accuracy of session recall in varied execution contexts.
Sources: AGENTS.md:144-149, eval/run-dogfood-eval.ts:40-60
System Architecture
The framework is composed of a specialized runner, a schema-validated configuration format, and a core evaluation engine.
Core Components
- Dogfood Runner (
run-dogfood-eval.ts): The entry point that orchestrates the execution. It parses JSONL golden files, executescxs findandread-range/read-pagecommands via child processes, and aggregates results. - Schema and Parser (
dogfood-schema.ts): Defines the structure of "Golden" entries and handles the parsing of.jsonlfiles. It ensures that every test case has a valid intent, query, and expected assertions. - Evaluation Core (
dogfood-eval-core.ts): Contains the business logic for matching CLI results against expectations. It determines if a specific session was found within the allowedtopKrank and validates if the retrieved context contains required phrases.
Sources: eval/run-dogfood-eval.ts:1-20, eval/dogfood-schema.ts:1-30, eval/dogfood-eval-core.ts:1-15
Evaluation Workflow
The evaluation follows a linear process from input parsing to report generation:
The diagram shows the data flow from raw golden definitions to the final generated scorecard.
Sources: eval/run-dogfood-eval.ts:44-78
Data Models and Schema
The framework relies on a strictly typed schema to define search expectations. Test cases are stored in JSONL format, where each line represents a DogfoodGolden object.
The DogfoodGolden Interface
| Field | Type | Description |
|---|---|---|
id | string | Unique identifier for the test case. |
query | string | The search string to pass to cxs find. |
intent | string | Description of the user's goal for this search. |
status | DogfoodStatus | One of candidate, hard, or stale. |
find | DogfoodFindOptions | Optional overrides for sort, limit, and selector. |
expected | DogfoodExpected | Assertions including topK, acceptableSessionUuids, and context. |
Sources: eval/dogfood-schema.ts:39-48
Status Levels and Blocking
The framework uses the status field to determine how failures are handled in the developer workflow:
- Hard: Represents a critical retrieval case. Failure of a "hard" item results in a non-zero exit code for the runner, blocking the acceptance gate.
- Candidate: Represents newly identified cases or potential issues. Failures are reported but do not block the gate.
- Stale: Items that are no longer applicable; these are skipped during the evaluation run.
Sources: AGENTS.md:154-159, eval/run-dogfood-eval.ts:85-87
Evaluation Logic and Predicates
The system evaluates results by checking a series of predicates against the FindResult array returned by the CLI.
Hit Selection
The function selectDogfoodHit attempts to find one of the acceptableSessionUuids within the first topK results (defaulting to 5). If found, that result becomes the basis for further context evaluation. If not found, the top-ranked result is used for evaluation (leading to a failure).
Sources: eval/dogfood-eval-core.ts:54-63, eval/dogfood-eval-core.test.ts:7-14
Predicate Types
| Predicate | Logic |
|---|---|
| session_uuid | Checks if the selected hit's UUID is in the acceptable list and within topK. |
| cwd | Validates if the hit's cwd contains a specific substring. |
| match_source | Ensures the hit came from the expected source (e.g., message vs session). |
| context | Scans the text retrieved by read-range or read-page for required "mustContain" strings. |
Sources: eval/dogfood-eval-core.ts:77-118
Context Retrieval Logic
The framework automatically decides whether to use read-range or read-page based on the match type:
- If the hit has a
matchSeq, it usesread-rangeto pull a window of messages. - If the hit is at the
sessionlevel (no sequence), it falls back toread-page.
Sources: eval/run-dogfood-eval.ts:127-145, eval/dogfood-eval-core.ts:65-71
Execution and Reporting
The runner is typically invoked via npm run eval:dogfood -- <path_to_goldens>.
Output Artifacts
Each run creates a timestamped directory in data/cxs-dogfood-eval/ containing:
- README.md: A summary table of pass/fail marks and detailed breakdowns for every query.
- scorecard.json: A machine-readable JSON object containing the scoreboard and per-query evaluation details.
- Command Logs: Raw
.find.json,.find.txt, and context files for every attempt to aid in debugging.
Sources: eval/run-dogfood-eval.ts:36-50, eval/run-dogfood-eval.ts:184-210
Multi-Attempt Support
A single golden entry can define multiple queries. The runner iterates through each, seeking a "pass" mark. This is useful for testing query refinement or synonym performance.
The sequence diagram illustrates the retry logic when multiple queries are provided for a single intent.
Sources: eval/run-dogfood-eval.ts:65-80, eval/run-dogfood-eval.ts:98-124
Summary
The Dogfood Evaluation Framework serves as the primary mechanism for maintaining high retrieval quality in the cxs project. By grounding tests in real user data and historical "archeology" cases, it ensures that changes to the ranking heuristic or FTS weights do not regress the ability to find critical information. The system's use of blocking "hard" statuses and detailed diagnostic logs makes it a robust acceptance gate for CI/CD and local development.