Testing & Benchmarking
Relevant source files
The following files were used as context for generating this wiki page:
Testing and benchmarking in the Oh My Pi project are centered around ensuring the surgical precision of the coding-agent when applying code edits and maintaining the reliability of core native and utility components. The primary benchmarking tool is the typescript-edit-benchmark, which evaluates LLM performance across various difficulty levels—Easy, Medium, Hard, and Nightmare—to test how effectively models handle surgical patches in complex codebases.
General testing guidelines emphasize defending concrete, externally observable contracts rather than implementation details. The project mandates the use of bun test for execution, prohibits mock.module() due to global registry pollution risks, and enforces strict rules against source-grepping in tests.
Sources: packages/typescript-edit-benchmark/src/generate.ts:16-29, AGENTS.md:144-159, AGENTS.md:179-188
Benchmarking Architecture
The benchmarking system is designed to evaluate edit precision rather than general bug-finding. It uses a specialized generator to create mutation cases from real-world TypeScript sources.
Mutation Generation & Difficulty
The generator selects files based on structural characteristics such as line count, repeated lines, and code density. It categorizes difficulty based on the presence of these traits to challenge the model's ability to locate and edit the correct code block.
| Difficulty | Characteristics |
|---|---|
| Easy | Short files, unique lines, explicit line numbers provided in prompt. |
| Medium | Medium length files, function context provided. |
| Hard | Long files, similar code blocks, no location hints provided. |
| Nightmare | Long files, repeating target lines, minimal info provided. |
Sources: packages/typescript-edit-benchmark/src/generate.ts:16-29, packages/typescript-edit-benchmark/src/generate.ts:250-264
Case Generation Flow
The following diagram illustrates the lifecycle of a benchmark case, from source analysis to the generation of a testable tarball fixture.
The generator uses diffLines to ensure mutations are precise and formatContent to normalize code before comparison. Sources: packages/typescript-edit-benchmark/src/generate.ts:109-123, packages/typescript-edit-benchmark/src/generate.ts:503-513
Benchmark Execution & Reporting
The bench:edit CLI (implemented in packages/typescript-edit-benchmark/src/index.ts) handles the execution of benchmark tasks. It supports configurable runs per task, thinking levels (e.g., low, medium, high), and task concurrency.
Metric Aggregation
Results are aggregated based on the "best run" of each task. Key metrics include:
- Task Success Rate: Percentage of tasks where at least one run succeeded.
- Edit Success Rate: Percentage of individual edit tool calls that were successful.
- Token Usage: Aggregated input, output, and reasoning tokens across successful runs.
- Indent Score: A measure of formatting consistency in the generated edits.
Sources: packages/typescript-edit-benchmark/src/report.ts:98-120, packages/typescript-edit-benchmark/src/index.ts:391-408
Verification Logic
Verification ensures that the model's output matches the expected fixture. It normalizes line endings and handles whitespace differences using a dedicated verification suite.
Testing Conventions & Tooling
The project follows strict engineering principles to ensure test suite safety and reliability.
Core Rules
- Execution: Always use
bun test. The use oftscfor testing or checking is prohibited in favor ofbun check. - Observability: Tests must assert observable contracts. Mocks should not be created; instead, behavior, output shapes, and state transitions should be tested directly.
- Isolation: No long-lived mutations of
process.envorBun.env. Global state should be handled viavi.spyOnandvi.restoreAllMocks. - No Source-Grep: Tests are forbidden from reading implementation files to assert on their text content (e.g., checking for specific function calls in source).
Sources: AGENTS.md:144-162, AGENTS.md:179-183
Native Component Testing
Native components, particularly those using the Rust-based pi-natives, are tested for performance and correctness in operations like grep, glob, and astMatch.
| Test Category | Description | Source File |
|---|---|---|
| Grep | Validates pattern matching, literal searches, and .gitignore respect. | packages/natives/test/native.test.ts |
| Glob | Tests file discovery with pattern matching and scan cache invalidation. | packages/natives/test/native.test.ts |
| AST | Evaluates structural code discovery and metavariable matching. | packages/natives/test/native.test.ts |
| Spacing | Validates .editorconfig parsing and indent derivation. | packages/utils/test/spacing.test.ts |
Sources: packages/natives/test/native.test.ts:180-205, packages/natives/test/native.test.ts:400-415, packages/utils/test/spacing.test.ts:25-45
Conclusion
Testing and Benchmarking in the Oh My Pi project provide a rigorous framework for validating the coding-agent. By combining high-difficulty benchmarks that simulate complex real-world code structures with a strict, contract-focused unit testing policy, the project ensures that both high-level agentic behavior and low-level native performance remain reliable and surgically precise.