Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Testing Strategy for the project centers on a multi-layered verification approach designed to ensure the reliability of autonomous agent behaviors, code modification integrity, and overall system security. It encompasses unit testing of core components, complex integration flows simulating agent decision-making, and automated quality gates that run post-implementation to validate changes.
The strategy is anchored by the CodeQualityPipeline, which acts as a verification engine for the agent's work, and a comprehensive suite of Vitest-based specifications that cover everything from individual action execution to full-loop "Auto-mode" sessions. This ensures that every file modification or command execution is governed by safety checks and verified against project-specific quality standards.
Core Testing Architecture
The project utilizes a tiered testing architecture to maintain stability across different levels of abstraction. The system differentiates between unit-level logic validation and high-level behavioral simulation.
Testing Layers
| Layer | Focus | Key Components |
|---|---|---|
| Unit Tests | Logic of individual classes and functions. | ActionExecutor, CodeQualityPipeline, AutomodeState |
| Integration Tests | Full agent lifecycles and multimodal flows. | Agent Flow, Vision Model detection, Image Attachment |
| Security Testing | Prevention of credential leaks and resource abuse. | SecurityScanner, Secret detection regex, Pre-commit gates |
| Automated Gates | Post-implementation project health checks. | lint, typecheck, test, build scripts |
Sources: tests/integration/agent-flow.spec.ts:13-17, src/core/CodeQualityPipeline.ts:21-24, tests/actionExecutor.spec.ts:100-105
Code Quality Pipeline
The CodeQualityPipeline is a critical component of the testing strategy, serving as an automated quality gate that the agent triggers after performing file modifications in implementation mode. It auto-detects project scripts from package.json and executes them to ensure that no regressions were introduced.
Execution Flow
The pipeline follows a specific execution order to optimize feedback speed:
- Parallel Phase: Executes
lintandtypecheckscripts simultaneously as they are typically independent. - Sequential Phase: Executes
testscripts, followed bybuildscripts, as these often depend on successful type resolution.
Script Detection Patterns
The pipeline identifies scripts using a primary name and several fallback alternatives:
| Type | Primary Name | Alternatives |
|---|---|---|
| Lint | lint | lint:check, eslint |
| Typecheck | typecheck | type-check, tsc, types, check:types |
| Test | test | test:unit, vitest, jest |
| Build | build | build:prod, compile |
Sources: src/core/CodeQualityPipeline.ts:68-73
Integration and Flow Testing
Integration tests simulate the "ReAct" loop of the agent, focusing on intent detection and state transitions. These tests ensure the agent correctly identifies whether a request is Diagnostic (read-only) or Implementation (requires mutations and quality checks).
Intent-Based Flow Testing
Testing confirms that the system handles instructions differently based on detected intent:
- Diagnostic Mode: Verified to skip environment bootstrap and restrict destructive tools like
write_fileordelete_path. - Implementation Mode: Verified to trigger the
EnvironmentBootstrapand run theCodeQualityPipelineafter changes.
Sources: tests/integration/agent-flow.spec.ts:25-45, src/core/agent.ts:1010-1030
Vision and Multimodal Testing
The strategy includes specific specs for multimodal capabilities, ensuring image file paths and base64 data are correctly detected and attached to LLM payloads.
Security and Mutation Testing
The testing strategy mandates strict verification of all file mutations and command executions through the ActionExecutor.
Security Gates
A dedicated SecurityScanner is integrated into the testing and execution flow to prevent the commit of sensitive data. The strategy tests these gates by:
- Injecting dummy secrets (e.g.,
sk-proj-...for OpenAI) into staged diffs. - Verifying that
git_commitactions are blocked whenhigh-severitysecrets are detected. - Confirming that low-severity findings result in warnings but do not block the workflow.
Sources: tests/integration/agent-flow.spec.ts:225-255, tests/actionExecutor.spec.ts:235-250
Mutation Safety
Mutation tools are tested for "Dry Run" compliance. The specs verify that when dryRun is enabled, tools like write_file, append_file, and run_command skip execution and return a descriptive placeholder instead of modifying the filesystem.
Sources: tests/actionExecutor.spec.ts:450-485
Auto-mode and Session Stability
Testing for "Auto-mode" sessions focuses on the "Circuit Breaker" logic and state persistence. This prevents the agent from entering infinite loops or repeating failed approaches.
Iteration Stability Testing
The AutomodeState specs verify that:
- Iteration Counting: The system correctly increments counters and logs actions for every turn.
- Circuit Breaker: The session triggers failure if no file progress is made for a defined threshold (default 3 iterations) or if the same error occurs repeatedly.
- Completion Promises: The system detects specific
<promise>DONE</promise>tags in LLM output to signal task end.
Sources: tests/automode.spec.ts:135-165, tests/automode.spec.ts:180-210
Summary
The Testing Strategy provides a robust framework for autonomous software engineering. By combining rigorous unit tests for the ActionExecutor, integration tests for agent intent, and an automated CodeQualityPipeline for post-change verification, the project ensures that the AI agent operates within safe, predictable, and high-quality boundaries. High-priority security gates for secret detection and circuit breakers for autonomous loops further safeguard the project environment during long-running tasks.