Testing Tools (Mutation, Impact, Coverage)

Relevant source files

The following files were used as context for generating this wiki page:

• src/tools/mutation-test.ts
• src/agents/architect.ts
• src/agents/test-engineer.ts (Referenced via identity and tasking in architect.ts and AGENTS.md)
• AGENTS.md
• src/tools/test-runner.ts
• src/tools/tests/mutation-test.test.ts

Testing tools within the Opencode Swarm project provide an automated pipeline for verifying code quality, security, and behavioral correctness. These tools are orchestrated primarily by the Architect and executed by specialized agents like the Test Engineer, utilizing specific tools such as mutation_test and test_runner.

The testing system is integrated into a multi-stage Quality Assurance (QA) gate that enforces standards for every code change before it is merged. This includes unit testing, adversarial security testing, and advanced verification techniques like mutation testing to identify gaps in test suites.
Sources: src/agents/architect.ts:108-165, AGENTS.md:58-69

Mutation Testing

Mutation testing is used to evaluate the effectiveness of the test suite by injecting small faults (mutants) into the source code and verifying if the existing tests can detect (kill) them.

Mutation Execution Flow

The mutation_test tool applies pre-generated patches to target files and executes a specified test command to calculate the "kill rate." A quality gate then evaluates this rate against pass/warn thresholds.
Sources: src/tools/mutation-test.ts:16-21, src/tools/tests/mutation-test.test.ts:133-145

The diagram shows the logic within the mutation engine where patches are applied and tested to determine the suite's robustness.
Sources: src/tools/mutation-test.ts:121-133

Tool Configuration

The mutation_test tool accepts the following parameters:

Parameter	Type	Description	Default
patches	Array	Objects containing id, filePath, patch (diff), etc.	Required
files	Array	Test file paths to run against mutants.	Required
test_command	Array	Command strings (e.g., ["npx", "vitest"]).	Required
pass_threshold	Number	Kill rate for a 'pass' verdict.	0.80
warn_threshold	Number	Kill rate for a 'warn' verdict.	0.60
working_directory	String	Project root for execution.	CWD

Sources: src/tools/mutation-test.ts:24-60, src/tools/mutation-test.ts:107-108

Test Execution and Coverage

The project utilizes bun:test for all test execution. The Architect manages the testing lifecycle, ensuring that coverage and regression checks are performed during the Phase implementation.

Test Runner Constraints

The test_runner tool is used for targeted validation. However, strict safety limits are enforced to prevent performance degradation:

• Max Safe Files: 50. If a request exceeds this, the tool returns a scope_exceeded outcome with a SKIP instruction.
• Broad Scopes: Large scopes like all, graph, or impact are discouraged for interactive use to avoid stalling the plugin host.
  Sources: AGENTS.md:71-79, src/tools/test-runner.ts:26

Coverage Enforcement

The Architect monitors test coverage reported by the Test Engineer. If coverage falls below 70%, the Architect may delegate an additional pass to target uncovered paths.
Sources: src/agents/architect.ts:167-171

This sequence illustrates the interaction between the Architect and Test Engineer to satisfy coverage requirements.
Sources: src/agents/architect.ts:167-171, AGENTS.md:81-88

Adversarial and Security Testing

Security-sensitive code (identified by SECURITY_KEYWORDS) triggers mandatory adversarial testing.

Adversarial Testing Pipeline

The Test Engineer is tasked with specifically identifying attack vectors such as malformed inputs, boundary violations, and injection attempts.
Sources: src/agents/architect.ts:178, src/agents/architect.ts:420-425

Feature	Description
Security Keywords	password, secret, token, auth, injection, etc.
TIER 3 Changes	Architect/auth/security files requiring high-scrutiny Stage B gates.
Target Vectors	Buffer overflows, injection, auth bypass, concurrency issues.

Sources: src/agents/architect.ts:135-144, tests/unit/agents/agent-audit-p2.test.ts:153-170

Testing Invariants

The project enforces several non-negotiable invariants regarding test writing and execution:

1. Framework: All tests must use bun:test exclusively.
2. Isolation: mock.module must be used carefully as it leaks across tests; Dependency Injection (DI) is preferred for isolation.
3. Portability: Tests must use os.tmpdir() and path.join() to ensure they run on Windows, macOS, and Linux.
4. Tool Use: Broad repo validation should use shell commands rather than the test_runner tool to avoid context bloat.
   Sources: AGENTS.md:81-88, AGENTS.md:71-79

Testing in Opencode Swarm is a multi-layered process ensuring that code is not only functionally correct but also robust against mutations and secure against adversarial attacks. By combining automated tool gates (Stage A) with agent-driven verification (Stage B), the system maintains high engineering standards throughout the software development lifecycle.
Sources: src/agents/architect.ts:114-144, AGENTS.md:12-25