Mutation Testing Guide

Relevant source files

The following files were used as context for generating this wiki page:

• apps/cli/CLAUDE.md
• apps/server/workers/src/durable-objects/handlers.spec.ts
• apps/server/workers/src/lib/image-processing.spec.ts
• apps/server/workers/src/tests/auth-mutations.spec.ts
• apps/server/workers/src/durable-objects/ConnectionManager.spec.ts
• apps/server/workers/src/tests/uploads-drizzle.spec.ts

Mutation testing is a high-level software testing technique used to validate the quality and effectiveness of an existing test suite. In this project, it is primarily implemented using Stryker Mutator within the happy-cli wrapper and targeted test suites in the happy-server-workers module. The process involves introducing small, intentional changes (mutants) to the production source code and verifying if the existing tests fail (killing the mutant) or pass (allowing the mutant to survive).

Sources: apps/cli/CLAUDE.md, apps/server/workers/src/tests/auth-mutations.spec.ts:1-20

Core Concepts and Methodology

Mutation testing evaluates the "detectability" of logic changes. A high mutation score indicates that the test suite is robust enough to catch subtle bugs.

Mutation Status Types

When a mutant is generated, it results in one of several statuses based on test outcomes:

Status	Meaning
Killed	A test failed after the mutation, correctly identifying the logic change.
Survived	All tests passed despite the mutation, indicating a gap in test coverage or weak assertions.
No Coverage	No tests executed the mutated line of code.
Timeout	The mutation caused an infinite loop or significant performance degradation.
Compile Error	The mutation resulted in invalid TypeScript that failed to compile.

Sources: apps/cli/CLAUDE.md

Mutation Testing Workflow

The typical workflow involves running mutations against specific logic blocks, especially critical business logic like authentication, encryption, and API handlers.

This diagram illustrates the feedback loop where surviving mutants drive the creation of more precise test cases.
Sources: apps/cli/CLAUDE.md, apps/server/workers/src/tests/auth-mutations.spec.ts:1-15

Execution Commands (CLI)

Mutation testing for the happy-cli project is managed via Yarn scripts.

Command	Purpose
yarn mutate	Executes a full mutation test run and generates an HTML report.
yarn mutate:incremental	Recommended for local development; only mutates changed files using a cache.
yarn mutate:dry-run	Verifies the test runner setup without actually performing mutations.
yarn mutate:report	Opens the interactive HTML report in the default browser.

Sources: apps/cli/CLAUDE.md

Implementation and Configuration

Configuration Settings

The project uses stryker.config.mjs for configuration. Key settings include:

• Mutate: Limited to production source files, excluding tests, UI modules (Ink/React), and server-dependent modules.
• Test Runner: Uses Vitest with a specialized vitest.stryker.config.ts.
• Thresholds: Enforces specific mutation scores (80% high, 60% low, 40% break). Failure to meet the "break" threshold causes CI pipelines to fail.
• In-Place: Enabled to support workspace path aliases.

Sources: apps/cli/CLAUDE.md

Target Mutation Areas

Developers are encouraged to focus on "surviving" mutants in critical infrastructure:

1. Authentication: Signature verification and token handling.
2. Encryption: AES-256-GCM logic and nonce management.
3. API Handlers: Concurrency control and database operations.

Sources: apps/cli/CLAUDE.md

Improving Test Quality to Kill Mutants

To address surviving mutants, tests must move beyond simple coverage to "assertive coverage." This involves verifying exact values and boundary conditions.

String Literal and Exact Value Assertions

Mutants often survive when tests check for the existence of a value rather than its specific content. Testing should enforce exact string literals for error messages and event types.

// Example of killing a StringLiteral mutant in handler tests
it('handleSessionMetadataUpdate should return exact error message', async () => {
    const result = await handleSessionMetadataUpdate(ctx, { sid: 'missing' });
    expect(result.response?.message).toBe('Session not found'); // Exact string check
});

Sources: apps/server/workers/src/durable-objects/handlers.spec.ts:850-870, apps/server/workers/src/routes/connect.spec.ts:1000-1020

Arithmetic and Equality Operator Testing

Specialized tests target arithmetic operators (e.g., + becoming -) and equality operators (e.g., === becoming !==). This is vital in dimensions assembly and rate-limiting logic.

This sequence shows how boundary testing kills mutations in limit-checking logic.
Sources: apps/server/workers/src/lib/image-processing.spec.ts:900-950, apps/server/workers/src/tests/uploads-drizzle.spec.ts:1000-1050

Logic Branch Coverage

Mutation testing identifies "NoCoverage" areas where conditional branches are not exercised. For example, ensuring rate limiting "fails closed" in production environments when the key-value store is unavailable requires explicitly testing the ENVIRONMENT === 'production' branch.

Sources: apps/server/workers/src/tests/auth-mutations.spec.ts:100-130

Best Practices for Mutation Testing

• Run Incremental First: Use yarn mutate:incremental before major commits to catch regressions quickly.
• Focus on Logic, not UI: Exclude UI-only components (like Ink or React views) from mutation testing as they often require complex rendering contexts that are not suitable for high-frequency mutation runs.
• Verify Exact Structure: For API responses, verify every property in an object literal. Survived mutations often occur in omitted optional properties.
• Kill Conditional Mutations: Ensure both the true and false paths of ternary operators or if statements result in distinct, verified outcomes.

Sources: apps/cli/CLAUDE.md, apps/server/workers/src/durable-objects/handlers.spec.ts:1100-1150, apps/server/workers/src/tests/auth-mutations.spec.ts:1-20

Conclusion

Mutation testing serves as the ultimate "test for the tests" in the Magic-Agent project. By systematically introducing mutants into critical modules like authentication and image processing, the project ensures that its test suite is not just a metric of lines covered, but a reliable safeguard against logic errors and security vulnerabilities. Developers should use the Stryker HTML reports to identify gaps and implement the exact-value assertions detailed in this guide.