Scoring & Reporting

Relevant source files

The following files were used as context for generating this wiki page:

• lib/mutineer/reporter.rb
• rake/docs_contract.rb
• test/json_reporter_test.rb
• test/html_reporter_test.rb
• skills/mutineer/SKILL.md
• AGENTS.md

Scoring and reporting in Mutineer provides developers and CI systems with a quantitative measure of test suite effectiveness. The system calculates a mutation score based on the ratio of killed mutants to total attempted mutants, then renders these findings into human-readable or machine-parsable formats. This module is critical for determining if a test suite can detect code regressions and for gating builds based on a defined quality threshold.

Sources: lib/mutineer/reporter.rb:11-17, skills/mutineer/SKILL.md:5-10

Mutation Scoring Model

The mutation score represents the percentage of covered mutations that the test suite successfully identified and "killed." The calculation strictly follows a specific mathematical model to ensure consistency across different reporting formats.

Calculation Logic

The score is calculated using the formula:
score = killed / (killed + survived)

Mutineer excludes several categories of mutants from the denominator to avoid diluting the score with mutants that did not or could not produce a valid verdict:

• no_coverage: Code that was never reached by the test suite.
• uncapturable: Mutants where the test harness failed to execute.
• unplaceable: Mutants where the class cannot be named statically.
• ran_at_load: Mutants on lines that executed during the application load phase.
• ignored: Mutants explicitly suppressed via configuration.
• errored / timeout: Mutants that crashed or exceeded time limits.

If the denominator is zero (no covered mutants), the score is nil rather than 0.0.

Sources: lib/mutineer/reporter.rb:510-525, AGENTS.md:128-130

Completeness Gate

Mutineer enforces a completeness check when a --threshold is provided. A run fails if it produces too many mutants without a verdict (errors, timeouts, or uncapturable mutants).

• Broken Share Limit: 10% (0.10). If "no verdict" mutants exceed 10% of total attempted mutants, the gate fails even if the score is high.
• Broken Floor: 1 mutant. A single timeout does not fail small runs (e.g., in a PR diff).

Sources: lib/mutineer/reporter.rb:20-27, lib/mutineer/reporter.rb:535-540

Reporting Formats

The Mutineer::Reporter class routes results to three primary formats: human, json, and html.

Format Characteristics

Format	Target Audience	Primary Features
Human	Terminal / Developers	Summary block, score line, and per-file survivor diffs.
JSON	CI / AI Agents	Machine-readable schema (v1.8) including stable mutant IDs and full matrix data.
HTML	Browsers / Reports	Self-contained file with inline CSS, colorized diffs, and per-source breakdown.

Sources: lib/mutineer/reporter.rb:50-70, lib/mutineer/reporter.rb:200-240

Reporting Data Flow

The following diagram illustrates how the Reporter consumes results and produces output.

The Reporter coordinates data from AggregateResult and KillMatrix to generate formatted output based on user-specified options.

Sources: lib/mutineer/reporter.rb:40-55

Exit Codes Contract

Mutineer uses standardized exit codes to communicate run outcomes to CI environments and automated scripts. This contract is single-sourced to prevent documentation drift.

Code	Meaning
0	Score ≥ threshold (or no gate) AND no baseline regression.
1	Score below threshold, OR broken harness (too many errors/timeouts), OR baseline regression, OR runtime error.
2	Usage error (invalid flags, bad paths, or unreadable baseline).

Sources: rake/docs_contract.rb:61-68, skills/mutineer/SKILL.md:45-51

JSON Report Schema

The JSON report (current version 1.8) follows a strict machine-readable contract. This allows AI agents and CI tools to parse results reliably.

Key Schema Blocks

• summary: Contains aggregate counts and the final score.
• survivors: An array of mutants that tests failed to catch, including unified diff blocks.
• no_verdict: Attempted mutants that failed to produce a result due to error, timeout, or uncapturable status.
• matrix: (Optional) The kill matrix showing which specific tests ran against which mutants.
• baseline: (Optional) Comparisons against a previously stored run, highlighting regressions.

The JSON schema provides a comprehensive view of the mutation run, enabling granular analysis by external tools.

Sources: lib/mutineer/reporter.rb:130-175, rake/docs_contract.rb:13-25

Implementation Details

Diff Generation

The Reporter generates line-aligned unified diffs for survivors. It uses byte offsets from the Prism parser to extract original and mutated source blocks. For files without a trailing newline, it appends the \ No newline at end of file marker to ensure compatibility with git apply.

Sources: lib/mutineer/reporter.rb:440-470, test/json_reporter_test.rb:185-200

Stream Discipline

• stdout: Receives the actual report content (Human, JSON, or HTML).
• stderr: Receives diagnostics, warnings, and gate failure explanations.
  This separation allows users to redirect the report to a file (e.g., mutineer run ... > report.json) without polluting it with diagnostic logs.

Sources: lib/mutineer/reporter.rb:11-15, lib/mutineer/reporter.rb:84-90

Mutation reporting provides the final feedback loop for the mutation testing process, transforming raw test results into actionable data for developers and automated systems. Through strict exit codes and versioned JSON output, it ensures that mutation testing remains a reliable gate in production software pipelines.

Sources: lib/mutineer/reporter.rb:580-600, AGENTS.md:120-130