Baseline Management

Relevant source files

The following files were used as context for generating this wiki page:

• lib/mutineer/baseline.rb
• lib/mutineer/reporter.rb
• lib/mutineer/cli.rb
• test/baseline_test.rb
• rake/docs_contract.rb
• test/cli_test.rb

Baseline Management provides continuous integration (CI) gating by comparing current mutation test results against a stored JSON report. The system identifies regressions, such as new surviving mutants or a decrease in the mutation score. It utilizes stable mutant identifiers to ensure comparisons remain accurate across different test runs and environments.

Sources: lib/mutineer/baseline.rb:10-15, lib/mutineer/reporter.rb:58-75

Architecture and Components

The baseline system consists of two primary data structures: the Baseline class and the Delta structure. The Baseline object loads and holds the reference data, while the Delta object stores the results of the comparison.

Data Structures

Component	Description
Mutineer::Baseline	Loads a prior JSON report and extracts survivors and mutation scores.
Mutineer::Baseline::Delta	Stores new survivors, fixed survivors, and score changes identified during a diff.
survivors[]	An array of mutant objects that the test suite failed to kill in a previous run.
score	The mutation percentage calculated as killed / (killed + survived).

Sources: lib/mutineer/baseline.rb:25-36, lib/mutineer/reporter.rb:110-140

The diagram shows the relationship between the Baseline loading logic, the resulting Delta comparison, and the Reporter that presents these results.

Sources: lib/mutineer/baseline.rb:10-36, lib/mutineer/reporter.rb:45-55

Baseline Comparison Logic

The comparison logic calculates the difference between current results and the baseline. It performs two main checks: survivor identity matching and score threshold validation.

Survivor Matching

The system compares mutants using stable identifiers. A mutant regresses if its ID exists in the current run but is absent from the baseline. If a baseline was created with an older ID format (pre-version 1.4), the system attempts to match mutants using a combination of the old ID and the normalized file path.

Sources: lib/mutineer/baseline.rb:98-115, test/baseline_test.rb:210-225

Score Gating and Epsilon

The diff method calculates the score drop. If the current score is lower than the baseline score minus a user-defined epsilon value, the run is marked as regressed. The epsilon parameter allows for minor float jitter without failing the CI gate.

Sources: lib/mutineer/baseline.rb:135-144, lib/mutineer/cli.rb:75-80

This flowchart illustrates the logic path taken during a baseline comparison, highlighting how scoped runs bypass score comparisons.

Sources: lib/mutineer/baseline.rb:118-144, test/baseline_test.rb:85-100

Scoped Run Handling

Runs performed with the --since flag are considered "scoped." A scoped run only targets mutants on changed lines. Because the denominator for the mutation score differs from a full run, the system automatically skips the score-drop check for scoped comparisons to prevent false negatives.

• Score Comparison: The system skips the score-drop check if either the baseline or the current run is scoped.
• Survivor Gating: The system still identifies new survivors by ID, as ID stability is maintained across scopes.
• Informational Scores: Both the baseline score and current score are reported, but the score_comparable flag is set to false.

Sources: lib/mutineer/baseline.rb:56-65, lib/mutineer/baseline.rb:122-132, test/baseline_test.rb:105-120

Reporting and Exit Codes

The Reporter module appends a "Baseline comparison" section to the output. This section lists new survivors by their qualified subject name and location.

Reporting Metadata

The JSON report includes specific fields to manage baselines:

• summary.scoped: A boolean indicating if the report is a partial diff.
• summary.id_format: An integer identifying the version of mutant IDs used.
• baseline.new_survivors[]: A list of mutants that regressed.

Sources: lib/mutineer/reporter.rb:150-180, lib/mutineer/reporter.rb:400-415

CLI Configuration

Users configure baseline behavior via specific CLI flags:

Flag	Type	Default	Description
--baseline FILE	String	nil	Path to a prior JSON report to use as a baseline.
--baseline-epsilon FLOAT	Float	0.0	Tolerance for score drops before failing.
--format json	String	"human"	Required format for generating a reusable baseline.

Sources: lib/mutineer/cli.rb:60-75, test/cli_test.rb:550-565

Exit Code Logic

The Reporter#exit_code method integrates the baseline result into the final process status. If regressed is true in the Delta, the process exits with status 1, even if the mutation score is above the standard threshold.

Sources: lib/mutineer/reporter.rb:125-145, lib/mutineer/cli.rb:460-470

Summary

Baseline Management enables strict regression testing by comparing current survivors and scores against a known good state. It handles the complexity of scoped runs and legacy ID formats to ensure that CI gates remain reliable. By using the --baseline flag, developers ensure that new code does not introduce untested vulnerabilities or decrease the overall quality of the test suite.

Sources: lib/mutineer/baseline.rb:10-20, lib/mutineer/cli.rb:475-485