Flaky Test Management

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• QUICK_START.md
• src/index.ts
• tests/basic.test.ts
• package.json

Flaky Test Management is a specialized system within the Enhanced Security Scanner designed to ensure the reliability and repeatability of security test suites. It addresses the inherent instability often found in security scanning environments—such as network timeouts, asynchronous operations, and infrastructure inconsistencies—by implementing automated detection, isolation, and recovery strategies.

The system aims to maintain high test quality assurance by providing self-checking mechanisms and ensuring that security tests remain isolated and repeatable across different environments. By managing flakiness, the scanner maintains a high confidence level in its results, supporting the goal of an industry-leading <5% false positive rate.

Sources: README.md:167-184, src/index.ts:10

Architecture and Components

The flaky test management logic is centered around the FlakyTestManager class, which orchestrates detection, quarantine, and retry logic. It operates as part of the core testing infrastructure to validate results before they impact the main build or security reports.

Key Components

Component	Description
Detection Engine	Monitors failure rates within a single build (intra-run) and across multiple historical builds (inter-run).
Quarantine System	Isolates identified flaky tests from the main CI/CD pipeline to prevent non-deterministic build failures.
Smart Retry Logic	Executes selective retries with exponential backoff for tests identified as potentially transient.

Sources: README.md:170-193, src/index.ts:10, tests/basic.test.ts:14-17

Implementation Logic

The FlakyTestManager is initialized with specific thresholds and windows to define what constitutes a "flaky" test.

const flakyManager = new FlakyTestManager({
  detection: {
    enabled: true,
    intraRunThreshold: 0.1,    // 10% failure rate within build
    interRunThreshold: 0.2,    // 20% failure rate across builds
    monitoringWindow: 7        // Track for 7 days
  },
  quarantine: {
    enabled: true,
    autoQuarantine: true,
    maxQuarantineDays: 30
  },
  retry: {
    enabled: true,
    maxRetries: 3,
    backoffMs: 1000,
    selectiveRetry: true
  }
});

Sources: README.md:172-192

Data Flow: Test Execution and Recovery

The following diagram illustrates the lifecycle of a security test when processed through the Flaky Test Management system.

The system prioritizes retries for tests that meet the "selective retry" criteria, ensuring that genuinely failing code is not ignored while transient environment issues are mitigated.

Sources: README.md:185-193, QUICK_START.md:330-340

Detection and Monitoring

Flakiness is detected through statistical analysis of test outcomes. The system categorizes causes into four primary areas to assist in remediation.

Common Causes of Flakiness

• Timing issues: Network timeouts or race conditions in async operations.
• Environment problems: Resource constraints or version mismatches in the CI/CD environment.
• Infrastructure issues: Instability within the runner or cloud provider.
• Tool integration: Availability issues with external security tools like Semgrep or TruffleHog.

Sources: README.md:200-204, QUICK_START.md:507-515

Monitoring Thresholds

The system utilizes a monitoring window (typically 7 days) to calculate failure percentages.

Metric	Threshold (Default)	Purpose
intraRunThreshold	0.1 (10%)	Detects flakiness occurring within a single execution batch.
interRunThreshold	0.2 (20%)	Detects persistent flakiness across different pipeline runs.
monitoringWindow	7 Days	The timeframe for which historical data is maintained.

Sources: README.md:175-178

Quarantine Management

Tests that consistently fail or exhibit high flakiness are moved into a quarantine state. This prevents them from failing the build while still allowing them to run and collect data.

CLI Operations for Quarantine

Users can interact with the quarantine system via the CLI to manage the lifecycle of unreliable tests.

# Detect flaky security tests over a 7-day window
enhanced-scanner test --detect-flaky --monitor-days 7

# Run scans with automated retry strategy
enhanced-scanner scan --retry-flaky --max-retries 3

# Manage quarantine list
enhanced-scanner quarantine --list
enhanced-scanner quarantine --release <test-name>

Sources: README.md:208-215

Significance in Security Scanning

In the context of security, flaky tests are particularly dangerous as they can lead to "security bankruptcy"—where developers ignore scanning results because they are perceived as unreliable. By implementing this management system, the project ensures:

• Fast Execution: Security test suites are optimized to run in <30s.
• Isolation: Security tests have no dependencies on each other to prevent cascading failures.
• Confidence: Only validated, consistent findings are presented to the user, supporting the core mission of reduced false positives.

Sources: README.md:195-200, QUICK_START.md:214-220

Summary

Flaky Test Management is a critical reliability layer that sits between raw test execution and security reporting. Through a combination of statistical detection, smart retries, and quarantine isolation, it ensures that the scanner's output remains a trusted source of truth for developers and security teams, minimizing the impact of transient environment or infrastructure failures on the security posture.