Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• CONTRIBUTING.md
• jest.config.js
• tests/basic.test.ts
• QUICK_START.md
• README.md
• package.json
• SECURITY_REVIEW.md

The Testing Strategy for the Unified Security Scanner is designed to ensure the reliability, accuracy, and performance of a multi-phase security analysis engine. It encompasses a multi-layered approach including unit testing, integration testing for third-party adapters, and advanced automated flaky test management to maintain a high-quality codebase with a target code coverage of over 90%.

The strategy focuses on validating the three-phase scanning logic—Traditional SAST, AI-enhanced validation, and Targeted Deep Dives—while maintaining a strict false positive rate of less than 5%. It integrates directly into CI/CD pipelines to act as a security gate, blocking deployments based on configurable severity thresholds.

Sources: README.md:14-25, CONTRIBUTING.md:65-72, QUICK_START.md:214-230

Core Testing Architecture

The project utilizes Jest as its primary testing framework, configured to support TypeScript and ES Modules. The architecture divides tests into package-specific suites to maintain isolation between the scanning engine (core) and the command-line interface (cli).

Test Execution Flow

The following diagram illustrates the lifecycle of a test execution within the project environment:

The test runner uses ts-jest to handle ESM and TypeScript files, collecting coverage from all source files while excluding type definitions and distribution artifacts.

Sources: jest.config.js:1-17, CONTRIBUTING.md:65-80, package.json:9-25

Test Categorization and Requirements

The project mandates specific testing types for different components to ensure comprehensive coverage.

Test Type	Focus Area	Mandatory Requirement
Unit Tests	Core engine logic, orchestrator, and utility functions.	Required for all new features.
Integration Tests	Third-party tool integrations (TruffleHog, Semgrep).	Must validate adapter communication and result parsing.
False Positive Regression	FP reduction engine and rule accuracy.	Must maintain < 5% false positive rate.
Self-Scanning	The scanner's ability to scan its own source code.	Used to identify vulnerabilities within the scanner itself.

Sources: CONTRIBUTING.md:65-80, README.md:387-395, SECURITY_REVIEW.md:5-15

Automated Flaky Test Management

To ensure CI/CD stability, the system implements a FlakyTestManager. This component tracks test failures across multiple runs to distinguish between genuine regressions and environmental instability.

Flaky Detection Logic

The manager uses specific thresholds to monitor test health, such as an intra-run threshold of 0.1 (10% failure rate) and an inter-run threshold of 0.2 (20% failure rate) over a 7-day monitoring window.

Sources: README.md:280-305, tests/basic.test.ts:13-16, src/index.ts:10

Performance Benchmarking and Validation

Testing also extends to performance metrics to ensure the scanner meets its target speed of approximately 1,400 files per second. Validation includes real-world testing against large open-source repositories to benchmark accuracy and speed.

Performance Comparison Targets

Metric	Traditional SAST	Enhanced Scanner v2.0
Scan Speed	57 files/sec	1,400 files/sec
False Positive Rate	88.4%	< 5%
Parallelization	4 workers	16 workers

Sources: README.md:370-385, QUICK_START.md:520-530

Implementation Details

Jest Configuration

The project uses a centralized jest.config.js to enforce ESM standards and coverage tracking.

export default {
  preset: 'ts-jest/presets/default-esm',
  extensionsToTreatAsEsm: ['.ts'],
  testEnvironment: 'node',
  collectCoverageFrom: [
    'src/**/*.ts',
    '!src/**/*.d.ts',
  ],
  coverageReporters: ['text', 'lcov', 'html'],
};

Sources: jest.config.js:1-17

Basic Test Structure

Tests are written in TypeScript and leverage describe/test blocks to validate core functionality, including the successful instantiation of the scanner and management modules.

describe('Enhanced Security Scanner', () => {
  test('should import successfully', async () => {
    const { EnhancedSecurityScanner } = await import('../src/index.js');
    expect(EnhancedSecurityScanner).toBeDefined();
  });
});

Sources: tests/basic.test.ts:5-12

Summary

The Testing Strategy is a rigorous framework combining traditional unit/integration tests with AI-aware validation and automated reliability management. By enforcing a 90% coverage minimum and utilizing a quarantine system for flaky tests, the project ensures that security findings are both accurate and reproducible across different development and CI/CD environments.