Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Fixops testing strategy is designed to provide comprehensive verification of the platform's ability to handle complex security data across multi-cloud environments. The strategy shifts focus from merely passing tests to identifying bugs, functional gaps, and data inconsistencies across 291 API endpoints and 14 simulated applications. It ensures that the Decision and Verification Engine can canonicalize data from diverse security tools (SAST, DAST, SCA, CNAPP) and provide deterministic risk scoring.
Sources: tests/e2e/README.md, tests/e2e/comprehensive_e2e_test.py:10-20
E2E Execution Lifecycle
The end-to-end (E2E) testing process follows a multi-phase lifecycle that simulates a complete customer onboarding and operational journey. This lifecycle ensures that infrastructure setup, data ingestion, and pipeline execution work in concert to produce accurate security decisions.
The testing lifecycle begins by simulating three distinct customers (Acme Financial, MedTech Healthcare, and GameZone Entertainment) using AWS, Azure, and GCP architectures respectively.
Sources: tests/e2e/comprehensive_e2e_test.py:410-430, tests/e2e/README.md
Test Execution Modes
Fixops supports three primary modes for different deployment scenarios:
| Mode | Purpose | Success Criteria |
|---|---|---|
PLATFORM_READINESS | Fresh install validation | API availability; empty responses are PASS. |
ONBOARDING_VALIDATION | Post-data ingestion check | Data completeness; empty responses are GAP. |
FULL | Comprehensive analysis | Detailed classification of all bugs and gaps. |
Sources: tests/e2e/comprehensive_e2e_test.py:18-35, tests/e2e/README.md
API and Pipeline Validation
The testing strategy utilizes a FixOpsClient to interact with the API surface, covering endpoints ranging from inventory management to advanced penetration testing results.
Ingestion Workflows
The pipeline is validated by submitting various security artifacts and ensuring the InputNormalizer correctly processes them into standardized formats.
Chunked Upload Testing
For enterprise scalability, the strategy includes a specific workflow for testing chunked file uploads, which is critical for handling large SBOM or SARIF files.
- Initialization:
POST /inputs/sbom/chunks/start - Transfer: Sequential
PUTrequests with specific offsets. - Completion:
POST /inputs/sbom/chunks/{session_id}/completeto trigger server-side assembly and validation.
Sources: tests/test_comprehensive_e2e.py:145-175
Security Architect Validation
A dedicated validation suite ensures the platform meets the structural and algorithmic standards expected of security architecture. This includes verifying the existence of proprietary modules and the sophistication of the implementation.
Structural and Algorithmic Checks
Testing scripts use Python's ast module to statically analyze the codebase for:
- Sophistication: Presence of Graph Algorithms (CFG), Taint Analysis (Source/Sink patterns), and ML/Statistical patterns (z-scores).
- Extensiveness: Ensuring critical modules like
iast_advanced.pyandproprietary_analyzer.pymeet minimum line count thresholds (e.g., 500+ lines). - Module Integrity: Verifying that critical security components (IAST, RASP, Container Analysis) are importable and contain valid class/function definitions.
Sources: tests/e2e/test_real_functionality.py:15-60, scripts/validate_fixops.py:100-130
Consistency and Determinism
The strategy concludes with consistency checks to verify that data flows correctly between different API modules and remains deterministic.
Cross-Endpoint Consistency
The system verifies that:
- Inventory Consistency: Registered applications are accurately reflected in
GET /api/v1/inventory/applications. - Analytics Consistency: Ingested SARIF findings appear in the analytics dashboard totals.
- Pipeline Reflection: The security pipeline summary matches the record counts from ingestion phases.
Sources: tests/e2e/comprehensive_e2e_test.py:1050-1120
Functional Simulations
The ComprehensiveValidator runs simulations to test complex security logic:
- Transitive Risk Propagation: Validating that vulnerabilities in deep dependencies are correctly attributed to the parent application.
- Typosquatting Detection: Testing the identification of malicious packages based on properties like
malicious_package_suspected. - Deterministic Scoring: Ensuring that repeated runs with identical inputs produce identical risk scores down to three decimal places.
Sources: simulations/e2e_validation/comprehensive_validation.py:160-250, simulations/e2e_validation/comprehensive_validation.py:320-350
Result Classification and Reporting
The testing suite categorizes every test case into a specific result type to facilitate rapid debugging and roadmap planning.
| Result | Definition |
|---|---|
| PASS | Valid response with expected data. |
| BUG | Server error (5xx) or unexpected validation failure (422). |
| GAP | Incomplete implementation or missing permission scope. |
| NEEDS-SEEDING | Empty response expected for unseeded data (Fresh install). |
| NOT-APPLICABLE | Disabled features or multipart upload requirements. |
Sources: tests/e2e/comprehensive_e2e_test.py:50-65, tests/e2e/README.md
Reports are generated in multiple formats, including a visual HTML report, a JSON results file for CI/CD consumption, and a "Bugs and Gaps" Markdown summary for developer triage.