The Ingestion Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Ingestion Pipeline is the entry point for all security and architectural data within the FixOps platform. It provides a standardized mechanism for accepting disparate artifact formats—such as Software Bill of Materials (SBOMs), static analysis results (SARIF), and vulnerability feeds—normalizing them into a canonical internal representation for downstream risk assessment and decision automation.
The system is designed to handle high-volume data through a combination of synchronous REST endpoints, chunked file upload management for large enterprise artifacts, and an orchestration layer that cross-references ingested findings with business context and threat intelligence.
Sources: apps/api/app.py, apps/api/pipeline.py:1-20, CLI_API_INVENTORY.md
Pipeline Architecture and Workflow
The ingestion architecture follows a multi-stage approach where artifacts are first received by the API, validated for content-type and size, and then passed to format-specific normalizers. Once normalized, the artifacts are persisted in an artifact archive before being consumed by the PipelineOrchestrator.
High-Level Data Flow
The following diagram illustrates the lifecycle of an artifact from initial upload to the generation of a pipeline result.
This flow ensures that all inputs, regardless of source tool or format, are unified before analysis begins.
Sources: apps/api/app.py:410-450, apps/api/pipeline.py:30-60, CLI_API_INVENTORY.md
Artifact Ingestion Endpoints
FixOps exposes dedicated endpoints for various security artifact types. Each endpoint performs specific content-type validation and applies upload limits defined in the system's OverlayConfig.
| Artifact Type | API Endpoint | Supported Formats | Required Parser |
|---|---|---|---|
| SBOM | /inputs/sbom | CycloneDX (JSON/XML), SPDX, Syft JSON | load_sbom |
| SARIF | /inputs/sarif | Static Analysis Results Interchange Format | load_sarif |
| CVE Feed | /inputs/cve | Vulnerability Feed (JSON/ZIP) | load_cve_feed |
| Design | /inputs/design | Threat Model Context (CSV) | _process_design |
| VEX | /inputs/vex | Vulnerability Exploitability eXchange | load_vex |
| CNAPP | /inputs/cnapp | Cloud-Native Findings | load_cnapp |
Sources: apps/api/app.py:585-750, apps/api/normalizers.py:10-50, CLI_API_INVENTORY.md
Large File Management (Chunked Uploads)
To accommodate large enterprise artifacts (e.g., massive SBOMs or scan results exceeding 100MB), the pipeline includes a ChunkUploadManager. This system allows clients to break files into manageable segments, ensuring reliable transmission over unstable networks or through restrictive proxies.
Chunked Upload Sequence
The sequence involves initializing a session, appending chunks with offset tracking, and final completion which triggers the normalization process.
Normalization and Canonical Data Models
The InputNormalizer class is responsible for converting tool-specific schemas into the internal FixOps model. This is critical for the PipelineOrchestrator to perform cross-tool correlation.
Key Normalization Components
- SBOM Normalization: Processes components, versions, and PURLs. It handles CycloneDX v1.5+ (including AI/ML-BOMs) and SPDX v2.3.
- SARIF Normalization: Extracts rule IDs, severity levels, and physical locations (file/line) from SAST/DAST tools.
- Business Context: Ingests CSV or YAML files containing service criticality, data classification, and internet exposure metadata.
Internal Data Structures
The normalized artifacts are held in memory (and persisted to disk) using specialized classes:
NormalizedSBOM: Contains a list of components and tool metadata.NormalizedSARIF: Contains a list of findings, tool names, and location data.NormalizedBusinessContext: Maps SSVC (Stakeholder-Specific Vulnerability Categorization) factors to specific components.
Sources: apps/api/normalizers.py:100-300, apps/api/app.py:500-580
Pipeline Orchestration
The final stage of ingestion is the execution of the pipeline. The PipelineOrchestrator consumes the normalized artifacts and performs "Crosswalk" analysis.
Evidence Bundling
Every successful pipeline run generates an evidence_bundle. This is a cryptographically signed package (RSA-SHA256) containing all stage inputs and checksums, intended for audit trails and compliance verification.
Sources: apps/api/pipeline.py:100-250, apps/api/app.py:840-900, tests/e2e/comprehensive_e2e_test.py
Conclusion
The Ingestion Pipeline serves as the foundational data ingestion engine for FixOps. By decoupling artifact reception from normalization and analysis, the system maintains flexibility to support evolving security standards like CycloneDX 1.5 and provides robust handling for enterprise-scale data through chunked uploads and asynchronous orchestration. This structured approach ensures that the subsequent decision engine operates on high-fidelity, canonically structured data.
Sources: apps/api/app.py, apps/api/pipeline.py