Feature: Data Ingestion and Normalization

Relevant source files

The following files were used as context for generating this wiki page:

• apps/api/normalizers.py
• core/cli.py
• apps/api/app.py
• lib4sbom/normalizer.py
• apps/api/pipeline.py
• tests/test_normalizers.py
• artifacts/upstream_symbol_index.json
• artifacts/upstream_doc_index.json

The Data Ingestion and Normalization feature is a foundational component of the Aldeci platform. Its primary purpose is to consume a wide variety of security and development artifacts—such as Software Bills of Materials (SBOMs), Static Analysis Results Interchange Format (SARIF) files, and CVE advisories—from multiple sources. These heterogeneous inputs are then transformed into a consistent, standardized set of data models.

This normalization process is critical as it decouples the core analysis and decision-making logic from the specific output formats of various security tools. By creating a unified and deterministic data representation, this feature enables reliable downstream processing by the Pipeline Orchestrator, which handles risk scoring, compliance checks, and automation. Ingestion is supported through both a FastAPI-based REST API for push-style uploads and a command-line interface (CLI) for local and CI/CD workflows.

Sources: artifacts/upstream_doc_index.json

Architecture and Data Flow

The ingestion and normalization process follows a clear data flow, starting from the user or a CI/CD pipeline and feeding into the main processing orchestrator. Raw artifacts are received, validated, and then transformed into Pydantic-based models that the rest of the system can consume.

The following diagram illustrates the high-level data flow for ingestion and normalization.

Sources: artifacts/upstream_symbol_index.json, artifacts/upstream_doc_index.json

Ingestion Mechanisms

The platform provides two primary methods for ingesting security artifacts, offering flexibility for different use cases.

Sources: artifacts/upstream_doc_index.json

REST API

A set of FastAPI endpoints allows for push-style data uploads. The API validates inputs based on MIME type, enforces size limits, and requires an API key for authentication. This is the primary method for integrations with external tools and services.

Endpoint	HTTP Method	Description
/inputs/sbom	POST	Uploads SBOM files in formats like CycloneDX or SPDX.
/inputs/sarif	POST	Uploads scanner findings in SARIF format.
/inputs/cve	POST	Uploads CVE or CISA KEV (Known Exploited Vulnerabilities) advisories.
/inputs/design	POST	Uploads design context, typically from a CSV file.

Sources: artifacts/upstream_doc_index.json, artifacts/upstream_symbol_index.json (for apps/api/app.py)

Command-Line Interface (CLI)

For local development, testing, and integration into shell-based CI/CD pipelines, a CLI provides parity with the API. The core/cli.py module contains the logic for handling ingestion through the stage-run command, which processes local artifact files.

Sources: artifacts/upstream_symbol_index.json (for core/cli.py), artifacts/upstream_doc_index.json

Normalization Layer

The normalization layer is responsible for transforming the raw, ingested data into a set of standardized Pydantic models. This ensures that all downstream components operate on a consistent and predictable data structure, regardless of the source tool.

Sources: artifacts/upstream_symbol_index.json, artifacts/upstream_doc_index.json

Core Components

The normalization process is orchestrated by several key classes and modules that work together to parse and standardize different artifact types.

Sources: artifacts/upstream_symbol_index.json (for apps/api/normalizers.py, lib4sbom/normalizer.py, apps/api/pipeline.py)

Normalized Data Models

The output of the normalization process is a collection of Pydantic models, each representing a specific type of security data in a standardized format.

Model Class	Description	Source File
NormalizedSBOM	A standard representation of an SBOM, with a list of SBOMComponent objects.	apps/api/normalizers.py
NormalizedSARIF	A standard representation of SARIF findings, with a list of SarifFinding objects.	apps/api/normalizers.py
NormalizedCVEFeed	A standard representation for a feed of CVE records.	apps/api/normalizers.py
NormalizedVEX	A standard representation for a Vulnerability Exploitability eXchange (VEX) document.	apps/api/normalizers.py
NormalizedCNAPP	A standard representation for Cloud Native Application Protection Platform (CNAPP) findings.	apps/api/normalizers.py
NormalizedBusinessContext	A standard representation for business context information.	apps/api/normalizers.py

Sources: artifacts/upstream_symbol_index.json (for apps/api/normalizers.py)

Special Handling: Snyk to SARIF Conversion

To improve interoperability with a wide range of tools, the normalization layer includes logic to handle proprietary formats. A notable example is the _convert_snyk_payload_to_sarif function within apps/api/normalizers.py. This utility function takes a Snyk-native JSON payload and transforms it into the industry-standard SARIF format before it undergoes the standard SARIF normalization process. This capability is validated in tests/test_normalizers.py.

Sources: artifacts/upstream_symbol_index.json (for apps/api/normalizers.py._convert_snyk_payload_to_sarif and tests/test_normalizers.py)

Supported Data Formats

The system is designed to ingest and normalize data from a variety of security and development artifacts. The primary supported formats are identified by keywords used throughout the codebase.

Format / Concept	Keyword	Primary Normalizer Class	Description
SBOM	SBOM	NormalizedSBOM	Software Bill of Materials (CycloneDX, SPDX).
SARIF	SARIF	NormalizedSARIF	Static Analysis Results Interchange Format.
KEV / CVE	KEV	NormalizedCVEFeed	CISA Known Exploited Vulnerabilities and general CVE data.
EPSS	EPSS	NormalizedCVEFeed	Exploit Prediction Scoring System data, enriching CVEs.
Provenance	provenance	N/A (Handled by services/provenance)	SLSA-compliant build and source provenance information.
Evidence	evidence	N/A (Handled by evidence/packager.py)	Bundled artifacts for audit and compliance.

Sources: tools/importer/mine_docs.py, artifacts/upstream_symbol_index.json

Conclusion

The Data Ingestion and Normalization feature is a cornerstone of the Aldeci architecture. By abstracting away the differences between various security data formats, it creates a clean, consistent, and reliable foundation for the platform's advanced analysis capabilities. This modular approach ensures that Aldeci can easily adapt to new tools and standards while maintaining the integrity and determinism of its core risk scoring, compliance, and decision-making logic.