Data Flow

Relevant source files

The following files were used as context for generating this wiki page:

• apps/api/normalizers.py
• risk/scoring.py
• services/graph/graph.py
• core/stage_runner.py
• core/cli.py
• README.md

The Aldeci platform processes various security and development artefacts through a multi-stage pipeline to produce contextual risk assessments, compliance evidence, and automated actions. The data flow is designed to be modular and configurable, driven by overlay files that define the behavior for different operational modes, such as demo or enterprise. Data can be ingested via a FastAPI-based API or a functionally equivalent command-line interface (CLI).

At its core, the system normalizes disparate input formats (e.g., SBOM, SARIF, CVEs) into a unified structure. This normalized data is then processed through a series of stages corresponding to the software development lifecycle (SDLC), such as build, test, and deploy. Each stage generates its own output, which is then synthesized in a final decision stage. This final stage produces explainable verdicts, compliance rollups, and auditable evidence bundles.

Sources: README.md, core/stage_runner.py

High-Level Architecture

The data flow follows a clear sequence: ingestion, normalization, staged processing, and output generation. The StageRunner class is the central orchestrator, coordinating the execution of each stage and managing the canonical inputs and outputs for a given run.

This diagram shows the high-level data flow through the Aldeci system.

Ingestion and Normalization

Data enters the system either through the FastAPI endpoints or the CLI. The StageRunner handles the canonical I/O for each stage, ensuring that inputs are saved and outputs are written to a structured directory based on an application ID (app_id) and a run ID (run_id).

Sources: core/stage_runner.py:110-150

Ingestion Points

The system accepts security artefacts via POST requests to specific API endpoints or as file inputs to the CLI.

Method	Endpoint / CLI Command	Description
API	POST /inputs/sbom	Uploads SBOM (Software Bill of Materials) JSON files.
API	POST /inputs/sarif	Uploads scanner findings in SARIF format.

• API | POST /inputs/cve | Uploads CVE or CISA KEV advisories. |
  | CLI | python -m core.cli stage-run --stage <name> --input <file> | Runs a specific pipeline stage using a local file. |

Sources: README.md, core/cli.py:311-316

Data Normalization

Once ingested, raw data is processed by the InputNormalizer class. This component is responsible for parsing various formats (e.g., CycloneDX, SPDX, Snyk) and converting them into standardized Pydantic models. This ensures that downstream services consume a consistent and predictable data structure.

Key normalized data structures include:

Class	File	Description
NormalizedSBOM	apps/api/normalizers.py	A standard representation of an SBOM, containing a list of SBOMComponent objects.
NormalizedSARIF	apps/api/normalizers.py	A standard representation of SARIF findings.
NormalizedCVEFeed	apps/api/normalizers.py	A standard representation of a CVE feed.
NormalizedVEX	apps/api/normalizers.py	A standard representation of a VEX (Vulnerability Exploitability eXchange) document.

The InputNormalizer can also handle special cases, such as converting Snyk's proprietary JSON format into a standard SARIF document if a dedicated converter is not available.

Sources: apps/api/normalizers.py:25-200, tests/test_normalizers.py:42-47

Staged Processing Pipeline

The StageRunner class orchestrates the entire processing pipeline. It executes a series of stages, each corresponding to a phase of the SDLC. Each stage processor (_process_<stage>) takes the normalized input, performs its analysis, and produces a canonical JSON output.

Sources: core/stage_runner.py:110-150

The sequence diagram below illustrates how a user command triggers the StageRunner to execute a specific stage.

Key Stages

The pipeline is defined by a set of canonical stages, each with a specific input and output.

Stage	Input Filename	Output Filename	Description
requirements	requirements-input.csv	requirements.json	Parses feature requirements and derives an SSVC anchor.
design	design-input.json	design.manifest.json	Processes application design context and calculates a design risk score.
build	sbom.json	build.report.json	Normalizes an SBOM, identifies risk flags (e.g., log4j), and calculates a build risk score.
test	scanner.sarif	test.report.json	Analyzes scanner findings, calculates severity counts, and assesses test coverage.
deploy	tfplan.json	deploy.manifest.json	Analyzes infrastructure-as-code plans for posture issues like public S3 buckets.
operate	ops-telemetry.json	operate.snapshot.json	Blends operational telemetry with KEV/EPSS feeds to assess exploit pressure.
decision	decision-input.json	decision.json	Synthesizes all prior stage outputs into a final verdict and generates an evidence bundle.

Sources: core/stage_runner.py:65-83

Decision Stage Flow

The decision stage is the culmination of the pipeline. It aggregates the outputs from all previous stages to make a final determination (ALLOW or DEFER), identify top risk factors, and generate a comprehensive evidence bundle.

This flow is visualized in the diagram below.

Supporting Services

The data flow relies on several key services for enrichment and analysis.

Risk Scoring

The risk scoring engine, detailed in risk/scoring.py, is responsible for calculating a composite risk score. It blends exploit probability from EPSS feeds, known exploitation status from CISA's KEV catalog, and other heuristics like version lag and exposure flags. This provides a "FixOpsRisk" score that helps prioritize remediation efforts.

Sources: risk/scoring.py, README.md

Provenance Graph

The services/graph/graph.py module implements a provenance graph, which is backed by SQLite. This service builds a graph that links commits, CI jobs, build artefacts, SBOM components, and CVEs. This linked data structure allows for powerful queries to trace lineage, identify KEV regressions, and detect security anomalies like version downgrades across releases. The graph is populated by a background worker and queried via the API.

Sources: services/graph/graph.py, README.md, tools/docgen/api_cli_map.py:480-481

Outputs

The primary outputs of the data flow are JSON artefacts and evidence bundles.

• JSON Artefacts: Each stage produces a canonical JSON file (e.g., build.report.json, risk.json). These files are stored in a run-specific directory and can be consumed by UIs, dashboards, or other automated tooling.
• Evidence Bundles: The decision stage creates a evidence_bundle.zip file. This archive contains all the input and output documents from the pipeline run, along with a signed manifest.json that checksums the contents. This bundle serves as an immutable, auditable record for compliance and security reviews.

Sources: core/stage_runner.py:495-519, README.md

Summary

The Aldeci data flow is a structured, stage-based pipeline that transforms raw security artefacts into actionable intelligence. By normalizing inputs, orchestrating a series of modular processing stages, and generating auditable outputs, the system provides a comprehensive and automated solution for contextual risk management and compliance. The StageRunner acts as the central controller, ensuring consistency between CLI and API-driven executions, while services for risk scoring and provenance graphing provide deep, contextual enrichment.