Creating Custom Scenarios

Relevant source files

The following files were used as context for generating this wiki page:

• tools/scenario/generate.py
• tools/scenario/validate.py
• tools/scenario/schema.yaml
• scenarios/test/test_0000.yaml
• tools/e2e/run_matrix.py
• tools/scenario/catalog.json
• scenarios/design/design_0200.yaml
• core/stage_runner.py
• tools/fixloop/triage_and_fix.py

Custom scenarios are a core component of the Aldeci project's testing strategy. They are defined in YAML files and are used to drive the end-to-end (E2E) matrix harness, ensuring that various features and command chains work as expected under different conditions. These scenarios are designed to be synthesized from templates, validated against a schema, and executed across different configurations (overlays and backends).

The scenario corpus is generated by the tools/scenario/generate.py script, which creates a large set of YAML files under the scenarios/<stage>/ directories. A central catalog, tools/scenario/catalog.json, indexes all available scenarios, making them discoverable for the E2E test runner. This system allows for comprehensive and repeatable testing of the Aldeci CLI and its underlying logic.

Sources: tools/scenario/generate.py:5-10, tools/e2e/run_matrix.py:1-6, tools/scenario/catalog.json

Scenario Schema

All scenario files must conform to a defined JSON schema. This ensures consistency and validity across the entire test corpus. The schema defines the required fields, data types, and structure of a scenario file.

Sources: tools/scenario/schema.yaml, tools/scenario/validate.py:17-18

Schema Diagram

The following diagram illustrates the high-level structure of a scenario object as defined in the schema.

Sources: tools/scenario/schema.yaml:3-60

Key Fields

The table below describes the main properties of a scenario object.

Field	Type	Required	Description
id	string	Yes	A unique identifier for the scenario, e.g., design_0200.
stage	string	Yes	The development lifecycle stage this scenario belongs to. Must be one of: design, build, test, deploy, operate.
description	string	Yes	A human-readable description of the scenario's purpose.
tags	array	No	A list of strings for categorizing the scenario (e.g., topology:mesh, epss:medium).
inputs	object	Yes	A map of input artifacts required for the scenario. Each key is a logical name, and the value specifies the source path and format.
commands	array	Yes	An ordered list of CLI commands to be executed.
expected	object	No	Defines the expected outcomes, such as the final exit_code.
invariants	array	No	A list of named invariants to be enforced by the test harness.
skip_if_gap	array	No	A list of command names that, if they result in a "gap" (exit code 12), will cause the rest of the scenario to be skipped.

Sources: tools/scenario/schema.yaml:5-60, scenarios/design/design_0200.yaml

Scenario Generation

Scenarios are automatically generated by the tools/scenario/generate.py script. This script synthesizes a large corpus of tests by combining various templates, inputs, and configurations.

Sources: tools/scenario/generate.py:5-10

Generation Process

The generation process follows these steps:

1. Load Upstream Samples: If available, the script scans the .upstream/Fixops directory for sample artifacts like SBOMs and SARIF files.
2. Synthesize Inputs: It creates a canonical mapping of inputs, such as sample_sbom and sample_policy, by selecting from the available upstream samples or using default examples.
3. Define Templates: A list of templates is created based on different topologies (e.g., linear, diamond), epss_bands, and other variations like version-lag.
4. Build Scenarios: The script iterates through the defined stages (design, build, etc.) and randomly picks templates to create a specified number of unique scenarios. The build_scenario function assembles the final scenario object, including generating the correct command-line arguments.
5. Write Files: Each generated scenario is written to a YAML file in the corresponding scenarios/<stage>/ directory.
6. Update Catalog: Finally, tools/scenario/catalog.json is updated with metadata for each new scenario, including its ID, stage, path, and tags.

This process is illustrated in the flowchart below.

Sources: tools/scenario/generate.py:32-337

Command Generation

The command_args function in generate.py is responsible for constructing the arguments for each command in a scenario's command chain. It uses the scenario_id and a template to create workspace-specific paths for inputs and outputs, ensuring that each scenario run is isolated.

The default command chain is:

1. ingest sbom
2. risk score
3. graph lineage
4. evidence bundle
5. gate

Sources: tools/scenario/generate.py:34-40, 148-208

Scenario Structure Example

A scenario is a YAML file that orchestrates a series of CLI commands. Below is a condensed example of a generated scenario.

id: design_0200
stage: design
description: 'Design scenario #200 covering mesh topology'
tags:
- epss:medium
- stage:design
- topology:mesh
inputs:
  sbom:
    source: samples/fixops_import/example.sbom.json
    format: sbom
  sarif:
    source: samples/fixops_import/example.sarif.json
    format: sarif
commands:
- name: ingest sbom
  args:
  - --in
  - samples/fixops_import/example.sbom.json
  - --out
  - artifacts/e2e/design_0200/normalized_sbom.json
- name: risk score
  args:
  - --sbom
  - artifacts/e2e/design_0200/normalized_sbom.json
  # ... other args
- name: gate
  args:
  - --policy
  - samples/fixops_import/example.policy.json
  # ... other args
expected:
  exit_code: 0
invariants:
- kev_policy
- epss_monotonic
- idempotence
- manifest_integrity
skip_if_gap: []

Sources: scenarios/design/design_0200.yaml, scenarios/test/test_0000.yaml

Scenario Validation and Execution

To maintain the integrity of the test suite, scenarios are validated and then executed by a dedicated test harness.

Validation

The tools/scenario/validate.py script ensures that all scenario files in the scenarios/ directory adhere to the official schema defined in tools/scenario/schema.yaml. It uses the jsonschema library to perform this validation. If jsonschema is not available, it performs a basic check for the presence of required fields.

Sources: tools/scenario/validate.py

Execution

The E2E test matrix is run by tools/e2e/run_matrix.py. This script discovers all scenarios, iterates through them, and executes them against different combinations of overlays (e.g., demo, enterprise) and backends (e.g., local, http).

The execution flow for a single scenario is as follows:

1. Load Scenario: The script loads the scenario's YAML definition.
2. Run Commands: For each command in the commands array, it constructs the full CLI command and executes it as a subprocess.
3. Check Status: It checks the exit code of each command. An exit code of 12 is treated as a "gap" (an expected unimplemented feature), while any other non-zero code is a failure.
4. Evaluate Invariants: After all commands complete, it calls tools.e2e.invariants.evaluate_invariants to check for conditions that must hold true for the run.
5. Report Results: The results, including command outputs and invariant checks, are written to a result.json file in the reports/e2e/ directory.

The following sequence diagram illustrates this process.

Sources: tools/e2e/run_matrix.py:100-165

Handling Failing Scenarios

The project includes a utility to streamline the process of addressing failing scenarios. The tools/fixloop/triage_and_fix.py script is designed to help developers triage a failure and prepare a fix.

When run with a path to a failing result.json, the script:

1. Creates a new git branch named fix/<scenario_id>.
2. Scaffolds a regression test file under tests/scenarios/test_<scenario_id>_regression.py.
3. Scaffolds a patch skeleton file in patches/fixops/ for the specified upstream file.
4. Generates a markdown report in reports/fixloop/ detailing the failure and next steps.

This automated workflow provides a consistent starting point for debugging and patching issues identified by the scenario matrix.

Sources: tools/fixloop/triage_and_fix.py

Summary

The scenario system is a powerful and extensible framework for E2E testing in Aldeci. By defining tests as data in YAML files, the project can easily generate, validate, and execute a vast matrix of test cases covering different stages, command chains, and configurations. The tooling around generation, validation, execution, and triage provides a robust foundation for maintaining code quality and ensuring predictable behavior across the entire platform.