Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• tools/e2e/run_matrix.py
• tests/conftest.py
• pyproject.toml
• tools/fixloop/triage_and_fix.py
• core/stage_runner.py
• artifacts/upstream_doc_index.json
• tests/test_cli.py

The Aldeci project employs a comprehensive testing strategy centered around pytest. The strategy encompasses unit tests, integration tests, and a structured end-to-end (E2E) testing matrix to ensure code quality, prevent regressions, and validate functionality across different configurations. The testing process is integrated into the development workflow, with tools to automate both test execution and the initial triage of failures.

Code coverage is actively monitored for the core and apps modules, with reports generated in both XML format for CI/CD systems and terminal output for developers. This ensures that new code is adequately tested and maintains the overall quality of the codebase.

Test Framework and Configuration

The project uses pytest as its primary test runner. Configuration is managed within the pyproject.toml file, which defines dependencies, test paths, and default command-line options.

Sources: pyproject.toml

Core Dependencies

The testing toolkit is defined as a development dependency group.

Tool	Purpose
pytest	The core test execution framework.
pytest-cov	Measures code coverage during test runs.
pytest-mock	Provides fixtures for mocking objects and functions.

Sources: pyproject.toml:100-103

Pytest Configuration

The [tool.pytest.ini_options] section in pyproject.toml specifies the standard behavior for running tests.

[tool.pytest.ini_options]
minversion = "6.2"
addopts = "-ra -q --cov=core --cov=apps --cov-report=xml:reports/coverage/coverage.xml --cov-report=term-missing"
testpaths = [
    "tests",
]

• minversion: Specifies the minimum required version of pytest.
• addopts: Sets default options for every pytest run.
  • -ra -q: Reports a summary for all test outcomes except passes, in a quiet format.
  • --cov=core --cov=apps: Specifies that coverage should be measured for the core and apps directories.
  • --cov-report: Defines the output formats for the coverage report (XML and terminal).
• testpaths: Directs pytest to discover tests within the tests directory.

Sources: pyproject.toml:110-115

Test Fixtures and Environment

Global test configurations, fixtures, and hooks are defined in tests/conftest.py. This file is central to setting up the test environment and providing reusable components for tests.

Sources: tests/conftest.py

End-to-End Test Flag

A custom command-line option, --e2e, is added to pytest to specifically enable end-to-end tests. These tests are marked with an e2e marker and are skipped by default to separate fast unit/integration tests from slower, more comprehensive E2E tests.

# tests/conftest.py:17-20
def pytest_addoption(parser):
    """Add a --e2e option to pytest."""
    parser.add_option(
        "--e2e", action="store_true", default=False, help="run end-to-end tests"
    )

This hook allows developers to run a quick verification with pytest or a full E2E suite with pytest --e2e.

Sources: tests/conftest.py:17-22

Environment Fixtures

Fixtures are used to prepare the environment for tests. For example, the signing_env fixture ensures that the necessary environment variables for artifact signing are set before a test that requires them is executed.

# tests/conftest.py:9-15
@pytest.fixture
def signing_env(monkeypatch):
    """Provide a signing environment for tests."""
    monkeypatch.setenv("FIXOPS_SIGNING_KEY", "dummy-key")
    monkeypatch.setenv("FIXOPS_SIGNING_KID", "dummy-kid")
    yield

This pattern is used to manage dependencies on external states like environment variables in a clean and isolated manner.

Sources: tests/conftest.py:9-15

End-to-End (E2E) Testing

E2E testing is orchestrated by tools/e2e/run_matrix.py, which executes a predefined matrix of test scenarios. This ensures that the application behaves correctly across different configurations.

Sources: tools/e2e/run_matrix.py

Test Matrix

The script defines a MATRIX that lists each E2E scenario, specifying the overlay (demo or enterprise) and backend (core or enterprise) to be used for the test run.

Property	Description	Example Values
name	A unique identifier for the scenario.	demo-core, enterprise-enterprise
overlay	The configuration overlay to apply.	demo, enterprise
backend	The backend implementation to use.	core, enterprise

Sources: tools/e2e/run_matrix.py:16-29

Execution Flow

The run_matrix.py script iterates through the matrix, sets the appropriate environment variables for each scenario, and invokes pytest with the --e2e flag. The results, including exit code and output, are captured in a result.json file for each scenario.

This diagram illustrates the E2E test matrix execution flow.

Regression and Bug-Fixing Workflow

The project has an automated workflow for triaging failures discovered by the E2E test matrix, managed by the tools/fixloop/triage_and_fix.py script. This tool standardizes the initial response to a bug, ensuring a regression test is created and a patch file is scaffolded.

Sources: tools/fixloop/triage_and_fix.py

Triage Process

When a result.json from a failed E2E run is provided, the script performs the following actions:

1. Creates a Git Branch: A new branch named fix/<scenario_id> is created.
2. Scaffolds a Regression Test: A new test file is generated in tests/scenarios/ to isolate and reproduce the failure.
3. Scaffolds a Patch File: A placeholder .patch file is created in patches/fixops/ for the developer to implement the fix.
4. Records a Report: A markdown report detailing the failure and next steps is saved in reports/fixloop/.

This sequence diagram shows the automated bug triage and fix preparation process.

CI/CD Integration

Testing is a core component of the CI/CD pipeline. The artifacts/upstream_doc_index.json file indicates the presence of GitHub Actions workflows that execute tests.

• qa.yml: This workflow is responsible for running unit tests, formatters, linters, and coverage checks on code changes.
• release-sign.yml: This workflow likely runs verification checks on generated artifacts and evidence bundles as part of the release process.

This integration ensures that all code merged into the main branch meets the project's quality standards.

Sources: artifacts/upstream_doc_index.json