Testing Infrastructure

Relevant source files

The following files were used as context for generating this wiki page:

• dev/guidelines/testing.md
• tests/AGENTS.md
• dev/constitution.md
• AGENTS.md
• dev/guides/creating-a-module.md
• dev/guides/debugging-sanity-failures.md

The testing infrastructure provides a multi-tiered validation framework for the opsmill.infrahub Ansible collection. It ensures module compliance, logic correctness, and end-to-end functionality against Infrahub instances. The system utilizes Docker-based execution to provide consistent environments across local development and CI pipelines.

The infrastructure categorizes tests into Sanity, Unit, and Integration tiers. Each tier addresses specific quality gates, from documentation formatting to complex state management and API interaction.

Sources: dev/guidelines/testing.md:3-9, tests/AGENTS.md:7-12, dev/constitution.md:65-71

Test Hierarchy and Orchestration

The project organizes testing into three distinct types, orchestrated primarily through invoke tasks or direct Docker Compose commands.

Test Type Summary

Type	Target	Focus	Tools
Sanity	sanity	Standards compliance, imports, and boilerplate.	ansible-test sanity
Unit	unit	Isolated function and class logic with mocked SDK.	pytest, mock
Integration	integration	End-to-end playbooks against live Infrahub.	ansible-playbook

Sources: dev/guidelines/testing.md:7-12, tests/AGENTS.md:7-12, dev/constitution.md:65-71

Execution Flow

The following diagram illustrates how testing tasks trigger Docker-based execution:

The testing system uses multi-stage Docker builds to isolate dependencies and execution environments for each test tier.
Sources: dev/guidelines/testing.md:27-43, tests/AGENTS.md:16-29

Sanity Testing

Sanity tests validate that the collection adheres to Ansible's strict module standards. This tier is the primary gate for ensuring the collection is technically valid for distribution.

Validation Scope

ansible-test sanity performs the following checks:

• Boilerplate Requirements: Presence of __metaclass__ = type and specific __future__ imports.
• Documentation: Format and completeness of DOCUMENTATION, EXAMPLES, and RETURN blocks.
• Import Correctness: Verification that Python imports function within an isolated environment.
• Plugin Interface: Compliance with Ansible's internal plugin APIs.

Sources: dev/guidelines/testing.md:45-56, dev/guides/debugging-sanity-failures.md:21-41, dev/constitution.md:14-23

Sanity Constraints

Sanity testing skips the standard pep8 test, as the project utilizes Ruff for style and formatting enforcement.
Sources: dev/guidelines/testing.md:52-56, tests/AGENTS.md:34-36

Unit Testing

Unit tests exercise individual functions and classes in isolation. These tests run without network access to an Infrahub instance.

Mocking Strategy

Because these tests run in isolation, they must mock the Infrahub SDK. The project standardizes on unittest.mock for this purpose.

• SDK Client: Mock InfrahubClientSync to prevent real API calls.
• Ansible Module: Mock AnsibleModule to control input parameters (params) and intercept results (exit_json, fail_json).

Sources: dev/guidelines/testing.md:83-111, tests/AGENTS.md:50-53, dev/constitution.md:68-69

Directory Structure

Unit tests mirror the plugins/ directory structure under tests/unit/:

tests/
  unit/
    plugins/
      module_utils/
        test_infrahub_utils.py
      modules/
        test_node.py

Sources: dev/guidelines/testing.md:71-81, tests/AGENTS.md:41-48

Integration Testing

Integration tests validate the full path from the Ansible module to the Infrahub API. These tests require a running Infrahub instance and network connectivity.

Integration Patterns

The infrastructure supports two integration methods:

1. Ansible Targets: Playbooks located in tests/integration/targets/ that exercise module functionality.
2. Pytest Containers: Tests in tests/integration/processor/ or inventory/ that use infrahub-testcontainers to spin up a temporary Infrahub instance.

Sources: dev/guidelines/testing.md:121-137, tests/AGENTS.md:68-68, pyproject.toml:39-43

Network Configuration

Integration containers connect via a dedicated integration_network defined in docker-compose.yml to facilitate communication with Infrahub.
Sources: dev/guidelines/testing.md:121-123, tests/AGENTS.md:68-68

Quality Gates and CI

The project enforces quality through specific gates in the CI pipeline (triggered on every PR to develop).

CI Workflow Steps

1. Linter Job: Runs Ruff (check and format), yamllint, rumdl, and Vale.
2. Type Check: mypy validates type hints, although it currently excludes infrahub_utils.py.
3. Ansible Tests: Executes Sanity and Unit tests within Docker.

Sources: dev/guidelines/testing.md:169-176, AGENTS.md:27-46, dev/constitution.md:72-73

Environment Variables for Test Configuration

Developers can tune test execution using environment variables passed to the Docker containers.

Variable	Purpose	Default
PYTHON_VER	Python version for the Docker build.	3.12
ANSIBLE_SANITY_ARGS	Extra arguments for ansible-test sanity.	(empty)
ANSIBLE_UNIT_ARGS	Extra arguments for pytest unit tests.	(empty)
ANSIBLE_INTEGRATION_ARGS	Extra arguments for integration playbooks.	(empty)

Sources: dev/guidelines/testing.md:36-43

Best Practices

• Collection Verification: Always confirm the test runner detects a new test using pytest --collect-only.
• Mutation Checking: Before finalizing a test, temporarily break the code it protects to ensure the test fails.
• Budget Derivation: For integration tests asserting request counts, derive budgets from pagination_size rather than hardcoding values.
• Sanity First: Run sanity tests after any change to plugin files, as they catch issues linters cannot detect.

Sources: dev/guidelines/testing.md:139-167, tests/AGENTS.md:83-83

Testing infrastructure ensures the reliability of the opsmill.infrahub collection by combining strict Ansible standards with robust unit and integration suites. The reliance on Docker multi-stage builds guarantees that all contributors test against a consistent environment, minimizing "it works on my machine" discrepancies.