Testing Strategy

The Ansible project employs a comprehensive, multi-layered testing strategy to ensure code quality, stability, and correctness. The primary tool for executing all types of tests is ansible-test, a unified testing utility. The strategy is built around three main pillars: Sanity, Unit, and Integration tests, which are executed in isolated environments, preferably using Docker.

This strategy is crucial for maintaining a large codebase with numerous plugins and modules, ensuring that changes are validated against a wide range of standards and use cases. All pull requests are subjected to this testing regimen via an Azure Pipelines-based CI system, which helps developers identify and fix issues before merging.

Sources: AGENTS.md:31-34, AGENTS.md:70-73

The ansible-test Tool

ansible-test is the command-line interface for running all tests within the Ansible repository. It provides a consistent way to execute different test suites and manage test environments.

Test Execution Environments

Tests can be run in different isolated environments to ensure consistency and avoid pollution from the host system. The preferred method is using Docker containers.

The default container is used for sanity and unit tests, while integration tests require distro-specific containers like ubuntu2204 or ubuntu2404.

Sources: AGENTS.md:80-83, AGENTS.md:86-89

Common Commands

The following diagram shows the typical developer workflow for running tests locally.

Test Categories

Sanity Tests

Sanity tests are static analysis checks that enforce coding standards, style, and other quality metrics without executing any Ansible code. They include linters like pylint and pep8, as well as custom validation scripts.

Key sanity tests include:

• validate-modules: A critical test that performs extensive validation on module and plugin files. It checks for schema correctness in DOCUMENTATION, EXAMPLES, and RETURN blocks, and verifies that the documented argument specification matches the one implemented in the code.
• ansible-doc: This test ensures that documentation for plugins can be correctly parsed and rendered by the ansible-doc tool. It validates documentable plugins and checks for errors in the documentation structure.
• pylint: Checks for programming errors and enforces a coding standard.
• pep8: Checks for style guide conformance.

A developer can run all sanity tests or specific ones:

# Run all sanity tests
ansible-test sanity -v --docker default

# Run a specific test (e.g., ansible-doc)
ansible-test sanity -v --docker default --test ansible-doc

Sources: AGENTS.md:41, test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/main.py, test/lib/ansible_test/_internal/commands/sanity/ansible_doc.py

Ignoring Sanity Tests

In some cases, it is necessary to ignore specific sanity test errors for a particular file. This is managed through the test/sanity/ignore.txt file. Each line in this file specifies a path, the test to ignore, and optionally a code to ignore.

Example from test/sanity/ignore.txt:

# lib/ansible/modules/file.py has an undocumented parameter
lib/ansible/modules/file.py validate-modules:undocumented-parameter

This entry tells the validate-modules test to ignore the undocumented-parameter error for the lib/ansible/modules/file.py module.

Sources: test/sanity/ignore.txt:14

Unit Tests

Unit tests are designed to test individual components or functions in isolation. They are located in the test/units/ directory, which mirrors the structure of the lib/ directory. They are written using the pytest framework and should be functional rather than tightly coupled to mocking.

To run all unit tests:

ansible-test units -v --docker default

To run a specific unit test file:

ansible-test units -v --docker default test/units/modules/test_command.py

Sources: AGENTS.md:47-48, AGENTS.md:120-121

Integration Tests

Integration tests verify the public API and end-to-end functionality of modules and plugins. They are organized into "targets," with each target residing in a subdirectory under test/integration/targets/. Each target typically tests a specific module or feature.

An integration test target directory contains tasks, handlers, and other files needed to execute a test scenario. The main entry point is often a tasks/main.yml file.

The following diagram shows the structure of a typical integration test target.

Integration tests are run against specific container images, not the default one.

# Run all integration tests on Ubuntu 22.04
ansible-test integration -v --docker ubuntu2204

# Run a specific integration test target
ansible-test integration -v --docker ubuntu2204 file

Sources: AGENTS.md:50-51

Continuous Integration (CI)

Ansible uses Azure Pipelines for its CI process. When a developer opens a pull request, a series of automated checks are triggered, including the sanity, unit, and integration tests.

Diagnosing CI Failures

When a CI check fails, developers can use the following workflow to diagnose the issue:

The ansibot provides quick feedback directly in the PR comments, while the gh pr checks command gives direct links to the detailed logs in Azure Pipelines for deeper analysis.

Sources: AGENTS.md:92-114

Incidental Coverage Reporting

The project uses a tool to report on incidental test coverage from integration tests. This helps identify which parts of the code are exercised by tests that are not explicitly designed for them. The hacking/azp/incidental.py script is used to process coverage data downloaded from Azure Pipelines. It can combine, filter, and compare coverage reports to find exclusive or missing coverage for specific test targets.

Sources: hacking/azp/incidental.py:21-23, hacking/azp/incidental.py:202-211

Conclusion

Ansible's testing strategy is a robust system designed to maintain high quality and stability. It combines static analysis (sanity tests), isolated component validation (unit tests), and end-to-end functional verification (integration tests). The ansible-test tool provides a unified interface for developers, while the automated CI system ensures that all contributions are thoroughly vetted before being merged into the devel branch. This structured approach is fundamental to the project's development lifecycle.

Sources: AGENTS.md:31-34, README.md:80-82