Using `ansible-test`
Relevant source files
The following files were used as context for generating this wiki page:
- test/lib/ansible_test/_internal/cli/actions.py
- test/lib/ansible_test/_internal/commands/sanity/init.py
- test/lib/ansible_test/_internal/commands/integration/init.py
- test/lib/ansible_test/_internal/commands/units/init.py
- AGENTS.md
- test/lib/ansible_test/_internal/commands/sanity/ansible_doc.py
- test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/main.py
The ansible-test command is the unified testing tool for the Ansible project. It provides a consistent interface for running various types of tests, including sanity, unit, and integration tests. This tool is essential for developers to ensure code quality, correctness, and adherence to project standards before submitting contributions. It is designed to be run in isolated environments, typically using Docker, to guarantee reliable and reproducible test results.
The testing infrastructure is located primarily within the test/ directory, with ansible-test itself serving as the main entry point for all test executions. The tool supports running tests against different Python versions and in different environments, making it a versatile utility for both local development and continuous integration (CI) pipelines.
Sources: AGENTS.md:32-101, pyproject.toml:92-92
Core Test Types
ansible-test categorizes tests into three main types, each serving a distinct purpose in the validation process.
- Sanity Tests: These are static analysis checks that inspect the code without executing it. They include linting (e.g.,
pylint,pep8), documentation validation (ansible-doc), and other code style and quality checks. - Unit Tests: These tests validate individual components or functions in isolation. They are typically written using
pytestand focus on the logic of Python code within thelib/directory. - Integration Tests: These tests validate the functionality of modules and plugins in a more realistic environment. They execute Ansible playbooks against targets (often Docker containers) to ensure that plugins behave as expected.
Sources: AGENTS.md:32-55
The following diagram illustrates the general workflow when using ansible-test.
This diagram shows the process of selecting a test type and execution environment with ansible-test.
Sources: AGENTS.md:32-61
Basic Commands
The following table summarizes the most common ansible-test commands for each test type.
| Command | Description |
|---|---|
ansible-test sanity -v --docker default | Run all sanity tests. |
ansible-test sanity --list-tests | List all available sanity tests. |
ansible-test sanity -v --docker default --test <test_name> | Run a specific sanity test (e.g., pylint). |
ansible-test sanity -v --docker default <path/to/file.py> | Run sanity tests on specific files. |
ansible-test units -v --docker default | Run all unit tests. |
ansible-test units -v --docker default <path/to/test.py> | Run a specific unit test file. |
ansible-test units -v --docker default --coverage | Run unit tests with code coverage reporting. |
ansible-test integration -v --docker ubuntu2204 | Run all integration tests using a specific OS container. |
ansible-test integration -v --docker ubuntu2204 <target_name> | Run a specific integration test target. |
Sources: AGENTS.md:32-55
Test Execution Environments
ansible-test provides options to run tests in isolated environments to ensure consistency and avoid pollution from the host system.
| Option | Description |
|---|---|
--docker | (Preferred) Runs tests inside a Docker or Podman container. This is the most reliable method for isolated testing. Different containers are used for different test types. |
--venv | A fallback option for when containers are not available. It creates a Python virtual environment for running tests, but results may be less reliable due to host environment differences. |
Sources: AGENTS.md:57-61
Test Categories in Detail
Sanity Tests
Sanity tests are crucial for maintaining code quality and consistency across the project. They are implemented as subclasses of SanitySingleVersion or similar base classes.
Sources: test/lib/ansible_test/_internal/commands/sanity/ansible_doc.py:44-44
An example is the ansible-doc sanity test, which verifies that module and plugin documentation can be correctly parsed. It does this by executing ansible-doc commands for the relevant plugins and checking for any errors in the output.
The structure of a sanity test class like AnsibleDocTest is shown below.
This diagram shows that a specific sanity test like AnsibleDocTest inherits from a base class and implements filter_targets and test methods.
Sources: test/lib/ansible_test/_internal/commands/sanity/ansible_doc.py:44-142
Another significant sanity test is validate-modules. This test performs a deep analysis of module files, checking for:
- Correct license headers.
- Proper documentation structure and schema.
- Consistency between the
DOCUMENTATIONblock and theargument_specin the code. - Deprecation correctness.
- Banned imports or function calls (e.g.,
sys.exit).
The validate-modules test is implemented in the ModuleValidator class, which contains numerous methods to perform these specific checks.
Unit Tests
Unit tests are located in test/units/ and are designed to mirror the structure of the lib/ directory. They are executed using the ansible-test units command. These tests should be functional and avoid tight coupling with extensive mocking. Running unit tests with the --coverage flag is recommended to assess how much of the code is exercised by the tests.
Sources: AGENTS.md:43-46, 52-53, 116-119
Integration Tests
Integration tests are located in test/integration/targets/. Each target directory represents a specific feature or plugin to be tested. These tests are executed via ansible-test integration.
A critical aspect of integration testing is the selection of the correct Docker container. Unlike sanity and unit tests which can use --docker default, integration tests require a distribution-specific container (e.g., --docker ubuntu2204, --docker centos9) to properly simulate a real-world environment. The base and default containers should not be used for integration tests.
Sources: AGENTS.md:48-51, 26-28
Diagnosing CI Failures
When a pull request fails in CI, ansible-test is the primary tool for local reproduction and debugging. The recommended workflow is as follows:
- Check
ansibotComments: Theansibotoften posts comments on the PR with direct links to failure logs and explanations for sanity test failures. - Get CI URLs: Use
gh pr checks <number>to get a list of all CI checks and direct links to the Azure Pipelines logs for failed jobs. - Analyze Logs: Examine the logs of the failed jobs in Azure Pipelines to find the specific error messages.
- Reproduce Locally: Use the information from the logs to run the exact same
ansible-testcommand locally to reproduce the failure for debugging.
Sources: AGENTS.md:63-95
This flowchart visualizes the CI failure diagnosis process.
This diagram outlines the steps to diagnose and resolve a CI failure using GitHub tools and local ansible-test execution.
Sources: AGENTS.md:63-95
Summary
ansible-test is the cornerstone of the Ansible testing strategy. It provides a comprehensive suite of tools for developers to validate their changes through static analysis (sanity), isolated component tests (units), and end-to-end functionality checks (integration). By standardizing the testing process and encouraging the use of isolated containerized environments, ansible-test helps maintain the high quality and reliability of the Ansible codebase. Proficiency with this tool is essential for any developer contributing to the project.
Sources: AGENTS.md:32-61