Running the Test Suite

Relevant source files

The following files were used as context for generating this wiki page:

• tools/run_tests.py
• tests/test_list.txt
• README.md
• tools/checkstyle.py
• tests/config_with_sensor.py
• pyrightconfig.json

The picamera2 project includes an automated test suite to ensure code quality, stability, and prevent regressions. The primary tool for this is the tools/run_tests.py script, which executes a series of Python scripts defined in test list files. These tests cover a wide range of functionalities, from basic capture operations to complex configurations and application-level examples.

Contributors are expected to ensure all automated tests are passing before submitting a pull request. This is a critical step in the development process to maintain the reliability of the library. The testing framework is designed to be straightforward, running self-checking Python scripts and reporting a summary of the results.

Sources: README.md:46-48, tools/run_tests.py:5-11

Test Runner Script: run_tests.py

The core of the testing system is the run_tests.py script. It automates the process of setting up the environment, executing individual test scripts, and evaluating their outcomes.

Sources: tools/run_tests.py

Command-Line Arguments

The script's behavior can be customized with several command-line arguments.

Argument	Short	Default Value	Description
--dir	-d	/home/pi/picamera2_tests	Specifies the temporary directory where tests are run.
--picamera2-dir	-p	/home/pi/picamera2	The location of the cloned picamera2 repository.
--test-list-files	-t	test_list_drm.txt, test_list.txt	A comma-separated list of files containing the tests to run.

Sources: tools/run_tests.py:141-148

Execution Flow

The script follows a well-defined sequence to execute the test suite. It begins by parsing command-line arguments, loads the lists of tests, and then iterates through each test, running it in a clean, isolated environment.

The following diagram illustrates the high-level execution flow of the run_tests.py script.

Test Configuration

Test Lists

Tests are defined in simple text files, with one test script path per line. The default files are test_list_drm.txt and test_list.txt.

• Lines starting with # are treated as comments and ignored.
• The special value EOL stops the script from reading any further tests from the file.
• By convention, test list files with "drm" in their name are run without an X server, which involves switching to a different virtual terminal (VT 1 instead of VT 7).

Sources: tools/run_tests.py:25-29, tools/run_tests.py:44-51, tools/run_tests.py:169-170, tests/test_list.txt

Test Environment

Each test is executed in a controlled environment:

• Working Directory: A temporary directory (default: /home/pi/picamera2_tests) is used. This directory is cleaned (all files removed) before each test run.
• Timeout: Each individual test script has a 90-second timeout. If a test exceeds this limit, it is considered to have timed out.
• Type Checking: The tests directory is explicitly excluded from type checking analysis by pyright.

Sources: tools/run_tests.py:19-21, tools/run_tests.py:54-57, tools/run_tests.py:91, pyrightconfig.json:3

Test Outcomes and Reporting

The run_tests.py script determines the outcome of each test by executing it as a subprocess and monitoring its exit code and output.

Outcome Evaluation

The script uses a try...except block to handle different execution outcomes. The standard output (stdout) of the test is also scanned for specific keywords.

The following sequence diagram shows the interaction between the test runner and a single test script.

Outcome Conditions

Outcome	Condition
PASSED	The script exits with code 0 and its output contains no "error" strings (after filtering known non-fatal messages).
SKIPPED	The script exits with code 0 and its output contains the word "skipped".
FAILED	The script exits with a non-zero status code (subprocess.CalledProcessError).
TIMED OUT	The script execution time exceeds 90 seconds (subprocess.TimeoutExpired).
ERROR	The script exits with code 0, but its output contains the string "error" (case-insensitive).

Sources: tools/run_tests.py:92-126

The script specifically ignores certain known non-fatal error messages that can be emitted by underlying libraries:

• test pattern modes
• qxcbconnection
• xdg_runtime_dir
• unable to set controls

Sources: tools/run_tests.py:102-109

An example of how a test script can report an error is by printing a line containing "ERROR":

# From tests/config_with_sensor.py
if config['main']['size'] != min_mode['size']:
    print("ERROR: main stream size mismatch")

Sources: tests/config_with_sensor.py:33-34

Code Style and Quality Checks

In addition to functional tests, the project emphasizes code style consistency. The README.md encourages conforming to the PEP 8 style guide.

• checkstyle.py: A script based on clang-format and other tools (pycodestyle, shellcheck) is available in tools/checkstyle.py to check for style issues in patches. It can identify issues like incorrect include orders, PEP 8 violations, and trailing whitespace.
• Pre-commit Hook: The repository recommends installing a pre-commit hook which uses flake8 to lint changes before they are committed, further automating style enforcement.

Sources: README.md:60-66, tools/checkstyle.py

Summary

The testing framework for picamera2 is a robust system designed to maintain high code quality. It relies on the run_tests.py script to execute a comprehensive suite of self-contained tests. By automating test execution, result evaluation, and even code style checks, the project provides a stable and reliable library for developers and users. Adherence to this testing process is a key requirement for any contributions to the codebase.

Sources: README.md:46-48, tools/run_tests.py:5-11