Testing: Twister & Ztest
Relevant source files
The following files were used as context for generating this wiki page:
The Zephyr project utilizes a multi-layered testing strategy centered around the Ztest framework and the Twister test runner. Ztest provides a structured C-based environment for creating unit and integration tests, offering assertion macros, test suites, and fixtures to validate software behavior. Twister acts as the orchestration engine, responsible for discovering tests, building binaries for various hardware or emulation targets, and aggregating results.
This testing infrastructure is critical for maintaining the high quality and safety standards required by the project. For safety-critical development, a high test coverage (ideally 100% for entry points, statements, and branches) is a prerequisite for auditable codebases aimed at certifications like IEC 61508.
The Ztest Framework
Ztest (Zephyr Test Framework) is the primary tool for developer-level testing. It supports both integration testing, where the full RTOS is involved, and unit testing, where specific modules are isolated and tested on a host machine using a unit_testing board.
Test Suite Structure
A test suite is defined using the ZTEST_SUITE macro. This macro registers a group of tests and can optionally include setup, teardown, and predicate functions to control execution flow.
The diagram above illustrates the execution lifecycle of a Ztest suite. Predicates allow the suite to run only when specific global states are met, while setup and teardown manage suite-level resources.
Sources: doc/develop/test/ztest.rst:11-40
Key Ztest Macros
Tests are added to suites using specific macros that define the execution context (e.g., userspace vs. kernel space).
| Macro | Description |
|---|---|
ZTEST(suite, test) | Standard test case within a suite. |
ZTEST_USER(suite, test) | Runs the test in a userspace thread if CONFIG_USERSPACE is enabled. |
ZTEST_F(suite, test) | Includes a fixture variable of type <suite_name>_fixture. |
ZTEST_EXPECT_FAIL | Annotates a test that is expected to fail; Ztest marks it as passed if it fails. |
Z_TEST_SKIP_IFDEF | Skips a test based on Kconfig definitions. |
Sources: doc/develop/test/ztest.rst:43-61, doc/develop/test/ztest.rst:115-136, doc/develop/test/ztest.rst:200-215
Twister Test Runner
Twister is a Python-based script (scripts/twister) that parses the Zephyr tree to find test scenarios and applications. It identifies tests by looking for testcase.yaml (or sample.yaml) files and scanning source code for Ztest case names.
Discovery and Execution Flow
Twister maps test scenarios defined in YAML files to specific platforms (hardware or emulators like QEMU). It handles the complexities of building for different architectures and managing the output for traceability.
The sequence diagram shows how Twister coordinates between the source code, build system, and target execution to produce results.
Sources: doc/develop/test/ztest.rst:180-195, samples/sample_definition_and_criteria.rst:37-65
Blackbox Testing
Twister itself is tested using "blackbox" tests written in Python with the pytest library. These tests invoke Twister as a user would, checking standard output (stdout), standard error (stderr), and generated files like testplan.json or twister.xml.
Sources: doc/develop/test/twister/twister_blackbox.rst:10-40
Unit Testing vs. Integration Testing
Zephyr distinguishes between testing modules in isolation (Unit) and testing them within the OS environment (Integration).
Unit Testing on Host
Unit tests are built with BOARD=unit_testing and run as native applications on the host (primarily Linux). This mode bypasses the kernel and allows direct function calls, often requiring stubs or mocks.
- Requirements: Must set
type: unitintestcase.yaml. - Mocking: Integrated with the FFF (Fake Function Framework) to simulate complex object behavior.
Sources: doc/develop/test/ztest.rst:222-245, doc/develop/test/ztest.rst:400-410
Integration Testing
Integration tests run on real hardware or emulators. They verify that components work together within the RTOS. Every sample application in Zephyr must have a YAML file so that Twister can verify its buildability and, if a harness is configured, its runtime output.
Sources: doc/develop/test/ztest.rst:155-175, samples/sample_definition_and_criteria.rst:37-45
Advanced Testing Features
Stress Testing (Ztress)
The Ztress framework executes user functions in multiple priority contexts to validate preemption resilience. It tracks execution counts and preemptions, adjusting sleep periods to maximize CPU load and the probability of preemption.
Sources: doc/develop/test/ztest.rst:318-340
Traceability and Documentation
For safety and quality compliance, tests are documented using Doxygen.
- Verification: The
@verify{@req{ID}}tag links a test case to a specific requirement. - Grouping: All test documentation must be grouped under the
all_testsDoxygen group, with subgroups prefixed bytests_.
Sources: doc/project/documentation.rst:30-65
Test Sequence Randomization
To find hidden dependencies between tests, users can enable CONFIG_ZTEST_SHUFFLE to randomize the test execution order. If a failure occurs, the seed is displayed so the failure can be reproduced.
Sources: doc/develop/test/ztest.rst:445-455
Summary of Quality Goals
The testing infrastructure supports several key software quality goals:
- Structural Coverage: For SIL 3/SC 3 targets, 100% coverage is required for entry points, statements, and branches.
- Regressions: High test coverage ensures that faulty changes are detected quickly.
- Traceability: Documentation and Twister reports provide evidence that requirements are implemented and verified.
Sources: doc/safety/safety_overview.rst:110-135, doc/contribute/contributor_expectations.rst:95-115