Testing Strategy & Fixtures
Relevant source files
The following files were used as context for generating this wiki page:
The flext-cli project employs a rigorous, multi-tiered testing strategy designed to ensure that the CLI facade, utility modules, and configuration systems remain robust under change. The strategy is built on the principle of "Fixed-Forward" engineering and "Zero-Tolerance" for unvalidated code, where tests, golden files, and executable documentation must remain valid even when global settings or configurations are modified.
The testing architecture prioritizes Railway-Oriented Programming (ROP) patterns, using Result types to handle fallible flows without relying on traditional exception-based control flow in the test logic. Coverage is driven through public interfaces to ensure that the internal implementation remains refactorable without breaking the public contract.
Sources: AGENTS.md:15-30, tests/unit/test_cli_params.py:10-20
Core Testing Principles
The project mandates several key principles that govern how tests are written and maintained:
- P0 - Configuration Validation: Tests must validate configuration changes by construction. They are prohibited from hardcoding or freezing values that are owned by
config/*.yamlor settings. - SSOT (Single Source of Truth): Tests only validate the canonical owner of a fact. If a test breaks due to a legitimate configuration change, the test is considered defective, not the change.
- Continuous Green: No completion of tasks is permitted while the project or environment is broken. This includes mandatory gates for Ruff, Pyrefly, Pyright, Mypy, and Pytest coverage.
- Execution Evidence: All claims of success require evidence including the exact command, working directory, exit status, and decisive output.
Sources: AGENTS.md:20-50, AGENTS.md:120-130
Test Tier Architecture
The project organizes its validation into several tiers, ranging from unit tests to integrated smoke tests using the public facade.
Unit Testing & Coverage
Unit tests target specific utility modules like _utilities/files.py and _utilities/yaml.py. These tests aim for 100% coverage via public interfaces, using parameterized cases to verify edge behaviors and error handling.
The diagram shows how various utility tests feed into a unified Result-based validation pattern.
Sources: tests/unit/_cases/test_files_cov/testsflextclifilescov_part_01.py:1-15, tests/unit/test_yaml_cov.py:1-10
Integrated Smoke Tests
Smoke tests verify that the public CLI facade (flext_cli.cli) correctly orchestrates underlying services. These tests often use the examples/ directory as a source of executable documentation to ensure that provided usage patterns remain functional.
Sources: tests/unit/_cases/test_examples_smoke/testsflextcliexamplessmoke_part_01.py:10-25
Testing Infrastructure & Tooling
The project utilizes a specific set of tools and libraries to enforce its quality gates, as defined in the managed configuration.
| Tool | Purpose | Configuration Highlights |
|---|---|---|
| Pytest | Primary test runner | Strict markers, coverage thresholds, and duration tracking. |
| Ruff | Linting & Formatting | Comprehensive ruleset with minimal ignores for test trees. |
| Pyrefly/Pyright | Static Analysis | Strict typing and member access verification. |
| Coverage.py | Code Coverage | Configured to fail if coverage drops below 15% (project baseline). |
Sources: pyproject.toml:90-110, pyproject.toml:220-250, pyproject.toml:500-550
Test Markers
The project uses standard Pytest markers to categorize and filter test execution:
unit: Isolated logic tests.integration: Tests covering multiple components.edge_cases: Specific focus on boundary conditions.slow: Tests that exceed standard execution time limits.
Sources: pyproject.toml:520-535
Fixtures and Helper Utilities
Testing is supported by a robust set of utilities (aliased as u, tm, and c) that provide consistent assertion patterns and mock environments.
The tm (Test Matcher) Pattern
Assertion logic typically uses the flext_tests.tm module, which provides a declarative syntax for checking results:
tm.ok(result): Asserts aResultis successful.tm.fail(result): Asserts aResultcontains an error.tm.that(value, eq=...): Fluent assertion for value equality.
Sources: tests/unit/_cases/test_files_cov/testsflextclifilescov_part_01.py:20-40, tests/unit/test_version.py:15-30
File and Environment Mocking
Tests frequently use tmp_path and monkeypatch to simulate filesystem structures and environment variables without leaking state between tests.
The sequence shows a typical environment-isolated test for configuration snapshots.
Sources: tests/unit/test_cmd.py:65-85
Configuration and Parameter Validation
A significant portion of the test suite is dedicated to validating CLI parameters and their application to global settings using the Railway-Oriented pattern.
Parameter Application Flow
The apply_to_config method is tested to ensure that CLI flags (like --verbose or --debug) correctly update the internal Settings model without side effects.
def test_apply_to_config_trace_requires_debug(self) -> None:
"""Test trace requires debug - Railway pattern validation."""
config_result = u.Tests.create_test_settings()
settings = config_result.value
result = cli.apply_to_config(settings, trace=True)
tm.fail(result)
error_msg = result.error.lower() if result.error else ""
tm.that(error_msg, has="trace mode requires debug mode")
Sources: tests/unit/test_cli_params.py:48-60
Conclusion
The testing strategy of flext-cli emphasizes stability through strict adherence to SSOT and Railway-Oriented patterns. By treating tests and examples as first-class code governed by the same quality gates as production source, the project ensures that any change to the core configuration or CLI surface is automatically validated against the intended runtime reality.
Sources: AGENTS.md:100-115, README.md:40-50