Strict Testing Patterns

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• pyproject.toml
• scripts/cmd/check/docker_standardization.py
• scripts/tests/test_registry_contract.py
• tests/unit/docker_quality_mock_tests.py
• docker/README.md

Strict Testing Patterns in the FLEXT ecosystem represent a set of inviolable engineering laws and automated quality gates designed to ensure system integrity, configuration validity, and architectural compliance. These patterns move beyond simple functional verification to enforce "Continuous Green" states, where tests validate configuration changes by construction and prohibit anti-patterns like hardcoding or bypasses.

The scope of these patterns encompasses all code tiers, including production source, examples/, scripts/, and tests/, ensuring they all adhere to the same rigor of linting, typing, and coverage gates. This governance is primarily codified in the AGENTS.md engineering law and enforced through specialized tools like ruff, pyrefly, pyright, and custom contract tests.
Sources: AGENTS.md:15-18, AGENTS.md:46-48, pyproject.toml:681-683

P0: Validation by Construction

The core principle of FLEXT testing is that tests must validate configuration and settings changes by construction. This means tests are prohibited from hardcoding values that the project owns through its configuration system. If a legitimate configuration change breaks a test, it is considered a defect in the test itself, which must be fixed rather than freezing the configuration.

Core P0 Requirements

• SSOT Alignment: Expected values must be read from the same Single Source of Truth (SSOT) used by production.
• Immutable Exceptions: Literal expectations are reserved exclusively for immutable external protocol contracts.
• Dynamic Validation: Tests must prove behavior for arbitrary valid values within the configured range.
  Sources: AGENTS.md:15-30

Continuous Quality Gates

The project employs a multi-layered gate system defined in pyproject.toml and governed by the AGENTS.md law. Every contribution must pass these gates before being declared "Done."

Gate Architecture

The following diagram illustrates the flow of a code change through the mandatory quality gates:

This flow ensures that no completion is permitted while the environment is broken or missing real-use QA.
Sources: AGENTS.md:46-52, pyproject.toml:680-685

Mandatory Test Markers

FLEXT uses specific pytest markers to categorize and enforce testing depth:

Marker	Description
unit	Unit level verification
integration	Integration between components
docker	Tests requiring Docker infrastructure
e2e	End-to-end integration tests
performance	Performance and benchmark tests
resilience	Testing system stability under failure
contract	Behavior-pinning contract tests
Sources: pyproject.toml:696-706, scripts/tests/test_registry_contract.py:1-6

Docker Standardization Patterns

Testing involving Docker must adhere to strict centralization rules. Prohibited patterns include creating local docker-compose.yml files or local fixtures in project directories. All Docker artifacts must reside in the centralized docker/ directory.

Docker Compliance Checks

The docker_standardization.py script enforces these patterns through the following checks:

def check_duplicate_compose(self) -> None:
    self._check_no_outside_files(
        "docker-compose file(s)",
        "docker-compose*.yml",
        ("docker",),
    )

def check_duplicate_dockerfiles(self) -> None:
    self._check_no_outside_files(
        "Dockerfile(s)",
        "Dockerfile*",
        ("docker/images",),
    )

Sources: scripts/cmd/check/docker_standardization.py:68-80, docker/README.md:144-155

Lifecycle Management via tk

The tk fixture is the mandatory API for container management. It provides:

1. Isolation: Clean state for each test.
2. Automatic Cleanup: Reverting changes post-test.
3. Dirty State Tracking: Containers failing tests are marked "dirty" and recreated from scratch on the next run.
   Sources: docker/README.md:112-126, scripts/cmd/check/docker_standardization.py:108-115

Contract and Behavior Pinning

Contract tests are used to pin the public behavior of critical systems, such as the registry-driven Make dispatcher. These tests ensure that the command framework remains consistent.

Dispatcher Contract Requirements

• Mutation Opt-in: Every mutating command must declare a required APPLY=Y parameter.
• Default Commands: Every discovered verb must include a WHAT=all command.
• Domain Integrity: A single verb must not span multiple domains (e.g., mixing "quality" and "build").
  Sources: scripts/tests/test_registry_contract.py:44-67, scripts/tests/test_registry_contract.py:38-42

Strict Mocking and Isolation

To prevent leaky abstractions and ensure environment independence, strict mocking rules are enforced, particularly in containerized environments.

Forbidden Patterns in Mocking

• No Mock Copies: Quality Dockerfiles are forbidden from copying specific internal packages like flext_core or flext_observability directly; they must use workspace installs.
• Isolated Sandboxes: Validations that write data must run in isolated sandboxes (tmp_path, pytester) to avoid mutating the active workspace.
  Sources: tests/unit/docker_quality_mock_tests.py:16-19, AGENTS.md:203-208

Summary of Technical Constraints

Constraint Type	Rule	Enforcement
Bypasses	Zero-tolerance for hardcodes, stubs, or fallbacks.	AGENTS.md R0, R2
Typing	No Any or bare object allowed.	pyrefly, pyright
Imports	__init__.py must be export-only (lazy imports).	tests/__init__.py
Evidence	Tasks are "Done" only with command + exit code + output.	AGENTS.md R4
Sources: AGENTS.md:129-145, tests/init.py:12-25

Strict Testing Patterns in FLEXT serve as the primary mechanism for maintaining architectural purity. By treating tests and examples as first-class code subject to the same rigorous gates as production logic, the system ensures that "Continuous Green" is a verifiable reality rather than an optimistic goal.