Pytest Diagnostics
Relevant source files
The following files were used as context for generating this wiki page:
The Pytest Diagnostics system is a robust extraction service designed to parse and structure test execution data from multiple sources, primarily JUnit XML and raw log outputs. Its purpose is to provide structured reports on test failures, errors, warnings, skips, and performance bottlenecks (slow tests) to facilitate build automation and quality gate enforcement within the FLEXT ecosystem.
This system serves as a critical component of the project's quality gates, enabling automated analysis of test results during CI/CD cycles or local development via make verbs.
Sources: src/flext_infra/validate/pytest_diag.py:1-12, README.md:38-40
Architecture and Extraction Logic
The diagnostics service employs a multi-layered extraction strategy. It prioritizes structured JUnit XML data for accuracy in counts and traces, but automatically falls back to regex-based log parsing if XML files are missing or malformed.
Extraction Flow
The extraction process follows a hierarchical logic to ensure that even in partial failure scenarios (where XML might not be generated due to a crash), relevant diagnostics are still captured from the standard output logs.
The diagram shows the fallback mechanism where log parsing supplements or replaces XML data.
Sources: src/flext_infra/validate/pytest_diag.py:127-142, tests/unit/validate/pytest_diag_tests.py:53-70
Key Components
| Class / Model | Responsibility |
|---|---|
FlextInfraPytestDiagExtractor | Main service class for extracting diagnostics and executing the CLI flow. |
FlextInfraPytestDiagXmlMixin | Specialized logic for parsing JUnit XML nodes. |
_DiagResult | Internal mutable container for accumulating extraction state. |
m.Infra.PytestDiagnostics | The canonical Pydantic model representing structured test results. |
Sources: src/flext_infra/validate/pytest_diag.py:27-31, src/flext_infra/validate/pytest_diag.py:112-124
Data Extraction Strategies
JUnit XML Parsing
The system parses JUnit XML to obtain definitive counts and traces. It extracts:
- Failures: Detailed failure messages and stack traces.
- Errors: Setup or teardown errors distinct from test logic failures.
- Timing: Test durations used to identify the slowest test cases.
- Skips: Cases explicitly marked as skipped.
Sources: tests/unit/validate/pytest_diag_tests.py:38-51, src/flext_infra/validate/pytest_diag.py:137-138
Regex-Based Log Parsing
When XML is unavailable, the extractor uses predefined regex patterns to scan the raw pytest log. This is particularly useful for capturing warnings and "slowest durations" summaries rendered by pytest at the end of a run.
# Regex headers used for log section identification
PYTEST_SLOWEST_HEADER_RE = r"=== slowest durations ==="
PYTEST_WARNINGS_HEADER_RE = r"=== warnings summary ==="
PYTEST_FAILURES_OR_ERRORS_RE = r"=== (FAILURES|ERRORS) ==="
Sources: src/flext_infra/validate/pytest_diag.py:53-70, src/flext_infra/validate/pytest_diag.py:73-86
Configuration and Tooling Integration
Pytest behavior is standardized across FLEXT projects via pyproject.toml and a managed tool_config.yml. This ensures consistent diagnostic output formats (like durations=10 and strict-markers) required for reliable extraction.
Standard Pytest Configuration
The following configuration options are enforced by the dependency modernizer to ensure diagnostics compatibility:
| Option | Value / Description |
|---|---|
minversion | 8.0 |
addopts | Includes --durations=10, --markdown-docs, --strict-markers |
python_files | *_test.py, *_tests.py, test_*.py |
markers | Standardized categories: unit, integration, slow, docker, e2e, etc. |
Sources: pyproject.toml:344-365, src/flext_infra/deps/tool_config.yml:517-542
Protected Edit Validation
The infrastructure includes a "Protected Edit" workflow that uses Pytest Diagnostics to validate code changes (e.g., refactors). It runs pytest in a isolated manner and checks for failures before committing changes.
CLI Execution and Reporting
The FlextInfraPytestDiagExtractor.execute() method allows the diagnostic service to be used as a standalone tool. It accepts paths for various report types and writes filtered results to disk.
CLI Parameters
| Parameter | Type | Description |
|---|---|---|
junit | Path | Input path to the JUnit XML file. |
log | Path | Input path to the raw pytest log. |
failed | Path? | Output file for failed case summaries. |
errors | Path? | Output file for error traces. |
warnings | Path? | Output file for warning lines. |
slowest | Path? | Output file for slow test entries. |
Sources: src/flext_infra/validate/pytest_diag.py:34-51, src/flext_infra/validate/pytest_diag.py:145-177
Summary
The Pytest Diagnostics module provides the essential bridge between raw test execution data and structured quality gate reporting. By combining structured XML parsing with robust log fallback and standardized tool configurations, it ensures that test health is always observable and actionable within the FLEXT infrastructure.
Sources: src/flext_infra/validate/pytest_diag.py:1-12, README.md:38-40