Quality Gates
Relevant source files
The following files were used as context for generating this wiki page:
Quality Gates in the flext-dbt-ldap project represent a multi-layered verification system designed to ensure data integrity, code standards, and operational reliability during LDAP data transformation. These gates encompass development-time linting, automated testing, and runtime data quality validations implemented through both Python services and dbt (data build tool) SQL tests.
The system enforces a "Zero-Tolerance / Strict-Total" policy where failures at any gate—whether in coding standards, schema compliance, or data completeness—must be addressed as root-cause fixes rather than via workarounds or suppressions.
Sources: README.md:37-39, AGENTS.md:16-24
Engineering Conduct and Development Gates
The project adheres to the Universal Agent Law, a set of inviolable rules for engineering conduct. These laws act as the primary quality gate for all contributors (human or AI), ensuring that code changes are generic, clean, and validated.
Core Engineering Principles
- SSOT (Single Source of Truth): Reference authoritative sources rather than duplicating facts.
- Fix-Forward-Only: Shared work trees require fixing the current state rather than reverting.
- Strict Typing: Use of
Anyorobjectis forbidden without documentation; most-restrictive types are required. - Make Contract: Canonical
makeverbs (check,test,val,docs) serve as the entry point for quality verification.
Sources: AGENTS.md:16-86, README.md:37-39
Technical Linting and Compliance
The project uses extensive Python-based quality tools configured in pyproject.toml to enforce standards before code reaches the runtime environment:
| Tool | Purpose | Configuration Detail |
|---|---|---|
| Ruff | Comprehensive linting | Target version Python 3.13; excludes __pycache__, .venv, etc. |
| Mypy / Pyright | Static type checking | Strict equality, no implicit re-exports, strict type mode. |
| Coverage | Testing depth | fail_under = 20 (minimum requirement for coverage). |
| Codespell | Documentation quality | Checks filenames and content for common misspellings. |
Sources: pyproject.toml:98-120, pyproject.toml:292-315, pyproject.toml:500-520
Data Quality Validation Service
At the application level, the Quality protocol and FlextDbtLdapClientMixin implement runtime quality gates that evaluate data extracted from LDAP before it is processed by dbt models.
Validation Workflow
The validation service calculates a quality_score based on the presence of required attributes and the validity of Distinguished Names (DNs). If the score falls below the min_quality_threshold, the pipeline execution is halted.
This diagram illustrates the sequence of checks performed during the validate_ldap_data operation.
Sources: src/flext_dbt_ldap/services/client.py:108-142, src/flext_dbt_ldap/protocols.py:165-175
Validation Metrics
Metrics are captured in the ValidationMetrics model to provide visibility into data health:
total_entries: Count of all evaluated LDAP entries.valid_dns: Count of entries with a validdnattribute.valid_entries: Count of entries satisfying all configuredrequired_attributes.quality_score: A normalized float representing the ratio of valid entries to total entries.
Sources: src/flext_dbt_ldap/_models/results.py:16-32, src/flext_dbt_ldap/services/client.py:121-131
Generic dbt Data Quality Gates
Once data enters the warehouse, a series of SQL-based quality gates are applied to the staging and dimension tables. These generic tests ensure referential integrity and logical consistency.
Automated SQL Tests
The following table summarizes the generic quality tests defined in the project:
| Test Target | Logic | Potential Failure |
|---|---|---|
| UID Uniqueness | Checks for duplicate uid in stg_users. | Multiple users sharing the same identifier. |
| DN Format | Applies validate_dn_format macro. | Improperly structured LDAP Distinguished Names. |
| Email Validity | Checks dim_users for null domains where email exists. | Invalid or malformed email addresses. |
| Referential Integrity | Left joins fact_memberships to user and group dims. | Orphaned memberships referencing non-existent users/groups. |
| Hierarchy Validity | Recursive CTE checks OU parent-child relationships. | Circular references or infinite loops in OU structure. |
| Freshness | Compares modified_at to current_timestamp. | Future-dated timestamps in LDAP metadata. |
Sources: tests/generic/test_data_quality.sql:3-64
Data Flow for Quality Checks
The integration between LDAP extraction and dbt transformation includes explicit quality gates at the transition points.
This diagram depicts how the project coordinates Python-based validation with downstream dbt SQL testing.
Sources: src/flext_dbt_ldap/services/client.py:65-98, tests/generic/test_data_quality.sql:1-15
Monitoring and Reporting
Quality gate outcomes are monitored through the Monitoring protocol, which tracks the health of the integration and the freshness of the data.
- Freshness Monitoring: Configured via
DbtSourceFreshnessto define warning and error thresholds for data latency. - Performance Tracking: Captures
execution_time,rows_processed, andmemory_usageto ensure the transformation process meets performance gates. - Service Status: Provides a
ServiceStatussummary to indicate if the integration isOPERATIONAL.
Sources: src/flext_dbt_ldap/protocols.py:228-245, src/flext_dbt_ldap/_models/configuration.py:63-70, src/flext_dbt_ldap/_models/results.py:70-85
Conclusion
Quality Gates in flext-dbt-ldap provide a robust framework for maintaining the high standards required for enterprise identity data. By combining strict engineering conduct, early-stage Python validation, and comprehensive dbt SQL testing, the system ensures that only high-quality, schema-compliant, and logically consistent LDAP data is utilized for downstream analytics.