Data Quality & Generic Tests

Relevant source files

The following files were used as context for generating this wiki page:

• tests/generic/test_data_quality.sql
• tests/test_macros.sql
• src/flext_dbt_ldap/services/sync.py
• src/flext_dbt_ldap/services/client.py
• src/flext_dbt_ldap/protocols.py
• src/flext_dbt_ldap/_models/results.py

Data Quality and Generic Tests in the flext-dbt-ldap project ensure the integrity, consistency, and accuracy of LDAP data as it is transformed from raw directory entries into structured data warehouse models. The system employs a multi-layered validation approach, combining pre-transformation Python-based checks with post-transformation SQL-based dbt generic tests.

Sources: src/flext_dbt_ldap/services/client.py:126-163, tests/generic/test_data_quality.sql:1-55

Validation Architecture

The data quality framework operates across two primary stages: extraction-time validation and warehouse-time testing. During extraction, the system calculates metrics such as DN validity and attribute completeness. Once in the warehouse, generic dbt tests enforce referential integrity and schema compliance.

The diagram above shows the data flow from the LDAP source through initial Python-based validation into the dbt modeling layer and final SQL testing.
Sources: src/flext_dbt_ldap/services/client.py:84-110, src/flext_dbt_ldap/services/sync.py:165-179

Python-Based Validation

The FlextDbtLdapClientMixin provides the validate_ldap_data method, which performs immediate checks on extracted entries before they are processed by dbt models. This ensures that the pipeline fails early if the source data does not meet minimum quality thresholds.

Key Validation Metrics

The system tracks several key performance indicators during the validation phase:

Metric	Description	Source
total_entries	Total number of LDAP entries evaluated.	results.py:15
valid_dns	Count of entries containing a non-null Distinguished Name.	results.py:18
valid_entries	Count of entries satisfying all configured required attributes.	results.py:21
quality_score	Ratio of valid entries to total entries.	results.py:25
validation_passed	Boolean flag indicating if quality_score meets min_quality_threshold.	results.py:28

Sources: src/flext_dbt_ldap/_models/results.py:13-31, src/flext_dbt_ldap/services/client.py:134-154

SQL-Based Generic Tests

The project implements specialized SQL tests to validate LDAP-specific data structures, such as hierarchical Organizational Units (OUs) and Distinguished Name (DN) formats.

Data Consistency Checks

The following generic tests are defined in test_data_quality.sql:

• Unique UIDs: Ensures uid values in stg_users are not duplicated.
• DN Format: Uses the validate_dn_format macro to verify LDAP DN strings.
• Referential Integrity: Checks fact_memberships to ensure all linked users and groups exist in their respective dimensions.
• Future Timestamps: Validates that modified_at timestamps are not ahead of the current system time.

Sources: tests/generic/test_data_quality.sql:4-14, tests/generic/test_data_quality.sql:24-30, tests/generic/test_data_quality.sql:50-57

Hierarchical Integrity

For Organizational Units, a recursive CTE test ensures that the OU hierarchy forms a valid tree structure starting from the configured ldap_base_dn. This prevents infinite loops and orphaned nodes in the hierarchy.

with recursive ou_check as (
    select dn, parent_ou_dn, 1 as depth
    from {{ ref('stg_org_units') }}
    where parent_ou_dn = '{{ var("ldap_base_dn") }}'
    union all
    select o.dn, o.parent_ou_dn, oc.depth + 1
    from {{ ref('stg_org_units') }} o
    join ou_check oc on o.parent_ou_dn = oc.dn
    where oc.depth < 10
)

Sources: tests/generic/test_data_quality.sql:33-44

Macro Validation Tests

The project includes a suite of dbt macro tests in test_macros.sql to verify the logic of transformation and validation helpers.

Macro	Test Focus
parse_dn	Verifies RDN extraction and parent DN identification.
validate_dn_format	Ensures strings adhere to standard LDAP DN key=value pairing.
normalize_array_field	Checks conversion of single strings or nulls into JSON arrays.
ldap_timestamp_to_timestamp	Validates parsing of LDAP-specific UTC strings (e.g., 20240101120000Z).
extract_email_domain	Tests regex-based extraction of domains from email strings.

Sources: tests/test_macros.sql:4-15, tests/test_macros.sql:43-60, tests/test_macros.sql:102-117

Quality Protocols

The FlextDbtLdapProtocols define the interface for quality-related services, ensuring consistent implementation of completeness checks and anomaly detection.

The diagram illustrates the standardized methods required for any service implementing data quality or monitoring features.
Sources: src/flext_dbt_ldap/protocols.py:164-192, src/flext_dbt_ldap/protocols.py:221-246

Summary

Data Quality & Generic Tests in flext-dbt-ldap provide a robust safety net for LDAP-to-Warehouse pipelines. By integrating Python-level validation during extraction and deep SQL-level testing in the warehouse, the system ensures that complex LDAP relationships—such as group memberships and OU hierarchies—are maintained correctly throughout the transformation lifecycle.