E2E Testing Workflow

Relevant source files

The following files were used as context for generating this wiki page:

• tests/e2e/conftest.py
• tests/e2e/sql/02-insert-test-data.sql
• tests/test_macros.sql
• tests/generic/test_data_quality.sql
• pyproject.toml
• src/flext_dbt_ldap/services/sync.py
• src/flext_dbt_ldap/services/client.py

The E2E (End-to-End) Testing Workflow in flext-dbt-ldap ensures the integrity of the full data pipeline, from LDAP entry extraction to data warehouse transformation. This workflow validates that dbt models correctly process raw LDAP data into refined dimensions and facts, maintaining data quality across the entire lifecycle.

The testing suite is integrated into the project's quality gates, utilizing pytest for orchestration and SQL-based tests for data validation within the warehouse environment. The workflow covers initial data seeding, macro validation, and generic data quality checks across multiple entities including users, groups, and organizational units.
Sources: pyproject.toml:292-303, tests/e2e/sql/02-insert-test-data.sql

Testing Architecture

The workflow follows a sequential process where test data is injected into a raw landing zone, processed via dbt models, and then validated using specialized SQL test files.

This diagram illustrates the high-level flow from initial data seeding through model execution to the final validation steps.
Sources: tests/e2e/sql/02-insert-test-data.sql, tests/test_macros.sql, tests/generic/test_data_quality.sql

Test Data Initialization

E2E tests begin by populating the raw_ldap schema with representative LDAP objects. This includes diverse entry types such as active employees, service accounts, and complex group memberships.

Target Schemas and Tables

Table Name	Description	Key Attributes
raw_ldap.users	Raw user entry landing table	dn, uid, mail, member_of, object_class
raw_ldap.groups	Raw group entry landing table	dn, cn, member, member_uid
raw_ldap.organizational_units	Raw OU structure table	dn, ou, business_category

Sources: tests/e2e/sql/02-insert-test-data.sql:4-13, tests/e2e/sql/02-insert-test-data.sql:56-61, tests/e2e/sql/02-insert-test-data.sql:103-108

Macro Validation

The project utilizes specialized Jinja/SQL macros to handle LDAP-specific logic such as DN parsing and timestamp conversion. The test_macros.sql file contains unit-style tests for these components.

Core Macros Under Test

• parse_dn: Validates the extraction of RDNs, attribute types, and parent paths from a Distinguished Name string.
• ldap_timestamp_to_timestamp: Ensures standard UTC conversion for LDAP GeneralizedTime formats (e.g., 20240101120000Z).
• normalize_array_field: Validates that both JSON arrays and single-value strings are correctly coerced into standard array formats.
• extract_email_domain: Tests the regex or string logic used to isolate domains from LDAP mail attributes.

Sources: tests/test_macros.sql:4-14, tests/test_macros.sql:43-55, tests/test_macros.sql:28-41, tests/test_macros.sql:75-90

Data Quality (DQ) Gates

Generic tests are applied to the resulting dbt models (stg_users, dim_users, fact_memberships, etc.) to ensure business logic consistency.

The sequence diagram shows the interaction between the test orchestrator and the warehouse during the quality gate phase.
Sources: src/flext_dbt_ldap/services/client.py:90-112, tests/generic/test_data_quality.sql

Automated Quality Checks

• Unique UIDs: Ensures no duplicate uid values exist in stg_users.
• DN Integrity: Uses validate_dn_format to check all dn fields in staging.
• Referential Integrity: Verifies that every record in fact_memberships successfully joins to both dim_users and dim_groups.
• Hierarchy Validation: A recursive CTE test ensures that organizational units form a valid tree under the ldap_base_dn without infinite loops.
• Temporal Logic: Validates that modified_at timestamps are not set in the future relative to current_timestamp.

Sources: tests/generic/test_data_quality.sql:4-10, tests/generic/test_data_quality.sql:17-21, tests/generic/test_data_quality.sql:24-42, tests/generic/test_data_quality.sql:48-54

Execution Service Integration

The FlextDbtLdapClientMixin and FlextDbtLdapSyncMixin provide the runtime methods that execute these workflows programmatically. The validate_warehouse_data_quality method triggers the execution of selected models and returns ValidationMetrics.

# From src/flext_dbt_ldap/services/sync.py
def validate_warehouse_data_quality(
    self,
    model_names: t.StrSequence | None = None,
) -> p.Result[m.DbtLdap.ValidationMetrics]:
    """Validate data quality in the warehouse."""
    try:
        run_result = self._run_selected_models(model_names)
        if run_result.failure:
            return r[m.DbtLdap.ValidationMetrics].fail(
                run_result.error or "Data quality validation failed",
            )
        return r[m.DbtLdap.ValidationMetrics].ok(
            m.DbtLdap.ValidationMetrics(validation_passed=True),
        )

Sources: src/flext_dbt_ldap/services/sync.py:165-181, src/flext_dbt_ldap/_models/results.py:17-31

The E2E Testing Workflow is critical for ensuring that LDAP data transformations remain accurate as schemas and business rules evolve. By combining static SQL data seeding with dynamic dbt testing, the project maintains a high quality score and reliable synchronization between identity providers and the data warehouse.