Execution Flow

The execution flow of the flext-dbt-ldap integration governs the lifecycle of data as it moves from an LDAP directory through a validation and transformation pipeline into a data warehouse via dbt. This process is primarily orchestrated by service mixins that handle extraction, quality gates, and synchronization states.

The system is designed as a structured pipeline that ensures data integrity through schema matching and quality scoring before executing dbt models to materialize final dimensions and facts.

Sources: README.md:1-15, src/flext_dbt_ldap/services/client.py:80-110

Pipeline Orchestration

The core execution logic is encapsulated in the run_full_pipeline method. This method acts as a synchronous controller that sequences the extraction, validation, and transformation phases.

Core Pipeline Phases

1. Extraction: Fetches raw entries from LDAP using the FlextLdap client.
2. Validation: Evaluates entries against required attributes and quality thresholds.
3. Transformation: Maps LDAP attributes to dbt-compatible formats and triggers dbt model execution.

The following diagram illustrates the high-level sequential flow of the pipeline:

Data Synchronization and State Management

Synchronization workflows extend the basic pipeline to support incremental updates and targeted syncing of specific LDAP objects such as users, groups, and memberships.

Incremental Logic

Synchronization utilizes bookmarks based on modifyTimestamp to filter LDAP queries. The FlextDbtLdapSyncMixin manages these bookmarks in a local JSON state file.

Sources: src/flext_dbt_ldap/services/sync.py:75-155, src/flext_dbt_ldap/services/sync.py:175-195

Sequence of Full Sync

A full warehouse sync orchestrates three distinct sub-syncs in a specific order to maintain referential integrity.

Data Transformation and Mapping

Before dbt models are executed, raw LDAP data undergoes attribute mapping and schema matching. This ensures that multi-valued LDAP attributes are flattened and entries are categorized into correct warehouse tables (e.g., users, groups, org units).

Mapping Logic

The _map_entry_attributes function translates LDAP-specific keys into the names expected by dbt models based on the ldap_attribute_mapping configuration. It also handles the Distinguished Name (DN) as a primary identifier.

Schema Matching

Entries are filtered into schemas based on their objectClass values:

• Users: Matched against person, inetOrgPerson, etc.
• Groups: Matched against groupOfNames, posixGroup, etc.
• Org Units: Matched against organizationalUnit.

Sources: src/flext_dbt_ldap/services/client.py:166-215, src/flext_dbt_ldap/_utilities/integration.py:25-50

Quality Gates and Validation

The execution flow includes a mandatory quality gate that calculates a quality_score based on the presence of required_attributes and valid DNs.

# Logic for calculating quality metrics
quality_score = valid_entries / total_entries if total_entries > 0 else 0.0
metrics = m.DbtLdap.ValidationMetrics(
    total_entries=total_entries,
    valid_dns=valid_dns,
    valid_entries=valid_entries,
    quality_score=round(quality_score, 3),
    validation_passed=quality_score >= self.settings.min_quality_threshold,
)

Sources: src/flext_dbt_ldap/services/client.py:140-155, src/flext_dbt_ldap/_models/results.py:14-30

Conclusion

The flext-dbt-ldap execution flow provides a robust bridge between directory services and data warehouses. By combining stateful incremental synchronization with configurable attribute mapping and strict quality validation, it ensures that LDAP data is accurately reflected in dbt-managed dimensions and facts.