Module Integration Points
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldap module serves as a specialized integration layer within the FLEXT ecosystem, designed to facilitate the extraction of LDAP directory data and its subsequent transformation into data warehouse-ready formats using dbt (data build tool). It operates at the intersection of LDAP directory services and modern data engineering pipelines, providing typed interfaces and automated synchronization workflows.
The module's architecture is built upon a Method Resolution Order (MRO) chain that inherits capabilities from flext-core, flext-ldap, and flext-meltano. This allows it to leverage standardized CLI patterns, LDAP connection management, and Singer-compatible data extraction protocols while providing specific dbt modeling logic for LDAP entities such as users, groups, and organizational units.
Sources: README.md:9-13, src/flext_dbt_ldap/protocols.py:16-18, pyproject.toml:31-36
Architecture and Protocol Inheritance
The module defines a rigorous set of protocols that extend the base functionality of the FLEXT ecosystem. The primary protocol, FlextDbtLdapProtocols, inherits from both flext_meltano and FlextLdapProtocols, ensuring that any implementation supports both LDAP operations and ELT pipeline standards.
Integration Hierarchy
The following diagram illustrates how the dbt-ldap module integrates with the broader FLEXT service hierarchy.
Service Integration Points
The module facilitates integration through three primary service mixins and base classes: FlextDbtLdapClientMixin, FlextDbtLdapSyncMixin, and the core FlextDbtLdapServiceBase. These components handle the lifecycle of data from the LDAP source to the final dbt transformation.
LDAP Extraction Integration
The FlextDbtLdapClientMixin provides the interface to the flext-ldap API. It handles the instantiation of the LDAP client using project-specific settings and manages the execution of searches against the directory.
create_ldap_api: Initializes aFlextLdapinstance by cloning global settings and injecting credentials (bind DN and password) from the dbt-ldap configuration.extract_ldap_entries: Wraps the synchronous LDAP search operation, providing logging and result handling specifically for dbt-oriented data flows._prepare_ldap_data_for_dbt: A critical internal integration point that maps raw LDAP entries into a structured dictionary format where keys correspond to dbt target tables.
Sources: src/flext_dbt_ldap/services/client.py:17-40, src/flext_dbt_ldap/services/client.py:177-195
Data Warehouse Sync Workflow
The FlextDbtLdapSyncMixin manages the end-to-end synchronization process. This involves maintaining state via "bookmarks" to support incremental loads.
| Integration Component | Function | Persistence Method |
|---|---|---|
| Incremental Sync | Filters LDAP queries based on modifyTimestamp. | State file: .flext_dbt_ldap_sync_state.json |
| User Sync | Synchronizes LDAP user objects to staging and dimension tables. | Target Models: stg_users, dim_users |
| Group Sync | Synchronizes LDAP group objects to staging and dimension tables. | Target Models: stg_groups, dim_groups |
| Membership Sync | Maps relationships between users and groups. | Target Model: fact_memberships |
Sources: src/flext_dbt_ldap/services/sync.py:53-150, src/flext_dbt_ldap/services/sync.py:206-208
Data Flow and Transformation
The integration process follows a defined pipeline from the external LDAP directory through an internal validation layer to the dbt execution engine.
Pipeline Execution Flow
The following sequence diagram shows the data movement through the run_full_pipeline method.
Transformation Utilities
The module uses the FlextDbtLdapUtilitiesIntegration class to perform typed transformations. This utility bridges the gap between raw Ldif.Entry objects and the typed models used in the dbt pipeline.
transform_users: Converts LDAP user entries intoUserDimensionobjects.transform_memberships: Extracts membership facts from both group attributes (e.g.,member) and user attributes (e.g.,memberOf)._extract_group_memberships: Iterates throughMEMBERSHIP_ATTRIBUTESto create direct relationship facts between user DNs and group DNs.
Sources: src/flext_dbt_ldap/_utilities/integration.py:32-68, src/flext_dbt_ldap/_utilities/integration.py:84-100
Configuration Integration
The module integrates with dbt via specific configuration models that define how the dbt project should be structured and executed.
dbt Project Configuration
The DbtProjectConfig model defines the paths and schemas that the integration uses to find and build resources.
| Parameter | Default Source | Description |
|---|---|---|
model_paths | c.DbtLdap.DEFAULT_MODEL_PATHS | Directories containing dbt models. |
macro_paths | c.DbtLdap.DEFAULT_MACRO_PATHS | Directories containing dbt macros for LDAP logic. |
target_schema | c.DbtLdap.DEFAULT_DBT_SCHEMA | The default database schema for resulting relations. |
materialized | c.Meltano.DBT_MATERIALIZATION_TABLE | Default strategy for creating warehouse objects. |
Sources: src/flext_dbt_ldap/_models/configuration.py:23-53
Conclusion
The flext-dbt-ldap module provides a comprehensive integration framework that automates the transition of identity data from directory services to analytical environments. By leveraging a structured pipeline of extraction, validation, and dbt-driven transformation, it ensures that LDAP data is consistent, high-quality, and properly related within a data warehouse schema.