Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldap module provides a structured framework for extracting data from LDAP directories and transforming it into a data warehouse using dbt (data build tool). It acts as an integration layer that bridge the gap between hierarchical LDAP structures and relational data warehouse models, utilizing the FLEXT ecosystem's core capabilities for synchronization, data validation, and modeling.
The system is built upon a mixin-based architecture where services inherit capabilities for LDAP client interactions and data warehouse synchronization workflows. This allows for modular execution of data pipelines, ranging from simple attribute extraction to full incremental synchronization of users, groups, and organizational memberships.
Sources: src/flext_dbt_ldap/services/client.py:12-20, src/flext_dbt_ldap/services/sync.py:14-20, README.md:9-12
Core Components and Service Layer
The architecture is divided into specialized mixins and service bases that define the operational logic for the LDAP-to-dbt pipeline.
Service Mixins
- FlextDbtLdapClientMixin: Manages the low-level interactions with the LDAP server via the
flext-ldapAPI. It handles entry extraction, attribute mapping, and initial data preparation for dbt ingestion. - FlextDbtLdapSyncMixin: Orchestrates high-level synchronization workflows, including full warehouse syncs and incremental updates based on stateful bookmarks.
Logic Flow Diagram
This diagram illustrates the flow of data from the LDAP source through the extraction and transformation stages.
Data Synchronization and State Management
The FlextDbtLdapSyncMixin manages persistence of the synchronization state to support incremental loads. It uses a JSON-based state file containing UTC bookmarks to filter LDAP entries modified after the last successful run.
Sync Components
| Component | Description | Search Filter Used |
|---|---|---|
| Users | Syncs user accounts, UIDs, and attributes. | c.DbtLdap.FILTER_USER |
| Groups | Syncs LDAP groups and common names. | c.DbtLdap.FILTER_GROUP |
| Memberships | Maps relationships between users and groups. | c.DbtLdap.FILTER_MEMBERSHIP |
Incremental Logic
The system calculates incremental filters by appending a modifyTimestamp check to base filters.
def _build_incremental_filter(self, base_filter: str, bookmark: str | None) -> str:
if bookmark is None:
return base_filter
return f"(&{base_filter}(modifyTimestamp>={bookmark}))"
Sources: src/flext_dbt_ldap/services/sync.py:151-180, src/flext_dbt_ldap/services/sync.py:195-198
Data Quality and Validation
The architecture enforces strict data quality thresholds before allowing data to proceed to the dbt transformation layer. This is handled by the validate_ldap_data method, which evaluates entries against a configurable min_quality_threshold.
Validation Metrics
The system tracks several key metrics during the validation phase:
- Total Entries: The count of entries extracted from LDAP.
- Valid DNs: Entries containing a valid Distinguished Name attribute.
- Valid Entries: Entries that contain all
required_attributesdefined in settings. - Quality Score: A ratio of valid entries to total entries.
Relational Schema Mapping
Data extracted from LDAP is prepared for the warehouse by mapping hierarchical attributes to a relational format. This involves categorizing entries by objectClass (e.g., Users vs. Groups) and applying transformation rules.
Raw Table Structures
The following SQL structures represent the landing zone for raw LDAP data before dbt processing:
| Table | Primary Key | Key Attributes |
|---|---|---|
raw_ldap.users | dn | uid, cn, mail, employee_number, modify_timestamp |
raw_ldap.groups | dn | cn, description, member, object_class |
raw_ldap.organizational_units | dn | ou, description, business_category |
Sources: tests/e2e/sql/01-create-raw-tables.sql:4-72, src/flext_dbt_ldap/services/client.py:209-224
Configuration and Protocols
The system relies on Pydantic-based models for configuring the dbt execution environment and the LDAP query parameters.
Configuration Hierarchy
- DbtConfig: Defines
target,profiles_dir, andproject_dir. - DbtProjectConfig: Contains metadata like
model_paths,macro_paths, andtarget_schema. - LdapQuery: Specifies
base_dn,filter_str, and searchscope.
Sources: src/flext_dbt_ldap/_models/configuration.py:15-46, src/flext_dbt_ldap/_models/schema.py:76-88
Protocols
The FlextDbtLdapProtocols class defines the service contracts for various domains:
- Dbt: Operations for compiling, running, and testing dbt models.
- LdapIntegration: Methods for extracting, transforming, and syncing LDAP data.
- Modeling: Specialized logic for creating dimensions (User/Group) and hierarchies.
- Quality: Checks for data completeness and schema compliance.
Sources: src/flext_dbt_ldap/protocols.py:28-105
Summary
The flext-dbt-ldap architecture provides a robust, stateful pipeline for integrating LDAP identity data into a modern data stack. By combining mixin-based service logic with structured configuration models and comprehensive validation protocols, it ensures that LDAP data is accurately transformed into actionable relational models while maintaining data integrity through quality gates and incremental synchronization.
Sources: src/flext_dbt_ldap/services/sync.py:75-84, src/flext_dbt_ldap/services/client.py:121-125