Project Overview
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldap project provides dbt (data build tool) models and Python service integrations for transforming LDAP (Lightweight Directory Access Protocol) data within a data warehouse environment. It serves as an integration bridge that extracts raw directory data—typically users, groups, and organizational units—and applies structured SQL transformations to create business-ready dimensions and fact tables.
The system is built on the FLEXT framework, leveraging flext-core, flext-ldap, and flext-meltano to handle the end-to-end pipeline from directory extraction to warehouse persistence and data quality validation. It supports both full and incremental synchronization using LDAP modifyTimestamp bookmarks.
Sources: README.md:1-12, pyproject.toml:25-30, src/flext_dbt_ldap/services/sync.py:76-85
Core Architecture
The architecture consists of a service-oriented Python layer that orchestrates data movement and a SQL layer comprised of dbt models.
System Components
- Service Facades: Python classes like
FlextDbtLdapSyncMixinandFlextDbtLdapClientMixinmanage the execution logic, including LDAP searching, data validation, and dbt command invocation. - dbt Model Layers: The project organizes SQL transformations into staging, intermediate, and marts layers, allowing for progressive refinement of raw LDAP entries into analytics-ready tables.
- Configuration Management: Pydantic-based models define dbt project settings, source freshness requirements, and LDAP-to-DBT attribute mappings.
Sources: src/flext_dbt_ldap/services/sync.py:16, src/flext_dbt_ldap/services/client.py:18, dbt_project.yml:18-35, src/flext_dbt_ldap/_models/configuration.py:16
Data Flow Transformation Pipeline
The following diagram illustrates the sequence from raw LDAP directory extraction to the final data mart production.
The data flow starts with entry extraction, followed by attribute mapping and quality gates, before entering the multi-stage dbt transformation layers in the warehouse.
Sources: src/flext_dbt_ldap/services/client.py:68-100, dbt_project.yml:22-35
Service Operations
The service layer provides high-level methods to manage the lifecycle of directory data synchronization.
Data Extraction and Validation
The FlextDbtLdapClientMixin handles the extraction of entries from the directory service. It includes a quality gate that calculates a quality_score based on the presence of required attributes defined in the configuration.
Sources: src/flext_dbt_ldap/services/client.py:40-66, src/flext_dbt_ldap/services/client.py:126-155
Synchronization Strategies
The FlextDbtLdapSyncMixin provides methods for synchronizing specific directory objects. It supports incremental logic by building LDAP filters that target entries modified since the last successful sync.
| Component | Sync Method | Key Logic |
|---|---|---|
| Users | sync_users_to_warehouse | Filters by c.DbtLdap.FILTER_USER and maps to STG_USERS/DIM_USERS. |
| Groups | sync_groups_to_warehouse | Uses FILTER_GROUP and persists state via JSON bookmarks. |
| Memberships | sync_memberships_to_warehouse | Direct pipeline to FACT_MEMBERSHIPS. |
Sources: src/flext_dbt_ldap/services/sync.py:76-158, src/flext_dbt_ldap/services/sync.py:175-181
Modeling and Data Structure
The project transforms hierarchical directory data into relational structures across defined dbt schemas.
dbt Schema Organization
| Layer | Schema | Materialization | Description |
|---|---|---|---|
| Base | ldap | table | Foundational LDAP entity models. |
| Staging | ldap_staging | view | Initial cleanup and renaming of raw fields. |
| Intermediate | ldap_intermediate | view | Business logic and relationship mapping. |
| Marts | ldap_marts | table | Business-ready entities (Users, Groups, Hierarchies). |
Sources: dbt_project.yml:21-35
Transformation Sequence
The transformation process is represented as a sequence of synchronous calls through the LdapIntegration and Modeling protocols.
The sequence demonstrates the orchestration between the LDAP client, the internal validation logic, and the final dbt execution.
Sources: src/flext_dbt_ldap/protocols.py:64-90, src/flext_dbt_ldap/services/client.py:68-100
Configuration and Metadata
Configurations are defined via Pydantic models in _models/configuration.py. These models control the environment paths and execution targets for dbt.
Key Configuration Fields
- DbtProjectConfig: Defines paths for models, seeds, macros, and snapshots. Default paths include
models,tests, andmacros. - DbtConfig: Specifies the
targetdatabase (defaulting to the configured target) and theprofiles_dir. - LdapAttributeMapping: A mapping within
FlextDbtLdapSettings(used by the client mixin) that dictates how directory attributes translate to warehouse columns.
Sources: src/flext_dbt_ldap/_models/configuration.py:20-56, src/flext_dbt_ldap/services/client.py:161-177
Data Quality and Results
The system produces detailed metrics for every pipeline run, enabling monitoring and automated failure handling.
Validation Metrics
The ValidationMetrics model tracks:
total_entries: Count of entries evaluated.quality_score: Float between 0.0 and 1.0 representing data integrity.validation_passed: Boolean flag determined by comparing score againstmin_quality_threshold.
Sources: src/flext_dbt_ldap/_models/results.py:16-30, src/flext_dbt_ldap/services/client.py:126-155
Performance Analysis
For large-scale directory synchronizations, the PerformanceAnalysis model tracks execution time, rows processed, and peak memory usage to provide tuning recommendations.
Sources: src/flext_dbt_ldap/_models/results.py:61-75
Conclusion
flext-dbt-ldap provides a robust framework for integrating enterprise directory services with modern data warehouses. By combining Python-based orchestration with dbt's transformation capabilities, it ensures that LDAP data is not only synchronized but also validated and structured for complex organizational analytics.