Examples & Use Cases

Relevant source files

The following files were used as context for generating this wiki page:

• examples/README.md
• src/flext_dbt_ldap/services/sync.py
• src/flext_dbt_ldap/services/client.py
• tests/test_macros.sql
• tests/generic/test_data_quality.sql
• tests/e2e/sql/01-create-raw-tables.sql

The flext-dbt-ldap project provides a robust framework for synchronizing LDAP directory data with data warehouses and performing complex SQL-based transformations using dbt. This module bridges the gap between hierarchical LDAP structures and relational data models, enabling enterprise-grade identity analytics and reporting.

The primary use cases involve extracting directory entries (Users, Groups, Organizational Units), persisting them into raw warehouse tables, and applying dbt models to create cleaned dimensions and facts. The system supports both full and incremental synchronization logic to maintain data freshness efficiently.

Sources: src/flext_dbt_ldap/services/sync.py, src/flext_dbt_ldap/services/client.py

Warehouse Synchronization Use Cases

Full Data Warehouse Sync

A primary use case is the complete synchronization of an LDAP directory to a target data warehouse. This process orchestrates the extraction and loading of multiple LDAP components including users, groups, and memberships into a unified relational schema.

The synchronization workflow is managed by the run_full_data_warehouse_sync method, which coordinates individual sync tasks and returns an aggregated SyncResult.

Sources: src/flext_dbt_ldap/services/sync.py:53-76

Incremental Synchronization

To optimize performance, the system supports incremental updates using LDAP modifyTimestamp bookmarks. This use case allows the system to only fetch and transform records that have changed since the last successful execution.

Component	Sync Key	Logic
Users	users	Filter: (&(objectClass=person)(modifyTimestamp>=TIMESTAMP))
Groups	groups	Filter: (&(objectClass=group)(modifyTimestamp>=TIMESTAMP))

The service maintains state in a local file (e.g., .flext_dbt_ldap_sync_state.json) to track the high-water mark for each entity type.

Sources: src/flext_dbt_ldap/services/sync.py:78-105, src/flext_dbt_ldap/services/sync.py:175-181

Data Transformation Examples

LDAP-to-Relational Pipeline

The project implements a standard ETL/ELT pipeline where hierarchical LDAP entries are flattened into relational formats suitable for dbt models.

Raw Table Schema Examples

For end-to-end (e2e) testing and implementation, the project defines standard SQL schemas for raw LDAP data. These tables host the data extracted from LDAP before dbt transformations are applied.

-- Example Raw Users Table Structure
CREATE TABLE IF NOT EXISTS raw_ldap.users (
    dn TEXT PRIMARY KEY,
    uid TEXT,
    cn TEXT,
    mail TEXT,
    object_class TEXT[],
    member_of TEXT[],
    create_timestamp TIMESTAMP WITH TIME ZONE,
    modify_timestamp TIMESTAMP WITH TIME ZONE
);

Sources: tests/e2e/sql/01-create-raw-tables.sql:4-29

Data Quality & Validation Use Cases

Custom SQL Macro Tests

The project includes specialized dbt macros for validating LDAP-specific data formats, such as Distinguished Names (DNs) and generalized timestamps.

Macro Name	Purpose	Example Validation
validate_dn_format	Ensures DN strings follow X.500 standards	uid=jdoe,ou=users,dc=com (Valid)
ldap_timestamp_to_timestamp	Converts 20240101120000Z to SQL timestamp	Year=2024, Hour=12
extract_ou_from_dn	Parses the OU component from a full DN	ou=users from full path

Sources: tests/test_macros.sql:4-46

Generic Data Quality Audits

Automated tests are used to ensure the integrity of the synchronized identity data, such as checking for unique UIDs and valid organizational hierarchies.

Summary of Usage

The flext-dbt-ldap module is designed for developers and data engineers who need to:

1. Extract: Pull directory data using standard LDAP filters and base DNs.
2. Synchronize: Maintain stateful, incremental copies of directory data in a warehouse.
3. Transform: Use dbt macros and models to normalize complex attributes (like member arrays and timestamps).
4. Validate: Apply specialized data quality tests to ensure identity data remains consistent across systems.

Sources: examples/README.md, src/flext_dbt_ldap/services/sync.py, src/flext_dbt_ldap/services/client.py