Data Transformation Rules
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Data Transformation Rules in the flext-dbt-ldap project define the logic and protocols for converting raw LDAP directory entries into structured data warehouse models. The system facilitates the transition from hierarchical LDAP attributes to relational dbt models, focusing on user dimensions, group dimensions, and membership facts. Sources: README.md, src/flext_dbt_ldap/protocols.py:1-15
The transformation process involves extracting data via LDAP queries, validating the quality of the retrieved entries, mapping attributes according to specific configuration rules, and executing dbt models to finalize the relational structure. Sources: src/flext_dbt_ldap/services/client.py:65-101, src/flext_dbt_ldap/_utilities/integration.py:23-40
Core Transformation Logic
The transformation logic is implemented through a combination of Python-based utilities and SQL-based dbt macros. The system categorizes LDAP entries into specific schemas (Users, Groups, Organizational Units) based on their objectClass attributes. Sources: src/flext_dbt_ldap/services/client.py:180-192
Transformation Workflow
The following diagram illustrates the flow from raw LDAP extraction to the final dbt transformation:
The workflow ensures that only data meeting a minimum quality threshold proceeds to the transformation stage. Sources: src/flext_dbt_ldap/services/client.py:65-101, src/flext_dbt_ldap/services/client.py:133-162
Classification Rules
Entries are matched to target tables based on their LDAP object classes. The system identifies standard LDAP classes to route data:
- Users:
inetOrgPerson,organizationalPerson,person. - Groups:
groupOfNames,groupOfUniqueNames. - Organizational Units:
organizationalUnit.
Sources: src/flext_dbt_ldap/services/client.py:183-189, src/flext_dbt_ldap/_utilities/integration.py:23-70
Transformation Components
1. Attribute Mapping
The FlextDbtLdapClientMixin handles the translation of LDAP-specific attribute names to dbt-compatible column names. If an attribute mapping is not explicitly defined in settings, the system defaults to the original LDAP attribute name. Sources: src/flext_dbt_ldap/services/client.py:164-178
| Component | Description | Source File |
|---|---|---|
UserDimension | Logic to transform user entries into a relational user dimension. | src/flext_dbt_ldap/_utilities/integration.py:72-80 |
GroupDimension | Logic to transform group entries into a relational group dimension. | src/flext_dbt_ldap/_utilities/integration.py:23-31 |
MembershipFact | Logic to extract group/user relationships into a fact table. | src/flext_dbt_ldap/_utilities/integration.py:33-58 |
2. SQL Macros
The project uses dbt macros to handle complex string parsing and data normalization directly in SQL. Sources: tests/test_macros.sql:1-5
parse_dn: Breaks down a Distinguished Name (DN) into its components (RDN, parent DN). Sources: tests/test_macros.sql:4-15ldap_timestamp_to_timestamp: Converts LDAP generalized time strings (e.g.,20240101120000Z) into standard SQL timestamps. Sources: tests/test_macros.sql:47-60normalize_array_field: Ensures multi-valued LDAP attributes are consistently represented as JSON arrays. Sources: tests/test_macros.sql:32-45
Transformation Configuration
Transformation behavior is governed by TransformationConfig and TransformationRule models. These configurations allow developers to define specific expressions for column-level changes and apply filters before the transformation occurs. Sources: src/flext_dbt_ldap/_models/schema.py:36-54
class TransformationConfig(m.Value):
"""Transformation configuration."""
source_table: str
transformations: t.StrMapping # Column transformation expressions
filters: t.StrSequence # Filter expressions
Sources: src/flext_dbt_ldap/_models/schema.py:36-47
Model Naming Conventions
The system uses standardized naming for the resulting dbt models, categorized into staging (stg), dimension (dim), and fact (fact) layers. Sources: src/flext_dbt_ldap/_constants/transformation.py:18-24
| Model Constant | Default Table Name |
|---|---|
STG_USERS | stg_users |
DIM_USERS | dim_users |
STG_GROUPS | stg_groups |
DIM_GROUPS | dim_groups |
FACT_MEMBERSHIPS | fact_memberships |
Sources: src/flext_dbt_ldap/_constants/transformation.py:18-24
Summary
Data Transformation Rules in flext-dbt-ldap provide a robust framework for normalizing complex LDAP structures. By combining Pythonic attribute mapping with SQL macros for specialized LDAP data types (like DNs and timestamps), the system ensures that identity data is reliably transformed into a high-quality relational format suitable for enterprise analytics. Sources: src/flext_dbt_ldap/protocols.py:84-114, src/flext_dbt_ldap/services/client.py:86-101