Wiki Home
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The flext-dbt-ldif project provides dbt (data build tool) models and Python services specifically designed for the transformation and analytics of LDIF (LDAP Data Interchange Format) data. It acts as an integration bridge that allows enterprise LDAP data to be processed, validated, and transformed into structured SQL models for business intelligence and data warehousing purposes. Sources: README.md:1-10, pyproject.toml:25-35
The system is built on top of the FLEXT framework, utilizing flext-core and flext-meltano for infrastructure. It offers a complete workflow from raw LDIF parsing to dbt transformation, including data quality assessments and automated model generation. Sources: pyproject.toml:25-35, tests/unit/test_services.py:165-175
System Architecture
The project follows a layered architecture combining Python-based orchestration services with dbt-based SQL transformations.
High-Level Workflow
The following diagram illustrates the primary data flow from an LDIF file through the validation and transformation stages.
The workflow includes parsing the source file, assessing data quality (scoring), generating dbt models (staging and analytics), and executing the final SQL transformations. Sources: tests/unit/test_services.py:28-60, tests/unit/test_services.py:165-185
Transformation Layers
The dbt project is structured into four primary layers, as defined in the project configuration:
| Layer | Materialization | Schema | Tag |
|---|---|---|---|
| Base | table | ldif | ldif, enterprise |
| Staging | view | ldif_staging | staging, ldif |
| Intermediate | view | ldif_intermediate | intermediate, ldif |
| Marts | table | ldif_marts | marts, ldif, business |
Sources: dbt_project.yml:31-50
Core Components
Python Service Layer
The Python API, centered around FlextDbtLdif, delegates operations to specific mixins and core classes.
- Service Mixin (
FlextDbtLdifServiceMixin.Service): Manages the high-level workflow, including therun_complete_workflowmethod which orchestrates parsing, validation, and transformation. Sources: tests/unit/test_services.py:19-25, tests/unit/test_services.py:165-218 - Model Generator (
FlextDbtLdifCore.Core.ModelGenerator): Responsible for producing dbt staging and analytics models. The default staging model is namedstg_ldif_entries, and the default analytics model isanalytics_ldif_insights. Sources: tests/unit/test_core.py:21-45 - Analytics (
FlextDbtLdifCore.Core.Analytics): Analyzes entry patterns, such as counting total entries and identifying unique Distinguished Names (DNs). Sources: tests/unit/test_core.py:48-80
dbt Code Generation (Codegen)
The project includes macros to automate dbt configuration:
generate_source: Automatically inspects database schemas to generate source YAML files, including column names and data types. Sources: dbt_packages/codegen/macros/generate_source.sql:17-30generate_model_yaml: Generates documentation YAML for existing models, optionally pulling descriptions from upstream dependencies. Sources: dbt_packages/codegen/macros/generate_model_yaml.sql:35-45
Configuration and Dependencies
Environment and Requirements
The project is designed for Python 3.13+. It depends on several core FLEXT libraries and Pydantic for data validation. Sources: pyproject.toml:28-44
| Dependency | Purpose |
|---|---|
flext-core | Base infrastructure and lazy loading |
flext-ldif | LDIF processing logic |
flext-meltano | Data integration orchestration |
pydantic | Data models and configuration |
dbt Project Settings
The dbt configuration is managed via dbt_project.yml. Key variables include:
ldif_schema: Defaulted toraw.database_name: Defaulted toldif_analytics.profile:flext_ldif.
Sources: dbt_project.yml:6-10, dbt_project.yml:64-75
Development Guidelines
The project enforces strict engineering standards via the Universal Agent Law. These rules include:
- Zero-Tolerance: Always fix the root cause; no dead code or TODOs. Sources: AGENTS.md:15-25
- Fix-Forward-Only: Reverting shared work is forbidden; agents must coordinate through the committed task board. Sources: AGENTS.md:27-35
- Strict Typing: Use the most restrictive types possible;
Anyis forbidden without documentation. Sources: AGENTS.md:83-86
Quality is verified through canonical make verbs: check, test, val, and docs. Sources: README.md:38-40
Summary
flext-dbt-ldif provides a robust, typed framework for managing LDAP data within a modern data stack. By combining automated SQL model generation with Python-based data quality validation, it ensures that LDIF data is reliably transformed into actionable business insights. Sources: README.md:9-12, tests/unit/test_services.py:215-220