LDIF Transformation Flow
Relevant source files
The following files were used as context for generating this wiki page:
The LDIF Transformation Flow is a structured orchestration process within the flext-dbt-ldif library designed to ingest Lightweight Data Interchange Format (LDIF) data, validate its quality, and generate the necessary dbt (data build tool) models for downstream analytics. It serves as an integration layer between raw LDAP directory exports and enterprise data warehouse environments.
The flow is managed primarily by the FlextDbtLdifServiceMixin.Service and exposed through a public facade, FlextDbtLdif. It automates the lifecycle from raw file parsing to the execution of dbt transformations, ensuring that LDAP data is materialized into structured tables and views suitable for business intelligence.
Sources: README.md:9-12, src/flext_dbt_ldif/services/service.py:19-25, src/flext_dbt_ldif/api.py:27-33
Architecture and Orchestration
The system uses a mixin-based architecture where a central Service class orchestrates specialized components: the FlextDbtLdifClient for low-level data operations and the FlextDbtLdifUnifiedService for dbt artifact generation.
Workflow Orchestration Diagram
The following flowchart illustrates the end-to-end execution path when run_complete_workflow is invoked.
Data Processing Stages
The transformation flow is divided into three primary functional stages: Ingestion/Validation, Model Generation, and Transformation.
1. Ingestion and Validation
The service first parses the raw LDIF file using the client. It converts the entries into a standard JsonMapping sequence and performs a quality assessment. This assessment results in a quality_score and a validation_status which determine if the data is fit for transformation.
Sources: src/flext_dbt_ldif/services/service.py:76-96, src/flext_dbt_ldif/models.py:59-64
2. dbt Model Generation
If enabled, the UnifiedService generates two tiers of dbt models:
- Staging Models: Materialized as views in the
ldif_stagingschema. These models perform initial selection from raw source tables. - Analytics Models: Materialized as tables in the
ldif_martsschema. These models depend on the staging layer and provide business-ready insights.
3. Transformation Execution
The final stage involves executing dbt commands to materialize the data. The service tracks the number of records processed and the status of the dbt run (e.g., "success").
Sources: src/flext_dbt_ldif/services/service.py:133-146, src/flext_dbt_ldif/models.py:66-72
Data Models and Schema Configuration
The project configures dbt through dbt_project.yml to ensure data is partitioned correctly across schemas.
| Component | Materialization | Schema | Tags |
|---|---|---|---|
| Staging | View | ldif_staging | ["staging", "ldif"] |
| Intermediate | View | ldif_intermediate | ["intermediate", "ldif"] |
| Marts (Business) | Table | ldif_marts | ["marts", "ldif", "business"] |
| Sources: dbt_project.yml:35-49 |
Key Data Structures
The internal logic relies on Pydantic-based models for state management between stages.
| Model Class | Description | Key Fields |
|---|---|---|
DbtModel | Represents a single SQL artifact. | name, sql_content, dependencies |
LdifValidationResult | Quality metrics for the input file. | total_entries, quality_score |
WorkflowResult | Summary of the end-to-end execution. | workflow_status, transformation_status |
| Sources: src/flext_dbt_ldif/models.py:17-57, src/flext_dbt_ldif/models.py:79-92 |
Implementation Details
The run_complete_workflow method in FlextDbtLdifServiceMixin.Service demonstrates the integration of these components:
# From src/flext_dbt_ldif/services/service.py:99-148
def run_complete_workflow(
self,
ldif_file: Path | str,
*,
generate_models: bool = True,
run_transformations: bool = True,
model_names: t.StrSequence | None = None,
) -> p.Result[m.DbtLdif.WorkflowResult]:
# 1. Parse and Validate
parse_result = self.client.parse_ldif_file(ldif_file)
entries = t.json_mapping_sequence_adapter().validate_python(parse_result.value)
validation = self.client.validate_ldif_data(entries)
# 2. Generate dbt Artifacts
if generate_models:
model_result = self.generate_and_write_models(entries)
# 3. Transform
if run_transformations:
transform = self.client.transform_with_dbt(transform_payload, model_names)
Sources: src/flext_dbt_ldif/services/service.py:99-148
Conclusion
The LDIF Transformation Flow provides a robust, automated pipeline for migrating directory service data into a structured analytical environment. By combining Pydantic-driven validation with dynamic dbt model generation, the system ensures that LDIF data is not only processed but verified for quality before entering the data warehouse.