System Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldif system is an integration framework designed to transform LDAP Data Interchange Format (LDIF) data into structured analytics using dbt (data build tool). It provides a bridge between raw LDIF file parsing and structured SQL-based modeling, allowing for automated generation of staging, intermediate, and marts layers within a data warehouse.
The architecture is built upon the flext-core and flext-ldif foundations, utilizing a service-oriented approach to manage the lifecycle of LDIF data processing, validation, and dbt model generation.
Sources: README.md:10-15, pyproject.toml:23-29
Core Service Layer
The system's logic is primarily orchestrated through the FlextDbtLdifServiceMixin.Service class. This component acts as the unified controller for high-level workflows, delegating specific tasks to specialized clients and generators.
Workflow Orchestration
The primary entry point for processing is the run_complete_workflow method, which executes a multi-stage pipeline:
- Parsing and Validation: Ingests raw LDIF files and assesses data quality.
- Model Generation: Dynamically creates dbt SQL models and YAML configuration files based on the LDIF schema.
- Transformation: Triggers dbt runs to materialize data into the warehouse.
The following sequence diagram illustrates the typical workflow execution:
dbt Transformation Architecture
The project implements a standard layered dbt architecture to manage the lifecycle of data from raw ingestion to business-ready reports. Each layer is configured with specific materialization strategies and target schemas.
Model Layer Definitions
| Layer | Materialization | Schema | Description |
|---|---|---|---|
| Base | Table | ldif | Foundational LDIF entity models. |
| Staging | View | ldif_staging | Raw LDIF data cleaning and casting. |
| Intermediate | View | ldif_intermediate | Complex business logic and joins. |
| Marts | Table | ldif_marts | Business-ready entities (Dimension/Fact tables). |
The following flow diagram depicts the data movement through these layers:
Sources: dbt_project.yml:33-55
Schema and Profile Configuration
The project is configured to use the flext_ldif profile and targets a database defined by the database_name variable (default: ldif_analytics). Schema names are dynamically determined using dbt variables like ldif_schema, staging_schema, and marts_schema.
Sources: dbt_project.yml:10-15, dbt_project.yml:71-82
Automated Model Generation (Codegen)
A critical component of the architecture is the use of the codegen package to automate the creation of dbt artifacts. This reduces manual overhead and ensures consistency between the LDIF source and the dbt models.
YAML Generation Logic
The system includes macros for generating dbt sources.yml and models.yml files by inspecting the database adapter's schema.
generate_source: Scans a schema and generates a YAML definition including table names, columns, and optionally data types and descriptions.generate_model_yaml: Generates YAML configuration for existing dbt models, facilitating documentation and test definition.
-- Example logic for generating source YAML
{% macro generate_source(schema_name, database_name=target.database, ...) %}
{% set tables = codegen.get_tables_in_schema(schema_name, database_name) %}
-- Logic to append table and column definitions to YAML list
{% endmacro %}
Sources: dbt_packages/codegen/macros/generate_source.sql:15-88, dbt_packages/codegen/macros/generate_model_yaml.sql:24-65
CTE Generation
To maintain high SQL quality and follow dbt best practices, the system provides a generate_model_import_ctes macro. This tool uses regular expressions to parse existing SQL files and refactor direct table/source references into modular Common Table Expressions (CTEs) at the top of the file.
Sources: dbt_packages/codegen/macros/generate_model_import_ctes.sql:5-242
Data Quality and Validation
Data integrity is enforced at the service level before transformation occurs. The LdifValidationResult model tracks:
- Total Entries: The number of records parsed from the source.
- Quality Score: A float representation of data health (e.g., 0.9).
- Validation Status: Status strings such as "passed" or "failed".
Sources: tests/unit/test_services.py:53-57, tests/unit/test_services_and_api.py:68-75
Summary
The flext-dbt-ldif architecture provides a robust, automated pipeline for LDIF-to-Warehouse transformations. By combining a Python-based service orchestrator with a highly structured dbt project and automated code generation macros, the system ensures that complex LDAP data structures are accurately and efficiently translated into actionable analytics.