dbt Project Configuration
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldif project is a specialized dbt (data build tool) implementation designed for transforming LDIF (LDAP Data Interchange Format) data into structured enterprise analytical models. The project configuration defines the environment, resource locations, and the multi-layered transformation architecture required to process complex LDAP directory data.
At its core, the project utilizes a hierarchical modeling approach—ranging from raw staging to business-ready marts—supported by Python-based orchestration services that manage model generation and execution workflows.
Sources: dbt_project.yml:1-5, README.md:7-9
Core Project Settings
The dbt project is identified as flext_dbt_ldif and uses the flext_ldif profile for connection management. It is configured to operate within a specific directory structure and utilizes several external packages to enhance its capabilities.
Resource Paths
The project defines standard directories for its assets:
- Models: Located in
models/ - Macros: Located in
macros/ - Tests: Located in
tests/ - Seeds: Located in
seeds/ - Snapshots: Located in
snapshots/
Sources: dbt_project.yml:1-17
External Package Dispatch
The project configures macro dispatching to prioritize specific namespaces, ensuring that helper utilities from the codegen and dbt_utils packages are correctly resolved.
dispatch:
- macro_namespace: dbt_utils
search_order: ["dbt_utils"]
- macro_namespace: codegen
search_order: ["codegen"]
Sources: dbt_project.yml:92-97
Model Hierarchy and Materialization
The transformation logic is organized into four distinct layers, each with specific materialization strategies and schema targets. This structure ensures a clean separation between raw data ingestion and finalized business reporting.
| Layer | Materialization | Schema | Tags | Purpose |
|---|---|---|---|---|
| Base/Default | table | ldif | ldif, enterprise | Foundational LDIF entity models. |
| Staging | view | ldif_staging | staging, ldif | Raw LDIF data cleaning and casting. |
| Intermediate | view | ldif_intermediate | intermediate, ldif | Business logic and complex joins. |
| Marts | table | ldif_marts | marts, ldif, business | Business-ready reporting models. |
Sources: dbt_project.yml:31-52
Data Flow Architecture
The following diagram illustrates how data flows through the configured dbt layers:
This flow represents the logical progression from raw file data to structured database tables as defined in the project configuration.
Sources: dbt_project.yml:31-52, src/flext_dbt_ldif/services/service.py:53-73
Variable Configuration
The project utilizes global and LDIF-specific variables to maintain flexibility across different environments (e.g., target schemas and database names).
| Variable Name | Default Value | Description |
|---|---|---|
start_date | "2024-01-01" | Reference date for incremental processing. |
ldif_schema | "raw" | Source schema for raw LDIF data. |
database_name | "ldif_analytics" | The target database for dbt operations. |
staging_schema | {{ target.schema }} | Dynamic schema for staging models. |
marts_schema | {{ target.schema }}_marts | Dynamic schema for mart models. |
Sources: dbt_project.yml:61-75
Orchestration and Model Metadata
The project includes a Python-based service layer that interacts with the dbt configuration to programmatically generate and run models. This is facilitated by the DbtModel domain model and the FlextDbtLdifServiceMixin.
DbtModel Structure
The project tracks dbt models as metadata objects before they are materialized in the database.
| Field | Type | Description |
|---|---|---|
name | str | The unique name of the dbt model. |
dbt_model_type | str | Category (e.g., staging, analytics). |
materialization | str | Strategy (view vs table). Defaults to view. |
sql_content | str | The actual SQL code for the model. |
dependencies | StrSequence | List of upstream models. |
Sources: src/flext_dbt_ldif/models.py:16-43
Workflow Execution
The orchestration service manages the lifecycle of a dbt run for LDIF data.
Project Hooks and Flags
To ensure operational visibility, the project uses dbt hooks and specific configuration flags to control behavior during execution.
- On-Run-Start: Logs the initiation of LDIF data processing.
- On-Run-End: Logs the completion of the transformation workflow.
- Flags:
send_anonymous_usage_stats:false(Privacy focus)use_colors:true(CLI readability)warn_error:false(Allows execution to proceed on warnings)
Sources: dbt_project.yml:99-114
Summary
The flext-dbt-ldif dbt project configuration establishes a robust, multi-layered transformation framework. By combining traditional dbt YAML configurations with dynamic Python orchestration and strict schema isolation, the system provides a scalable method for turning unstructured LDAP directory exports into high-quality analytical data sets.