dbt Project Configuration
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The flext-dbt-ldap project configuration defines the structure, transformation logic, and execution environment for transforming raw LDAP directory data into structured data warehouse models. This configuration orchestrates the movement of data through various stages—from staging and intermediate business logic to final reporting marts—ensuring data quality and referential integrity throughout the pipeline.
Sources: dbt_project.yml:1-43, README.md:1-10
The configuration is managed through a combination of standard dbt YAML files and Pydantic-based Python models that define the runtime environment, directory structures, and LDAP-specific transformation rules. This dual approach allows for static project definitions while maintaining dynamic control over execution settings and data quality thresholds.
Sources: src/flext_dbt_ldap/_models/configuration.py:11-15, src/flext_dbt_ldap/settings.py:17-21
Core Project Structure
The dbt project is organized into functional directories and specific model layers. The main configuration file, dbt_project.yml, specifies the naming, profiling, and pathing for the transformation resources.
Directory Mapping
| Directory Path | Purpose | Configuration Field |
|---|---|---|
models | SQL and Python model definitions | model-paths |
analyses | Ad-hoc SQL analysis files | analysis-paths |
tests | Data quality and schema tests | test-paths |
data | CSV seed files for static data | seed-paths |
macros | Reusable Jinja/SQL logic | macro-paths |
snapshots | SCD (Slowly Changing Dimension) logic | snapshot-paths |
Sources: dbt_project.yml:1-15, src/flext_dbt_ldap/_models/configuration.py:17-45
Model Hierarchy and Materialization
The project implements a layered architecture for LDAP data processing, using different schemas and materialization strategies for each layer to optimize for performance and data accessibility.
Python Configuration Models
Beyond the YAML configuration, the system utilizes Python classes to define the execution state and configuration validation for the dbt runtime.
DbtProjectConfig
The DbtProjectConfig model represents the internal state of the dbt project settings used during service execution.
| Field | Type | Default Value | Description |
|---|---|---|---|
name | str | Required | The dbt project name |
profile | str | "" | The dbt profile name |
target_path | str | target | Directory for build artifacts |
target_schema | str | ldap | Default database schema |
materialized | str | table | Default materialization strategy |
Sources: src/flext_dbt_ldap/_models/configuration.py:17-48
Runtime Settings
Runtime behavior is controlled via FlextDbtLdapSettings, which manages environment-prefixed variables for both LDAP connectivity and dbt execution.
- Environment Prefix:
FLEXT_DBT_LDAP_ - Project Directory: Defaults to
. - LDAP Base DN: Used for search operations and referenced in dbt variables.
Sources: src/flext_dbt_ldap/settings.py:17-45, dbt_project.yml:45-49
Data Transformation Pipeline
The configuration facilitates a full pipeline from LDAP extraction to dbt transformation. This flow is managed by service mixins that utilize the defined configurations to map LDAP attributes to relational columns.
Attribute and Schema Mapping
The configuration allows developers to define how LDAP directory structures map to database tables and columns through settings.
ldap_attribute_mapping: A dictionary mapping LDAP-specific attribute names (e.g.,uid) to dbt model attribute names (e.g.,user_id).ldap_schema_mapping: Maps LDAP entity types (Users, Groups, Org Units) to specific destination tables in the raw schema.
Sources: src/flext_dbt_ldap/settings.py:59-75, src/flext_dbt_ldap/services/client.py:149-160
Validation and Quality Gates
Configuration includes thresholds and rules for ensuring data integrity before and after transformation.
- Quality Thresholds: Managed via
min_quality_threshold(default0.8or as configured inDbtLdap.DEFAULT_QUALITY_THRESHOLD). - Required Attributes: A list of LDAP attributes that must be present for an entry to be considered valid during the extraction phase.
- Freshness: Configured via
DbtSourceFreshnesswithwarn_afteranderror_afterthresholds.
Sources: src/flext_dbt_ldap/settings.py:48-57, src/flext_dbt_ldap/_models/schema.py:65-75, src/flext_dbt_ldap/_models/configuration.py:53-61
Conclusion
The dbt project configuration in flext-dbt-ldap provides a robust framework for managing enterprise identity data. By combining standard dbt project definitions with extensible Python-based configuration models, it enables a highly configurable and type-safe environment for transforming complex LDAP directory structures into actionable business intelligence in a data warehouse.