Configuration & Settings
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldap project utilizes a structured configuration system to manage the transformation of LDAP data into a data warehouse using dbt. This system encompasses runtime environment variables, dbt project-level settings, and state management for incremental synchronizations. The configuration is primarily driven by Pydantic models that enforce type safety and provide defaults for LDAP connectivity, dbt project structures, and data quality thresholds.
The configuration architecture is divided into runtime settings (environment-based), dbt project definitions (YAML-based), and internal data models that define how LDAP entities are mapped to relational tables.
Sources: src/flext_dbt_ldap/settings.py:1-20, src/flext_dbt_ldap/_models/configuration.py:1-15
Runtime Settings (FlextDbtLdapSettings)
Runtime settings are managed via the FlextDbtLdapSettings class, which inherits from FlextMeltanoSettings. It uses the FLEXT_DBT_LDAP_ environment variable prefix for automatic configuration loading.
LDAP Connectivity Settings
These settings define how the service connects to the source LDAP directory.
| Setting | Type | Description | Default |
|---|---|---|---|
ldap_host | str | LDAP server hostname | localhost |
ldap_port | int | LDAP server port | 389 |
ldap_use_tls | bool | Whether to use TLS | False |
ldap_bind_dn | SecretStr | Authentication DN | None |
ldap_bind_password | SecretStr | Authentication password | None |
ldap_base_dn | str | Base DN for searches | dc=example,dc=com |
Sources: src/flext_dbt_ldap/settings.py:23-50
Transformation & Quality Settings
These settings control how the data is validated and mapped during the transformation pipeline.
| Setting | Type | Description | Default |
|---|---|---|---|
dbt_project_dir | str | Path to dbt project | . |
min_quality_threshold | float | Min quality score (0.0-1.0) | 0.8 |
required_attributes | StrSequence | Attributes required for valid entries | [] |
ldap_attribute_mapping | StrMapping | LDAP to dbt attribute map | {} |
ldap_schema_mapping | StrMapping | LDAP schemas to dbt tables map | {} |
Sources: src/flext_dbt_ldap/settings.py:53-88
dbt Project Configuration
The project defines its relational structure and materialization strategies through dbt_project.yml and corresponding Python value models.
Project Structure
The DbtProjectConfig model mirrors the dbt project configuration, defining paths for models, seeds, and macros.
The diagram shows the logical grouping of project configuration fields within the model.
Sources: src/flext_dbt_ldap/_models/configuration.py:18-56, dbt_project.yml:1-20
Layered Materialization
The project implements a multi-stage transformation layer as configured in dbt_project.yml:
- Staging: Raw LDAP data transformation (Materialized as views, schema:
ldap_staging). - Intermediate: Business logic transformations (Materialized as views, schema:
ldap_intermediate). - Marts: Business-ready data models (Materialized as tables, schema:
ldap_marts).
Sources: dbt_project.yml:23-45
Sync & State Management
The FlextDbtLdapSyncMixin handles the persistence of synchronization states to enable incremental loads.
State File Persistence
Sync state is stored in a JSON file located at {dbt_project_dir}/.flext_dbt_ldap_sync_state.json. This file tracks bookmarks (usually timestamps) for different sync keys such as users or groups.
The diagram illustrates the flow of state retrieval and update during a synchronization workflow.
Sources: src/flext_dbt_ldap/services/sync.py:180-220
Incremental Logic
The system builds LDAP search filters dynamically based on the stored bookmark. If an incremental sync is requested for a key (e.g., users), it appends a modifyTimestamp check to the base filter.
Sources: src/flext_dbt_ldap/services/sync.py:171-178, src/flext_dbt_ldap/services/sync.py:222-231
Data Models for Schema & Transformation
The project uses internal Pydantic models to define the requirements and rules for data processing.
Data Validation Configuration
The DataValidationConfig class defines the constraints applied to LDAP entries before they are processed by dbt.
| Field | Type | Description |
|---|---|---|
min_quality_threshold | NonNegativeFloat | Minimum accepted score |
required_attributes | StrSequence | List of mandatory LDAP attributes |
validate_dns | bool | If True, DNs must be present |
columns | OperationAttributes | Per-attribute validation rules |
Sources: src/flext_dbt_ldap/_models/schema.py:53-68
Transformation Configuration
The TransformationConfig class defines how source tables are filtered and how columns are transformed using SQL-like expressions.
class TransformationConfig(m.Value):
source_table: str
transformations: t.StrMapping # e.g., {"dbt_col": "ldap_col::text"}
filters: t.StrSequence # e.g., ["status = 'active'"]
Sources: src/flext_dbt_ldap/_models/schema.py:33-45
Conclusion
The configuration system in flext-dbt-ldap ensures a decoupled architecture where LDAP source details, dbt project parameters, and data quality rules are managed through strongly-typed models. By combining environment-based runtime settings with persistent sync state, the project provides a robust foundation for enterprise-grade LDAP data integration and transformation.