dbt Profiles Setup
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The dbt Profiles Setup defines the connection configurations and environments required for flext-dbt-ldif to interact with various data warehouses. It facilitates the transformation of LDIF (LDAP Data Interchange Format) data into structured analytics models by mapping logical project requirements to physical database credentials.
Within this project, the profile configuration bridges the gap between the dbt transformation logic and the underlying infrastructure, supporting multiple adapters including Postgres, Redshift, BigQuery, and Snowflake. Sources: dbt_project.yml:6, dbt_packages/codegen/integration_tests/profiles.yml:8-53
Core Profile Configuration
The project is configured to use the flext_ldif profile by default. This profile acts as the primary container for connection outputs used during the transformation of LDAP data. Sources: dbt_project.yml:6
Project-to-Profile Mapping
The dbt_project.yml file establishes the link between the dbt project and its profile. It also defines the target schemas for different materialization layers, which the profile must support.
The relationship between project settings and profile definitions. Sources: dbt_project.yml:6-45, dbt_packages/codegen/integration_tests/profiles.yml:8-53
Environment-Specific Targets
The project utilizes environment variables to securely manage credentials across different database adapters. This approach is standardized across integration tests and production-like setups to ensure sensitive information like passwords and keyfiles are not hardcoded. Sources: dbt_packages/codegen/integration_tests/profiles.yml:3-6
Adapter Configurations
The following table summarizes the configuration parameters required for the supported database adapters:
| Adapter | Parameters | Credential Method |
|---|---|---|
| Postgres | host, user, pass, port, dbname, schema | Environment Variables |
| Redshift | host, user, pass, dbname, port, schema | Environment Variables |
| BigQuery | project, dataset, method | Service Account JSON |
| Snowflake | account, user, password, role, database, warehouse | Environment Variables |
Sources: dbt_packages/codegen/integration_tests/profiles.yml:10-53
Credential Management via Environment Variables
Integration tests rely on specific naming conventions for environment variables to inject secrets.
postgres:
type: "postgres"
host: "{{ env_var('POSTGRES_HOST') }}"
user: "{{ env_var('POSTGRES_USER') }}"
pass: "{{ env_var('DBT_ENV_SECRET_POSTGRES_PASS') }}"
port: "{{ env_var('POSTGRES_PORT') | as_number }}"
Sources: dbt_packages/codegen/integration_tests/profiles.yml:12-16
Integration with Services
The FlextDbtLdifServiceMixin orchestrates the execution of dbt workflows and relies on the correctly configured profile to run transformations. The service layer initializes a project_dir which dbt uses to locate the dbt_project.yml and, by extension, the required profile. Sources: src/flext_dbt_ldif/services/service.py:31-41
The flow of profile resolution during a service-triggered dbt transformation. Sources: src/flext_dbt_ldif/services/service.py:125-135, dbt_project.yml:6
Schema and Variable Overrides
While the profile handles the physical connection, the project defines logical schema variables that interact with the profile's target schema. This allows for dynamic schema naming (e.g., appending _marts to the base schema) based on the active profile target. Sources: dbt_project.yml:50-55
| Variable | Default / Formula | Description |
|---|---|---|
ldif_schema | "raw" | Source schema for raw LDIF data |
staging_schema | {{ target.schema }} | Staging layer schema from profile |
marts_schema | {{ target.schema }}_marts | Analytics layer schema |
Sources: dbt_project.yml:50-55
Summary
The dbt Profiles Setup for flext-dbt-ldif provides a flexible, environment-variable-driven configuration system. By separating connection details from the transformation logic in dbt_project.yml, the system supports a wide range of database adapters and ensures secure credential handling across development, testing, and production environments.