Configuration & Settings
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Configuration & Settings module provides a production-ready, typed infrastructure for managing the FLEXT Meltano environment. It facilitates Go ↔ Python bridge integration by orchestrating Meltano, Singer, and DBT technologies through a centralized settings model. The system ensures environment-aware configuration with strict validation, supporting various deployment stages from development to production.
The core of this system is the FlextMeltanoSettings class, which inherits from FlextSettingsBase and utilizes Pydantic for schema enforcement and environment variable mapping. This allows the platform to maintain a single source of truth (SSOT) for project paths, log levels, and tool versions across the enterprise data integration bridge.
Sources: src/flext_meltano/settings.py:15-18, src/flext_meltano/README.md:14-19
Core Settings Model
The FlextMeltanoSettings class defines the runtime parameters required for Meltano orchestration. It uses the FLEXT_MELTANO_ environment variable prefix for automated configuration loading.
Architecture Overview
The settings architecture follows a hierarchical structure where path definitions, runtime environments, and version requirements are encapsulated within a single model.
Explanation: This diagram shows the data flow from environment variables into the FlextMeltanoSettings model and its subsequent categorization into paths, versions, and runtime states.
Sources: src/flext_meltano/settings.py:15-110, pyproject.toml:340-350
Key Configuration Parameters
| Parameter | Type | Default / Alias | Description |
|---|---|---|---|
project_root | Path | MELTANO_PROJECT_ROOT | The base directory of the Meltano project. |
config_dir | Path | .meltano | Directory for Meltano internal configurations. |
environment | str | development | The active runtime environment (dev, testing, production). |
log_level | LogLevel | INFO | Verbosity of the runtime logs. |
meltano_version | str | 3.9.1 | The minimum required version of Meltano. |
singer_sdk_version | str | 0.48.0 | The minimum required Singer SDK version. |
Sources: src/flext_meltano/settings.py:24-85, src/flext_meltano/_constants/base.py:35-43
Project & Component Models
The system uses structured Pydantic models to validate project-level configurations and component-specific (Tap/Target) settings.
Project Maturity and Consistency
The PipelineProjectModel assesses the project's operational state based on the availability of environments. A project is considered MATURE if it has at least three environments, including a production marker.
Explanation: The diagram illustrates the logic for determining project maturity based on environment count and production markers.
Sources: src/flext_meltano/_models/projects.py:53-115
Component Configurations
Configurations for data sources and sinks are partitioned into TapConfig and TargetConfig. These models include built-in security features to protect sensitive data during serialization.
# Protecting sensitive configuration during serialization
@u.field_serializer("connection_config")
def serialize_connection_config(self, value: t.JsonMapping) -> t.JsonMapping:
"""Serialize connection settings with sensitive data protection."""
return FlextMeltanoModelsCore.protect_sensitive_config(value)
Sources: src/flext_meltano/_models/sources.py:46-50, src/flext_meltano/_models/sources.py:101-105
Path Resolution & Validation
Settings provide utility methods to resolve absolute paths for critical project artifacts, ensuring that the bridge can locate virtual environments and configuration files reliably.
Path Resolution Logic
Explanation: This sequence shows how the settings model interacts with the file system to validate that a Meltano project is correctly initialized.
Sources: src/flext_meltano/settings.py:112-145
Data Transformation Configuration (DBT)
DBT integration is managed via dbt_project.yml and specific Pydantic models like DbtManifest. Configuration includes directory paths and materialization strategies.
| Feature | Path / Value | Source File |
|---|---|---|
| Model Paths | dbt/models | dbt_project.yml |
| Target Path | dbt/target | dbt_project.yml |
| Materialized (Marts) | table | dbt_project.yml |
| Materialized (Staging) | view | dbt_project.yml |
Sources: dbt_project.yml:1-50, src/flext_meltano/_models/projects.py:17-45
Conclusion
The Configuration & Settings module in FLEXT Meltano establishes a robust, type-safe foundation for data pipeline orchestration. By centralizing path management, environment validation, and version control within the FlextMeltanoSettings model, the project ensures consistent behavior across its Go ↔ Python bridge. This architecture allows for secure, scalable data integration while providing developers with clear patterns for extending the platform's capabilities.