Environment Variables Mapping
Relevant source files
The following files were used as context for generating this wiki page:
Environment Variables Mapping in the airflow-ext project facilitates the seamless integration between Meltano and Apache Airflow. It allows Meltano's configuration settings to be translated into the environment variables that Airflow expects, ensuring that the Airflow instance running as a Meltano utility is correctly configured with project-specific paths, database connections, and logging settings.
This mapping is critical for maintaining portability across different environments (development, staging, production) by utilizing dynamic path expansion and standardized naming conventions for Airflow configuration options.
Core Mapping Architecture
The extension relies on a specific set of environment variables to determine the location of the Airflow home directory, configuration files, and DAG folders. These variables are typically defined in the meltano.yml file and consumed by the Airflow extension class.
Initialization and Path Resolution
The Airflow class in airflow_ext/wrapper.py performs the initial mapping of environment variables to internal path objects. If the primary environment variable is missing, it attempts to use an extension-specific fallback.
Sources: airflow_ext/wrapper.py:32-50
Environment Variable Table
The following table details the key environment variables mapped between Meltano settings and Airflow configuration:
| Meltano Setting | Airflow Environment Variable | Default / Example Value | Description |
|---|---|---|---|
database.sql_alchemy_conn | AIRFLOW__DATABASE__SQL_ALCHEMY_CONN | sqlite:///$MELTANO_PROJECT_ROOT/.../airflow.db | The connection string for the metadata database. |
core.dags_folder | AIRFLOW__CORE__DAGS_FOLDER | $MELTANO_PROJECT_ROOT/orchestrate/airflow/dags | Location where Airflow looks for DAG files. |
extension.airflow_home | AIRFLOW_HOME | $MELTANO_PROJECT_ROOT/orchestrate/airflow | The root directory for Airflow configuration. |
extension.airflow_config | AIRFLOW_CONFIG | $AIRFLOW_HOME/airflow.cfg | The specific path to the Airflow config file. |
webserver.web_server_port | AIRFLOW__API__PORT | 8080 | Port for the web/API server (Airflow 3.x). |
logging.base_log_folder | AIRFLOW__LOGGING__BASE_LOG_FOLDER | $MELTANO_PROJECT_ROOT/.../logs | Absolute path to the log storage directory. |
Sources: README.md:23-108, airflow_ext/wrapper.py:42-49
Execution Environment Configuration
During invocation, the extension ensures the execution environment is modified to support specific dependency requirements and logging behaviors.
Internal Environment Overrides
The extension explicitly sets certain variables at runtime to ensure compatibility, such as disabling GPL dependencies for certain libraries.
- Variable:
SLUGIFY_USES_TEXT_UNIDECODE - Value:
yes - Purpose: Configures the environment to make Airflow installable without GPL dependencies.
Sources: airflow_ext/wrapper.py:51-52
Logging and Metadata Variables
The extension uses Typer to map command-line options to environment variables for logging configuration, which are then parsed by the Meltano EDK.
Sources: airflow_ext/main.py:76-88
DAG Generation Environment
When the meltano_dag_generator.py (derived from airflow_ext/files/orchestrate/meltano.py) runs within Airflow, it utilizes environment variables to locate the Meltano binary and the project root to trigger scheduled jobs.
Key Variables in DAG Context
MELTANO_PROJECT_ROOT: Used to define the working directory forBashOperatortasks. If not set, it defaults to the current working directory.PATH: The generator falls back to searching the systemPATHfor themeltanoexecutable if the local project symlink (.meltano/run/bin/meltano) is missing.
Sources: airflow_ext/files/orchestrate/meltano.py:38-48, tests/conftest.py:51-53
Summary
The Environment Variables Mapping system acts as the bridge between Meltano's declarative configuration and Airflow's environment-driven runtime. By mapping settings like AIRFLOW_HOME and AIRFLOW__CORE__DAGS_FOLDER, the extension ensures that Airflow operates within the context of the Meltano project, utilizing local logs, databases, and DAG definitions.