Configuration Reference

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• airflow_ext/wrapper.py
• airflow_ext/main.py
• airflow_ext/files/orchestrate/meltano.py
• pyproject.toml
• airflow_ext/pass_through.py

The Airflow utility extension for Meltano provides a wrapper around Apache Airflow (versions 2.7+ and 3.x) to facilitate seamless orchestration of Meltano projects. It manages the lifecycle of Airflow configurations, database initialization, and automated DAG generation based on Meltano schedules.

This reference details the environment variables, settings, and internal configuration logic used by the extension to bridge the gap between Meltano's management layer and Airflow's operational environment.

Sources: README.md:1-10, pyproject.toml:1-25, airflow_ext/wrapper.py:27-35

Core Path Configuration

The extension relies on specific path resolutions to locate Airflow's home directory, configuration files, and DAG definitions. These are primarily derived from environment variables, which can be configured within the meltano.yml file.

Primary Path Variables

Setting Name	Environment Variable	Default Value (relative to $AIRFLOW_HOME)	Description
extension.airflow_home	AIRFLOW_HOME	$MELTANO_PROJECT_ROOT/orchestrate/airflow	The primary directory for Airflow files.
extension.airflow_config	AIRFLOW_CONFIG	airflow.cfg	Path to the Airflow configuration file.
core.dags_folder	AIRFLOW__CORE__DAGS_FOLDER	dags/	Directory where Airflow DAGs are stored.

Sources: README.md:92-113, airflow_ext/wrapper.py:46-60

Environment Initialization Logic

During initialization, the extension attempts to resolve the AIRFLOW_HOME directory. If it is not found in the environment, the extension will log an error and exit.

Airflow Settings and Mappings

Meltano users configure Airflow via the settings block in meltano.yml. These settings are mapped to Airflow-specific environment variables following the AIRFLOW__<SECTION>__<KEY> convention.

Database and Webserver

Meltano Setting	Airflow Environment Variable	Description
database.sql_alchemy_conn	AIRFLOW__DATABASE__SQL_ALCHEMY_CONN	Connection string for the metadata database.
webserver.web_server_port	AIRFLOW__API__PORT	Port for the web/API server (Note: changed in Airflow 3.x).

Logging Configuration

Meltano Setting	Airflow Environment Variable	Default Value
logging.base_log_folder	AIRFLOW__LOGGING__BASE_LOG_FOLDER	.../airflow/logs
logging.dag_processor_manager_log_location	AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION	.../logs/dag_processor_manager.log
scheduler.child_process_log_directory	AIRFLOW__SCHEDULER__CHILD_PROCESS_LOG_DIRECTORY	.../logs/scheduler

Sources: README.md:27-91

Automatic Configuration Generation

The extension automates the creation of the airflow.cfg file and the initialization of the metadata database during the initialize command.

airflow.cfg Creation Flow

The extension generates an initial configuration by invoking Airflow's own default listing command. It includes logic to strip pre-output warnings that might corrupt the .ini file structure.

Database Initialization by Version

The extension detects the Airflow version to determine the correct database initialization command.

• Airflow 2.x: Executes airflow db init
• Airflow 3.x: Executes airflow db migrate

Sources: airflow_ext/wrapper.py:152-180

DAG Generator Configuration

The meltano_dag_generator.py (derived from airflow_ext/files/orchestrate/meltano.py) is automatically deployed to the AIRFLOW__CORE__DAGS_FOLDER during initialization. It uses internal defaults for DAG execution.

Default DAG Arguments

The generator applies the following default arguments to all created DAGs:

Argument	Value
owner	"airflow"
catchup	False
retries	1
retry_delay	timedelta(minutes=5)
concurrency	1

Sources: airflow_ext/files/orchestrate/meltano.py:34-45, airflow_ext/wrapper.py:84-100

Schedule Resolution Logic

The generator executes meltano schedule list --format=json to identify tasks to be orchestrated.

Extension CLI and Logging Configuration

The extension provides a CLI built with typer to wrap Airflow commands.

Logging Environment Variables

The extension supports standard Meltano logging configurations via environment variables:

• LOG_LEVEL: Controls the verbosity (default: INFO).
• LOG_TIMESTAMPS: Boolean to show timestamps in logs.
• MELTANO_LOG_JSON: Boolean to enable Meltano-specific JSON log format.

Sources: airflow_ext/main.py:68-85, airflow_ext/pass_through.py:12-20

Extension Commands

Command	Execution Target	Description
initialize	ext.initialize()	Deploys DAG generator, creates config, inits DB.
invoke	ext.invoke()	Pass-through for standard Airflow CLI commands.
describe	ext.describe()	Returns EDK model description of the extension.

Sources: airflow_ext/main.py:27-66, airflow_ext/wrapper.py:102-124

Summary

The airflow-ext configuration is centered on bridging Meltano settings to Airflow environment variables and automating the setup of the Airflow environment. By managing the AIRFLOW_HOME and AIRFLOW_CONFIG automatically, it ensures that Meltano-specific DAGs (generated from Meltano schedules) are correctly registered and executed within a standard Airflow installation.