Installation Guide

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• pyproject.toml
• airflow_ext/wrapper.py
• airflow_ext/main.py
• airflow_ext/files/orchestrate/meltano.py
• airflow_ext/pass_through.py

The Meltano Airflow extension (airflow-ext) provides a utility wrapper for Apache Airflow within a Meltano project. It automates the configuration of Airflow settings, handles metadata database initialization across different Airflow versions, and deploys a DAG generator that dynamically translates Meltano schedules into Airflow DAGs.

This guide details the process of adding the extension to a Meltano project, initializing the environment, and configuring essential settings for local or production orchestration.

Sources: README.md:1-5, airflow_ext/wrapper.py:23-28

Core Installation Workflow

The installation process involves adding the utility to the meltano.yml project file, installing the dependencies, and running the initialization sequence to prepare the Airflow home directory.

The diagram shows the standard sequence from definition to verification.
Sources: README.md:95-104

Step 1: Configuration in meltano.yml

The extension requires specific environment variables and dependencies. The pip_url must include meltano.edk, the extension itself, and the target apache-airflow version with appropriate constraints.

Sources: README.md:11-19, pyproject.toml:23-29

Step 2: Plugin Initialization

Running meltano invoke airflow:initialize triggers the Airflow.initialize() method. This command performs several critical setup tasks:

1. Config Generation: Executes airflow config list --defaults to generate an initial airflow.cfg.
2. Database Migration: Detects the Airflow version. It runs db init for Airflow 2.x and db migrate for Airflow 3.x.
3. DAG Generator Deployment: Copies the bundled meltano_dag_generator.py (sourced from airflow_ext.files.orchestrate.meltano) into the configured DAGs folder.

Sources: airflow_ext/wrapper.py:75-99, airflow_ext/wrapper.py:157-183, airflow_ext/main.py:26-40

Environment Configuration

The extension relies on specific environment variables to locate the Airflow home and configuration files.

Variable	Description	Default / Example Value
AIRFLOW_HOME	Primary directory for Airflow files.	$MELTANO_PROJECT_ROOT/orchestrate/airflow
AIRFLOW_CONFIG	Path to the Airflow configuration file.	$AIRFLOW_HOME/airflow.cfg
AIRFLOW__CORE__DAGS_FOLDER	Directory where Airflow looks for DAGs.	$AIRFLOW_HOME/dags
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN	Connection string for the metadata DB.	sqlite:///$MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/airflow.db

Sources: README.md:25-45, airflow_ext/wrapper.py:41-55

Initialization Logic Flow

The extension handles the logic of determining which database commands to run based on the installed version of Airflow.

Deployment of the DAG Generator

During initialization, the extension installs a dynamic DAG generator. This generator is responsible for reading Meltano schedules and creating corresponding Airflow DAGs.

• File Path: airflow_ext/files/orchestrate/meltano.py is copied to the AIRFLOW__CORE__DAGS_FOLDER.
• Behavior: It executes meltano schedule list --format=json and iterates through both legacy (v1) and job-based (v2) schedules.
• Filtering: Schedules with an interval set to @once (where cron_interval is null) are explicitly skipped and do not result in a DAG.

Sources: airflow_ext/wrapper.py:82-93, airflow_ext/files/orchestrate/meltano.py:175-195

Service Management

Once installed and initialized, the Airflow services can be managed via the airflow_invoker (or the default airflow alias in Meltano).

# Create an admin user
meltano invoke airflow users create -u admin -p password --role Admin -e admin@example.com -f admin -l admin

# Start services
meltano invoke airflow scheduler &
meltano invoke airflow api-server

Sources: README.md:106-118, airflow_ext/pass_through.py:12-20

Summary

The airflow-ext utility simplifies Airflow deployment within Meltano by abstracting version-specific database setup and providing a standardized meltano.yml template. Successful installation is achieved by defining the utility, running initialize, and verifying the environment via the pass-through CLI.

Sources: README.md:1-8, airflow_ext/wrapper.py:121-133