Key Features

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• airflow_ext/wrapper.py
• airflow_ext/main.py
• airflow_ext/files/orchestrate/meltano.py
• airflow_ext/pass_through.py
• pyproject.toml

The airflow-ext project is a Meltano utility extension designed to simplify the orchestration of Meltano projects using Apache Airflow. It acts as a wrapper around the Airflow CLI, providing automated configuration management, database initialization, and a dynamic DAG generator that translates Meltano schedules into Airflow workflows. Sources: README.md, pyproject.toml:1-15

1. Automated Extension Lifecycle Management

The extension automates the setup and maintenance of an Airflow environment within a Meltano project context. This is primarily handled through the Airflow class which inherits from ExtensionBase. Sources: airflow_ext/wrapper.py:24-30

Initialization and Configuration

The initialize command performs several critical setup tasks:

• Configuration Generation: It invokes airflow config list --defaults to generate a base airflow.cfg file, ensuring that the environment is ready for use without manual configuration. Sources: airflow_ext/wrapper.py:113-140
• Database Migration: It detects the Airflow version (supporting both 2.x and 3.x) and executes the appropriate database command (db init for v2 or db migrate for v3). Sources: airflow_ext/wrapper.py:142-171
• Resource Deployment: It deploys the meltano_dag_generator.py and associated documentation to the configured Airflow DAGs folder. Sources: airflow_ext/wrapper.py:73-100

The diagram shows the sequence of operations performed during the extension's initialization phase. Sources: airflow_ext/wrapper.py:73-171

2. Dynamic DAG Generation

A core feature is the automatic generation of Airflow DAGs based on Meltano's internal schedules. The meltano.py script (deployed during initialization) serves as the DAG generator. Sources: airflow_ext/files/orchestrate/meltano.py:186-203

Supported Schedule Types

The generator supports both legacy (v1) and modern (v2) Meltano schedule formats:

• ELT Schedules: Creates a single-task DAG for legacy Extract-Load-Transform schedules using the BashOperator to execute meltano schedule run <name>. Sources: airflow_ext/files/orchestrate/meltano.py:61-120
• Job Schedules: Creates multi-task DAGs for Meltano Jobs. Each task within a job is mapped to an Airflow task, with upstream dependencies maintained between sequential tasks. Sources: airflow_ext/files/orchestrate/meltano.py:123-183

Feature	Description	Source
Schedule Discovery	Runs meltano schedule list --format=json to retrieve project schedules.	meltano.py:188
Catchup Control	catchup is set to False by default to prevent backfilling for non-incremental extractors.	meltano.py:53, 107
Versioning Support	Automatically adjusts DAG parameters (e.g., schedule vs schedule_interval) based on Airflow 2.x/3.x detection.	meltano.py:100-105
Task Mapping	Maps Meltano job steps to Airflow BashOperator instances.	meltano.py:168-175

3. CLI Passthrough and Invocation

The extension provides two primary entry points for interacting with Airflow:

• airflow_extension: Handles extension-specific management commands like initialize, describe, and invoke. Sources: airflow_ext/main.py:21-70
• airflow_invoker: A lightweight shim that passes commands directly to the underlying Airflow CLI, allowing users to run standard Airflow commands (e.g., users create, scheduler, webserver) within the Meltano-managed environment. Sources: airflow_ext/pass_through.py:12-21

This diagram illustrates how the airflow_invoker acts as a transparent proxy to the standard Airflow CLI. Sources: airflow_ext/pass_through.py:12-21, README.md:95-108

4. Environment-Driven Configuration

The extension heavily relies on environment variables to integrate Meltano project paths with Airflow's internal requirements. Sources: README.md:15-85

Environment Variable	Meltano Mapping	Purpose
AIRFLOW_HOME	$MELTANO_PROJECT_ROOT/orchestrate/airflow	Root directory for Airflow metadata and configs.
AIRFLOW_CONFIG	$AIRFLOW_HOME/airflow.cfg	Path to the generated config file.
AIRFLOW__CORE__DAGS_FOLDER	$MELTANO_PROJECT_ROOT/orchestrate/airflow/dags	Location where the DAG generator and custom DAGs reside.
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN	sqlite:///$MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/airflow.db	Default database connection string.

Sources: README.md:27-85, airflow_ext/wrapper.py:34-45

Conclusion

The meltano-airflow-ext streamlines the deployment of Airflow as an orchestrator for Meltano projects. By automating the boilerplate of configuration, database management, and DAG creation, it allows developers to focus on defining data integration workflows while benefiting from a fully compatible Airflow 2.7+ (and 3.x) environment. Sources: README.md:1-10, airflow_ext/wrapper.py:24-100