Orchestrators (Airflow)
Relevant source files
The following files were used as context for generating this wiki page:
Apache Airflow is integrated into the Meltano ecosystem as a primary orchestrator for managing complex data workflows. Its role is to schedule and execute jobs defined in the project, providing a robust interface for monitoring and managing the lifecycle of ELT (Extract, Load, Transform) pipelines. Within Meltano, Airflow is categorized under the orchestrators plugin type, though it is often configured as a utility extension to leverage its command-line interface and web server.
Sources: src/meltano/core/plugin/airflow.py:44, src/meltano/schemas/meltano.schema.json:442, integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json:1159
Plugin Architecture
The Airflow integration relies on two primary classes: Airflow and AirflowInvoker. These classes bridge the gap between Meltano's plugin management and Airflow's native execution environment.
AirflowInvoker
The AirflowInvoker is a specialized subclass of PluginInvoker responsible for preparing the runtime environment. It dynamically sets critical Airflow environment variables based on the project's layout and plugin configuration. Specifically, it points AIRFLOW_HOME to the plugin's run directory and AIRFLOW_CONFIG to the generated airflow.cfg file.
Sources: src/meltano/core/plugin/airflow.py:30-41
Airflow Plugin Class
The Airflow class inherits from BasePlugin and implements the core logic for managing the Airflow configuration and metadata database. It manages the unflattening of Meltano's flat configuration structure into the nested sections required by the airflow.cfg file.
This diagram illustrates the relationship between the core Meltano plugin interfaces and the Airflow-specific implementations.
Sources: src/meltano/core/plugin/airflow.py:30-51
Lifecycle Hooks and Initialization
Meltano utilizes asynchronous hooks to automate the setup and teardown of the Airflow environment.
Environment Preparation
During the before_install phase, the plugin sets the SLUGIFY_USES_TEXT_UNIDECODE environment variable to "yes" to ensure compatibility during installation without requiring specific GPL-licensed dependencies.
Sources: src/meltano/core/plugin/airflow.py:90-97
Configuration and Database Initialization
The before_configure hook is the most critical phase. It performs the following sequence:
- Generate Defaults: Invokes
airflow config list --defaultsto capture the standard configuration. - Apply Custom Config: Updates the
airflow.cfgfile with settings defined inmeltano.yml. - Detect Version: Executes
airflow versionto determine the correct database initialization command. - Init Database: Executes
airflow initdbfor versions prior to 2.0.0, orairflow db initfor modern versions.
The sequence of events during project configuration ensuring the Airflow metadata database is synced with the current configuration.
Sources: src/meltano/core/plugin/airflow.py:109-158
Configuration Management
Airflow configuration in Meltano is derived from environment variables and meltano.yml settings. The Airflow.process_config method takes the flat dictionary provided by Meltano's configuration service and nests it based on keys (e.g., core.dags_folder becomes [core] section, dags_folder key).
Sources: src/meltano/core/plugin/airflow.py:56-67
Core Settings and Commands
Commonly used settings and commands found in Meltano manifests include:
| Component | Setting/Command | Description |
|---|---|---|
| Command | initialize | Seeds the database and deploys the DAG orchestrator. |
| Command | create-admin | Creates an Airflow user with administrative privileges. |
| Command | ui | Starts the Airflow webserver (defaults to port 8080). |
| Setting | core.dags_folder | The directory where Airflow looks for DAG files. |
| Setting | database.sql_alchemy_conn | The URI for the Airflow metadata database. |
| Environment | AIRFLOW_HOME | Managed automatically by the invoker. |
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json:1159-1262, src/meltano/core/plugin/airflow.py:36
Configuration Constraints
Note that certain Airflow configuration values, such as core.executor, cannot be modified via meltano config set and must be managed through the extension's environment or the airflow.cfg directly if necessary.
Sources: CHANGELOG.md:104-105
Cleanup
To maintain project cleanliness, the before_cleanup hook automatically unlinks (deletes) the generated airflow.cfg file. This ensures that transient configurations do not persist between different execution contexts.
Sources: src/meltano/core/plugin/airflow.py:161-171
The Airflow integration in Meltano provides a seamless bridge between declarative project definitions and programmatic workflow orchestration. By managing the underlying configuration and database initialization automatically, it allows developers to focus on defining data pipelines rather than managing orchestration infrastructure.