Database Initialization and Setup

Relevant source files

The following files were used as context for generating this wiki page:

• airflow_ext/wrapper.py
• airflow_ext/main.py
• README.md
• airflow_ext/files/orchestrate/meltano.py
• pyproject.toml
• tests/conftest.py

Introduction

The Database Initialization and Setup process in airflow-ext is a critical lifecycle stage that prepares the Apache Airflow environment for use within a Meltano project. This extension automates the creation of necessary configuration files, the setup of the Airflow metadata database, and the deployment of orchestration scripts that allow Airflow to trigger Meltano schedules.

This system ensures that Airflow is correctly configured with environment-specific paths for logs, DAGs, and plugins, while handling version-specific database migration commands for both Airflow 2.x and 3.x.

Sources: airflow_ext/wrapper.py:61-101, README.md:93-98

Initialization Lifecycle

The initialization process is primarily handled by the initialize command, which triggers a sequence of setup tasks. This includes environment validation, configuration generation, database schema creation, and file deployment.

Execution Flow

When a user runs meltano invoke airflow:initialize, the extension performs the following steps:

1. Pre-Invoke Verification: Ensures AIRFLOW_HOME is set.
2. Config Generation: Generates an airflow.cfg based on default values if it doesn't exist.
3. Database Initialization: Runs the version-appropriate command to prepare the metadata database.
4. DAG Deployment: Creates the DAGs folder and installs the meltano_dag_generator.py.

The following diagram illustrates the sequence of operations during initialization:

Configuration Management

The extension manages the Airflow configuration by capturing default settings from the installed Airflow binary. This prevents hardcoding defaults that might change between Airflow versions.

Config Creation Logic

The _create_config method executes airflow config list --defaults via a subprocess. It specifically handles cases where Airflow might emit warning logs to stdout by stripping non-INI content before writing to airflow.cfg.

Sources: airflow_ext/wrapper.py:127-147

Environment Variables and Paths

Key paths are derived from environment variables, defaulting to locations within the AIRFLOW_HOME directory.

Variable	Description	Default Logic
AIRFLOW_HOME	Primary directory for Airflow files	Required; exits if not found
AIRFLOW_CONFIG	Path to the configuration file	$AIRFLOW_HOME/airflow.cfg
AIRFLOW__CORE__DAGS_FOLDER	Directory where DAGs are stored	$AIRFLOW_HOME/dags

Sources: airflow_ext/wrapper.py:38-51

Database Schema Setup

The _initdb method provides compatibility between different major versions of Apache Airflow. It inspects the output of airflow version to determine the correct CLI command for database setup.

• Airflow 2.x: Uses airflow db init.
• Airflow 3.x: Uses airflow db migrate.

    def _initdb(self) -> None:
        """Initialize the airflow metadata database."""
        # Airflow version 2 accepts "db init" while version 3 expects "db migrate"
        proc = self.airflow_invoker.run("version", stdout=subprocess.PIPE)
        version = proc.stdout.strip().splitlines()[-1]

        if version.startswith("2."):
            self.airflow_invoker.run("db", "init")
        elif version.startswith("3."):
            self.airflow_invoker.run("db", "migrate")

Sources: airflow_ext/wrapper.py:149-175

Orchestration Integration

A key part of the setup is the deployment of the Meltano DAG generator. This script is bundled with the extension and copied to the Airflow DAGs folder during initialization.

Files Deployed

1. meltano_dag_generator.py: Sourced from airflow_ext.files.orchestrate/meltano.py. This script dynamically generates Airflow DAGs for Meltano schedules by calling meltano schedule list --format=json.
2. README.md: A guidance file for the DAGs folder.

Sources: airflow_ext/wrapper.py:84-101, airflow_ext/files/orchestrate/meltano.py:146-160

Setup Commands

Users interact with the initialization system through the Meltano CLI.

Command	Action
meltano invoke airflow:initialize	Runs the full setup suite (config, db, dags)
meltano invoke airflow users create ...	Manual step post-initialization to create an admin user

Sources: README.md:93-104

Summary

Database initialization and setup in the meltano-airflow-ext project is an automated workflow that bridges the gap between a standard Airflow installation and a Meltano-managed utility. By dynamically generating configurations and handling version-specific database commands, it ensures a consistent environment for running Meltano jobs via the Airflow scheduler.