Logging and Directory Configuration

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• airflow_ext/wrapper.py
• airflow_ext/main.py
• airflow_ext/pass_through.py
• airflow_ext/files/orchestrate/meltano.py
• pyproject.toml

The Meltano Airflow extension utilizes a structured approach to manage directory structures and logging behaviors, ensuring compatibility between the Meltano ecosystem and Apache Airflow. It leverages environment variables and a custom wrapper to define where Airflow stores its configuration, logs, and Directed Acyclic Graphs (DAGs), while providing a unified logging interface through structlog.

Sources: README.md:1-10, airflow_ext/wrapper.py:33-54, airflow_ext/main.py:84-110

Directory Configuration Architecture

The extension mandates a specific directory structure within a Meltano project to maintain isolation and portability. This is primarily governed by the AIRFLOW_HOME environment variable, which serves as the root for all Airflow-related artifacts.

Key Directory Paths

The Airflow class in the extension wrapper initializes several critical paths based on environment variables or defaults relative to AIRFLOW_HOME.

Path Variable	Environment Variable	Default Value	Description
airflow_home	AIRFLOW_HOME	N/A (Required)	The root directory for Airflow configuration and files.
airflow_cfg_path	AIRFLOW_CONFIG	{AIRFLOW_HOME}/airflow.cfg	Location of the main Airflow configuration file.
airflow_core_dags_path	AIRFLOW__CORE__DAGS_FOLDER	{AIRFLOW_HOME}/dags	Directory where Airflow looks for DAG definitions.

Sources: airflow_ext/wrapper.py:39-54, README.md:83-96

Directory Initialization Flow

During the initialize command, the extension ensures that the required directory structure exists and populates it with necessary files, such as the Meltano DAG generator.

Logging Configuration

Logging in the extension is handled through a combination of standard Python logging (for the Airflow components) and structlog (for the extension wrapper).

Extension Wrapper Logging

The extension uses structlog for its internal operations, such as configuration generation and process invocation. Users can control the logging behavior via CLI options or environment variables.

• Log Level: Controlled by LOG_LEVEL (default: INFO).
• JSON Formatting: Can be toggled with --meltano-log-json or MELTANO_LOG_JSON, aligning output with Meltano's JSON logging standard.
• Timestamps/Levels: Optional display of timestamps and log levels in the console output.

Sources: airflow_ext/main.py:84-110, pyproject.toml:26

Airflow System Logging

Airflow's internal logging is configured via settings defined in meltano.yml which map to Airflow-specific environment variables.

Airflow Setting	Env Variable	Description
logging.base_log_folder	AIRFLOW__LOGGING__BASE_LOG_FOLDER	Absolute path where Airflow stores log files.
logging.dag_processor_manager_log_location	AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION	Specific path for DAG parser logs.
scheduler.child_process_log_directory	AIRFLOW__SCHEDULER__CHILD_PROCESS_LOG_DIRECTORY	Directory for logs from scheduler child processes.

Sources: README.md:58-79

Configuration Generation and Management

The extension manages the generation of the airflow.cfg file to prevent configuration corruption and ensure default settings are applied correctly within the Meltano environment.

Config File Creation Logic

The wrapper executes airflow config list --defaults to capture the base configuration. It includes a specific filtering mechanism to remove any stdout warnings (e.g., deprecation notices) that might precede the actual INI content.

# airflow_ext/wrapper.py:126-133
if proc.stdout:
    config = proc.stdout
    if not config.startswith("["):
        section_start = config.find("\n[")
        if section_start != -1:
            config = config[section_start + 1 :]

    with self.airflow_cfg_path.open("w") as f:
        f.write(config)

Sources: airflow_ext/wrapper.py:113-135

Database Initialization

The directory configuration is finalized by initializing the Airflow metadata database. The extension detects the Airflow version to use the appropriate command.

Summary

Logging and directory configuration in the airflow-ext project are designed to bridge the gap between Meltano's project-centric structure and Airflow's environment requirements. By strictly managing AIRFLOW_HOME and using specialized initialization logic to generate clean configuration files and directory paths, the extension ensures that Airflow logs and metadata remain contained within the Meltano project context.

Sources: README.md, airflow_ext/wrapper.py