Logging System

Relevant source files

The following files were used as context for generating this wiki page:

• meltano/edk/logging.py
• meltano/edk/extension.py
• meltano/edk/process.py
• pyproject.toml
• tests/test_meltano_edk.py

The Logging System in the Meltano Extension Developer Kit (EDK) provides a standardized framework for capturing, formatting, and distributing log events across extensions. It leverages structlog for structured logging and integrates with standard Python logging to ensure compatibility with both internal extension logic and external subprocess execution.

The system is designed to handle diverse logging needs, including human-readable console output, machine-readable JSON formats for production environments, and the capturing of stdout/stderr from wrapped subprocesses.

Sources: meltano/edk/logging.py:1-12, pyproject.toml:35

Core Architecture and Configuration

The logging architecture centers on a configurable pipeline of structlog processors. This pipeline manages log levels, timestamps, and formatting based on the execution context.

Configuration Methods

The EDK provides two primary ways to configure the logging system:

1. default_logging_config: A programmatical interface to set log levels, toggle timestamps, and choose between JSON or console rendering.
2. pass_through_logging_config: An environment-aware configuration that automatically reads settings from standard Meltano environment variables.

The diagram shows how environment variables flow into the configuration utility to initialize the structured logging pipeline.
Sources: meltano/edk/logging.py:79-151

Configuration Parameters

The following table describes the configuration options available through the environment and the default configuration function:

Option	Env Var	Type	Default	Description
Level	LOG_LEVEL	string	info	Logging severity (debug, info, warning, error, critical).
Timestamps	LOG_TIMESTAMPS	boolean	False	Whether to include ISO timestamps.
Levels	LOG_LEVELS	boolean	False	Whether to include the log level name in the output.
JSON Format	MELTANO_LOG_JSON	boolean	False	If True, outputs logs as JSON; otherwise, human-readable.

Sources: meltano/edk/logging.py:12-25, meltano/edk/logging.py:136-151

Structured Logging Processors

The system utilizes a specific sequence of processors to transform log events before they are written to the output stream.

• Filtering: structlog.stdlib.filter_by_level ensures logs below the configured threshold are discarded.
• Exception Handling: format_exc_info and ExceptionPrettyPrinter ensure that tracebacks are captured and formatted correctly, even in JSON mode.
• Rendering: The system switches between structlog.processors.JSONRenderer and structlog.dev.ConsoleRenderer based on configuration.

Sources: meltano/edk/logging.py:100-120

Subprocess Logging

A critical feature of the EDK is the ability to capture and log output from wrapped CLI applications. This is handled by the Invoker class, which manages subprocess execution.

Stream Redirection

When using run_and_log, the EDK uses asynchronous stream readers to monitor stdout and stderr of the subprocess, re-routing lines to the extension's logger as info level events.

This sequence illustrates the asynchronous capture and logging of subprocess output.
Sources: meltano/edk/process.py:100-155

Error Logging

The log_subprocess_error utility provides a specialized way to handle failed subprocesses by replaying captured stderr at the warning level before logging a final error message with the return code.

Sources: meltano/edk/process.py:22-38

Extension Interface Integration

The ExtensionBase class integrates logging into the lifecycle of an extension. The pass_through_invoker method specifically uses a BoundLogger to track the execution of pre_invoke, invoke, and post_invoke steps.

def pass_through_invoker(
    self,
    logger: structlog.BoundLogger,
    *command_args: ExecArg,
) -> None:
    logger.debug("pass through invoker called", command_args=command_args)
    try:
        self.pre_invoke(None, *command_args)
    except Exception:
        logger.exception("pre_invoke failed with uncaught exception")
        sys.exit(1)

Sources: meltano/edk/extension.py:100-116, tests/test_meltano_edk.py:100-108

Summary

The Logging System provides a robust foundation for Meltano extensions by combining structured logging via structlog with flexible configuration through environment variables. It ensures that both internal extension operations and external subprocess executions are observable, providing clear, formatted output to sys.stderr whether in a development console or a production JSON-log aggregator.