EDK Runtime Debugging

Relevant source files

The following files were used as context for generating this wiki page:

• meltano/edk/logging.py
• meltano/edk/process.py
• meltano/edk/extension.py
• meltano/edk/models.py
• tests/test_meltano_edk.py

EDK Runtime Debugging encompasses the mechanisms provided by the Meltano Extension Developer Kit (EDK) to observe, trace, and troubleshoot extension behavior during execution. This includes structured logging, subprocess output capture, and lifecycle hook monitoring. The system ensures that developers can diagnose issues occurring within the extension logic itself or within the wrapped CLI tool.

The debugging framework relies on structlog for structured observability and asynchronous process handling to stream internal tool logs back to the user. Key components include the ExtensionBase lifecycle methods and the Invoker class, which manages subprocess communication.

Logging Architecture and Configuration

The EDK uses structlog to provide configurable, structured logging. This allows extensions to output logs in either human-readable console formats or machine-readable JSON, which is essential for debugging in automated environments.

Structured Logging Flow

Logging is configured via default_logging_config, which sets up a pipeline of processors to enrich log entries with timestamps, levels, and exception information.

Environment-Based Configuration

The pass_through_logging_config function allows runtime debugging behavior to be adjusted using environment variables without code changes.

Environment Variable	Description	Default
LOG_LEVEL	Minimum severity level (debug, info, warning, error, critical).	INFO
LOG_TIMESTAMPS	Whether to include ISO timestamps in log entries.	False
LOG_LEVELS	Whether to include the log level label in the output.	False
MELTANO_LOG_JSON	If True, outputs logs as JSON; otherwise, uses human-readable text.	False

Sources: meltano/edk/logging.py:11-20, meltano/edk/logging.py:133-149

Subprocess and Runtime Invocation

Extensions often wrap external CLI tools. The EDK provides utilities to capture and log the standard output (stdout) and standard error (stderr) of these tools in real-time.

Asynchronous Log Streaming

The Invoker class utilizes asyncio to read from subprocess streams concurrently, preventing buffers from filling up and providing immediate feedback during long-running tasks.

Error Handling in Process Execution

When a subprocess fails, the log_subprocess_error utility extracts the available stderr and replays it through the extension's logger at the warning level before logging a final error summary.

def log_subprocess_error(
    cmd: str, err: subprocess.CalledProcessError, error_message: str
) -> None:
    if err.stderr:
        for line in err.stderr.split("\n"):
            log.warning(line, cmd=cmd, stdio_stream="stderr")
    log.error(
        f"error invoking {cmd}",
        returncode=err.returncode,
        error_message=error_message,
    )

Sources: meltano/edk/process.py:22-38

Lifecycle Hook Debugging

The ExtensionBase provides a structured execution flow. Debugging often involves tracing the transitions between pre_invoke, invoke, and post_invoke.

Pass-through Invoker Execution Trace

The pass_through_invoker method is a critical path for many extensions. It wraps the invocation lifecycle in a try-except block, logging specific failures for each stage.

Describe Command Inspection

The describe method provides metadata about the extension. For debugging, this can be formatted in multiple ways using describe_formatted to verify the extension's capabilities and command structure.

Format	Functionality
text	Uses devtools.prettier for human-readable debugging of objects.
json	Standard JSON output for programmatic inspection.
yaml	YAML output, often preferred for Meltano configuration files.

Sources: meltano/edk/extension.py:18-25, meltano/edk/extension.py:76-94

Summary

EDK Runtime Debugging is centered on high-fidelity visibility into extension execution. By combining structured structlog output with asynchronous subprocess management in the Invoker class, the EDK ensures that developers can trace logic through ExtensionBase lifecycle hooks and capture all output from wrapped CLI applications. The use of environment variables for logging configuration allows for flexible debugging levels without requiring code changes to the extension.

Sources: meltano/edk/logging.py, meltano/edk/process.py, meltano/edk/extension.py