Meltano Core Integration

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• meltano/edk/extension.py
• meltano/edk/process.py
• meltano/edk/models.py
• meltano/edk/logging.py
• meltano/edk/types.py

The Meltano Extension Developer Kit (EDK) provides a standardized framework for integrating external tools into the Meltano ecosystem. It serves as a middle layer that allows custom utilities, such as dbt or cron, to be wrapped and executed as Meltano extensions with consistent behavior regarding lifecycle management, logging, and command discovery.

Sources: README.md:1-10, meltano/edk/extension.py:1-20

Extension Interface and Lifecycle

At the heart of the integration is the ExtensionBase abstract class. This interface defines the standard lifecycle of an extension invocation, ensuring that any tool wrapped by the EDK adheres to a predictable execution flow.

Lifecycle Methods

Extensions built with the EDK implement several key methods that govern their behavior:

Method	Description	Implementation Requirement
pre_invoke	Actions to perform before the main command executes.	Optional
initialize	On-demand setup logic for the extension.	Optional
invoke	The primary execution logic for the wrapped tool.	Required (Abstract)
post_invoke	Cleanup or reporting actions after execution.	Optional
describe	Returns metadata about supported commands.	Required (Abstract)

Sources: meltano/edk/extension.py:32-90

Execution Flow

The EDK manages the transition between these lifecycle stages, particularly through the pass_through_invoker method, which provides a safe execution wrapper for external CLI tools.

This diagram illustrates the sequential flow of an extension invocation and the centralized error handling that triggers a process exit on failure.
Sources: meltano/edk/extension.py:105-147

Subprocess Invocation and Management

Extensions often wrap existing binary executables. The Invoker class in meltano/edk/process.py provides the utilities necessary to run these binaries as subprocesses while capturing their output for Meltano's logging system.

The Invoker Class

The Invoker is initialized with a binary path, working directory, and environment variables. It supports two primary execution modes:

1. Blocking Execution (run): A wrapper around subprocess.run that returns a CompletedProcess.
2. Streaming Execution (run_and_log): Uses asyncio to stream stdout and stderr to the logger in real-time.

Sources: meltano/edk/process.py:42-60

Asynchronous Stream Processing

When using run_and_log, the EDK utilizes asynchronous tasks to handle concurrent reading from multiple output streams.

The sequence shows how the Invoker concurrently reads from process pipes and forwards them to the standard logging interface.
Sources: meltano/edk/process.py:108-155

Metadata and Discovery

Meltano discovers extension capabilities through the describe method. The EDK uses structured data models to define what an extension can do.

Data Models

The integration relies on specific dataclasses to communicate capabilities:

• ExtensionCommand: Describes a command that follows the full EDK lifecycle.
• InvokerCommand: Describes a "pass-through" command where arguments are sent directly to the wrapped tool (using the :splat convention).
• Describe: The top-level container for all commands supported by the extension.

Sources: meltano/edk/models.py:7-50

Formatted Output

The ExtensionBase provides describe_formatted to export this metadata in text, json, or yaml formats, allowing Meltano core to programmatically parse extension capabilities.

Sources: meltano/edk/extension.py:23-29, meltano/edk/extension.py:92-103

Logging and Configuration

The EDK provides a standardized logging configuration via structlog to ensure that extension logs are compatible with Meltano Core's expectations, including support for JSON formatting and specific environment variables.

Environment Variable	Description
LOG_LEVEL	Sets the severity threshold (INFO, DEBUG, etc.).
LOG_TIMESTAMPS	If "True", includes ISO timestamps in logs.
LOG_LEVELS	If "True", includes the log level in output.
MELTANO_LOG_JSON	If "True", outputs logs in JSON format for machine parsing.

Sources: meltano/edk/logging.py:125-145

The pass_through_logging_config function automatically reads these variables to configure the global logger.
Sources: meltano/edk/logging.py:123-145

Conclusion

Meltano Core Integration via the EDK provides a robust framework for tool encapsulation. By standardizing the lifecycle through ExtensionBase, subprocess management through the Invoker, and capability discovery via Describe models, it allows developers to quickly build extensions that behave consistently within the larger Meltano ecosystem.