EDK Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

• meltano/edk/extension.py
• meltano/edk/process.py
• meltano/edk/models.py
• meltano/edk/logging.py
• meltano/edk/types.py
• meltano/edk/init.py
• README.md

The Meltano Extension Developer Kit (EDK) is a Python-based framework designed to simplify the creation of custom Meltano extensions. It serves as a wrapper for third-party CLI tools (e.g., dbt, cron), providing a standardized interface for Meltano to interact with these utilities as part of a data orchestration pipeline.

The EDK architecture centers on an abstract base class that defines the extension lifecycle, a robust process invoker for subprocess management, and a structured metadata system for describing extension capabilities. Unlike the Singer SDK, which focuses on data movement (taps and targets), the EDK is optimized for wrapping existing command-line applications.

Sources: README.md:1-12, meltano/edk/extension.py:1-25

Extension Interface and Lifecycle

The core of every EDK extension is the ExtensionBase class. This abstract base class enforces a specific lifecycle and set of capabilities that Meltano expects. Developers implement this interface to define how the wrapped application is initialized, described, and executed.

Lifecycle Methods

The EDK defines a clear execution flow through four primary hooks:

• pre_invoke: Executed before the main command. Used for environment setup or configuration validation.
• invoke: The primary execution method where the wrapped CLI is called.
• post_invoke: Executed after the main command, regardless of success or failure.
• initialize: An optional on-demand method for one-time setup tasks.

Sources: meltano/edk/extension.py:34-93

Extension Class Hierarchy

The following diagram illustrates the relationship between the base interface and the data models used to describe the extension.

Subprocess Management (Invoker)

The Invoker class is responsible for the actual execution of external binaries. It supports both standard synchronous execution and asynchronous streaming of logs from the subprocess to the extension's logger.

Execution Modes

1. Direct Run (run): A blocking wrapper around subprocess.run that returns a CompletedProcess object. It is used when the extension only needs the return code or captured output.
2. Streamed Run (run_and_log): An asynchronous execution mode that captures stdout and stderr in real-time, decoding the bytes and passing them to the structlog interface.

Sources: meltano/edk/process.py:41-168

Process Execution Flow

The sequence diagram below shows how the pass_through_invoker coordinates the extension lifecycle with the underlying process execution.

Metadata and Capabilities Modeling

The EDK uses a structured model system to communicate its capabilities back to Meltano. This is primarily handled through the describe method, which returns details about supported commands and whether the extension supports a "pass-through" CLI mode.

Command Types

Model	Description	Default Commands
ExtensionCommand	Standard EDK commands.	describe, invoke, pre_invoke, post_invoke, initialize
InvokerCommand	A pass-through command for the wrapped tool.	:splat (indicates wildcard arguments)

Sources: meltano/edk/models.py:6-45

Formatting Support

Extensions can output their description in multiple formats for different consumers:

• Text: Human-readable summary using devtools.prettier.
• JSON: Machine-readable format for integration.
• YAML: Standard format for Meltano configuration files.

Sources: meltano/edk/extension.py:27-32, meltano/edk/extension.py:100-112

Logging Configuration

The EDK provides a standardized logging utility based on structlog. It is designed to bridge the gap between internal extension logs and the logs generated by the wrapped subprocess.

Configuration Parameters

The default_logging_config function supports several environment-driven configurations:

• LOG_LEVEL: Standard levels (DEBUG, INFO, etc.). Defaults to INFO.
• LOG_TIMESTAMPS: Toggles ISO timestamps.
• LOG_LEVELS: Toggles inclusion of log levels in output.
• MELTANO_LOG_JSON: Switches between human-readable console rendering and JSON rendering for structured logging.

Sources: meltano/edk/logging.py:12-110

Technical Specifications

Data Types

The EDK defines a custom type alias for command-line arguments to ensure consistency across the process management layer.

• ExecArg: str | bytes

Sources: meltano/edk/types.py:1-7

Implementation Summary

Developers using the EDK typically focus on implementing the ExtensionBase and using the Invoker within the invoke method.

class MyExtension(ExtensionBase):
    def invoke(self, command_name, *command_args):
        invoker = Invoker(bin="my-tool")
        invoker.run_and_log(command_name, *command_args)

    def describe(self):
        return models.Describe(
            commands=[models.ExtensionCommand(name="my-ext")]
        )

Sources: meltano/edk/extension.py:73-86, meltano/edk/process.py:41-55

The EDK architecture ensures that while each extension wraps a unique tool, the interface remains consistent, allowing Meltano to manage various utilities through a unified extension protocol.