Introduction to Meltano EDK
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Extension Developer Kit (EDK) is a Python-based framework designed to facilitate the rapid development of custom Meltano extensions. While the Singer SDK is intended for building extractors and loaders, the EDK focuses on wrapping existing software (such as utilities like dbt or cron) to integrate them into the Meltano ecosystem as first-class plugins.
The EDK provides a standardized interface for extension lifecycle management, including initialization, invocation, and self-description. It also includes utilities for subprocess management and structured logging to ensure extensions behave consistently within a Meltano environment.
Sources: README.md:1-10, pyproject.toml:1-15, meltano/edk/extension.py:1-10
Core Architecture and Interface
The EDK's primary architectural component is the ExtensionBase abstract class. All extensions must inherit from this class and implement its abstract methods to define their behavior.
Extension Lifecycle Methods
The lifecycle of an extension invocation is managed through a series of hooks that allow for setup, execution, and cleanup.
| Method | Type | Description |
|---|---|---|
pre_invoke | Optional | Logic executed before the main command is called. |
initialize | Optional | On-demand setup (e.g., creating config files). |
invoke | Abstract | The core execution logic of the extension. |
post_invoke | Optional | Cleanup or reporting logic after execution. |
describe | Abstract | Returns metadata about the extension's capabilities. |
Sources: meltano/edk/extension.py:34-92, meltano/edk/models.py:20-35
The Invocation Flow
The following sequence diagram illustrates how the pass_through_invoker coordinates the execution of an extension command, ensuring that lifecycle hooks are triggered in the correct order.
The pass_through_invoker is specifically designed to wrap a CLI and pass arbitrary arguments through the extension's lifecycle hooks.
Sources: meltano/edk/extension.py:108-146, tests/test_meltano_edk.py:84-95
Extension Description and Metadata
Extensions use the describe method to communicate their UI and CLI capabilities back to Meltano. This metadata is structured using data classes defined in models.py.
Data Models
- Command: Represents a generic runnable command with a name and description.
- ExtensionCommand: A specialized command that defaults to supporting the standard EDK lifecycle (describe, invoke, etc.).
- InvokerCommand: Represents a "pass-through" style command where the extension simply passes arguments to an underlying binary.
- Describe: The top-level container holding the list of supported commands.
Sources: meltano/edk/models.py:7-46
Formatted Output
The EDK supports multiple output formats for descriptions, which can be retrieved via the describe_formatted method.
Subprocess Management (The Invoker)
The EDK provides an Invoker utility class in process.py to handle the execution of external binaries. This class supports both blocking runs and asynchronous execution with real-time logging.
Invoker Capabilities
- Standard Run: Wraps
subprocess.runwithcheck=True, using the extension's environment and working directory. - Run and Log: Uses
asyncioto execute a process and streamstdoutandstderrdirectly to the extension's logger as the data arrives. - Signal Handling: Automatically handles
SIGINT(on non-Windows platforms) to ensure child processes are terminated correctly if the extension is interrupted.
Sources: meltano/edk/process.py:38-158, tests/test_process.py:46-65
Logging and Configuration
The EDK enforces structured logging using structlog. This ensures that logs are machine-readable (JSON) when running in production/CI environments and human-readable in development.
Logging Configuration Variables
Logging behavior can be controlled via environment variables, surfaced through the pass_through_logging_config function.
| Environment Variable | Default | Description |
|---|---|---|
LOG_LEVEL | INFO | The minimum logging level (DEBUG, INFO, etc.). |
LOG_TIMESTAMPS | False | Whether to include ISO timestamps in logs. |
LOG_LEVELS | False | Whether to include the log level in the output. |
MELTANO_LOG_JSON | False | If True, outputs logs in JSON format. |
Sources: meltano/edk/logging.py:15-30, meltano/edk/logging.py:130-149
Project Initialization
Developers can bootstrap new extensions using the copier template included in the repository. The template prompts for key metadata to generate the project structure.
Key Template Variables
- extension_name: The display name of the extension.
- wrapper_target_name: The actual CLI command being wrapped (e.g.,
dbt). - extension_id: The unique identifier for the extension, usually suffixed with
-ext. - library_name: The Python package name generated for the extension.
Sources: README.md:16-30, copier.yml:1-35
Summary
The Meltano EDK provides a robust foundation for building Meltano utility extensions. By standardizing the interface through ExtensionBase, providing a managed Invoker for subprocesses, and integrating structured logging, the EDK ensures that custom extensions are maintainable and compatible with the broader Meltano plugin architecture. Developers primarily interact with the framework by implementing the lifecycle hooks and defining their extension's capabilities through the Describe model.