Customizing Extension Behavior
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Extension Developer Kit (EDK) provides a structured framework for building custom extensions by extending a base interface. Customizing extension behavior involves overriding specific lifecycle methods and configuring how the extension describes itself to Meltano. This allows developers to wrap existing CLI tools, add pre- and post-processing logic, and manage initialization routines.
Extensions are primarily defined by inheriting from ExtensionBase, an abstract base class that enforces a consistent API for command invocation and capability description. Through this interface, developers can control the execution flow of wrapped binaries and define how the extension interacts with the Meltano ecosystem.
Sources: meltano/edk/extension.py:31-33, README.md:1-10
Extension Lifecycle Methods
Behavior is customized by implementing or overriding methods in the ExtensionBase class. These methods represent different stages of an extension's execution, from setup to finalization.
Core Lifecycle Hooks
The following table describes the primary methods available for customization:
| Method | Type | Purpose |
|---|---|---|
pre_invoke | Optional | Logic executed before the main command. Used for setup or validation. |
invoke | Abstract | The primary execution logic. Must be implemented by the subclass. |
post_invoke | Optional | Logic executed after the main command. Used for cleanup or reporting. |
initialize | Optional | On-demand setup routine (e.g., creating config files). |
describe | Abstract | Returns metadata about the extension's commands and capabilities. |
Sources: meltano/edk/extension.py:34-92, tests/test_meltano_edk.py:16-31
Execution Flow
The pass_through_invoker method demonstrates the standard execution sequence when an extension wraps a CLI tool. It ensures that the lifecycle hooks are called in order and handles unhandled exceptions by exiting the process.
The diagram shows the synchronous execution flow of the extension lifecycle hooks.
Sources: meltano/edk/extension.py:112-148
Describing Capabilities
Extensions must inform Meltano of their available commands and settings via the describe method. This method returns a Describe model containing a list of Command objects.
Command Models
The EDK provides several dataclasses to model different types of commands:
ExtensionCommand: Represents the standard EDK CLI, including built-in operations likedescribeandinitialize.InvokerCommand: Represents a "pass-through" style command where the extension acts as a wrapper for another CLI tool (indicated bypass_through_cli=True).Command: A generic base for any runnable command.
Sources: meltano/edk/models.py:7-46
Formatted Output
The describe_formatted method allows the extension to export its capabilities in json, yaml, or text formats, which is essential for Meltano's discovery mechanism.
Sources: meltano/edk/extension.py:94-110, tests/test_meltano_edk.py:53-84
Subprocess Management and Invocation
For extensions that wrap external binaries, the Invoker class provides utilities for executing subprocesses while managing standard input/output and environment variables.
The Invoker Class
The Invoker supports both blocking execution and asynchronous streaming of logs.
The sequence diagram illustrates how the Invoker captures and logs output from a subprocess in real-time.
Sources: meltano/edk/process.py:46-136
Process Configuration
The Invoker can be configured with:
bin: The path to the binary to execute.cwd: The working directory.env: Environment variables (defaults toos.environ).
Sources: meltano/edk/process.py:53-64, meltano/edk/types.py:7
Logging Configuration
Customizing behavior also involves configuring how the extension logs information. The EDK provides a standard logging setup using structlog.
Configuration Options
Developers can use default_logging_config or pass_through_logging_config to standardize output.
| Parameter | Source | Description |
|---|---|---|
LOG_LEVEL | Env Var | Sets the minimum logging level (e.g., INFO, DEBUG). |
MELTANO_LOG_JSON | Env Var | Toggles between JSON and human-readable console rendering. |
LOG_TIMESTAMPS | Env Var | Includes ISO timestamps in log entries. |
Sources: meltano/edk/logging.py:84-142
Exception Handling in Subprocesses
The log_subprocess_error utility ensures that when a wrapped command fails, its stderr is replayed into the extension's logs for debugging.
Sources: meltano/edk/process.py:20-37
Summary
Customizing extension behavior in the Meltano EDK is achieved by implementing the ExtensionBase interface and leveraging the provided Invoker and logging utilities. By overriding lifecycle hooks (pre_invoke, invoke, post_invoke), developers can inject custom logic around wrapped CLI tools, while the describe system ensures these capabilities are correctly exposed to Meltano. This architecture promotes a clean separation between the extension's management logic and the underlying tools it orchestrates.