Extension Lifecycle

Relevant source files

The following files were used as context for generating this wiki page:

• meltano/edk/extension.py
• meltano/edk/process.py
• meltano/edk/models.py
• meltano/edk/logging.py
• tests/test_meltano_edk.py

The Extension Lifecycle in the Meltano Extension Developer Kit (EDK) defines the standard stages and execution flow for custom Meltano extensions. It provides a structured interface for initialization, discovery, and execution of commands, ensuring consistency across different extension implementations.

Extensions act as wrappers around third-party CLIs or custom logic, providing Meltano with a predictable way to interact with them through a set of lifecycle methods. Sources: meltano/edk/extension.py:1-25, README.md:1-15

Core Interface: ExtensionBase

All extensions must implement the ExtensionBase abstract base class. This class defines the hooks and methods that govern the lifecycle of an extension from initialization to completion.

Lifecycle Methods

Method	Type	Description
initialize	Optional	Called on-demand to set up the extension environment.
describe	Abstract	Returns a description of provided commands and capabilities.
pre_invoke	Optional	Logic executed immediately before the primary command invocation.
invoke	Abstract	The primary execution point for the extension's main functionality.
post_invoke	Optional	Cleanup or reporting logic executed after the command finishes.

Sources: meltano/edk/extension.py:32-84, meltano/edk/models.py:23-35

Execution Flow

The EDK supports two primary execution patterns: standard command invocation and the pass-through invoker.

Standard Invocation Flow

The standard flow involves a sequence of hooks that allow the extension to prepare its environment, execute its primary logic (often by wrapping a subprocess), and perform post-execution tasks.

Pass-through Invoker

The pass_through_invoker is a utility method used to bypass specific command logic and pass arguments directly to the wrapped CLI. It wraps the lifecycle hooks in exception handling to ensure that failures are logged and the process exits cleanly.

Process Management

Subprocess execution is managed by the Invoker class, which handles the transition from Python logic to external system processes.

Invoker Capabilities

• Blocking Execution: The run method uses subprocess.run with check=True, causing an immediate failure if the child process returns a non-zero exit code.
• Asynchronous Logging: The run_and_log method utilizes asyncio to capture stdout and stderr streams in real-time, forwarding them to the configured logger.
• Signal Handling: On non-Windows platforms, the invoker captures SIGINT and forwards it to the child process to ensure graceful shutdowns.

Sources: meltano/edk/process.py:44-142, tests/test_process.py:44-65

Extension Discovery (Describe)

The describe phase allows Meltano to discover what an extension can do. This is critical for the lifecycle as it informs the system of available subcommands and whether the extension supports pass-through CLI arguments.

Logging and Configuration

Logging is initialized before or during the extension lifecycle to ensure all phases are observable. The EDK provides pass_through_logging_config, which configures structlog based on environment variables.

Logging Configuration Environment Variables

Variable	Description	Default
LOG_LEVEL	Minimum log level (INFO, DEBUG, etc.)	INFO
LOG_TIMESTAMPS	Toggle ISO timestamps in logs	False
LOG_LEVELS	Toggle log level labels in output	False
MELTANO_LOG_JSON	Toggle JSON-formatted output	False

Sources: meltano/edk/logging.py:112-132, meltano/edk/process.py:15-35

Summary

The Meltano EDK lifecycle provides a standardized framework for wrapping external tools. By implementing the ExtensionBase interface, developers ensure their extensions support on-demand initialization, capability description, and a robust execution pipeline including pre- and post-invocation hooks. The underlying Invoker and logging utilities handle the complexities of subprocess management and observability, allowing the extension to integrate seamlessly into the Meltano ecosystem.

Sources: meltano/edk/extension.py, meltano/edk/process.py, README.md