Real-world Extension Examples
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Extension Developer Kit (EDK) provides a standardized framework for building custom Meltano extensions. These extensions act as wrappers around existing CLI tools, allowing them to be integrated seamlessly into Meltano projects as utilities. While the EDK itself is a library, its design is best understood through its implementation patterns, such as the dbt-ext and cron-ext examples, and the automated templates used to scaffold new projects.
The EDK architecture centers on the ExtensionBase class, which defines a lifecycle for command execution, and the Invoker class, which handles the technical details of subprocess management. This structure ensures that any wrapped tool follows the same operational patterns for initialization, invocation, and description.
Sources: README.md:1-29, meltano/edk/extension.py:30-41
Extension Lifecycle and Architecture
Real-world extensions built with the EDK follow a specific execution lifecycle. This lifecycle allows developers to inject logic before and after the wrapped command is executed.
Core Lifecycle Methods
The ExtensionBase class enforces or provides hooks for the following stages:
| Method | Purpose | Implementation Requirement |
|---|---|---|
initialize | Prepares the extension environment (e.g., config generation). | Optional (No-op by default) |
pre_invoke | Actions to take immediately before the tool runs. | Optional |
invoke | The primary execution logic for the wrapped command. | Required (Abstract) |
post_invoke | Cleanup or reporting actions after execution. | Optional |
describe | Returns metadata about supported commands and capabilities. | Required (Abstract) |
Sources: meltano/edk/extension.py:44-93, meltano/edk/models.py:27-40
Execution Flow Diagram
The following diagram illustrates how the EDK manages the transition between these lifecycle stages during a standard invocation.
The flow shows the optional initialization stage followed by the mandatory execution sequence used by the pass_through_invoker.
Sources: meltano/edk/extension.py:112-146, tests/test_meltano_edk.py:69-76
Implementation Example: Wrapper Extensions
A common real-world use case for the EDK is creating a "pass-through" wrapper for existing tools like dbt. This is achieved by implementing ExtensionBase and utilizing the Invoker utility to bridge the EDK and the target binary.
The Invoker Utility
The Invoker class handles subprocess execution, environment variable injection, and asynchronous logging of stdout and stderr.
run(): A blocking call for simple command execution.run_and_log(): An asynchronous execution that streams subprocess output to the extension's logger.
Sources: meltano/edk/process.py:45-135
Sequence of a Wrapped Invocation
This sequence diagram shows how an extension interacts with the Invoker to execute a third-party CLI command.
Scaffolding Real-world Projects
The EDK uses Copier to provide a template for new extensions. This template demonstrates the standard project structure required for a functional extension.
Template Configuration
The copier.yml file defines the variables necessary to bootstrap an extension that wraps a specific target command.
| Variable | Description | Example |
|---|---|---|
extension_name | Human-readable name | Dbt |
extension_id | Unique identifier for Meltano | dbt-ext |
wrapper_target_name | The actual CLI command being wrapped | dbt |
cli_prefix | The command prefix for the extension CLI | dbt |
Sources: copier.yml:1-42, tests/copier_template/test_generated.py:38-51
Generated Project Structure
When a developer uses copier copy gh:meltano/edk, the EDK generates a Python project with pyproject.toml dependencies and a class inheriting from ExtensionBase. The generated tests ensure that the new extension correctly implements the describe and invoke patterns.
Sources: README.md:15-25, tests/copier_template/test_generated.py:53-76
Summary
Real-world Meltano EDK extensions function as structured wrappers that standardize how external tools are called and logged. By implementing the ExtensionBase lifecycle and utilizing the Invoker for subprocess management, extensions provide a consistent interface for Meltano utilities. The provided Copier template serves as the primary "real-world" blueprint, ensuring that all new extensions follow the architectural requirements of the kit.