Generating a New Extension

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• copier.yml
• meltano/edk/extension.py
• meltano/edk/models.py
• meltano/edk/process.py
• pyproject.toml
• tests/copier_template/test_generated.py

The Meltano Extension Developer Kit (EDK) provides a standardized framework and scaffolding system for creating custom Meltano extensions. These extensions act as wrappers around existing CLI tools (such as dbt or custom utilities), allowing them to integrate seamlessly into Meltano workflows.

Generating a new extension involves utilizing a copier-based template that automates the creation of the project structure, dependency management, and the core Python classes required to implement the EDK interface.

Sources: README.md:1-12, pyproject.toml:1-15

Extension Generation Process

The generation process relies on copier to process a template stored within the EDK repository. This template prompts the user for specific configuration values and outputs a functional Python project.

Project Scaffolding

The primary method for starting a new project is the copier copy command directed at the EDK repository. This initializes a directory containing a pyproject.toml, a library structure, and the necessary boilerplate to implement the ExtensionBase class.

The diagram shows the sequential steps a developer takes to move from an empty environment to a ready-to-develop EDK extension project.
Sources: README.md:14-35, tests/copier_template/test_generated.py:34-52

Configuration Variables

The copier.yml file defines the variables required during generation. These variables determine the naming conventions and the CLI tool being wrapped.

Variable	Type	Description	Default
extension_name	string	The CamelCase name of the extension.	N/A
extension_id	string	The ID used for the wrapper (e.g., my-ext).	{{ extension_name_lower }}-ext
library_name	string	The Python package name (underscored).	{{ extension_id.replace('-', '_') }}
cli_prefix	string	The prefix for the generated CLI.	{{ extension_name_lower }}
wrapper_target_name	string	The binary or command being wrapped (e.g., dbt).	N/A

Sources: copier.yml:1-42

Core Architecture Components

Once generated, the extension is built upon several core EDK components that manage the lifecycle of the wrapped command.

Extension Lifecycle

Every generated extension must implement or utilize the ExtensionBase class. This class defines the lifecycle hooks that Meltano calls when executing the extension.

The sequence diagram illustrates the lifecycle of an extension execution, showing how the EDK manages the flow before and after the wrapped binary is called.
Sources: meltano/edk/extension.py:35-95, meltano/edk/process.py:143-165

Command Invocation

The EDK provides an Invoker utility in process.py to handle the actual execution of the wrapper_target_name.

• Invoker.run(): A blocking wrapper around subprocess.run that handles environment variables and working directories.
• Invoker.run_and_log(): An asynchronous implementation that streams stdout and stderr from the wrapped process directly to the extension's logs using structlog.

Sources: meltano/edk/process.py:53-110, meltano/edk/process.py:143-165

Capabilities and Metadata

The describe method is a mandatory implementation that returns a Describe model. This model informs Meltano about the commands and capabilities the extension supports.

@dataclass(slots=True)
class ExtensionCommand(Command):
    description: str = "The extension cli"
    pass_through_cli: bool = False
    commands: list[str] = field(
        default_factory=lambda: [
            "describe", "invoke", "pre_invoke", "post_invoke", "initialize"
        ]
    )

Sources: meltano/edk/models.py:22-35, meltano/edk/extension.py:82-90

Project Structure and Requirements

A newly generated extension includes a specific set of dependencies and metadata defined in the pyproject.toml template.

• Python Version: Requires Python >= 3.10.
• Core Dependencies: Includes structlog for logging, PyYAML for configuration, and devtools for formatting.
• Development Tools: Generated projects are pre-configured with ruff for linting and formatting, and pytest for testing.

Sources: pyproject.toml:7-35, tests/copier_template/test_generated.py:54-79

Summary

Generating a new extension using the Meltano EDK involves initializing a project via the copier template, which sets up a standardized Python environment. This environment includes the ExtensionBase framework for managing command lifecycles and the Invoker utility for executing wrapped binaries. By providing metadata such as the extension_name and wrapper_target_name, developers can quickly create robust integrations that follow Meltano's extension specifications.