Building the Extension Package

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• pyproject.toml
• copier.yml
• meltano/edk/extension.py
• meltano/edk/models.py
• meltano/edk/process.py
• meltano/edk/logging.py

The Meltano Extension Developer Kit (EDK) provides a standardized framework and scaffolding for building custom Meltano extensions. These extensions typically wrap existing CLI tools (such as dbt or cron) to make them compatible with Meltano's lifecycle and orchestration capabilities.

Building an extension involves using a copier template to initialize the project structure, implementing a subclass of ExtensionBase to define behavior, and configuring the package metadata via pyproject.toml. The EDK simplifies this by providing built-in utilities for subprocess management, logging, and capability description.
Sources: README.md:1-12, meltano/edk/extension.py:32-40

Project Initialization

Creating a new extension begins with the EDK's copier template. This template automates the creation of the necessary directory structure and configuration files required for a Python-based Meltano extension.

Scaffolding with Copier

Developers use the copier tool to initialize a project from the EDK repository. This process prompts for several configuration variables that define the extension's identity and the target CLI it wraps.

Variable	Description	Default / Example
extension_name	The name of the extension	MyExtensionName
extension_id	The ID used by Meltano	{{ extension_name_lower }}-ext
wrapper_target_name	The CLI command being wrapped	dbt
library_name	Python package name	myextensionname_ext
cli_prefix	Command prefix for the CLI	{{ extension_name_lower }}

Sources: copier.yml:1-35, README.md:18-32

The following diagram illustrates the initialization flow:

The initialization flow transitions from environment setup to project generation and dependency synchronization.
Sources: README.md:20-35, copier.yml:1-10

Extension Architecture

Extensions are built by inheriting from the ExtensionBase abstract class. This class defines the lifecycle methods that the EDK calls when the extension is invoked.

Core Lifecycle Methods

An extension implementation must define how it interacts with the wrapped tool through these primary hooks:

• pre_invoke: Logic executed before the main command.
• invoke: The primary execution logic where the wrapped tool is called.
• post_invoke: Cleanup or reporting logic executed after the command.
• describe: Returns a models.Describe object detailing the extension's capabilities and supported commands.
• initialize: An optional on-demand initialization method.

Sources: meltano/edk/extension.py:42-106, meltano/edk/models.py:38-42

Invocation Flow

When an extension is executed, the EDK manages the transition between lifecycle states. The pass_through_invoker utility provides a standard wrapper that ensures pre_invoke, invoke, and post_invoke are called in order, with error handling that exits the process on failure.

The sequence diagram shows the standard execution pattern managed by the EDK's invoker logic.
Sources: meltano/edk/extension.py:126-166, meltano/edk/process.py:53-70

Subprocess Management

The EDK includes an Invoker class in process.py to handle the execution of external binaries. This utility supports both blocking execution and asynchronous execution with real-time log streaming.

Invoker Capabilities

• run(): A synchronous wrapper around subprocess.run that uses check=True to raise CalledProcessError on failure.
• run_and_log(): An asynchronous execution method that captures stdout and stderr and redirects them to the extension's configured logger.
• Signal Handling: On non-Windows platforms, the invoker captures SIGINT and propagates it to the child process.

Sources: meltano/edk/process.py:53-155

Metadata and Configuration

The pyproject.toml file serves as the manifest for the extension package. It defines dependencies, build systems, and entry points.

Package Metadata

The EDK itself relies on hatchling as the build backend and hatch-vcs for dynamic versioning. Required dependencies for any extension using the EDK include structlog, PyYAML, and devtools.

[build-system]
requires = ["hatchling>=1,<2", "hatch-vcs"]
build-backend = "hatchling.build"

[project]
name = "meltano.edk"
dependencies = [
    "structlog>=21,<27",
    "PyYAML>=6,<7",
    "devtools>=0.9.0,<1",
]

Sources: pyproject.toml:1-55, pyproject.toml:116-120

Extension Capabilities (Describe)

Extensions must implement the describe method to return a Describe model. This metadata allows Meltano to understand what the extension can do.

Class	Attribute	Description
Describe	commands	List of supported Command objects
ExtensionCommand	pass_through_cli	Boolean indicating if arbitrary args are accepted
ExtensionCommand	commands	List of specific subcommands (default: describe, invoke, etc.)

Sources: meltano/edk/models.py:7-42

Logging Configuration

Standardized logging is provided through logging.py. The EDK supports structured logging via structlog, which can be configured to output either human-readable console text or machine-readable JSON.

The pass_through_logging_config() function automatically configures the logger based on environment variables commonly used in Meltano environments:

• LOG_LEVEL: Controls verbosity (DEBUG, INFO, etc.).
• MELTANO_LOG_JSON: Enables JSON formatting for production logging.
• LOG_TIMESTAMPS: Toggles ISO timestamps in log entries.

Sources: meltano/edk/logging.py:118-136, meltano/edk/logging.py:65-90

Summary

Building an extension package with the Meltano EDK involves scaffolding a project via copier, implementing the ExtensionBase lifecycle, and utilizing the Invoker for subprocess execution. By following this framework, developers ensure their extensions provide consistent metadata via the describe interface and follow Meltano's standard logging and error handling patterns.