Prerequisites and Setup
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Extension Developer Kit (EDK) provides a framework and a standardized template for building custom Meltano extensions. It is designed to wrap existing CLI tools, allowing them to integrate seamlessly into the Meltano ecosystem. This page covers the environmental requirements, project initialization steps, and configuration necessary to start developing with the EDK.
Sources: README.md:1-10, pyproject.toml:1-10, meltano/edk/extension.py:30-40
System Requirements
The EDK is built using Python and requires a modern environment to support its asynchronous processing and type hinting features.
Python Environment
The project specifies a minimum Python version of 3.10. It is officially tested and classified for versions 3.10 through 3.14.
Sources: pyproject.toml:11, pyproject.toml:18-22
Required Tools
- uv: Used for tool installation and dependency synchronization.
- copier: A templating tool used to initialize new projects from the EDK's blueprint.
Sources: README.md:16-20, pyproject.toml:58
Project Initialization
Creating a new extension involves using the copier template provided in the repository. This process automates the creation of the file structure and initial configuration.
Initialization Flow
The following diagram illustrates the sequence of commands used to set up a new extension project:
Sources: README.md:18-35
Copier Configuration Options
When running copier, several variables must be defined to customize the generated extension.
| Variable | Type | Description | Default / Example |
|---|---|---|---|
extension_name | string | The display name of the extension | MyExtensionName |
admin_name | string | Author name for pyproject.toml | Bob Loblaw |
extension_id | string | Unique ID for the wrapper | myextension-ext |
wrapper_target_name | string | The CLI command being wrapped | dbt |
cli_prefix | string | Prefix for the generated CLI | myextension |
Sources: copier.yml:1-35
Dependency Management
The EDK relies on a specific set of core dependencies for its functionality, including structured logging and YAML processing.
Core Dependencies
- structlog (>=21, <27): Used for structured, configurable logging.
- PyYAML (>=6, <7): Handles configuration and description output parsing.
- devtools (>=0.9.0, <1): Used for pretty-printing object descriptions.
Sources: pyproject.toml:31-35
Development Tools
For developers contributing to the EDK itself or running tests, a separate set of tools is required:
- pytest: For running the test suite.
- ruff: Used for linting and formatting the codebase.
- mypy: Provides static type checking.
Sources: pyproject.toml:57-65
Extension Architecture Setup
Once initialized, the extension must implement the ExtensionBase interface. This involves defining how the extension interacts with its wrapped subprocess.
Logging Configuration
Extensions should be initialized with standard logging configurations to ensure compatibility with Meltano's output expectations. This is handled via the pass_through_logging_config function, which reads environment variables.
| Environment Variable | Purpose |
|---|---|
LOG_LEVEL | Sets the verbosity (DEBUG, INFO, etc.) |
MELTANO_LOG_JSON | If "True", outputs logs in JSON format |
LOG_TIMESTAMPS | If "True", includes ISO timestamps in logs |
Sources: meltano/edk/logging.py:118-135
Subprocess Invoker Setup
The Invoker class is the primary utility for executing the wrapped CLI tool. It must be initialized with the path to the binary and optionally a working directory or environment variables.
# Example Invoker Initialization
from meltano.edk.process import Invoker
invoker = Invoker(
bin="dbt",
cwd="/path/to/project",
env={"DBT_PROFILES_DIR": "."}
)
Sources: meltano/edk/process.py:48-60
The EDK is designed to simplify the lifecycle of a Meltano extension by providing standardized project structures and robust utilities for subprocess management and logging. By following the prerequisites and utilizing the copier template, developers can focus on the specific logic of the tool they are wrapping.