Building CLI Wrappers
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano Extension Developer Kit (EDK) provides a standardized framework for building extensions that wrap existing Command Line Interface (CLI) tools. By using the EDK, developers can create "wrappers" that integrate seamlessly into the Meltano ecosystem, providing structured logging, configuration management, and a consistent lifecycle for command execution.
The core purpose of a CLI wrapper in the EDK is to bridge the gap between Meltano's requirements and the underlying tool's capabilities. This is achieved through the ExtensionBase class for lifecycle management and the Invoker class for robust subprocess handling.
Sources: README.md:1-12, meltano/edk/extension.py:27-30
Architecture and Components
Building a CLI wrapper involves several key architectural components that manage the definition, execution, and output of the wrapped tool.
The Extension Base Class
The ExtensionBase is an abstract base class that defines the interface every Meltano extension must implement. It provides the hooks necessary to manage the lifecycle of a command execution.
| Method | Description |
|---|---|
pre_invoke | Logic executed before the main command, such as validation or environment setup. |
invoke | The primary execution logic where the wrapped CLI is actually called. |
post_invoke | Cleanup or reporting logic executed after the command finishes. |
describe | Returns a Describe model explaining the extension's capabilities and commands. |
Sources: meltano/edk/extension.py:27-104
Subprocess Management
The Invoker class is the primary utility for executing the target CLI. It wraps Python's subprocess and asyncio modules to provide both blocking and non-blocking execution modes.
The Invoker ensures that environment variables and the working directory are correctly passed to the wrapped process.
Sources: meltano/edk/process.py:44-162
Command Invocation Flow
The EDK uses a structured flow to ensure that wrappers handle errors gracefully and provide consistent logging. The pass_through_invoker method is a high-level helper that orchestrates the entire lifecycle.
The Lifecycle Sequence
When a user invokes a command through the extension, the following sequence occurs:
If any stage of the lifecycle (pre-invoke, invoke, or post-invoke) raises an unhandled exception, the EDK is designed to log the failure and hard-exit the process with a status code of 1.
Sources: meltano/edk/extension.py:126-160, meltano/edk/process.py:157-162
Configuration and Models
Wrappers are defined using specific data models that inform Meltano about what the extension can do. These are primarily defined in models.py.
Command Types
There are three main models used to describe functionality:
- ExtensionCommand: Describes the standard CLI commands provided by the extension itself (e.g.,
describe,initialize). - InvokerCommand: Specifically used for "pass-through" functionality where arguments are passed directly to the wrapped tool (using
:splat). - Describe: A container model that holds the list of available commands.
| Field | Type | Description |
|---|---|---|
name | str | The name of the command. |
description | str | A brief explanation of the command. |
pass_through_cli | bool | Whether the command allows arbitrary arguments to be passed to the underlying CLI. |
Sources: meltano/edk/models.py:6-50
Logging Configuration
For a CLI wrapper to be useful, its output must be captured and formatted correctly. The EDK provides a pass_through_logging_config that configures structlog based on environment variables.
- LOG_LEVEL: Sets the minimum log level (default: INFO).
- MELTANO_LOG_JSON: If "True", outputs logs in JSON format for machine parsing.
- LOG_TIMESTAMPS: Includes ISO timestamps in log entries.
The Invoker class specifically handles the streaming of stdout and stderr from the wrapped CLI into the EDK's logging system, ensuring that output from the underlying tool is prefixed or formatted according to the extension's configuration.
Sources: meltano/edk/logging.py:120-137, meltano/edk/process.py:101-114
Project Initialization
Creating a new CLI wrapper is streamlined through a Copier template. This template prompts for key information required to build the wrapper:
- extension_name: The display name of the extension.
- wrapper_target_name: The actual binary name of the command being wrapped (e.g.,
dbtorterraform). - cli_prefix: The prefix used when calling the extension via Meltano.
Sources: copier.yml:1-35, README.md:18-35
Summary
Building a CLI wrapper with the Meltano EDK involves extending ExtensionBase to define lifecycle hooks and using the Invoker class to execute an underlying binary. This structure provides a robust foundation for error handling, standardized logging, and command discovery, allowing external tools to function as first-class citizens within a Meltano project.
Sources: meltano/edk/extension.py:27-30, meltano/edk/process.py:44-46