Configuration and State Management
Relevant source files
The following files were used as context for generating this wiki page:
Configuration and state management within the Meltano Extension Developer Kit (EDK) revolve around defining how an extension describes its capabilities, manages its runtime environment, and handles logging configurations. The EDK provides a structured interface via the ExtensionBase class and supporting models to ensure consistent behavior across different extension implementations.
This system facilitates the interaction between Meltano and custom extensions by providing mechanisms for command discovery (via describe), environment-aware process execution (via Invoker), and standardized logging through environment variables.
Sources: meltano/edk/extension.py:1-35, meltano/edk/models.py:1-45
Extension Capabilities and Metadata
The EDK uses a set of dataclasses to model the commands and capabilities an extension provides. This metadata allows Meltano to understand how to interact with the extension, whether it supports pass-through CLI arguments, or requires specific lifecycle hooks.
Data Models for Capabilities
The core of the description system is the Describe model, which contains a list of commands. Commands are categorized into generic commands, extension-specific commands, and invoker-style commands.
| Field | Type | Description |
|---|---|---|
name | str | The unique name of the command. |
description | str | A human-readable description of what the command does. |
commands | list[str] | A list of sub-commands or capabilities supported. |
pass_through_cli | bool | Indicates if the command accepts arbitrary CLI arguments. |
Sources: meltano/edk/models.py:7-45
The following class diagram illustrates the relationship between these models:
The diagram shows that ExtensionCommand and InvokerCommand inherit from the base Command class, providing default values for capabilities and lifecycle hooks.
Sources: meltano/edk/models.py:7-45
Lifecycle and Invocation State
State management during the execution of an extension is handled through a defined lifecycle. The ExtensionBase class enforces a structure that includes initialization, pre-invocation, invocation, and post-invocation phases.
Invocation Flow
When an extension is triggered, the pass_through_invoker method manages the execution flow, ensuring that hooks are called in the correct sequence.
The flow diagram represents the sequential execution of lifecycle hooks and the terminal state in case of unhandled exceptions.
Sources: meltano/edk/extension.py:86-130
Execution Environment
The Invoker class manages the state of the subprocess environment, including the working directory (cwd) and environment variables (env). If no environment is provided, it defaults to a copy of the current os.environ.
Sources: meltano/edk/process.py:42-59
Logging Configuration
Configuration for logging is primarily managed through environment variables, allowing Meltano to control the verbosity and format of extension logs without modifying the extension code.
Supported Logging Environment Variables
| Variable | Default | Description |
|---|---|---|
LOG_LEVEL | INFO | Sets the logging threshold (DEBUG, INFO, WARNING, ERROR, CRITICAL). |
LOG_TIMESTAMPS | False | Enables/disables ISO timestamps in logs. |
LOG_LEVELS | False | Enables/disables the inclusion of the log level in output. |
MELTANO_LOG_JSON | False | Toggles between human-readable (Console) and JSON formatting. |
Sources: meltano/edk/logging.py:157-175
Logging Initialization Logic
The pass_through_logging_config function retrieves these environment variables and applies them to the structlog and stdlib logging systems.
This diagram outlines how environment variables are parsed to configure processors and renderers for the extension's logging system.
Sources: meltano/edk/logging.py:84-154
Template Configuration
The EDK uses copier for project initialization, which involves a set of configuration questions that define the extension's initial state and identity.
Copier Variables
| Variable | Type | Description |
|---|---|---|
extension_name | str | The name of the extension (e.g., MyExtensionName). |
extension_id | str | The ID of the wrapper, defaults to extension_name_lower-ext. |
wrapper_target_name | str | The path or name of the CLI command being wrapped. |
library_name | str | The Python library name for the extension. |
Sources: copier.yml:1-38
Summary
Configuration and state in the Meltano EDK are handled through a combination of structured metadata classes, a defined execution lifecycle, and environment-driven logging. By implementing the ExtensionBase and using the Invoker utility, developers can ensure their extensions behave predictably within the Meltano ecosystem, while the logging system provides flexible observability controlled via standard environment variables.