Pass-through Execution
Relevant source files
The following files were used as context for generating this wiki page:
Pass-through execution in the superset-ext project allows Meltano to act as a transparent proxy for the underlying Apache Superset CLI. This feature enables users to execute any native Superset command directly through the Meltano interface, ensuring that the full functionality of the original tool remains accessible while benefiting from the environment management provided by the Meltano SDK.
The mechanism is implemented through a combination of an "invoker" script and a dedicated extension command. By leveraging the meltano.edk.process.Invoker class, the extension wraps the native superset binary, injecting the necessary environment variables and configuration files required for it to run within the Meltano project context.
Architecture and Components
The pass-through system consists of two primary entry points: a dedicated shim for direct invocation and a command within the Typer-based extension CLI.
The Invoker Shim
The superset_invoker is a lightweight entry point defined in pyproject.toml. It calls the pass_through_cli function, which initializes the Superset extension class and immediately forwards all system arguments to the underlying binary.
Extension Invocation Logic
Inside the Superset class (inheriting from ExtensionBase), the pass-through logic is managed by an Invoker instance. This instance is pre-configured with the environment variables prepared by the extension's utility functions.
| Component | Responsibility | Source |
|---|---|---|
superset_invoker | The CLI script name used by Meltano to wrap native commands. | pyproject.toml:57 |
pass_through_cli | Function that handles raw sys.argv and passes them to the invoker. | superset_ext/pass_through.py:11 |
Invoker | Meltano EDK class used to run subprocesses with specific environments. | superset_ext/extension.py:34 |
invoke method | A Typer command that allows calling the invoker via meltano invoke superset-ext:invoke. | superset_ext/main.py:75 |
Sources: superset_ext/extension.py:30-34, superset_ext/main.py:75-88
Data Flow and Environment Preparation
Before any command is passed through to the native Superset CLI, the extension must prepare the environment to ensure Superset points to the correct database and configuration files.
- Prefix Trimming: Environment variables starting with
SUPERSET_are gathered and the prefix is removed to create native Superset configuration keys. - Path Resolution: The
SUPERSET_HOMEandSUPERSET_CONFIG_PATHare derived from Meltano project defaults (e.g.,.meltano/utilities/superset) if not explicitly provided. - Config Generation: A
superset_config.pyfile is dynamically written to the filesystem, translating environment-based configuration into a Python module that Superset can import.
Command Configuration
The pass-through behavior is configured in the plugin definition, typically within a meltano.yml file. The executable is set to superset_invoker, which allows standard arguments to be passed directly.
Example Configuration
utilities:
- name: superset_ext
executable: superset_invoker
commands:
invoke:
executable: superset_extension
args: invoke
Sources: README.md:12-25
Handling Extra Arguments
The invoke command in main.py is specifically configured with allow_extra_args=True and ignore_unknown_options=True. This ensures that flags like --help or complex native Superset arguments (e.g., fab create-admin --username=admin) are captured as a list and passed to the invoker without the Typer parser attempting to interpret them.
Sources: superset_ext/main.py:75-77, superset_ext/extension.py:108-118
Summary
Pass-through execution provides a seamless bridge between Meltano and the native Superset CLI. By automating environment preparation and configuration file generation, the extension ensures that any command executed via superset_invoker or the invoke command operates within the context of the Meltano-managed project, while maintaining compatibility with the standard Superset interface.