Generating superset_config.py

Relevant source files

The following files were used as context for generating this wiki page:

• superset_ext/extension.py
• superset_ext/utils.py
• superset_ext/main.py
• README.md
• tests/test_superset_ext_utils.py

The generation of superset_config.py is a core function of the superset-ext Meltano extension. This process ensures that Apache Superset is correctly configured to run within a Meltano project by translating environment variables and Meltano configurations into a valid Python configuration file required by the Superset application.

This mechanism allows developers to manage Superset settings directly through the meltano.yml file or environment variables, maintaining a single source of truth for project configuration. The file is typically created during the initialization phase or when explicitly triggered by the extension's commands.

Sources: README.md:1-40, superset_ext/extension.py:44-51

Configuration Generation Workflow

The generation process is managed by the Superset extension class and utility functions that handle environment parsing and file I/O.

Environment Preparation

Before writing the file, the extension prepares a dictionary of configuration options. It filters environment variables starting with SUPERSET_, trims the prefix, and sets default values for essential paths such as SUPERSET_HOME and SQLALCHEMY_DATABASE_URI.

The workflow ensures that the environment is "Meltano-aware" by utilizing the MELTANO_PROJECT_ROOT to resolve absolute paths for the database and home directories.

Sources: superset_ext/utils.py:48-84, superset_ext/extension.py:34-42

The write_config Logic

The write_config function in superset_ext/utils.py is responsible for the actual file creation. It generates a Python script that dynamically injects the configuration dictionary into the module's global scope.

config_script_lines = [
    "import sys",
    "module = sys.modules[__name__]",
    f"config = {superset_config!r}",
    "for key, value in config.items():",
    "    if key.isupper():",
    "        setattr(module, key, value)",
    "",
    "# add any additional config here #",
    "",
]

Sources: superset_ext/utils.py:108-118

Key Components and Paths

The extension uses specific paths and environment variables to determine where the configuration is stored and what it contains.

Configuration Resolution Table

Parameter	Default Value	Description
SUPERSET_CONFIG_PATH	analyze/superset/superset_config.py	The target location for the generated file.
SUPERSET_HOME	$MELTANO_PROJECT_ROOT/.meltano/utilities/superset	The home directory for Superset data.
SQLALCHEMY_DATABASE_URI	sqlite:///$SUPERSET_HOME/superset.db	The connection string for the metadata database.
FLASK_APP	superset	Required environment variable for the underlying Flask app.

Sources: superset_ext/utils.py:65-82, superset_ext/extension.py:45-50

Extension Logic Flow

The following sequence diagram illustrates the interaction between the CLI entry point, the Extension class, and the utility functions during the initialization command.

Triggering Generation

There are two primary ways to trigger the generation of the superset_config.py file through the extension:

1. Initialize Command: Running meltano invoke superset_ext:initialize checks for the existence of the config file. If missing (or if --force is used), it writes the file and proceeds to run superset db upgrade and superset init.
2. Force Overwrite: The initialize function accepts a force boolean argument. When true, it bypasses the existence check and overwrites the current superset_config.py with the latest configuration derived from the environment.

Sources: superset_ext/main.py:27-33, superset_ext/extension.py:53-60, README.md:44-48

Summary of Logic

The generation mechanism effectively transforms a static Python dictionary into a dynamic configuration module. By iterating through the dictionary and using setattr(module, key, value), the extension ensures that any uppercase key provided in the Meltano configuration becomes a valid top-level configuration constant in Superset. This allows for seamless integration of settings like ROW_LIMIT, SECRET_KEY, and WEBSERVER_PORT defined in meltano.yml.

Sources: superset_ext/utils.py:111-115, README.md:28-35