Per-Backend Connection Settings

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/meltano_state_backend_clickhouse/backend.py
• pyproject.toml
• src/meltano_state_backend_clickhouse/init.py
• tests/test_backend.py

Per-backend connection settings allow for granular configuration of the ClickHouse state backend in Meltano. While the primary configuration is typically handled via a Connection URI, these individual settings provide a mechanism to override specific connection parameters such as host, port, credentials, and table names. This flexibility is critical for environments where URI-based configuration is insufficient or where specific ClickHouse-native features like TLS (Secure) or custom schemas are required.

Sources: README.md:23-28, src/meltano_state_backend_clickhouse/backend.py:84-142

Configuration Hierarchy and Resolution

The ClickHouse state backend resolves connection parameters by combining values from the MELTANO_STATE_BACKEND_URI and optional per-backend settings. Per-backend settings explicitly defined in meltano.yml or via environment variables take precedence over values parsed from the URI.

The diagram shows how URI-parsed values are combined with explicit overrides to form the final client configuration.
Sources: src/meltano_state_backend_clickhouse/backend.py:102-142, README.md:23-25

Connection Parameter Mapping

The function connection_params_from_uri is responsible for merging URI components with explicit overrides. It ensures that required fields—username and database—are present before returning a dictionary suitable for the clickhouse-connect client.

Sources: src/meltano_state_backend_clickhouse/backend.py:102-142

Setting Name	URI Component	Override Variable	Default	Kind
host	hostname	CLICKHOUSE_HOST	localhost	String
port	port	CLICKHOUSE_PORT	8123	Integer
database	path	CLICKHOUSE_DATABASE	(Required)	String
user	username	CLICKHOUSE_USER	(Required)	String
password	password	CLICKHOUSE_PASSWORD	""	String (Sensitive)
secure	N/A	CLICKHOUSE_SECURE	None	Boolean

Sources: src/meltano_state_backend_clickhouse/backend.py:84-135, pyproject.toml:30-37

State Storage Customization

Beyond connection credentials, the backend supports customizing where the state and lock data are persisted within ClickHouse. This is managed via two specific settings that do not originate from the URI.

Schema and Table Settings

• state_backend.clickhouse.schema: Defines the ClickHouse database/schema used for state storage. It defaults to meltano.
• state_backend.clickhouse.table: Defines the base table name for state storage. It defaults to state.

When these are configured, the backend automatically generates the necessary identifiers for both the state table and the associated lock table (e.g., <schema>.<table_name>_lock).

Sources: src/meltano_state_backend_clickhouse/backend.py:137-142, src/meltano_state_backend_clickhouse/backend.py:175-177

The class diagram illustrates how configuration settings are internalized by the manager to construct table identifiers.
Sources: src/meltano_state_backend_clickhouse/backend.py:151-180

Implementation Details

The settings are registered as Meltano entry points, allowing the Meltano core to recognize and provide them to the ClickhouseStateStoreManager during initialization.

Entry Point Definitions

[project.entry-points."meltano.settings"]
host = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_HOST"
port = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_PORT"
database = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_DATABASE"
user = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_USER"
password = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_PASSWORD"
secure = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_SECURE"
schema = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_SCHEMA"
table = "meltano_state_backend_clickhouse.backend:CLICKHOUSE_TABLE"

Sources: pyproject.toml:29-37

Manager Initialization

During the __init__ phase of ClickhouseStateStoreManager, the URI and all optional overrides are passed to the parameter builder. If no schema or table is provided, defaults are applied:

# From src/meltano_state_backend_clickhouse/backend.py:175-177
self.schema = schema or DEFAULT_SCHEMA_NAME
self.table = table or DEFAULT_TABLE_NAME
self.state_table = f"`{self.schema}`.`{self.table}`"
self.lock_table = f"`{self.schema}`.`{self.table}_lock`"

Sources: src/meltano_state_backend_clickhouse/backend.py:175-177

Summary

Per-backend connection settings for ClickHouse provide a robust way to override URI-based configurations. By defining settings for host, port, security, and storage location, users can tailor the backend to complex infrastructure requirements while maintaining default simplicity via the state_backend.uri parameter.

Sources: README.md:23-28, src/meltano_state_backend_clickhouse/backend.py:84-142