Scheme Registration (__init__.py)
Relevant source files
The following files were used as context for generating this wiki page:
The Scheme Registration module serves as the primary entry point for the Meltano ClickHouse state backend. Its main purpose is to expose the ClickhouseStateStoreManager to Meltano's plugin architecture, allowing users to persist pipeline state—such as incremental bookmarks and full-table markers—within a ClickHouse warehouse. This brings ClickHouse into parity with other supported backends like Postgres and Snowflake.
Sources: src/meltano_state_backend_clickhouse/init.py:1-6, README.md:4-7
Integration and Entry Points
The project utilizes Python entry points to register itself with Meltano. By defining these in the package configuration, Meltano can automatically discover the ClickHouse scheme when a URI starting with clickhouse:// is provided in the configuration.
Entry Point Configuration
The registration is split into two main categories: the state backend manager itself and the specific settings required to configure the connection.
| Entry Point Group | Name | Reference |
|---|---|---|
meltano.state_backends | clickhouse | meltano_state_backend_clickhouse.backend:ClickhouseStateStoreManager |
meltano.settings | host | meltano_state_backend_clickhouse.backend:CLICKHOUSE_HOST |
meltano.settings | port | meltano_state_backend_clickhouse.backend:CLICKHOUSE_PORT |
meltano.settings | database | meltano_state_backend_clickhouse.backend:CLICKHOUSE_DATABASE |
Sources: pyproject.toml:37-47
Component Relationship
The following diagram illustrates how the __init__.py file acts as the interface between the Meltano core system and the internal backend implementation.
The registration ensures that when a user sets MELTANO_STATE_BACKEND_URI="clickhouse://...", the ClickhouseStateStoreManager is instantiated to handle all state operations.
Sources: pyproject.toml:47, README.md:16-18, src/meltano_state_backend_clickhouse/init.py:5
State Store Management
The core component registered is the ClickhouseStateStoreManager. This class inherits from Meltano's StateStoreManager and implements the specific logic required to interact with ClickHouse's OLAP architecture.
Key Implementation Details
- Schema & Table Defaults: The backend defaults to using the
meltanodatabase and a table namedstate. - ReplacingMergeTree: Because ClickHouse does not support standard row-level updates, the backend uses the
ReplacingMergeTreeengine. This engine manages versioning via anupdated_atcolumn, where the newest record perstate_idwins during aFINALquery.
# From src/meltano_state_backend_clickhouse/backend.py
class ClickhouseStateStoreManager(StateStoreManager):
def __init__(
self,
uri: str,
# ... overrides
schema: str | None = None,
table: str | None = None,
**kwargs: Any,
) -> None:
super().__init__(**kwargs)
# ... initialization logic
self.schema = schema or DEFAULT_SCHEMA_NAME
self.table = table or DEFAULT_TABLE_NAME
Sources: src/meltano_state_backend_clickhouse/backend.py:165-197, README.md:28-32
Connection Parameter Handling
The registration process includes the connection_params_from_uri utility, which parses the Meltano backend URI and merges it with optional granular settings (host, port, user, etc.).
URI Parsing Logic
The following sequence diagram shows how connection parameters are resolved during the initialization triggered by the scheme registration.
Summary
The registration defined in __init__.py and supported by pyproject.toml effectively integrates ClickHouse as a first-class state backend for Meltano. By exporting ClickhouseStateStoreManager, the module enables Meltano to perform state operations (get, set, list, delete) using ClickHouse-specific optimizations like ReplacingMergeTree and best-effort advisory locking to handle concurrency in an OLAP environment.
Sources: src/meltano_state_backend_clickhouse/backend.py:5-15, README.md:31-36