Installation Guide
Relevant source files
The following files were used as context for generating this wiki page:
The meltano-state-backend-clickhouse package is a Meltano state backend add-on designed to persist pipeline state—such as incremental bookmarks and full-table markers—within a ClickHouse warehouse. This module brings ClickHouse support to parity with other major database backends like Postgres, Snowflake, and BigQuery.
Sources: README.md:1-7, pyproject.toml:9, src/meltano_state_backend_clickhouse/backend.py:1-4
The installation process involves adding the package to your environment and configuring Meltano to use the clickhouse scheme. This guide covers requirements, installation steps, and the necessary configuration settings to establish a connection between Meltano and your ClickHouse instance.
Sources: README.md:9-15, pyproject.toml:50-58
Prerequisites
Before installing, ensure your environment meets the following requirements:
- Python: Version 3.10 or higher.
- Meltano: Version 3.7 or higher.
- ClickHouse Connect: Version 0.7 or higher (automatically handled as a dependency).
Sources: pyproject.toml:20-30
Installation Steps
Standard Installation
Install the package directly from the GitHub repository using pip:
pip install "meltano-state-backend-clickhouse @ git+https://github.com/Meltano/meltano-state-backend-clickhouse.git%22
Sources: README.md:11-13
Dependency Graph
The package relies on clickhouse-connect for database communication and meltano for integration with the core platform.
Sources: pyproject.toml:29-31,43-46
Configuration
Meltano must be instructed to use ClickHouse as its state backend via the MELTANO_STATE_BACKEND_URI.
Connection URI
The primary method of configuration is through a URI string:
clickhouse://user:password@host:8123/database
Sources: README.md:19-21, src/meltano_state_backend_clickhouse/backend.py:126-140
Configuration Methods
You can apply configuration via environment variables or the meltano.yml file.
Environment Variable:
export MELTANO_STATE_BACKEND_URI="clickhouse://user:password@host:8123/meltano"
Sources: README.md:19-21
meltano.yml Configuration:
state_backend:
uri: clickhouse://user:password@host:8123/meltano
Sources: README.md:25-27
Available Settings
Individual settings can be used to override values parsed from the URI. These are mapped as Meltano settings.
| Setting Name | Type | Default | Description |
|---|---|---|---|
state_backend.clickhouse.host | String | localhost | ClickHouse server hostname |
state_backend.clickhouse.port | Integer | 8123 | ClickHouse HTTP interface port |
state_backend.clickhouse.database | String | (Required) | ClickHouse database name |
state_backend.clickhouse.user | String | (Required) | ClickHouse username |
state_backend.clickhouse.password | String | "" | ClickHouse password |
state_backend.clickhouse.secure | Boolean | False | Use HTTPS/TLS |
state_backend.clickhouse.schema | String | meltano | Database/Schema for state storage |
state_backend.clickhouse.table | String | state | Table name for state storage |
Sources: src/meltano_state_backend_clickhouse/backend.py:59-105, 175-180, pyproject.toml:32-41
System Architecture
The backend implements a ClickhouseStateStoreManager which handles table creation and state persistence. Because ClickHouse is an OLAP store, it uses specific engines to handle updates and locking.
Table Structure
The manager creates two tables automatically:
- State Table: Uses
ReplacingMergeTree(updated_at)to handle state updates via versioning. - Lock Table: Uses
ReplacingMergeTree(locked_at)with a TTL for advisory locking.
Initialization Flow
When the ClickhouseStateStoreManager is initialized, it parses connection parameters and ensures the required database and tables exist.
Verification
To verify the installation, you can run the provided test suite. The tests use testcontainers to spin up a ClickHouse instance for lifecycle validation.
pytest tests/test_backend.py
Sources: pyproject.toml:46, 57-59, tests/test_backend.py:1-15
The backend is considered correctly installed if it can perform a "set and get" round-trip, which involves inserting a MeltanoState object and retrieving it back with FINAL semantics to ensure the newest version is returned.
Sources: tests/test_backend.py:53-60, src/meltano_state_backend_clickhouse/backend.py:269-281