Local Development Setup

Relevant source files

The following files were used as context for generating this wiki page:

• pyproject.toml
• README.md
• src/meltano_state_backend_clickhouse/backend.py
• src/meltano_state_backend_clickhouse/init.py
• tests/test_backend.py

The meltano-state-backend-clickhouse project provides a ClickHouse-based state store for Meltano, enabling persistence for pipeline state such as incremental bookmarks and full-table markers. Because ClickHouse is an OLAP store, the development environment focuses on simulating row-level consistency through specific engine choices like ReplacingMergeTree and custom advisory locking logic.

Local development requires a Python environment (>= 3.10) and access to a ClickHouse instance for testing. The setup leverages hatch for building and pytest with testcontainers for integration testing.

Sources: README.md:5-10, pyproject.toml:13-20, src/meltano_state_backend_clickhouse/backend.py:3-14

Development Environment Dependencies

The project uses hatchling as the build backend and defines specific dependency groups for development and testing.

Category	Requirement	Description
Python	>=3.10	Supported versions up to 3.13
Build System	hatch-vcs, hatchling	Version control integration and build backend
Core Dependencies	clickhouse-connect>=0.7, meltano>=3.7	Interface for ClickHouse and Meltano base classes
Test Dependencies	pytest>=8, testcontainers[clickhouse]>=4.8	Testing framework and ephemeral container support

Sources: pyproject.toml:1-51

ClickHouse Infrastructure Requirements

For local development and testing, a ClickHouse instance is required. The test suite defaults to a local instance but can be overridden via environment variables.

Database Schema

The backend automatically initializes two tables within a specified schema (default: meltano):

1. State Table: Stores actual pipeline state.
2. Lock Table: Manages best-effort advisory locks.

Configuration for Development

Developers can configure the backend using a URI or individual environment variables. The ClickhouseStateStoreManager parses these settings to establish a connection via clickhouse-connect.

Connection Parameters

Parameter	Setting Name	Default
host	state_backend.clickhouse.host	localhost
port	state_backend.clickhouse.port	8123
database	state_backend.clickhouse.database	Required (no default)
user	state_backend.clickhouse.user	Required (no default)
secure	state_backend.clickhouse.secure	False

Sources: src/meltano_state_backend_clickhouse/backend.py:65-125, src/meltano_state_backend_clickhouse/backend.py:161-165

URI Example

For local testing against a default ClickHouse setup:

export MELTANO_STATE_BACKEND_URI="clickhouse://default:clickhouse@localhost:8123/meltano"

Sources: tests/test_backend.py:21-24, README.md:38-40

Testing Procedures

The test suite uses pytest and handles the lifecycle of test tables automatically.

Integration Tests Flow

The testing process involves creating unique tables for each test run to ensure isolation, especially since ClickHouse does not support standard transactions.

Running Tests

To run the full suite locally:

1. Ensure ClickHouse is running.
2. Set the CH_TEST_URI environment variable if not using defaults.
3. Execute pytest.

Sources: tests/test_backend.py:11-24, pyproject.toml:53-56

Advisory Locking Logic

A critical part of local development is understanding the "best-effort" lock mechanism. Because ClickHouse lacks INSERT uniqueness constraints, the backend implements a "settle-then-verify" pattern.

Local development on this backend requires careful attention to ClickHouse's eventual consistency and merge behavior, specifically how ReplacingMergeTree interacts with the FINAL keyword during state retrieval.

Sources: src/meltano_state_backend_clickhouse/backend.py:246-250, README.md:46-49