Local Development Setup
Relevant source files
The following files were used as context for generating this wiki page:
The meltano-state-backend-postgresql project is a Meltano extension designed to facilitate state storage within a PostgreSQL database. Local development setup focuses on establishing a Python environment compatible with Meltano 3.7+ and configuring a local PostgreSQL instance for integration testing and state management verification.
The environment utilizes uv for dependency management and tox for orchestrating multi-version Python testing. Developers interact with the PostgreSQLStateStoreManager class, which handles connection lifecycles, table initialization, and state persistence operations.
Sources: README.md:1-10, pyproject.toml:21-25, src/meltano_state_backend_postgresql/backend.py:126-128
Environment Initialization
Development requires Python 3.10 or higher. The project uses the hatchling build system and manages dependencies through specific groups defined in pyproject.toml, including dev, lint, tests, and types.
Dependency Management
To synchronize the local environment and install all necessary development dependencies, use the following command:
uv sync
Sources: README.md:95-97, pyproject.toml:1-55
Development Tooling
| Tool | Purpose | Source |
|---|---|---|
| uv | Fast Python package installer and resolver | README.md:96 |
| tox | Automation of testing across multiple Python versions | pyproject.toml:172-230 |
| ruff | Linting and code formatting | pyproject.toml:57-123 |
| mypy | Static type checking | pyproject.toml:130-141 |
| pytest | Test execution framework | pyproject.toml:147-156 |
Infrastructure and Database Setup
For local testing and development, the project relies on Dockerized PostgreSQL instances managed via testcontainers. This ensures a clean, isolated database environment for every test run.
Local Database Container
The integration test suite automatically provisions a postgres:16 container. Developers do not need to manually install PostgreSQL if Docker is available.
@pytest.fixture(scope="module")
def postgres_container() -> Generator[PostgresContainer, None, None]:
"""Start a PostgreSQL container for integration tests."""
with PostgresContainer("postgres:16", driver=None) as postgres:
yield postgres
Sources: tests/conftest.py:24-28, pyproject.toml:54
State Schema Representation
The backend automatically manages a specific table structure to store Meltano state. Upon initialization, the PostgreSQLStateStoreManager executes a CREATE TABLE IF NOT EXISTS command.
The diagram above represents the schema created by the _ensure_tables method.
Sources: src/meltano_state_backend_postgresql/backend.py:228-237
Configuration for Development
Developers can configure the backend using either a full connection URI or individual environment variables. The meltano.yml fixture provides a template for local development configuration.
Example Configuration (meltano.yml)
state_backend:
uri: postgresql://user:password@localhost:5432/meltano
postgresql:
sslmode: prefer
Sources: fixtures/project/meltano.yml:1-9
Environment Variables
The following environment variables can be used to override settings during local development:
MELTANO_STATE_BACKEND_POSTGRESQL_HOSTMELTANO_STATE_BACKEND_POSTGRESQL_PORTMELTANO_STATE_BACKEND_POSTGRESQL_USERMELTANO_STATE_BACKEND_POSTGRESQL_PASSWORDMELTANO_STATE_BACKEND_POSTGRESQL_DATABASEMELTANO_STATE_BACKEND_POSTGRESQL_SCHEMA
Sources: src/meltano_state_backend_postgresql/backend.py:48-111, tests/test_meltano.py:34-72
Testing and Verification
The project enforces high code quality through a strict testing regimen, including integration tests that run against real PostgreSQL instances and unit tests with mocked connections.
Test Execution Flow
The development workflow involves running linting, type checks, and functional tests. The project aims for 100% coverage.
Sources: README.md:101-110, pyproject.toml:163-167
Running Tests
To run the full suite for a specific Python version (e.g., 3.12):
uvx --with tox-uv --with tox-gh tox -e lint,types,3.12
Integration tests are restricted to Linux environments during CI but can be run locally on any system with Docker support.
Sources: README.md:103-105, tests/test_integration.py:22-25
Summary
The local development setup for meltano-state-backend-postgresql is designed for high automation using uv and tox. By leveraging testcontainers, the project ensures that developers can run complex integration tests against a PostgreSQL instance without manual database configuration, maintaining a robust and scalable state management solution for Meltano.
Sources: README.md:95-110, pyproject.toml:172-230