Meltano Integration Tests

Relevant source files

The following files were used as context for generating this wiki page:

• tests/test_meltano.py
• tests/test_integration.py
• tests/conftest.py
• tests/test_backwards_compat.py
• tests/test_state.py
• src/meltano_state_backend_postgresql/backend.py
• tests/test_conninfo.py

Introduction

The Meltano Integration Tests provide a comprehensive suite for validating the meltano-state-backend-postgresql extension. These tests ensure that the PostgreSQL backend correctly implements the Meltano StateStoreManager interface, allowing for persistent storage, retrieval, and locking of job states within a real database environment.

The testing architecture utilizes testcontainers to spin up ephemeral PostgreSQL instances, ensuring that tests are isolated and reproducible. The scope includes validating connection string parsing, schema and table creation, state CRUD operations, and backwards compatibility with Meltano's legacy systemdb schema.
Sources: tests/test_integration.py:27-31, tests/conftest.py:24-28, tests/test_backwards_compat.py:29-35

Test Environment Architecture

The integration test environment is built upon pytest fixtures that manage the lifecycle of a PostgreSQL container and the Meltano project context.

Database Orchestration

The primary fixture, postgres_container, uses testcontainers.postgres.PostgresContainer to launch a "postgres:16" image. This container provides the live database environment required for true integration testing.
Sources: tests/conftest.py:24-28

The diagram above illustrates how the test runner orchestrates the container and the state backend manager during a test session.
Sources: tests/test_integration.py:44-53, tests/conftest.py:24-28

Integration Test Scenarios

The TestIntegration class validates various configurations and operational states of the backend.

Schema and Table Configurations

Tests are parameterized to cover different combinations of schemas and table names, ensuring the table_identifier logic correctly handles custom inputs and URL-encoded search paths.

Parameter	Description	Source
schema	The PostgreSQL schema name to create and use.	tests/test_integration.py:34
table	Custom name for the state table (defaults to state).	tests/test_integration.py:35
schema_in_options	Boolean indicating if the schema is passed via ?options=-csearch_path=.	tests/test_integration.py:36

Sources: tests/test_integration.py:33-42, src/meltano_state_backend_postgresql/backend.py:175-182

State Operations Flow

The integration tests verify the end-to-end flow of state data from Meltano objects to PostgreSQL storage.

Key Test Categories

Backwards Compatibility

The TestBackwardsCompat class ensures that the new backend can read state written by Meltano's legacy systemdb (SQLAlchemy-based) backend. It achieves this by:

1. Running MigrationService to create legacy tables.
2. Writing state using the legacy manager.
3. Switching configuration to the new PostgreSQL backend.
4. Verifying the state is successfully retrieved from the shared schema.
   Sources: tests/test_backwards_compat.py:65-102

Concurrency and Locking

Tests validate the acquire_lock method, which uses PostgreSQL advisory locks (pg_try_advisory_lock). The suite ensures that:

• Locks are held during the execution of the context manager.
• Locks are successfully released (pg_advisory_unlock).
• StateIDLockedError is raised if a lock cannot be acquired within 30 seconds.
  Sources: src/meltano_state_backend_postgresql/backend.py:354-386, tests/test_state.py:133-176

Connection and Validation

tests/test_conninfo.py focuses on the robustness of the connection string building and validation logic.

Test Case	Validation Logic
test_missing_user_validation	Raises MissingStateBackendSettingsError if user is omitted.
test_uri_port_parsing	Validates port extraction from URI vs explicit kwargs.
test_uri_sslmode_from_query_param	Ensures sslmode query parameters are respected.

Sources: tests/test_conninfo.py:107-135, src/meltano_state_backend_postgresql/backend.py:112-136

Code Snippet: Table Identifier Validation

The integration suite uses a helper to verify that the generated SQL identifiers correctly match the expected PostgreSQL format based on input parameters:

# tests/test_integration.py:65-79
def _assert_table_identifier(self, identifier, schema, table, schema_in_options):
    match schema, table, schema_in_options:
        case None, None, _:
            assert identifier.as_string() == f'"{DEFAULT_TABLE_NAME}"'
        case None, table, _:
            assert identifier.as_string() == f'"{table}"'
        case schema, None, False:
            assert identifier.as_string() == f'"{schema}"."{DEFAULT_TABLE_NAME}"'
        case schema, None, True:
            assert identifier.as_string() == f'"{DEFAULT_TABLE_NAME}"'

Sources: tests/test_integration.py:65-79

Conclusion

The Meltano Integration Tests ensure the meltano-state-backend-postgresql extension provides a reliable, performant, and backwards-compatible state storage solution. By leveraging real database containers and covering edge cases like connection URI parsing and advisory locking, the test suite guarantees that job states are managed safely across various deployment environments.