Backwards Compatibility
Relevant source files
The following files were used as context for generating this wiki page:
Backwards compatibility in the meltano-state-backend-postgresql project focuses on ensuring a seamless transition and interoperability between the legacy Meltano System Database (SystemDB) state storage and the dedicated PostgreSQL state backend. This capability allows Meltano to access state data previously managed via SQLAlchemy-based migrations within the same PostgreSQL instance, provided the connection parameters align.
The primary mechanism for maintaining compatibility is the backend's ability to interface with existing PostgreSQL schemas and tables. The backend supports configurable search paths and table names, enabling it to point to the state table created by Meltano's internal MigrationService.
Sources: tests/test_backwards_compat.py:34-40, README.md:38-40
Architecture and Migration Flow
The compatibility layer ensures that state set by the legacy systemdb backend can be retrieved by the PostgreSQLStateStoreManager. This is achieved by aligning the database URI and schema settings. The following diagram illustrates the transition flow from a SystemDB-managed state to the new backend.
The diagram shows how the MigrationService initializes the database schema, which is then accessible by the new backend after configuration updates.
Sources: tests/test_backwards_compat.py:84-110, src/meltano_state_backend_postgresql/backend.py:228-232
Schema and Search Path Interoperability
The backend utilizes PostgreSQL's search_path to resolve table locations. When migrating from SystemDB, the legacy state is often stored in a specific schema created during migrations. The PostgreSQLStateStoreManager can be configured to target these specific locations using two primary methods:
1. URI Options
The search_path can be defined directly in the connection URI using the options parameter. This approach allows the backend to find the legacy state table without qualifying the table name in SQL queries.
Sources: README.md:33-35, tests/test_conninfo.py:46-52
2. Explicit Schema Configuration
The backend also supports an explicit schema setting. If provided, the backend generates qualified identifiers (e.g., "myschema"."state"), which overrides any search_path provided in the URI.
Sources: src/meltano_state_backend_postgresql/backend.py:192-196, tests/test_conninfo.py:73-82
| Configuration Item | Legacy SystemDB Context | New Backend Equivalent |
|---|---|---|
| Database URI | database_uri | state_backend.uri |
| Schema | -csearch_path in URI | state_backend.postgresql.schema |
| Table Name | state (default) | state_backend.postgresql.table |
Sources: tests/test_backwards_compat.py:91-105, README.md:55-65
Data Consistency and Type Mapping
To maintain compatibility with Meltano's internal MeltanoState object, the backend handles specific data type conversions between PostgreSQL's TEXT columns and Python dictionaries.
- JSON Handling: Legacy state stored as text in the database is automatically deserialized using
json.loads(). - Null Handling: The backend explicitly converts
NULLvalues from thepartial_stateandcompleted_statecolumns into empty Python dictionaries{}to prevent runtime errors in Meltano.
# From src/meltano_state_backend_postgresql/backend.py:276-281
return MeltanoState(
state_id=state_id,
partial_state=json.loads(partial_state) if partial_state is not None else {},
completed_state=json.loads(completed_state) if completed_state is not None else {},
)
Sources: src/meltano_state_backend_postgresql/backend.py:269-282, tests/test_integration.py:166-175
Dependency Requirements
The backend is designed to work with modern Meltano versions while maintaining structural compatibility with state created by older versions.
| Requirement | Value |
|---|---|
| Meltano Version | >= 3.7 |
| Python Version | >= 3.10 |
| Context Manager Support | Required for Meltano < 4.2.0 |
Sources: pyproject.toml:19-21, src/meltano_state_backend_postgresql/backend.py:330-333
Summary
Backwards compatibility is achieved through flexible configuration of PostgreSQL identifiers and robust handling of legacy database schemas. By allowing the PostgreSQLStateStoreManager to connect to schemas previously managed by Meltano's MigrationService and ensuring JSON data is correctly mapped to Python dictionaries, the backend facilitates a non-destructive transition from SystemDB to a standalone state backend.