Remote State Backends
Relevant source files
The following files were used as context for generating this wiki page:
Remote State Backends in Meltano provide a mechanism for persisting the incremental replication state of data pipelines in cloud storage rather than the default local system database. This allows for better scalability, multi-user collaboration, and persistence in ephemeral execution environments like Docker containers or CI/CD runners.
By offloading state management to services like AWS S3, Azure Blob Storage, or Google Cloud Storage, Meltano ensures that multiple instances of a project can safely access and update job progress using distributed locking mechanisms.
Architecture and Supported Providers
Meltano utilizes a pluggable architecture for state management where different storage providers implement a common interface. The appropriate backend is selected based on the URI scheme provided in the configuration.
The diagram above illustrates how the Meltano core interacts with specialized StateStoreManager implementations based on the configured URI scheme.
Sources: pyproject.toml:130-134, src/meltano/core/state_store/s3/backend.py, src/meltano/core/state_store/azure/backend.py
Key Classes
S3StateStoreManager: Handles state persistence in AWS S3 buckets. Supports customendpoint_urlfor S3-compatible services like MinIO.AZStorageStateStoreManager: Manages state within Azure Blob Storage containers.GCSStateStoreManager: Interfaces with Google Cloud Storage buckets.
Sources: src/meltano/core/state_store/s3/backend.py:27, src/meltano/core/state_store/azure/backend.py:34, src/meltano/core/state_store/google/backend.py:24
Configuration
Remote backends are configured primarily via the state_backend object in meltano.yml or through equivalent environment variables. The uri property determines which provider is utilized.
| Provider | URI Scheme | Key Settings |
|---|---|---|
| AWS S3 | s3://bucket/prefix | aws_access_key_id, aws_secret_access_key, endpoint_url |
| Azure Blob | azure://container/prefix | connection_string, storage_account_url |
| Google Cloud | gs://bucket/prefix | application_credentials (path to JSON) |
Sources: src/meltano/schemas/meltano.schema.json:76-135, integration/example-library/meltano-state-s3/index.md:46-52
Global Lock Settings
To prevent concurrent updates to the same state ID, remote backends implement a locking strategy:
lock_timeout_seconds: Duration (default 10s) a lock remains valid.lock_retry_seconds: Wait interval when retrying lock acquisition.
Sources: src/meltano/schemas/meltano.schema.json:85-92, src/meltano/core/bundle/settings.yml:43-52
Data Flow: State Retrieval and Update
When a Meltano job runs, it performs a synchronized dance with the remote backend to ensure state integrity.
The sequence diagram shows the typical workflow of locking, reading, and updating state in a remote store.
Sources: src/meltano/core/state_store/s3/backend.py:100-150, tests/meltano/core/state_store/test_state_store.py:382-420
Provider Specific Implementation Details
AWS S3 / S3-Compatible
The S3 backend uses the boto3 library. It supports a prefix to organize states within a bucket. The state for a specific ID is stored at /{prefix}/{state_id}/state.json.
# Example CLI configuration for S3
meltano config set meltano state_backend.uri "s3://my-meltano-bucket/production"
meltano config set meltano state_backend.s3.aws_access_key_id "AKIA..."
Sources: src/meltano/core/state_store/s3/backend.py:35-50, integration/example-library/meltano-state-s3/index.md:46-52
Azure Blob Storage
Azure requires a connection_string or a storage_account_url. If a connection string is not provided, the manager attempts to use DefaultAzureCredential for authentication.
Sources: src/meltano/core/state_store/azure/backend.py:43-70, tests/meltano/core/state_store/test_state_store.py:270-310
Google Cloud Storage
The GCS backend typically relies on a service account JSON file. It implements specialized path joining logic to ensure URI prefixes are handled correctly across different operating systems.
Sources: src/meltano/core/state_store/google/backend.py:32-45, tests/meltano/core/state_store/test_state_store.py:330-345
State Migration and Maintenance
Meltano provides interfaces for maintaining remote state.
- Migration: The
StateStoreManager.migrate()method allows implementations to fix legacy pathing issues, such as deduplicating nested prefixes in S3 keys. - Cleanup: State files and lock files can be managed via the
delete_filemethods implemented in each provider.
Sources: CHANGELOG.md:19, src/meltano/core/state_store/s3/backend.py:202-230, tests/meltano/core/state_store/test_state_store.py:422-450
Remote State Backends are critical for production Meltano deployments, providing the reliability and concurrency control needed for complex data platform architectures.