State Management & Merging
Relevant source files
The following files were used as context for generating this wiki page:
State management in Meltano is a critical system responsible for persisting progress indicators (bookmarks) between ELT pipeline runs. This ensures that data extractors (taps) only fetch new or updated data, enabling incremental replication and improving efficiency across various backends.
The system is built around a pluggable architecture where different StateStoreManager implementations handle storage logic for specific protocols (e.g., S3, GCS, Azure, or local filesystems). Meltano also provides CLI tools for manual state inspection, modification, and merging to handle complex synchronization scenarios.
State Storage Architecture
Meltano uses a centralized state backend configuration to determine where progress data is stored. By default, Meltano uses the system database (systemdb), but it can be configured to use various cloud storage providers or the local filesystem.
State Store Managers
The core logic is encapsulated in manager classes that inherit from a base StateStoreManager interface. These managers handle the retrieval, persistence, and listing of state objects.
The diagram shows the hierarchy of state store managers derived from the base manager interface.
Sources: tests/meltano/core/state_store/test_state_store.py:65-100, pyproject.toml:95-103
Backend Support and URIs
State backends are identified by URI schemes. Meltano uses entry points to map these schemes to specific manager classes.
| Scheme | Manager Implementation | Description |
|---|---|---|
systemdb | DBStateStoreManager | Uses the Meltano project database (SQLite/Postgres). |
file:// | LocalFilesystemStateStoreManager | Stores state as JSON files on the local disk. |
s3:// | S3StateStoreManager | Persists state to AWS S3 buckets. |
gs:// | GCSStateStoreManager | Persists state to Google Cloud Storage. |
azure:// | AZStorageStateStoreManager | Persists state to Azure Blob Storage. |
Sources: pyproject.toml:95-103, src/meltano/schemas/meltano.schema.json:80-140, src/meltano/core/bundle/settings.yml:25-30
Configuration and Settings
State management is configured via the state_backend object in meltano.yml or through environment variables. This includes the storage URI and locking parameters for concurrency control.
Core State Settings
| Setting | Type | Default | Description |
|---|---|---|---|
state_backend.uri | string | systemdb | The URI pointing to the state storage backend. |
state_backend.lock_timeout_seconds | integer | 10 | Valid duration for a lock on a state ID. |
state_backend.lock_retry_seconds | integer | 1 (or 5 in schema) | Wait time between lock acquisition retries. |
Sources: src/meltano/core/bundle/settings.yml:33-45, src/meltano/schemas/meltano.schema.json:80-95
Cloud-Specific Configuration
Each cloud backend has its own sub-configuration block for authentication and endpoint details. For example, the S3 backend supports aws_access_key_id, aws_secret_access_key, and endpoint_url for compatibility with MinIO.
Sources: src/meltano/schemas/meltano.schema.json:115-135, integration/example-library/meltano-state-s3/index.md:40-45
State Data Flow
During a meltano run or meltano elt operation, the state flows from the storage backend to the extractor and finally back to the storage backend once the loader confirms a successful write.
The flowchart illustrates the lifecycle of state during a pipeline execution.
Sources: integration/example-library/meltano-state-s3/index.md:50-60, CHANGELOG.md:145-155
State Merging and Manual Management
Meltano provides CLI commands to manually manage state, which is essential for troubleshooting or resetting progress.
State CLI Operations
- meltano state list: Lists all state IDs stored in the configured backend.
- meltano state get <state_id>: Retrieves the current JSON state for a specific ID.
- meltano state set <state_id>: Overwrites the state for an ID. This can be done via raw JSON string or an input file.
- meltano state edit <state_id>: Opens the state in a text editor for manual patching (added in v4.0.0).
- meltano state merge: Merges new state data into existing state, typically used to update specific bookmarks without overwriting the entire object.
Sources: integration/example-library/meltano-state-s3/index.md:65-75, CHANGELOG.md:145-150, integration/example-library/meltano-state-local/index.md:25-30
Locking Mechanism
To prevent data corruption during concurrent runs, state backends implement an acquire_lock method.
lock_timeout_secondsdefines how long a lock remains valid.lock_retry_secondsdefines the interval for re-attempting access if a state ID is currently locked by another process.
Sources: src/meltano/schemas/meltano.schema.json:85-93, CHANGELOG.md:25-30
Conclusion
The state management system in Meltano provides a robust, scalable way to handle pipeline progress. By supporting cloud-native backends like S3 and GCS with built-in locking and CLI-based manual intervention, it allows developers to maintain data integrity across distributed environments and complex ELT workflows. Progress is tracked via unique state IDs (often formatted as environment:tap-to-target), ensuring that multiple pipelines can run independently within the same project.
Sources: src/meltano/core/bundle/settings.yml:33-45, integration/example-library/meltano-state-s3/index.md:65-70