Meltano Cloud Deployment
Relevant source files
The following files were used as context for generating this wiki page:
Meltano Cloud Deployment refers to the architecture and configuration patterns used to run Meltano in managed or containerized cloud environments. Meltano is a declarative, code-first data integration engine designed to be highly portable, allowing it to transition from local development to production cloud environments through consistent configuration and manifest-based deployment.
Sources: README.md:1-2, src/meltano/schemas/meltano.schema.json
Deployment Architecture and Manifests
The core of Meltano's cloud deployment strategy is the Manifest File. A manifest is a compiled, self-contained JSON representation of a Meltano project that describes all plugins, configurations, and schedules for a specific environment (e.g., prod, staging). Unlike the standard meltano.yml, which may use include_paths to reference other files, a manifest is resolved and flattened at compilation time to serve as the single source of truth for the deployment runtime.
Manifest Hierarchy and Environment Handling
Manifest files are environment-specific. When Meltano is deployed to the cloud, it typically uses a file named meltano-manifest.<environment>.json. This file includes plugin lockfiles merged during compilation to ensure that the exact same plugin versions used in development are deployed to production.
The diagram shows the compilation of environment-specific manifests for cloud deployment.
Sources: src/meltano/core/manifest/init.py:1-40
Cloud Runtime Configuration
Cloud deployments rely heavily on environment variables and specialized backends for state and logging.
State Backends
In cloud environments, Meltano must store incremental replication state in a durable, shared location rather than a local SQLite database. Supported cloud-native backends include:
| Backend | Configuration Property | Required Authentication |
|---|---|---|
| AWS S3 | state_backend.s3 | aws_access_key_id, aws_secret_access_key |
| Google Cloud Storage | state_backend.gcs | application_credentials (path to JSON) |
| Azure Blob Storage | state_backend.azure | connection_string |
| System Database | state_backend.uri | Database connection string (e.g., PostgreSQL) |
Sources: src/meltano/schemas/meltano.schema.json:62-118, CHANGELOG.md:150-155
Containerization with Docker
Meltano provides official Docker images optimized for cloud environments. These are categorized into:
- Slim Images (
meltano/meltano:latest-slim): Optimized for size, includes cloud storage support for S3/GCS. - Full Images (
meltano/meltano:latest): Includes complete database connectors (MSSQL, PostgreSQL) and build tools.
Sources: README.md:52-64
Orchestration and Jobs
Cloud deployments typically automate data pipelines using the meltano run or meltano el commands, often triggered by an external orchestrator like Airflow or a native cloud scheduler.
Job and Task Flow
Jobs define ordered tasks that Meltano executes. In a cloud manifest, these are represented as an array of objects containing task strings (e.g., tap-snowflake target-snowflake dbt:run).
The sequence diagram illustrates a standard ELT job execution flow in a cloud environment using remote state.
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json:44-125, CHANGELOG.md:320-330
Logging and Metrics
Meltano Cloud Deployment utilizes structured logging and telemetry for observability.
- Logging Format: Configurable via
MELTANO_CLI_LOG_FORMAT, typically set tojsonorcoloredin cloud logs for ingestion into services like AWS CloudWatch or Datadog. - Metric Interception: Meltano can parse logs from Singer SDK-based plugins to extract row counts and execution metrics.
- Redirects: Project documentation and cloud-specific troubleshooting guides are managed via redirects in the deployment infrastructure (e.g., Netlify).
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json:28-35, netlify.toml:15-60, CHANGELOG.md:275-280
Infrastructure as Code (IaC) Settings
Key deployment-related settings found in the schema:
| Setting | Type | Description |
|---|---|---|
database_uri | String | URI for the project system database (e.g., postgresql://...) |
project_readonly | Boolean | Prevents Meltano from writing to the project files at runtime (recommended for prod). |
auto_install | Boolean | Whether to install plugins automatically on command invocation. |
requires_meltano | String | Version specifier to ensure infrastructure matches project requirements. |
Sources: src/meltano/schemas/meltano.schema.json:48-61, src/meltano/schemas/meltano.schema.json:240-250
Meltano's cloud deployment capability ensures that complex data integration projects can be deployed reliably using containerized environments and manifest files, maintaining a strict separation between code-based configuration and sensitive production credentials.