Containerization & Docker
Relevant source files
The following files were used as context for generating this wiki page:
Containerization and Docker support in Meltano provide a declarative way to execute plugins and manage project environments using isolated container images. This system allows Meltano to invoke plugin commands within Docker containers, ensuring consistent environments across different deployment targets and simplifying dependency management for complex plugins like dbt or specific database connectors.
Meltano provides official Docker images optimized for different use cases, including "slim" images for general use and "full" images for projects requiring extensive database connectors and build tools. The core architecture utilizes a container specification model to map Meltano's internal configurations and environment variables directly into Docker container runtime parameters.
Sources: README.md:126-133, src/meltano/core/container/container_spec.py
Docker Images and Distribution
Meltano is distributed via Docker Hub with two primary image variants. These images are maintained to support various Python versions, recently including support for Python 3.14.
Image Variants
| Variant | Tag Suffix | Description |
|---|---|---|
| Slim | :latest-slim | Optimized size, includes cloud storage support. Recommended for most projects. |
| Full | :latest | Includes all database connectors and build tools (e.g., MSSQL/PostgreSQL). |
Sources: README.md:126-133, CHANGELOG.md:164-165, CHANGELOG.md:200-201
Container Specification (ContainerSpec)
Meltano uses a ContainerSpec class to define how a command should be executed within a container. This model captures essential Docker runtime parameters, including image names, entrypoints, volume mounts, and port mappings.
Architecture of ContainerSpec
The following diagram illustrates how the ContainerSpec model structures data for Docker configuration:
The ContainerSpec inherits from Canonical and provides the get_docker_config method to transform Meltano's internal state into a dictionary format compatible with container engines.
Sources: src/meltano/core/container/container_spec.py:27-58
Environment Variable Mapping
Meltano translates project-level environment variables into container-compatible formats. The internal utility env_mapping_to_docker converts dictionaries into the standard <ENVVAR>=<VALUE> list format required by Docker.
Sources: src/meltano/core/container/container_spec.py:12-24
Plugin Configuration for Containers
Container execution can be configured per command within a plugin definition in meltano.yml. This allows a single plugin to have some commands run locally and others run via Docker.
Configuration Schema
The container_spec object within a plugin command supports the following properties:
| Property | Type | Description |
|---|---|---|
image | string | Required. The Docker image to use (e.g., ghcr.io/dbt-labs/dbt-postgres:latest). |
command | string | The command to execute within the container. |
entrypoint | string | Overrides the default entrypoint of the image. |
ports | object | Mapping of host ports to container ports. |
volumes | array | List of volume mounts, supporting variable expansion (e.g., $MELTANO_PROJECT_ROOT/transform/:/usr/app/). |
env | object | Environment variables to inject into the container. |
Sources: src/meltano/schemas/meltano.schema.json:448-520
Logic Flow for Docker Configuration Generation
The get_docker_config method performs several transformations to prepare the runtime environment:
The generation logic ensures that the host's project root is correctly mapped and that all necessary environment variables—including those dynamically provided at runtime—are present in the container.
Sources: src/meltano/core/container/container_spec.py:60-104
Practical Applications
Using Containers as State Backends
Docker is used in Meltano's integration ecosystem to provide infrastructure for services like state backends. For example, a MinIO container can be run to provide an S3-compatible backend for storing project state.
docker run -d --name minio -p 9001:9001 -p 9000:9000 \
quay.io/minio/minio server /data --console-address ":9001"
Sources: integration/example-library/meltano-state-s3/index.md:15-21
Execution Examples
Users can execute Meltano project commands directly through the official images without a local installation of the Meltano Python package.
# Verify version using the slim image
docker run --rm meltano/meltano:latest-slim --version
Sources: README.md:135-139
Summary
Containerization in Meltano is a first-class feature that enables portable, declarative pipeline execution. By using ContainerSpec, Meltano bridges the gap between its declarative plugin configuration and the Docker runtime API, allowing for complex volume mounting, port mapping, and environment variable injection. The availability of slim and full images ensures that project environments remain optimized for size and functionality.
Sources: src/meltano/core/container/container_spec.py, README.md:126-133