Installation & Quickstart
Relevant source files
The following files were used as context for generating this wiki page:
Meltano is a declarative, code-first data integration engine designed to build, run, and orchestrate ELT (Extract, Load, Transform) pipelines. It leverages a CLI-first approach to manage data workflows using a curated ecosystem of plugins from the Meltano Hub.
The platform is built on Python and follows a modular architecture where functionality is extended through various plugin types, including extractors, loaders, mappers, and orchestrators. This guide covers the primary methods for setting up a Meltano project and executing initial data synchronization tasks.
Sources: README.md:1-25, pyproject.toml:1-10, src/meltano/schemas/meltano.schema.json:1-10
Installation Methods
Meltano supports multiple installation environments, ranging from containerized Docker images to local Python-based setups.
Docker Installation
Meltano provides official Docker images optimized for different use cases:
- Slim images (
meltano/meltano:latest-slim): Optimized size, includes cloud storage support. - Full images (
meltano/meltano:latest): Includes all database connectors (e.g., MSSQL/PostgreSQL) and build tools.
Sources: README.md:43-57
Local Python Setup
Meltano requires Python 3.10 or higher. Dependency management is typically handled via uv or pip.
# Installing dependencies for development
uv sync --all-extras --all-groups
Sources: pyproject.toml:7, AGENTS.md:10-15
Project Initialization
A Meltano project is defined by a meltano.yml configuration file. When a project is initialized, a standard directory structure and configuration are created.
Project Structure
The initial project setup creates several directories and files to manage the ELT lifecycle:
meltano.yml: The primary project configuration file..meltano/: Internal directory for managed virtual environments and the system database.extract/,load/,transform/: Directories for storing plugin-specific assets..env: Used for managing sensitive environment variables.
Sources: src/meltano/core/bundle/initialize.yml:1-25, src/meltano/schemas/meltano.schema.json:44-50
Configuration Defaults
New projects are initialized with a set of default environments (dev, staging, prod) and an internal SQLite system database.
| Setting | Default Value | Description |
|---|---|---|
default_environment | dev | The active environment if none is specified. |
database_uri | sqlite:///.meltano/meltano.db | URI for the project system database. |
send_anonymous_usage_stats | true | Telemetry for development priority. |
venv.backend | virtualenv | Backend used for plugin isolation. |
Sources: src/meltano/core/bundle/initialize.yml:1-10, src/meltano/schemas/meltano.schema.json:20-100
Quickstart Workflow
The standard workflow for starting a data integration task involves installing plugins, configuring settings, and running the ELT pipeline.
Step 1: Install Plugins
Plugins are added to the project using the meltano add command and installed via meltano install. Installation creates isolated virtual environments for each plugin.
# Add an extractor and a loader
meltano add tap-gitlab target-jsonl
meltano install
Sources: integration/example-library/meltano-basics/index.md:5-10, integration/example-library/meltano-objects/index.md:12-15
Step 2: Configure Settings
Configuration can be adjusted via the CLI, which updates the meltano.yml file or the internal system database.
# Set configuration for an extractor
meltano config set tap-gitlab start_date 2022-04-25T00:00:01Z
Sources: integration/example-library/meltano-basics/index.md:14-20
Step 3: Run the Pipeline
Data movement is executed using the meltano run or meltano elt commands. The following diagram illustrates the data flow during a standard execution:
The data flow starts with identifying the environment context, ensuring plugins are installed, streaming data from the extractor (Tap) to the loader (Target), and finally persisting the state.
Sources: integration/example-library/meltano-basics/index.md:22-30, src/meltano/schemas/meltano.schema.json:230-250
Orchestration & Management
Meltano allows grouping tasks into "Jobs" and automating them through "Schedules".
Adding Jobs and Schedules
Jobs define a sequence of tasks, while schedules determine when those jobs should run using cron expressions.
# Add a job consisting of a tap-target pair
meltano job add gitlab_to_jsonl --tasks '["tap-gitlab target-jsonl"]'
# Schedule the job to run daily
meltano schedule add run_gitlab --job gitlab_to_jsonl --interval "@daily"
Sources: integration/example-library/meltano-objects/index.md:22-30, src/meltano/schemas/meltano.schema.json:580-630
Environment Management
Meltano environments allow for configuration overrides based on the deployment target (e.g., using different database credentials for QA vs. Production).
# Add a new environment
meltano environment add qa
Sources: integration/example-library/meltano-objects/index.md:17-20, src/meltano/schemas/meltano.schema.json:650-680
Summary
Meltano provides a robust framework for declarative data integration. By using Docker for deployment and the CLI for project management, users can quickly initialize projects, install Singer-compatible plugins, and orchestrate complex ELT workflows with minimal manual configuration. The system's reliance on a central meltano.yml and isolated virtual environments ensures consistency across development and production environments.
Sources: README.md, integration/example-library/meltano-basics/index.md