Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
Meltano is a declarative, code-first data integration engine designed to build, run, and orchestrate ELT (Extract, Load, Transform) pipelines. It serves as a single source of truth for data workflows by centralizing the configuration of disparate data tools—such as Singer taps and targets—into a unified project structure. The platform focuses on scalability and maintenance by providing a declarative layer over 600+ APIs and databases.
Sources: README.md:1-12, src/meltano/schemas/meltano.schema.json:4-6
The architecture is built around a core engine that manages plugin lifecycles, virtual environment isolation, and state persistence. It leverages a centralized configuration file (meltano.yml) or generated manifests to define the relationships between extractors, loaders, mappers, and transformers, while supporting environment-specific overrides for development, staging, and production.
Sources: src/meltano/schemas/meltano.schema.json:11-30, integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:30-40
Core System Components
The project core is organized around specialized service classes that handle specific domains of the Meltano lifecycle. New functionality is typically implemented by creating a new service class, such as the SettingsService.
Sources: AGENTS.md:46-48
Management Services
Recent architectural updates have centralized management logic into several key services:
- VirtualEnvService: Manages plugin isolation by delegating to
VenvBackendimplementations (e.g.,virtualenvoruv). - ProjectDirsService: Provides a unified API for project directory definitions and path management.
- StateStoreManager: Manages the persistence and migration of pipeline state across different backends.
Sources: CHANGELOG.md:42-45, CHANGELOG.md:36-37, CHANGELOG.md:18-20
Plugin System
Meltano utilizes a modular plugin architecture. Plugins are categorized by their role in the data pipeline:
| Plugin Type | Description |
|---|---|
| Extractors | Taps that pull data from sources (Singer-compatible). |
| Loaders | Targets that push data to destinations. |
| Mappers | Implement Stream Maps for inline data transformation. |
| Orchestrators | Manage job scheduling and execution (e.g., Airflow). |
| Transformers | Handle warehouse-level transformations (e.g., dbt). |
| Utilities | Arbitrary tools integrated into the Meltano project. |
This diagram illustrates the relationship between a Meltano project and its various functional components, highlighting the role of plugins and state management.
Configuration and Environments
Meltano's configuration follows a hierarchical resolution pattern, allowing properties to be defined globally, per environment, or via environment variables.
Environment Overrides
The environments key in the project configuration allows users to define specific configurations for dev, stage, or prod. This includes overrides for plugin settings, environment variables (env), and pipeline-specific parameters.
Sources: src/meltano/schemas/meltano.schema.json:115-121, integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:20-25
The configuration resolution flow ensures that environment-specific settings take precedence over global defaults.
Data Flow and Execution
Meltano pipelines typically follow a task-based execution model defined within jobs. A job consists of one or more tasks, which can be individual plugin invocations or sequences of plugins (e.g., a tap followed by a target).
Sources: src/meltano/schemas/meltano.schema.json:173-176, integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:42-48
Typical ELT Job Structure
A standard job in a manifest defines a name and a list of tasks. For example, a meltanohub_el job might include:
- Extraction and Loading:
tap-meltanohub target-snowflake - Transformation/Testing:
dbt-snowflake:test_source_meltanohub
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json:57-65
The sequence diagram shows the data flow from source to destination, including the interaction with the State Store to ensure incremental replication.
State and Persistence
Meltano manages the "state" of data syncs to support incremental extraction. The platform supports multiple backends for storing this state.
State Backends
The state_backend configuration defines where sync progress is persisted:
- systemdb: The default backend, using the internal SQLite database (usually at
.meltano/meltano.db). - Cloud Storage: Support for AWS S3, Google Cloud Storage (GCS), and Azure Blob Storage.
Sources: src/meltano/schemas/meltano.schema.json:52-105, CHANGELOG.md:18-20
Configuration Options
| Option | Type | Default | Description |
|---|---|---|---|
uri | String | systemdb | Location of the backend. |
lock_timeout_seconds | Integer | - | Valid duration for a state file lock. |
lock_retry_seconds | Integer | - | Interval between retrying lock acquisition. |
Sources: src/meltano/schemas/meltano.schema.json:56-68
Virtual Environment Management
Plugin isolation is a core architectural pillar. Meltano creates a dedicated virtual environment for each plugin to prevent dependency conflicts.
- Isolation: Every plugin's
pip_urlis installed into a separate venv. - Backends: Meltano supports
virtualenvanduvfor environment creation. Thevenv.backendsetting controls this behavior globally. - Automatic Installation: The
auto_installflag (defaulting to true) ensures plugins are installed upon command invocation if the virtual environment is missing.
Sources: src/meltano/schemas/meltano.schema.json:107-114, src/meltano/schemas/meltano.schema.json:281-285, CHANGELOG.md:36-37
Conclusion
Meltano's architecture provides a robust framework for managing data pipelines by treating them as managed code projects. By utilizing specialized services for directory management, virtual environments, and state persistence, the system ensures that complex ELT workflows remain isolated, reproducible, and easy to orchestrate across different environments.