Environments & Variables
Relevant source files
The following files were used as context for generating this wiki page:
- src/meltano/core/environment.py
- src/meltano/core/environment_service.py
- src/meltano/schemas/meltano.schema.json
- src/meltano/core/manifest/init.py
- src/meltano/core/bundle/initialize.yml
- integration/example-library/meltano-env-precedence/index.md
- integration/example-library/meltano-expand-envvars-in-array/index.md
Environments within the project provide a declarative way to manage configuration overrides and environment variables across different deployment stages, such as development, staging, and production. By defining an environment, users can switch the context of the entire project, ensuring that plugins and core settings adapt to the specific requirements of that stage without changing the primary project definition.
Environment variables serve as the backbone of dynamic configuration. They follow a strict hierarchical precedence and expansion logic, allowing values to be injected from the terminal, .env files, or specific project-level and plugin-level definitions. This system ensures that sensitive data can be kept out of version control while maintaining a single source of truth through the project's manifest system.
The Environment Model
The system defines an environment as a named entity containing specific configurations and environment variable mappings.
Environment Structure
An environment is primarily composed of a unique name and an optional dictionary of environment variables and configurations.
Sources: src/meltano/core/environment.py:10-18, src/meltano/schemas/meltano.schema.json:101-105
| Field | Type | Description |
|---|---|---|
name | String | The unique identifier for the environment (e.g., prod, dev). |
config | Object | Environment-specific overrides for plugin configurations. |
env | Object | A mapping of environment variable keys to their values. |
Sources: src/meltano/schemas/meltano.schema.json:964-1005
Default Environments
When a new project is initialized, a default set of environments is typically provided to encourage best practices.
Sources: src/meltano/core/bundle/initialize.yml:1-12
The diagram above illustrates the relationship between a Project and its defined Environments.
Environment Management Service
The EnvironmentService is the core component responsible for the lifecycle of environments within a project. It interacts directly with the project files to persist changes.
Key Operations
- List Environments: Retrieves all environment definitions currently stored in the configuration files.
- Add Environment: Appends a new environment definition to the project.
- Remove Environment: Deletes an existing environment by name.
This flowchart shows the high-level logic for managing environments via the service.
Environment Variable Precedence & Expansion
The project employs a strictly hierarchical process for merging and expanding environment variables. This hierarchy determines which value takes precedence when the same variable is defined in multiple locations.
Precedence Hierarchy
- Terminal Variables: Variables defined in the shell where the process is running.
- .env File: Variables loaded from a local
.envfile. - Job Level: Variables defined within a specific job context.
- Schedule Level: Variables defined for a specific scheduled task.
- Project Level: Global variables defined in the project configuration.
- Plugin Level: Variables specific to the plugin being executed.
Sources: src/meltano/core/manifest/init.py:47-60
Variable Expansion Logic
Expansion allows setting values to reference other variables using standard ${VAR} or $VAR syntax. This is particularly useful in arrays and complex configuration objects.
Sources: integration/example-library/meltano-expand-envvars-in-array/index.md
| Logic Element | Description |
|---|---|
| Expansion | Variables are expanded into manifest string fields during processing. |
| Strict Mode | A feature flag strict_env_var_mode can be enabled to exit with an error if an undefined variable is referenced. |
| Redaction | Sensitive settings are redacted in command output but remain available to the runtime environment. |
Sources: src/meltano/core/manifest/init.py:60-65, src/meltano/schemas/meltano.schema.json:284-290
The sequence diagram details how the process sequentially merges variable sources.
Manifest Integration
Manifest files serve as compiled, self-contained representations of a project for a specific environment. When a manifest is generated for an environment, the environments field is omitted because the environment selection is baked into the file name (e.g., meltano-manifest.<env>.json).
Characteristics of Manifests
- Self-contained: They include all necessary configurations, including merged plugin lockfiles.
- Environment Specific: A manifest represents the project with an active environment already applied.
- Secrets Security: Manifests should never contain secrets or values directly from external environment variables to prevent accidental exposure of sensitive data in the compiled JSON.
Sources: src/meltano/core/manifest/init.py:20-45
Conclusion
Environments and variables provide the flexibility required to maintain a single codebase while deploying across diverse infrastructures. The EnvironmentService manages the lifecycle of these contexts, while the manifest and expansion systems ensure that the final configuration used by plugins is predictable, hierarchical, and secure. This architecture allows developers to define complex data pipelines that are both declarative and dynamic.