Introduction & Getting Started
Relevant source files
The following files were used as context for generating this wiki page:
FLEXT Meltano is an enterprise-grade data integration platform designed to provide a production-ready bridge between Go services and Python-based data orchestration tools. It leverages Meltano, Singer, and DBT technologies to facilitate scalable ETL/ELT pipelines within the FLEXT ecosystem. The project is built on Python 3.13+ and implements strict typing, dependency injection, and structured logging to meet enterprise standards.
Sources: README.md:1-10, src/flext_meltano/README.md:14-20
Architecture Overview
The system is organized into a layered architecture that separates core logic from external integrations. This structure ensures high maintainability and clear abstraction boundaries between the Go service bridge and the underlying data tools.
Architectural Layers
| Layer | Responsibility | Key Components |
|---|---|---|
| Foundation | Configuration, exceptions, and base classes | base.py, common.py, container.py |
| Bridge Integration | Go service communication and CLI orchestration | simple_bridge.py, execution.py, cli.py |
| Core Operations | Service management and plugin discovery | core.py, discovery.py, installation.py |
| Singer Integration | Singer SDK bridge and protocol handling | singer.py, singer_unified.py |
| Data Transformation | DBT project management and execution | dbt.py |
Sources: src/flext_meltano/README.md:30-60
Operation Flow
The following diagram illustrates the high-level data flow from a FLEXT Service through the Python bridge to final data operations.
This flow ensures that all external requests are validated and translated into canonical Meltano or Singer commands before execution.
Sources: README.md:21-27, src/flext_meltano/README.md:104-110
Getting Started
Installation and Dependencies
FLEXT Meltano requires Python 3.13 and manages dependencies through pyproject.toml. It depends on several core FLEXT libraries and industry-standard data tools.
# Key dependencies from pyproject.toml
dependencies = [
"meltano @ git+https://github.com/flext-sh/meltano.git%40flext/relax-deps%22%2C
"singer-sdk>=0.53.7",
"dbt-core @ git+https://github.com/flext-sh/dbt-core.git%40flext/relax-deps#subdirectory=core",
"flext-core",
"flext-cli",
"pydantic>=2.13.3"
]
Sources: pyproject.toml:34-56
Basic Usage Example
The project provides a public facade via the meltano module, which exposes common operations like plugin discovery and component factory methods.
from flext_meltano import meltano
def simple_api_example() -> None:
# Fetch Meltano version
version_result = meltano.fetch_version()
# Discover installed plugins
discovery_result = meltano.discover_installed_plugins()
if discovery_result.success:
print(f"Plugins: {len(discovery_result.value)}")
# Access typed settings
print(f"Environment: {meltano.settings.environment}")
Sources: examples/01_simple_working.py:20-33, examples/01_simple_working.py:54-61
Core Configuration & Models
Project Configuration
The PipelineProjectModel validates the consistency of a Meltano project, including environment markers and maturity assessments.
| Field | Type | Description |
|---|---|---|
project_id | str | Unique identifier for the project |
project_root | Path | Directory containing meltano.yml |
default_environment | ProjectEnvironment | Default runtime environment (dev, staging, prod) |
environments | Sequence | List of allowed environments |
Sources: src/flext_meltano/_models/projects.py:50-80
Project Maturity Logic
The system automatically assesses project maturity based on the environment configuration:
- MATURE: Contains a production environment and at least 3 environments total.
- DEVELOPING: Contains at least 2 environments.
- BASIC: Default state for new projects.
Sources: src/flext_meltano/_models/projects.py:100-115, src/flext_meltano/_constants/base.py:165-170
Engineering Standards
The project follows a strict set of "Universal Agent Laws" and engineering principles to ensure code quality and stability.
Universal Principles
- SSOT (Single Source of Truth): One authoritative source per fact; configuration is environment-aware and validated.
- SOLID: Implementation depends on abstractions (protocols/interfaces) rather than concrete classes.
- Zero-Debt Baseline: Root causes must be fixed generically; no temporary workarounds are permitted.
- Restrictive Typing: Strict MyPy mode compliance with no
Anytypes allowed.
Sources: AGENTS.md:12-45, src/flext_meltano/README.md:90-100
Directory Constants
The system uses standardized paths for its internal operations:
.meltano: Configuration directory..pipeline: State persistence directory.logs: Execution logs.transform: DBT transformation source files.
Sources: src/flext_meltano/_constants/base.py:53-62
Summary
FLEXT Meltano provides a robust framework for integrating Python data pipelines into a broader Go-based service architecture. By standardizing on Meltano and Singer protocols while enforcing enterprise engineering standards, it enables reliable, observable, and maintainable data integration workflows. Developers should start by utilizing the meltano public facade for common orchestration tasks.