Core Architecture & DI Principles
Relevant source files
The following files were used as context for generating this wiki page:
FLEXT Meltano is an enterprise-grade data integration library designed to bridge the gap between Go services and Python-based data orchestration using Meltano, Singer, and DBT. The architecture is built on a Foundation Layer providing core configuration, dependency management, and schema definitions, and an Operations Layer that handles service execution and validation.
The system adheres to strict engineering principles, specifically Dependency Inversion (DIP) and Dependency Injection (DI). By depending on abstractions (protocols) rather than concrete implementations and injecting collaborators into services, the project ensures a decoupled, testable, and extensible codebase.
Sources: src/flext_meltano/README.md:12-23, AGENTS.md:73-77
Architectural Layers
The project is organized into five distinct architectural layers to maintain separation of concerns and technical scalability.
| Layer | Purpose | Core Modules |
|---|---|---|
| Foundation | Base classes, Configuration, DI Container | base.py, container.py, common.py |
| Bridge Integration | Go-to-Python communication | simple_bridge.py, execution.py |
| Core Operations | Service logic, discovery, and installation | core.py, validation.py, installation.py |
| Singer Integration | Singer SDK protocols and adapters | singer.py, singer_base.py, flext_singer.py |
| Data Transformation | DBT orchestration and project management | dbt.py |
Sources: src/flext_meltano/README.md:25-50
Dependency Injection (DI) & Inversion Principles
The project mandates the use of DI/DIP as a "Universal Engineering Principle." Services must not hard-wire construction inside business logic or depend on hidden globals. Instead, they must depend on protocols defined in the library.
Dependency Flow
The diagram shows how the DI Container resolves concrete implementations for services that only depend on defined Protocols.
Sources: AGENTS.md:73-77, src/flext_meltano/_protocols/project.py:15-32
Protocol-Based Contracts
The library defines runtime-checkable protocols to ensure type safety and strict interface compliance across the bridge. Key protocols include:
- Project: Defines the interface for Meltano project operations, including
root_dirandfind_plugins. - Adapter: Standardizes data source/sink connections via
connect(),disconnect(), andis_connectedstates. - IndexedPlugin: Manages plugin metadata such as
variants,default_variant, andlogo_url.
Sources: src/flext_meltano/_protocols/project.py:18-67
Core Service Model
FLEXT Meltano uses a result-oriented pattern (Railway Oriented Programming) for its core services. The FlextMeltanoAbstractionsBase class serves as the template for services that wrap the Meltano runtime, returning Result[T] objects to handle success or failure without raising unhandled exceptions.
Pipeline Execution Flow
This diagram illustrates the flow of a Singer pipeline execution initiated from the bridge, showing the delegation through the Abstractions Service to the CLI Executor.
Sources: src/flext_meltano/services/abstractions_base.py:101-137, src/flext_meltano/README.md:86-90
Configuration Management
The system utilizes Pydantic-based models for validated configuration. The PipelineProjectModel enforces project structure and maturity assessments.
| Field | Type | Description |
|---|---|---|
project_id | str | Unique identifier for the Meltano project. |
project_root | Path | Directory where meltano.yml resides. Defaults to CWD. |
default_environment | ProjectEnvironment | The environment used if none is specified (e.g., dev). |
schema_version | int | Version of the pipeline schema (Version 1 supported). |
Sources: src/flext_meltano/_models/projects.py:46-70
Project Maturity Assessment
The core architecture includes logic to assess the maturity of a project based on its environment configurations. A project is deemed MATURE if it contains a production environment and meets the environment count threshold (default is 3).
if has_prod and env_count >= c.Meltano.VALIDATION_MATURITY_MATURE_ENV_COUNT:
return str(c.Meltano.ProjectMaturity.MATURE)
Sources: src/flext_meltano/_models/projects.py:90-101
Conclusion
The core architecture of FLEXT Meltano is defined by its layered structure and rigorous adherence to SOLID principles, specifically Dependency Inversion. By utilizing runtime-checkable protocols and centralized dependency injection, the system provides a robust bridge for enterprise data integration. The use of Railway Oriented Programming (Result[T]) throughout the abstractions and executor layers ensures that the Go-to-Python bridge remains resilient and provides clear error context for orchestrated subprocesses.
Sources: src/flext_meltano/README.md:12-23, src/flext_meltano/services/abstractions_base.py:22-42