System Protocols & Interfaces
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The FLEXT Meltano System Protocols & Interfaces define the architectural blueprint for enterprise data integration. By leveraging Python's typing.Protocol, the system establishes a type-safe, contract-driven environment that decouples the core domain logic from specific implementations of Meltano, Singer, and DBT technologies. These protocols ensure that internal services and external integrations maintain a consistent interface for operations such as plugin discovery, pipeline execution, and data extraction.
This interface layer acts as a bridge between high-level orchestration services and low-level subprocess executions. It organizes the system into specialized domains—CLI Management, Singer SDK Backends, Project Operations, and Plugin Lifecycle—enabling strict type checking and runtime validation across the FLEXT ecosystem.
Sources: src/flext_meltano/protocols.py:1-25, src/flext_meltano/_protocols/services.py:1-20
Protocol Architecture & Domain Mapping
The system utilizes a unified protocol class, FlextMeltanoProtocols, which aggregates various domain-specific interfaces. This hierarchy ensures that any component implementing the Meltano domain protocol adheres to all required standards for plugins, projects, services, and Singer-based data operations.
The diagram shows the inheritance structure where the Meltano domain protocol composes specialized protocols for plugins, projects, services, and Singer operations.
Sources: src/flext_meltano/protocols.py:13-25, src/flext_meltano/_protocols/init.py:1-44
Core Service Interfaces
The service layer defines the primary execution engine for Meltano and DBT commands. The MeltanoExecutor protocol is central to this layer, providing methods for fetching plugins, executing pipelines, and managing DBT operations.
Meltano Executor Protocol
The executor serves as the primary interface for running data operations.
| Method | Purpose | Returns |
|---|---|---|
execute_meltano_command | Executes raw Meltano runtime commands | p.Result[CommandExecutionResult] |
fetch_project_plugins | Returns project-scoped plugin definitions | p.Result[Sequence[StrMapping]] |
execute_dbt_command | Executes specialized DBT commands | p.Result[CommandExecutionResult] |
execute_pipeline | Runs a complete ELT pipeline (Tap to Target) | p.Result[CommandExecutionResult] |
Sources: src/flext_meltano/_protocols/services.py:25-58
Component Facades
Specialized facades represent the public surface for various ELT components. These include properties to identify source (tap), sink (target), or transformation (DBT) components.
@runtime_checkable
class ComponentFacade(p.Service[t.JsonMapping], Protocol):
@property
def service_name(self) -> str: ...
@property
def source_name(self) -> str | None: ...
@property
def sink_name(self) -> str | None: ...
@property
def transformation_name(self) -> str | None: ...
Sources: src/flext_meltano/_protocols/services.py:78-106
Singer Protocol Implementations
The system defines specific protocols for Singer Taps and Targets to facilitate data extraction and loading. These protocols extend the base Service pattern to provide standardized execute, discover, and sync operations.
Data Extraction (Tap) and Loading (Target)
The Tap and Target protocols manage the lifecycle of data moving through the pipeline.
This flowchart illustrates the data flow from discovery through synchronization to the final sink handling.
Sources: src/flext_meltano/_protocols/singer.py:17-60
Message Handling Interfaces
Internal protocols like SingerTargetHandler ensure that incoming Singer messages (RECORD, SCHEMA, STATE) from standard input are processed correctly by domain-specific logic.
- handle_schema: Processes metadata and structure definitions.
- handle_record: Processes the actual data payloads.
- handle_state: Processes incremental state bookmarks.
Sources: src/flext_meltano/_protocols/singer.py:168-190
CLI and Manager Protocols
The CLI infrastructure is managed through a routing system and specific managers for different operational areas. The CommandRouterCli protocol composes various managers to provide a unified entry point.
CLI Manager Structure
| Manager Protocol | Responsibility | Key Methods |
|---|---|---|
CLIManager | Base command handling | handle_command |
SingerManager | Tap and Target lifecycle | handle_tap_command, handle_target_command |
StatusManager | Health checks and versions | handle_version_command, run_health_check |
PipelineCli | Pipeline orchestration | show_pipeline_help |
Sources: src/flext_meltano/_protocols/services.py:118-208, src/flext_meltano/_protocols/cli.py:13-53
Project & Adapter Protocols
The Project protocol provides type-safe access to Meltano project structures, allowing components to find plugins and identify the project root directory. Complementing this, the Adapter protocol manages the state of connections to data sources or sinks.
Connection Lifecycle
The sequence diagram demonstrates the connection management lifecycle defined in the Adapter protocol.
Sources: src/flext_meltano/_protocols/project.py:35-55
Summary
The FLEXT Meltano System Protocols & Interfaces provide a robust framework for decoupling data integration logic from its underlying implementation. By defining strict contracts for executors, managers, and Singer-based components, the system ensures high reliability and maintainability. The use of runtime-checkable protocols allows the project to maintain enterprise-grade type safety while remaining flexible enough to support a wide range of Meltano plugins and DBT transformations.