Library Runner Integration
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Library Runner Integration serves as a unified orchestration layer for executing data integration tasks within the FLEXT Meltano ecosystem. It provides a high-level Pythonic interface to manage ELT (Extract, Load, Transform) pipelines and DBT (Data Build Tool) transformations, abstracting the complexities of underlying subprocess execution and Meltano CLI interactions.
This system is designed as an "Enterprise-Grade Data Integration Bridge," facilitating communication between Go services and Python-based data operations. By utilizing a mixin architecture for MRO (Method Resolution Order) composition, the Library Runner allows for seamless integration of pipeline execution and transformation logic into the broader FlextMeltano service framework.
Sources: src/flext_meltano/README.md:16-18, src/flext_meltano/services/library_runner.py:15-18
Architecture and Components
The FlextMeltanoLibraryRunner is a service mixin that inherits from FlextMeltanoServiceBase. Its primary responsibility is coordinating the FlextMeltanoExecutor to perform specific Meltano and DBT commands.
Core Orchestration Flow
The runner coordinates three primary execution scenarios:
- Complete ELT Pipeline: Executes a source (tap), a destination (target), and optionally follows up with DBT transformations.
- Standard ELT Pipeline: Executes only the Extract and Load phases using specified tap and target configurations.
- DBT Transformations: Triggers data transformations using DBT models within a Meltano project context.
Key Classes and Mixins
| Component | Type | Description |
|---|---|---|
FlextMeltanoLibraryRunner | Class (Mixin) | The primary interface for orchestrating ELT and DBT operations. |
FlextMeltanoExecutor | Attribute | A private instance (_elt_executor) used to handle the low-level execution logic. |
FlextMeltanoServiceBase | Base Class | Provides common service functionality such as logging and settings management. |
Sources: src/flext_meltano/services/library_runner.py:15-22, src/flext_meltano/services/init.py:47-49
Execution Logic and Data Flow
ELT Pipeline Execution
The integration manages the lifecycle of a pipeline run, converting high-level requests into structured payloads. It uses runtime utilities to build standardized execution payloads that include status, exit codes, and timing metrics.
DBT Transformation Orchestration
The run_dbt_transformation method allows for flexible execution of DBT models. It can either use the default project root or a dynamically provided project_dir by creating a temporary copy of the executor with updated settings.
Key transformation parameters:
- models: A sequence of strings identifying specific DBT models to run.
- project_dir: An optional
Pathto override the Meltano project location.
Sources: src/flext_meltano/services/library_runner.py:66-93
Payload Normalization
The system ensures that output from the Meltano CLI is normalized into a standard "Enterprise-Grade" format suitable for Go ↔ Python communication.
Standard Execution Fields
| Field | Type | Description |
|---|---|---|
success | boolean | Indicates if the command finished with exit code 0. |
output | string | Standard output captured from the execution. |
error | string | Standard error captured from the execution. |
exit_code | integer | The process exit code. |
status | string | Normalized status (e.g., "success" or "error"). |
execution_time | float | Total duration of the operation in seconds. |
Sources: src/flext_meltano/_utilities/runtime.py:255-276, src/flext_meltano/services/library_runner.py:46-52
Implementation Example
The following snippet demonstrates how the Library Runner functionality is typically exposed through the public meltano facade in user scripts.
from flext_meltano import meltano
# Example of executing a runtime command through the facade
# which internally utilizes the executor and runner logic
result = meltano.execute_meltano_command([
"meltano",
"version",
])
if result.success:
print(f"Execution payload: {result.value}")
Sources: examples/01_simple_working.py:44-55
Conclusion
The Library Runner Integration is a critical component of the flext-meltano project, providing the necessary abstraction for reliable data pipeline orchestration. By centralizing execution logic and enforcing standardized result payloads, it enables robust integration between external FLEXT services and internal data transformation tools.
Sources: src/flext_meltano/README.md:10-14, src/flext_meltano/services/library_runner.py:15-18