Meltano Integration
Relevant source files
The following files were used as context for generating this wiki page:
Meltano Integration allows the flext-target-oracle to function as a native Singer-compliant target within a Meltano project. This integration enables automated data pipelines where Oracle serves as the destination for data extracted from various sources (taps). It utilizes a YAML-based configuration to define settings, environments, and orchestration schedules.
The integration is built upon the Singer specification, supporting standard message types such as SCHEMA, RECORD, STATE, and ACTIVATE_VERSION. By wrapping the core FlextTargetOracle logic, it provides a seamless interface for Meltano's workflow engine to manage data loading, state persistence, and performance tuning through bulk operations.
Sources: examples/meltano_integration/meltano.yml:1-5, src/flext_target_oracle/_utilities/client.py:12-25, README.md:9-11
Integration Architecture
The integration follows a standard Meltano plugin architecture where target-oracle is defined as a target plugin. It processes an input stream of JSON-formatted Singer messages provided by an extractor (tap).
System Data Flow
The following diagram illustrates how Meltano orchestrates data movement between a source tap and the Oracle target.
Meltano invokes the target plugin, which initializes the FlextTargetOracle client. This client routes messages to the FlextTargetOracleLoader to perform DDL (schema creation) and DML (data insertion) operations.
Sources: examples/meltano_integration/meltano.yml:13-17, src/flext_target_oracle/_utilities/client.py:27-35, src/flext_target_oracle/_utilities/loader.py:34-55
Configuration Settings
The integration defines several configurable settings in meltano.yml that map directly to the FlextTargetOracleSettings model. These settings control connection parameters, loading strategies, and performance optimizations.
Core Configuration Options
| Setting Name | Type | Default | Description |
|---|---|---|---|
oracle_host | string | - | Oracle database hostname or IP address. |
oracle_port | integer | 1521 | Oracle database port number. |
oracle_service | string | - | Oracle service name for connection. |
oracle_user | string | - | Oracle database username. |
oracle_password | password | - | Oracle database password. |
default_target_schema | string | MELTANO_DATA | Default schema for table creation. |
load_method | options | INSERT | Data loading strategy (INSERT, MERGE, BULK_INSERT, BULK_MERGE). |
batch_size | integer | 1000 | Number of records per batch for processing. |
use_bulk_operations | boolean | true | Enables Oracle bulk operations for performance. |
Sources: examples/meltano_integration/meltano.yml:25-78, src/flext_target_oracle/_utilities/client.py:190-195
Singer Message Processing
The FlextTargetOracle class serves as the primary entry point for Singer messages. It implements a routing logic to handle different message types received from Meltano.
Message Processing Sequence
The following sequence diagram shows how the target handles the lifecycle of a typical data stream.
Key Message Handlers
_handle_schema: Usesloader.ensure_table_existsto verify or create destination tables based on the Singer schema. It maps JSON types to Oracle-native types (e.g.,CLOB,TIMESTAMP,NUMBER)._handle_record: Buffers incoming data. When thebatch_sizethreshold is met, it triggers a flush to the database._handle_state: Captures bookmark information to ensure Meltano can resume syncs from the last successful point.
Sources: src/flext_target_oracle/_utilities/client.py:146-177, src/flext_target_oracle/_utilities/loader.py:465-490
Performance and Production Patterns
For enterprise environments, the integration supports advanced loading methods and health monitoring as demonstrated in production examples.
Loading Methods
The load_method setting determines the SQL strategy used by the loader:
- INSERT: Standard row-by-row or batch insertion.
- MERGE/BULK_MERGE: Updates existing records based on
key_propertiesor inserts new ones. - BULK_INSERT: Utilizes
execute_manyfor high-performance ingestion.
Health and Monitoring
The integration supports runtime health snapshots, providing metrics on:
oracle_connectivity: Verification of database availability.records_processed: Total count of successful vs. failed records.processing_duration_seconds: Performance tracking of the sync job.
Sources: examples/03_production_setup.py:53-75, examples/meltano_integration/meltano.yml:55-70, src/flext_target_oracle/_utilities/loader.py:560-580
Usage Example
A standard Meltano job configuration to sync CSV data to Oracle is defined in the jobs section of the configuration file.
jobs:
- name: csv-to-oracle
tasks:
- tap-csv target-oracle
schedules:
- name: daily-oracle-sync
interval: "@daily"
job: csv-to-oracle
This configuration establishes a daily schedule for the csv-to-oracle job, which pipes data from tap-csv directly into the target-oracle plugin.
Sources: examples/meltano_integration/meltano.yml:85-95
Conclusion
Meltano Integration for flext-target-oracle provides a robust, configuration-driven approach to Oracle data loading. By adhering to the Singer protocol and providing extensive performance tuning options like bulk operations and custom batch sizes, it facilitates the creation of enterprise-grade data pipelines within the Meltano ecosystem.