Core API Interface
Relevant source files
The following files were used as context for generating this wiki page:
The Core API Interface of flext-target-oracle provides a structured, protocol-driven approach to loading Singer-compliant data into Oracle databases. It acts as the primary orchestration layer, bridging the gap between incoming Singer messages (SCHEMA, RECORD, STATE) and the underlying Oracle DDL/DML operations. The interface is built upon flext-core principles, utilizing railway-oriented programming (Result patterns) for robust error handling and strict type safety.
This system is designed around a clear separation of concerns: the FlextTargetOracle class handles high-level message routing and state, while the FlextTargetOracleLoader manages the technical execution of database operations via the flext-db-oracle API.
Sources: CONTRIBUTING.md:104-121, src/flext_target_oracle/_utilities/loader.py:33-50, examples/01_basic_usage.py:126-160
Protocol-Driven Architecture
The project employs runtime_checkable protocols to define strict contracts for various components. These protocols ensure that services for connection management, schema enforcement, and batch processing remain interchangeable and verifiable.
Key Service Protocols
| Protocol | Purpose | Key Methods |
|---|---|---|
ConnectionService | Manages Oracle connectivity and info retrieval. | test_connection(), get_connection_info() |
SchemaService | Handles table creation and DDL enforcement. | ensure_table_exists() |
BatchService | Orchestrates record buffering and flushing. | add_record(), flush_all_batches() |
RecordService | Transforms Singer records into Oracle-ready payloads. | transform_record() |
Sources: src/flext_target_oracle/_protocols/base.py:104-150
Interaction Flow
The following diagram illustrates how the API processes a typical sequence of Singer messages through these protocol-defined services.
The Core API Interface routes messages based on their type, ensuring tables are prepared before records are buffered and eventually flushed to the database.
Sources: src/flext_target_oracle/_utilities/loader.py:444-460, examples/01_basic_usage.py:126-160
Data Loading Logic (The Loader)
The FlextTargetOracleLoader is the primary implementation of the database interaction logic. It inherits from FlextMeltanoServiceBase and utilizes the FlextDbOracleApi as its source of truth for Oracle operations.
Column Mapping and Ordering
The loader handles the complex task of mapping Singer JSON schemas to Oracle-compatible data types. It supports:
- Alphabetical Ordering: Grouping columns by category (Primary Keys, Regular, Audit, SDC).
- Audit Columns: Automatic injection of
_SDC_EXTRACTED_ATand_SDC_LOADED_AT. - Storage Modes: Support for
flattened,json, orhybridstorage of nested data structures.
Batch Processing and Merging
Records are not written individually but are stored in typed buffers (_record_buffers). Once a buffer reaches the configured batch_size, a flush operation is triggered.
| Operation | Logic |
|---|---|
| Flush | Normalizes records, builds Oracle parameters, and executes a batch insert. |
| Merge (Upsert) | If sdc_mode is set to merge, the loader executes a delete of existing records matching primary keys before inserting new ones. |
| Truncate | Optionally clears the table before the load starts if truncate_before_load is enabled. |
Sources: src/flext_target_oracle/_utilities/loader.py:476-520, src/flext_target_oracle/_utilities/loader.py:546-570, tests/integration/test_oracle.py:106-125
Error Handling and Result Patterns
The interface strictly follows Railway-Oriented Programming (ROP) using p.Result (or r in implementation). Operations return a result object that explicitly indicates success or failure, containing either the value or an error context.
Oracle Exception Hierarchy
Custom exceptions inherit from flext-core bases but add Oracle-specific metadata such as stream_name, record_count, or service_name.
OracleConnectionError: Includesdsn,user, andservice_name.SchemaError: Includestable_nameandvalidation_errors.ProcessingError: Includesoperationanderror_records.
Sources: src/flext_target_oracle/_utilities/errors.py:38-120
Configuration and Initialization
The core interface is configured via FlextTargetOracleSettings, which utilizes Pydantic for validation. Key configuration parameters influence how the API behaves during data ingestion.
| Parameter | Type | Description |
|---|---|---|
batch_size | Integer | Number of records to buffer before flushing to Oracle. |
sdc_mode | String | Data strategy: append or merge. |
storage_mode | String | How nested JSON is handled: flattened, json, or hybrid. |
column_mappings | Dictionary | Maps Singer field names to specific Oracle column names. |
Sources: src/flext_target_oracle/_utilities/loader.py:134-160, examples/01_basic_usage.py:53-70, tests/integration/test_oracle.py:255-275
Connection Lifecycle
The target initialization process typically follows these steps:
- Validation: Check configuration domain rules.
- Initialization: Create the
FlextTargetOracleinstance. - Connection Test: Verify the DSN and credentials.
- Execution: Start processing the stream of Singer messages.
Sources: examples/01_basic_usage.py:126-150, src/flext_target_oracle/_utilities/loader.py:410-430
Conclusion
The Core API Interface provides a high-level, safe, and efficient gateway to Oracle data loading. By abstracting the complexities of Oracle SQL and Singer protocol message handling into a series of services and protocols, it ensures that the system is both maintainable and performant. Its reliance on flext-core patterns provides a consistent developer experience across the FLEXT ecosystem.