Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
The flext-target-oracle project is a Singer-compliant target designed for loading data into Oracle databases. It is built upon the FLEXT ecosystem, leveraging unified interfaces for message routing, state management, and high-performance database operations. The system prioritizes type safety, railway-oriented error handling, and clean separation between Singer protocol logic and database-specific DDL/DML execution.
Sources: README.md:1-10, CONTRIBUTING.md:105-110, src/flext_target_oracle/_utilities/loader.py:15-20
Key Components
The architecture is divided into three primary layers: the Target interface, the Loader engine, and the Configuration/Exception utilities.
Component Summary
| Component | Responsibility | Relevant Files |
|---|---|---|
| Target Class | Main Singer implementation; routes messages (SCHEMA, RECORD, STATE). | target.py |
| Loader Class | Core data loading logic; handles batching, DDL, and DML. | loader.py |
| Configuration | Pydantic-based settings validation and domain rules. | settings.py |
| Oracle API | Abstraction layer for Oracle-specific SQL execution via flext-db-oracle. | flext-db-oracle dependency |
Sources: CONTRIBUTING.md:112-132, src/flext_target_oracle/_utilities/loader.py:23-40
System Architecture and Data Flow
The system follows a Top-Down data flow where Singer messages are ingested by the Target, processed into batches by the Loader, and finally persisted to Oracle.
The FlextTargetOracleLoader maintains internal buffers for each stream to optimize throughput using batch operations. When the batch_size (defined in settings) is reached, the buffer is flushed using execute_many through the FlextDbOracleApi.
Sources: src/flext_target_oracle/_utilities/loader.py:46-60, src/flext_target_oracle/_utilities/loader.py:530-550, tests/integration/test_oracle.py:270-285
Database Loading Logic
The loading process involves complex transformations of Singer schemas into Oracle-compatible DDL and the mapping of JSON records into SQL parameters.
Schema Mapping and DDL
The loader dynamically generates DDL by mapping Singer types to Oracle types (e.g., string to VARCHAR2, integer to NUMBER). It also supports owner-managed SDC (Singer Data Capture) audit columns: _SDC_EXTRACTED_AT and _SDC_LOADED_AT.
Batch Processing Sequence
The following diagram illustrates the sequence of events during a batch flush:
Configuration and Storage Modes
The target supports multiple storage strategies to handle different data structures, particularly nested JSON objects.
- Flattened Mode: Nested JSON properties are flattened into individual Oracle columns.
- JSON Mode: The entire record is stored as a CLOB/JSON column in Oracle.
- Hybrid Mode: Key properties are extracted into columns while the full payload is preserved in a JSON column.
Sources: src/flext_target_oracle/_utilities/loader.py:155-165, tests/integration/test_oracle.py:175-190
Error Handling Framework
The project utilizes "Railway-Oriented Programming" via the r result pattern. This ensures that exceptions are caught and wrapped in structured Result objects rather than crashing the pipeline.
Specific error types include:
- OracleConnectionError: Issues with host, port, or service name.
- SchemaError: Invalid Singer schemas that cannot be mapped to Oracle.
- ProcessingError: Failures during the DML execution phase, including record counts and stream names in the context.
Sources: src/flext_target_oracle/_utilities/errors.py:28-40, src/flext_target_oracle/_utilities/errors.py:75-90, CONTRIBUTING.md:134-138
Conclusion
The flext-target-oracle architecture provides a robust, enterprise-ready bridge between Singer taps and Oracle databases. By delegating low-level SQL concerns to flext-db-oracle and focusing on Singer protocol compliance and batching efficiency, it achieves high performance while maintaining strict type safety and error traceability.
Sources: README.md:1-15, CONTRIBUTING.md:105-110