Advanced Configuration
Relevant source files
The following files were used as context for generating this wiki page:
Advanced Configuration in the flext-target-oracle-wms project encompasses enterprise-grade features designed for high-performance data loading, custom business logic transformation, and robust observability. This module allows developers to move beyond basic Singer protocol implementations to handle complex Oracle WMS Cloud SaaS requirements, such as custom type conversions, performance tuning, and mission-critical error handling patterns.
Sources: examples/README.md:92-95, examples/05_advanced_configuration.py:1-12
Data Transformation and Type Conversion
The system utilizes a specialized WMSTypeConverter and WMSDataTransformer to prepare Singer RECORD messages for the Oracle WMS environment. This involves converting scalar values into Oracle-friendly formats and ensuring that data structures comply with the destination's schema requirements, such as key casing.
Type Conversion Logic
The WMSTypeConverter handles the translation of Singer types (string, integer, number, object, array, boolean) into formats acceptable by the Oracle WMS payload. For complex types like object or array, the converter serializes the data into JSON strings.
The flow above illustrates how different input types are normalized before being sent to the Oracle WMS API.
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:46-68, tests/unit/test_wms_patterns.py:53-100
Record Transformation
The WMSDataTransformer applies the conversion logic across entire records. A critical requirement for Oracle WMS integration is the uppercasing of all record keys to match expected database/API field names.
| Operation | Component | Description |
|---|---|---|
| Key Normalization | WMSDataTransformer | Converts all dictionary keys to uppercase (e.g., qty becomes QTY). |
| Type Resolution | WMSTypeConverter | Maps Singer schema properties to specific Oracle data types. |
| Schema Validation | WMSSchemaMapper | Builds a normalized catalog entry including table_name mappings. |
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:86-113, tests/unit/test_sinks.py:92-101
Performance and Batch Tuning
For high-volume enterprise environments, the target supports advanced performance tuning through batch processing and load method selection.
Load Methods
The project defines specific strategies for how data is applied to the Oracle WMS target. These are configured via the load_method setting.
- APPEND_ONLY: Adds new records without checking for existing entries.
- UPSERT: Updates existing records or inserts new ones based on key properties.
- MERGE: Synchronizes data using merge logic.
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:32-44, tests/unit/test_quality.py:58-64
Batch Configuration
Batch sizes can be tuned to optimize network throughput and memory usage. Performance benchmarks indicate significant scaling improvements as batch sizes increase.
| Configuration Parameter | Default Value | Description |
|---|---|---|
batch_size | 1000 | Number of records to process in a single batch. |
bulk_insert_mode | False | Enables optimized bulk insertion patterns. |
parallel_processing | False | Allows concurrent processing of multiple streams. |
Sources: examples/05_advanced_configuration.py:24, examples/README.md:144-162, tests/unit/test_quality.py:66-68
Monitoring and Observability
Advanced configuration integrates flext-observability to provide production-grade monitoring. This is achieved through the FlextObservabilityMonitor and function decorators.
@flext_monitor_function(monitor)
def run_advanced_example() -> t.Scalar:
"""Run advanced configuration example with monitoring."""
logger.info("Starting advanced configuration example")
# Implementation logic...
return True
Sources: examples/05_advanced_configuration.py:27-35, examples/01_basic_usage.py:38-41
Error Handling and Resilience
For mission-critical deployments, advanced configuration includes resilience patterns to handle transient network failures or Oracle WMS API rate limits.
Circuit Breaker and Retries
The system can be configured with exponential backoff and circuit breaker logic.
Sources: examples/README.md:103-125
Configuration Validation
The Validation helper ensures that the target does not attempt to execute with an incomplete or invalid configuration.
| Required Field | Description |
|---|---|
base_url | The URL of the Oracle WMS Cloud instance (e.g., https://example.wms.oracle.com). |
username | API authentication username. |
password | API authentication password. |
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:27-44, examples/01_basic_usage.py:27-31
Advanced Configuration allows for the fine-tuning of the flext-target-oracle-wms to meet specific enterprise data requirements, focusing on transformation accuracy, performance at scale, and operational visibility.