Extraction Performance Tuning
Relevant source files
The following files were used as context for generating this wiki page:
Extraction Performance Tuning in the FLEXT Tap Oracle WMS project involves optimizing the data retrieval process from Oracle Warehouse Management Systems. The system utilizes a REST-based architecture with Singer SDK patterns, requiring careful balance between throughput, memory usage, and API rate limits. Performance is primarily managed through configuration settings that control pagination, concurrency, and request timing.
The core tuning mechanisms are implemented within the FlextTapOracleWms class and its associated stream definitions, which interface with the flext-oracle-wms base library. Developers can fine-tune these parameters to handle various data volumes, ranging from small configuration entities to large-scale inventory and order records.
Performance Architecture and Data Flow
The performance architecture relies on the interaction between the Singer Tap, the WMS Client, and the underlying pagination logic. Data flows from the Oracle WMS REST API through a paginator that handles HATEOAS-based navigation.
The diagram above illustrates the flow of data extraction and the specific control points where performance tuning is applied.
Sources: src/flext_tap_oracle_wms/tap.py:270-305, tests/performance/test_extraction_performance.py:35-120
Key Tuning Parameters
Performance is controlled through specific settings defined in the Pydantic models and the Singer configuration schema.
Pagination Control
Pagination is the most significant factor in extraction speed. The page_size parameter determines how many records are fetched per request. Oracle WMS supports a maximum limit of 1250 records per page.
| Parameter | Type | Default | Impact |
|---|---|---|---|
page_size | Integer | 100 | Higher values increase throughput but raise memory usage. |
pagination_mode | Option | offset | Supports offset or cursor based navigation. |
max_pages_per_stream | Integer | null | Limits the depth of extraction for testing or sampling. |
Sources: meltano.yml:100-115, src/flext_tap_oracle_wms/tap.py:46-50, tests/integration/test_functional.py:164-170
Request and Concurrency Management
To prevent overwhelming the Oracle WMS instance or hitting environment-specific API limits, the tap provides concurrency and rate-limiting controls.
- Request Concurrency: Controls the number of simultaneous requests allowed. Production environments typically support higher concurrency (e.g., 5) compared to dev environments.
- Rate Limiting: Integrated rate limiting via
enable_rate_limitingandmax_requests_per_minuteto adhere to API quotas. - Timeouts: Configurable
request_timeoutto ensure the tap doesn't hang on slow API responses.
Sources: meltano.yml:38-45, tests/performance/test_extraction_performance.py:93-110, tests/integration/test_wms_connection.py:38-40
Optimization Strategies
Memory Management
Large extractions can lead to significant memory spikes if records are buffered excessively before being processed. The system is designed to use generators (get_records) to stream data, but developers should monitor RSS (Resident Set Size) during tuning. Performance tests aim for less than 100MB of memory increase during large (1,000+ record) extractions.
Sequence of streaming data to maintain a low memory footprint.
Sources: tests/performance/test_extraction_performance.py:78-91, src/flext_tap_oracle_wms/tap.py:280-295
Discovery Performance
Catalog discovery can be time-consuming for large WMS implementations with many custom entities. The project targets a discovery time of under 10 seconds. The discovery process utilizes the describe method (API Metadata Only) rather than sampling data, which significantly reduces the performance overhead of schema generation.
Sources: tests/performance/test_extraction_performance.py:47-55, meltano.yml:126-132, tests/e2e/test_e2e.py:230-245
Tuning Examples
The following JSON configurations demonstrate optimized settings for different use cases.
Performance Optimized Configuration
This setup maximizes throughput for production data syncs.
{
"page_size": 500,
"request_concurrency": 5,
"enable_rate_limiting": true,
"max_requests_per_minute": 300,
"pagination_mode": "cursor"
}
Sources: meltano.yml:35-45, tests/performance/test_extraction_performance.py:57-65
Resource Constrained Configuration
Used for shared development environments to minimize impact on the WMS server.
{
"page_size": 50,
"request_concurrency": 1,
"max_pages_per_stream": 10,
"log_level": "DEBUG"
}
Sources: meltano.yml:18-25, tests/integration/test_functional.py:168-170
Conclusion
Extraction performance tuning in flext-tap-oracle-wms is a multi-dimensional task involving pagination optimization, concurrency management, and memory efficiency. By leveraging the built-in performance benchmarks and the flexible configuration model, operators can scale the tap from lightweight development tasks to high-throughput enterprise data warehouse integrations while maintaining compliance with Oracle WMS API constraints.