Running Performance Benchmarks
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Performance benchmarks in the FLEXT Tap Oracle WMS project are designed to validate the efficiency, scalability, and resource utilization of data extraction workflows. These benchmarks focus on measuring critical metrics such as catalog discovery time, data throughput (records per second), and memory consumption during large-scale extractions. The performance suite ensures that the tap meets production-grade requirements, such as maintaining memory usage under specific limits and achieving high throughput even with complex Oracle WMS data structures.
Benchmarks are categorized as part of the broader testing strategy, which includes Unit Testing Strategy and E2E Testing Strategy. Currently, several performance tests are marked as integration-level benchmarks that require either a live Oracle WMS instance or a comprehensive mocking environment to execute successfully.
Sources: tests/performance/test_extraction_performance.py:12-14, tests/e2e/README.md:213-220, tests/README.md:14-25
Benchmark Architecture and Components
The performance testing framework utilizes pytest along with psutil for resource monitoring and time for latency benchmarking. It is structured to evaluate the tap's behavior under different configurations, such as varying page sizes and rate-limiting settings.
Execution Flow
The following diagram illustrates the lifecycle of a performance benchmark execution:
The diagram shows the standard flow from environment loading to final validation of performance thresholds.
Sources: tests/performance/test_extraction_performance.py:27-50, tests/e2e/README.md:244-260
Key Performance Indicators (KPIs)
Benchmarks evaluate the tap against specific technical thresholds to ensure operational stability.
| Metric | Target Threshold | Description |
|---|---|---|
| Discovery Time | < 10.0 seconds | Time taken to perform discovercatalog_typed() for all WMS entities. |
| Memory Increase | < 100 MB | Maximum allowable RSS memory growth during a 1,000-record extraction. |
| Throughput | > 100 records/sec | Minimum records processed per second during extraction. |
| CPU Usage | < 80% | Maximum CPU utilization during peak extraction workflows. |
Sources: tests/performance/test_extraction_performance.py:59-62, tests/performance/test_extraction_performance.py:108-112, tests/e2e/README.md:237-241, tests/e2e/README.md:267-271
Performance Test Scenarios
Catalog Discovery Benchmarking
This scenario measures the time required for the tap to initialize and discover the full metadata schema of the Oracle WMS instance. It uses the discovercatalog_typed() method and asserts that the operation completes in under 10 seconds.
Sources: tests/performance/test_extraction_performance.py:54-62
Pagination and Throughput
The performance suite benchmarks different page_size configurations (e.g., 10, 50, 100, 200) to determine the optimal setting for data transfer. It specifically targets high-volume streams like inventory.
Sources: tests/performance/test_extraction_performance.py:64-84
Resource Usage and Memory Leak Detection
To prevent memory exhaustion, benchmarks monitor the Resident Set Size (RSS) using psutil. A large extraction (typically 1,000 records) is performed while comparing the initial_memory to final_memory.
Sources: tests/performance/test_extraction_performance.py:101-115
def test_memory_usage_during_large_extraction(self, tap: FlextTapOracleWms) -> None:
"""Test memory usage during large extractions."""
process = psutil.Process()
initial_memory = process.memory_info().rss / 1024 / 1024
tap.initialize()
streams = tap.discover_streams()
# ... extraction logic ...
final_memory = process.memory_info().rss / 1024 / 1024
memory_increase = final_memory - initial_memory
assert memory_increase < 100
Sources: tests/performance/test_extraction_performance.py:101-115
Rate Limiting Impact
The system compares performance metrics with enable_rate_limiting set to True vs False. This ensures that while the tap respects API limits (e.g., max_requests_per_minute=60), it does not introduce unnecessary overhead.
Sources: tests/performance/test_extraction_performance.py:121-149
Execution Environment Setup
Performance tests require specific environment variables to be defined in a .env file to establish connections with the target WMS or mock server.
| Variable | Default Value | Description |
|---|---|---|
ORACLE_WMS_BASE_URL | https://localhost | The endpoint for the Oracle WMS API. |
ORACLE_WMS_PAGE_SIZE | 100 | The number of records per page requested during benchmarks. |
ORACLE_WMS_API_VERSION | v10 | The version of the WMS API being benchmarked. |
Sources: tests/performance/test_extraction_performance.py:23-40, tests/README.md:120-128
Running the Benchmarks
Benchmarks are executed using pytest with specific markers to isolate performance tests from unit or functional tests.
# Run only performance benchmarks
pytest tests/performance/test_extraction_performance.py -v -m performance
# Run performance tests with standard output enabled (for real-time metrics)
pytest tests/performance/ -s -m performance
Sources: tests/performance/test_extraction_performance.py:152, tests/unit/README.md:126-128
Summary
The performance benchmark suite provides a critical quality gate for the FLEXT Tap Oracle WMS, focusing on efficiency and resource safety. By measuring discovery latency, extraction throughput, and memory stability across various configurations, the project ensures the tap is capable of handling enterprise-scale data volumes without degrading system performance. Current development efforts are focused on implementing a comprehensive E2E mock environment to allow these benchmarks to run autonomously in CI/CD pipelines.