Pagination & Sequencing
Relevant source files
The following files were used as context for generating this wiki page:
The "Pagination & Sequencing" system within the FLEXT Tap Oracle WMS module provides the logic and configuration necessary to handle large data extractions from Oracle Warehouse Management Systems. Given the high volume of records in WMS environments (such as items, inventory, and orders), the system implements robust mechanisms to fetch data in chunks while maintaining data integrity through sequencing.
The core of this functionality is built upon the Singer SDK's RESTStream patterns, specifically utilizing a custom WMSPaginator designed to handle Oracle WMS HATEOAS-based pagination and offset-based or cursor-based strategies. This ensures that the tap can scale to millions of records without exceeding memory limits or API timeout thresholds.
Sources: src/flext_tap_oracle_wms/README.md:120-125, meltano.yml:148-155, tests/e2e/test_e2e.py:280-290
Pagination Strategies
The system supports multiple pagination modes to accommodate different WMS API capabilities and performance requirements. These modes are configurable via the tap settings and applied during the stream's record extraction process.
Offset-based Pagination
This is the default strategy where the tap requests records using limit and page (or offset) parameters. The tap calculates the next request based on the current page number and the configured page_size.
Cursor-based Pagination
For more performant extraction in high-volume environments, cursor-based pagination uses a unique identifier (cursor) provided by the WMS response to fetch the next set of records. This avoids the performance degradation often associated with deep offsets in large datasets.
Configuration Parameters
The following table outlines the configuration options available for managing pagination:
| Parameter | Type | Default | Description |
|---|---|---|---|
page_size | Integer | 100 | Number of records to request per API call (Max 1250). |
pagination_mode | Option | offset | Strategy to use: offset or cursor. |
max_pages_per_stream | Integer | null | Limit the number of pages fetched for testing or partial syncs. |
request_concurrency | Integer | 1 | Number of concurrent requests allowed for extraction. |
Sources: meltano.yml:148-180, tests/integration/test_functional.py:165-175, src/flext_tap_oracle_wms/tap.py:35-50
Data Flow for Paginated Requests
The data flow for a paginated request involves the stream object building operation arguments, which include the pagination parameters. The Oracle WMS API returns a response containing the records and metadata (such as next_url or total_pages) used by the WMSPaginator to determine if more data exists.
The diagram shows the iterative loop where parameters are updated for each subsequent request until the WMS indicates no further records are available.
Sources: tests/integration/test_functional.py:165-180, src/flext_tap_oracle_wms/README.md:120-125, tests/e2e/test_e2e.py:380-395
Sequencing and Ordering
Sequencing ensures that data is extracted in a predictable order, which is critical for incremental replication and maintaining state. The tap utilizes the WMS ordering parameter to sort records, typically by a primary key or a timestamp-based replication key.
Incremental Replication Sequencing
For streams configured with INCREMENTAL replication, the tap applies filters based on the replication_key (e.g., last_updated_ts). The sequencing logic ensures that the tap requests records in ascending order of this key so that the state (bookmark) can be safely updated as progress is made.
Full Table Sequencing
In FULL_TABLE replication mode, the tap may apply descending order (-id) to fetch the most recent records first or maintain a stable sort to ensure no records are missed if the dataset changes during extraction.
This sequence illustrates how the ordering parameter is maintained across multiple paginated requests to ensure consistent data delivery.
Sources: tests/integration/test_functional.py:215-240, tests/e2e/test_e2e.py:130-155, meltano.yml:255-275
Implementation Details
The implementation of these features relies on the FlextTapOracleWmsStream class, which handles the generation of URL parameters.
Key Logic in _build_operation_kwargs
This method is responsible for injecting the correct pagination and sequencing parameters into the request. It validates that the limit does not exceed the Oracle WMS maximum of 1250 and applies filtering logic for incremental bookmarks.
# From tests/integration/test_functional.py:175-180
# Illustrating the validation of pagination parameters in the stream
url_params = stream._build_operation_kwargs(page=1, context=None)
assert "limit" in url_params
assert isinstance(url_params["limit"], int)
assert url_params["limit"] <= 1250 # Oracle WMS constraint
Replication Method Mapping
Replication and sequencing behavior is defined in the catalog metadata. The project supports both FULL_TABLE and INCREMENTAL methods, as seen in the configuration.
| Stream | Replication Method | Replication Key |
|---|---|---|
inventory | INCREMENTAL | last_updated_ts |
order_hdr | INCREMENTAL | last_updated_ts |
item | FULL_TABLE | N/A |
Sources: tests/integration/test_functional.py:170-185, meltano.yml:255-275, tests/e2e/test_e2e.py:135-150
Conclusion
Pagination and sequencing are fundamental to the FLEXT Tap Oracle WMS, enabling reliable data extraction from high-volume enterprise environments. By combining HATEOAS-aware pagination with strict ordering based on replication keys, the system ensures that data is synchronized efficiently and incrementally without duplication or loss.
Sources: src/flext_tap_oracle_wms/README.md:120-125, tests/e2e/test_e2e.py:380-400