Dynamic Date Windowing

Dynamic Date Windowing is a configuration-driven mechanism within the FLEXT Tap Oracle WMS project that allows for flexible data extraction based on temporal boundaries. This feature enables developers and operators to define extraction periods using either static ISO-8601 timestamps or dynamic relative keywords like "yesterday" or "today," facilitating automated incremental loads and historical backfills.

The system integrates these time-based boundaries into the stream's request parameters, ensuring that the Oracle WMS API only returns records within the specified window. This is primarily achieved through the validation and mapping of start_date and end_date parameters within the tap's configuration settings.
Sources: meltano.yml:81-86, src/flext_tap_oracle_wms/settings.py

Configuration Parameters

The Dynamic Date Windowing feature is controlled by two primary configuration keys: start_date and end_date. These parameters are defined in the project's Pydantic-based settings model and exposed via meltano.yml for orchestration.

Date Window Settings

Sources: meltano.yml:81-86, meltano.yml:195-196, tests/unit/test_config.py:100-105

Logic and Flow

The implementation relies on strict validation rules to ensure the temporal window is logical. The FlextTapOracleWmsSettings class enforces business rules, such as ensuring the start_date does not occur after the end_date.

Date Processing Flow

The diagram shows how date configuration moves from raw input to API request parameters, passing through format and logical validation gates.
Sources: tests/unit/test_config.py:118-124, tests/unit/test_config.py:148-154

Implementation Details

Validation Rules

The configuration module includes specific logic to prevent extraction errors caused by invalid date ranges. If a user provides a start_date that is later than the end_date, the system raises a ValidationError. Furthermore, dates must adhere to the ISO-8601 format (e.g., YYYY-MM-DDTHH:MM:SSZ).
Sources: tests/unit/test_config.py:107-116, tests/unit/test_config.py:155-161

Application to Streams

During extraction, the FlextTapOracleWmsStream utilizes the configured dates to build operation keywords. These values are mapped to WMS-specific filtering parameters, often leveraging timestamp-based replication keys like last_updated_ts for incremental syncs.

# Example date-based filtering logic in tests
context = {"replication_key_value": "2024-01-01T00:00:00Z"}
url_params = stream._build_operation_kwargs(page=1, context=context)
kwargs_filter = url_params.get("filter")
# Verification of timestamp filters
assert ">=" in str(kwargs_filter) or "<" in str(kwargs_filter)

Sources: tests/integration/test_functional.py:228-234, meltano.yml:217-226

Use Case Examples

The project provides several configuration examples demonstrating how to implement different windowing strategies:

1. Relative Backfill (Yesterday): Configures the tap to extract data exclusively from the previous day.
   • Source: examples/configs/test_config_simple_dates_yesterday.json
2. Trailing Window (7 Days): Sets the window from 7 days ago until "today."
   • Source: examples/configs/test_config_simple_dates_today_7d.json
3. Mixed Static/Dynamic: Combines a static starting point with a dynamic endpoint.
   • Source: examples/configs/test_config_simple_dates_mixed.json

Summary

Dynamic Date Windowing provides the FLEXT Tap Oracle WMS with the necessary temporal control to perform both historical loads and precise incremental updates. By leveraging Pydantic validation and ISO-8601 standards, the system ensures that date ranges are valid before initiating expensive API requests, thereby improving the reliability of the data pipeline.
Sources: tests/unit/test_config.py:165-175, tests/integration/test_functional.py:277-290