High-Level Tap Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_tap_oracle_wms/tap.py
• src/flext_tap_oracle_wms/README.md
• src/flext_tap_oracle_wms/settings.py
• meltano.yml
• pyproject.toml

The FLEXT Tap Oracle WMS is a Singer-compliant tap designed for extract-transform-load (ETL) operations from the Oracle Warehouse Management System (WMS). Built on top of the singer-sdk, it facilitates the extraction of enterprise data into a standardized Singer format for downstream consumption by targets (e.g., Snowflake, BigQuery, Postgres). The architecture prioritizes strict type safety using Python 3.13 features, Pydantic-based configuration management, and deep integration with the FLEXT ecosystem libraries such as flext-core, flext-oracle-wms, and flext-meltano.

The system's scope encompasses entity discovery, automated schema generation, and data extraction with support for both full-table and incremental replication methods. It leverages a dedicated WMS client for low-level API interactions while maintaining high-level Singer protocol compliance through the FlextTapOracleWms class.

Sources: src/flext_tap_oracle_wms/README.md:16-24, src/flext_tap_oracle_wms/tap.py:33-35, pyproject.toml:21-30

Core Architectural Components

The architecture is divided into several functional layers that handle different stages of the data lifecycle.

Component Relationship Overview

The following diagram illustrates the relationship between the main Tap class, the configuration settings, and the underlying WMS client.

The FlextTapOracleWms class serves as the central orchestrator, managing the lifecycle of the WMS client and stream generation based on the validated configuration.

Sources: src/flext_tap_oracle_wms/tap.py:33-55, src/flext_tap_oracle_wms/README.md:92-95

Key Classes and Roles

Component	Responsibility	Source Reference
FlextTapOracleWms	Primary entry point; orchestrates discovery and sync operations.	tap.py:33
FlextTapOracleWmsSettings	Pydantic model for validating environment variables and Meltano config.	settings.py:18
FlextTapOracleWmsStream	Handles individual entity extraction, pagination, and filtering.	tap.py:30
OracleWms.Client	Low-level HTTP client wrapper for Oracle WMS API requests.	tap.py:181

Discovery and Schema Mapping

Discovery is the process by which the tap identifies available entities in the Oracle WMS and maps them to Singer streams.

Entity Discovery Logic

The tap uses the wms_client.discover_entities() method to fetch a list of available resources. For every entity found, it builds a SingerCatalogEntry with a default JSON schema of type OBJECT.

1. Client Start: The WMS client is initialized with credentials and a base URL.
2. Entity Discovery: The client identifies entities like inventory, items, and orders.
3. Catalog Building: discovercatalog_typed converts these into a Singer-compatible catalog.
4. Metadata Attachment: Each stream is tagged with metadata specifying default replication methods (typically FULL_TABLE by default during initial discovery).

Sources: src/flext_tap_oracle_wms/tap.py:181-224, meltano.yml:197-208

Configuration and Validation

Configuration management is handled by FlextTapOracleWmsSettings, which uses Pydantic to enforce data types and business rules before any network connection is attempted.

Configuration Parameters

Parameter	Type	Default	Description
base_url	str / AnyUrl	Empty	The Oracle WMS instance root URL.
username	str	Empty	WMS API Username.
password	SecretStr	Empty	WMS API Password (masked in logs).
page_size	int	100	Number of records per API request.
max_retries	int	5	Maximum retry attempts for failed requests.
include_entities	list[str]	[]	Explicit list of entities to sync.
enable_rate_limiting	bool	True	Whether to throttle requests to the API.

Sources: src/flext_tap_oracle_wms/settings.py:18-100, meltano.yml:68-120

Business Rule Validation

The settings class implements a validate_business_rules method that ensures logical consistency:

• Mutual Exclusion: An entity cannot be in both include_entities and exclude_entities.
• Date Range: If both are provided, start_date must be less than or equal to end_date.
• ISO Format: Dates must adhere to the ISO 8601 standard via regex validation.

Sources: src/flext_tap_oracle_wms/settings.py:165-177

Data Extraction Flow

The extraction flow involves setting up the SDK's RESTStream implementation (via FlextTapOracleWmsStream) and processing records based on the replication method.

Replication Methods

The tap supports two primary replication strategies as defined in the metadata of meltano.yml:

• FULL_TABLE: Entire entity data is fetched in every sync (e.g., item, location).
• INCREMENTAL: Only records changed since the last replication_key value (e.g., last_updated_ts) are fetched. This is primarily used for inventory and order_hdr.

Sources: meltano.yml:197-217, src/flext_tap_oracle_wms/tap.py:228-245

Runtime Execution

When the tap is executed for a sync, it performs the following steps:

1. Initialize: Validates the flext_config.
2. Client Setup: Starts the wms_client.
3. Stream Generation: Builds a sequence of FlextTapOracleWmsStream objects from the catalog.
4. Sync: Iterates through each stream, performing paginated requests to the Oracle WMS API and emitting Singer messages.

Sources: src/flext_tap_oracle_wms/tap.py:181-196, src/flext_tap_oracle_wms/tap.py:257-261

Conclusion

The High-Level Tap Architecture of flext-tap-oracle-wms provides a robust, type-safe bridge between Oracle WMS and the Singer ecosystem. By centralizing logic in a Singer-compliant Tap class and offloading low-level API concerns to the flext-oracle-wms library, the project ensures high maintainability and protocol compliance. Its rigorous configuration validation via Pydantic and flexible discovery mechanism allow it to adapt to varying WMS environments while providing reliable data pipelines.