Meltano Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Meltano integration for the flext-tap-oracle-wms project provides a structured orchestration layer for extracting data from Oracle Warehouse Management Systems (WMS). By leveraging the Singer SDK, the project integrates as an "extractor" within the Meltano ecosystem, allowing for automated data pipelines, environment-specific configurations, and scheduled synchronization jobs.
This integration facilitates the mapping of WMS entities into Singer-compliant streams, handling authentication, pagination, and state management. It serves as the primary interface for users to deploy the tap within a modern data stack, supporting various target loaders like Postgres, Snowflake, or BigQuery.
Sources: src/flext_tap_oracle_wms/README.md:14-25, meltano.yml:1-10
Plugin Configuration
The tap-oracle-wms is defined as a Meltano extractor plugin. It includes metadata for discovery and defines the execution environment, including the required Python version and dependencies.
Extractor Definition
The extractor is configured with specific capabilities that allow Meltano to interact with the tap's underlying Singer logic:
- Catalog: Supports generating and consuming Singer catalogs.
- Discover: Enables automatic discovery of available WMS entities.
- State: Supports incremental replication by maintaining state between runs.
Sources: meltano.yml:46-55
Dependency Mapping
The project relies on several flext ecosystem libraries to function within Meltano, as defined in the project metadata.
| Dependency | Purpose |
|---|---|
flext-core | Foundation patterns, logging, and result handling. |
flext-oracle-wms | Low-level Oracle WMS API client and data models. |
flext-meltano | Bridge logic for Singer/Meltano orchestration. |
flext-cli | Command-line interface registration and execution. |
Sources: pyproject.toml:31-38, src/flext_tap_oracle_wms/README.md:83-88
Configuration Settings
Meltano manages the tap configuration through environment variables and a structured settings block in meltano.yml.
Connection and Auth Settings
The following table outlines the core configuration parameters required for the WMS integration.
| Setting Name | Type | Secret? | Description |
|---|---|---|---|
base_url | string | No | WMS instance URL (e.g., Oracle Cloud tenant URL). |
auth_method | options | No | Either basic or oauth2. |
username | string | No | WMS username for authentication. |
password | password | Yes | WMS password for basic authentication. |
company_code | string | No | Specific WMS company identifier. |
facility_code | string | No | Specific WMS facility identifier. |
Sources: meltano.yml:60-104
Data Extraction Controls
Users can tune the performance and scope of the extraction using optional settings.
| Setting Name | Default | Description |
|---|---|---|
page_size | 100 | Number of records per API request (max 1250). |
pagination_mode | offset | Strategy for navigating large datasets (offset or cursor). |
start_date | 2024-01-01 | ISO8601 date for the beginning of incremental data. |
request_concurrency | 1 | Number of simultaneous requests to the WMS API. |
retry_limit | 3 | Number of retries for failed network requests. |
Sources: meltano.yml:126-155, src/flext_tap_oracle_wms/tap.py:34-45
Orchestration Flow
The integration follows the standard Meltano workflow where the Tap discovers entities, generates a catalog, and then extracts data based on that catalog.
The diagram above shows the high-level flow from Meltano invocation to data loading into a target warehouse.
Sources: src/flext_tap_oracle_wms/README.md:21-45, meltano.yml:235-245
Internal Tap Implementation
The FlextTapOracleWms class implements the SingerTapBase to facilitate these operations.
The class structure illustrates how the main Tap class manages the WMS client and spawns individual streams for discovered entities.
Sources: src/flext_tap_oracle_wms/tap.py:28-115, src/flext_tap_oracle_wms/README.md:47-50
Replication and State Management
The integration supports different replication methods defined in the Meltano metadata, allowing for efficient data transfer.
Replication Methods
- FULL_TABLE: Used for static or small configuration entities (e.g.,
item,location,company). - INCREMENTAL: Used for high-volume transactional data (e.g.,
inventory,order_hdr,order_dtl).
State Logic
For incremental streams, the tap uses a replication-key (typically last_updated_ts). During execution, the tap generates STATE messages that Meltano stores. In subsequent runs, these bookmarks are passed back to the tap to filter WMS requests for records newer than the last bookmark.
Sources: meltano.yml:195-215, src/flext_tap_oracle_wms/tap.py:182-212
Environment Management
meltano.yml defines specific environments (dev, staging, prod) with varying configurations to optimize performance and logging.
| Environment | Log Level | Page Size | Purpose |
|---|---|---|---|
dev | DEBUG | 100 | Rapid testing with small max_pages_per_stream. |
staging | INFO | 500 | Validation with higher throughput and retries. |
prod | WARNING | 100 | Stable extraction with concurrency and cursor pagination. |
Sources: meltano.yml:7-35
Conclusion
The Meltano integration for the FLEXT Tap Oracle WMS provides a robust framework for enterprise data extraction. By combining the flexibility of Meltano's environment management with the Singer SDK's standardized protocol, the system ensures scalable and maintainable data pipelines into various modern data warehouses. The architecture leverages a modular "flext" ecosystem to maintain a separation of concerns between WMS API communication and Singer orchestration logic.
Sources: src/flext_tap_oracle_wms/README.md:8-13, README.md:12-15