Singer Target Implementation
Relevant source files
The following files were used as context for generating this wiki page:
The Singer Target Implementation for Oracle WMS provides a typed, production-grade interface for receiving and processing Singer protocol messages (SCHEMA, RECORD, and STATE) and loading them into Oracle Warehouse Management System (WMS) Cloud. This system leverages the flext-* ecosystem to ensure strict typing, observability, and efficient data transformation.
The implementation is centered around a runtime client that manages stream metadata via a Catalog Manager, performs data transformations to match Oracle-friendly payloads, and handles the orchestration of the Singer lifecycle from setup to cleanup.
Sources: README.md:9-11, src/flext_target_oracle_wms/_utilities/client.py:90-105
Architecture and Components
The system architecture is modular, separating the protocol handling from the business logic of data transformation and table management. The primary entry point is the Target class within the FlextTargetOracleWmsUtilitiesClient namespace.
Core Components
| Component | Description |
|---|---|
Target | Orchestrates the Singer lifecycle, handles message dispatching, and maintains stream state. |
CatalogManager | Manages in-memory storage for Singer stream metadata and schema definitions. |
StreamProcessor | High-level processor that coordinates between table lookups and record transformations. |
WMSTableManager | Maintains the mapping between Singer stream names and Oracle WMS table names (usually uppercased). |
WMSDataTransformer | Converts Singer record payloads into Oracle-friendly formats using a type converter. |
WMSTypeConverter | Handles scalar conversions (e.g., Booleans to strings, objects to JSON strings). |
Sources: src/flext_target_oracle_wms/_utilities/client.py:27-105, src/flext_target_oracle_wms/_utilities/helpers.py:38-125
Relationship Diagram
The following diagram illustrates the relationship between the main runtime classes and their responsibilities.
The Target class acts as the central coordinator, delegating specific processing tasks to specialized helper classes.
Sources: src/flext_target_oracle_wms/_utilities/client.py:27-130, src/flext_target_oracle_wms/utilities.py:22-38
Data Processing Flow
The target processes data by reading JSON lines from stdin, parsing them into Singer messages, and dispatching them based on their type.
Message Handling Lifecycle
- SCHEMA: The target registers the stream in the
CatalogManagerand initializes the table mapping via theWMSTableManager. - RECORD: The target validates that a schema has been registered for the stream, then passes the record to the
StreamProcessorfor transformation. Keys are typically uppercased to match Oracle WMS conventions. - STATE: The target logs the state value, which is used for bookmarking by the Singer source (tap).
The flow ensures that data is only processed for known streams and that all records undergo necessary transformations before being sent to the destination.
Sources: src/flext_target_oracle_wms/_utilities/client.py:155-188, src/flext_target_oracle_wms/_utilities/helpers.py:75-105, tests/unit/test_workflow.py:53-70
Configuration and Validation
Target execution requires valid configuration settings, primarily focused on Oracle WMS authentication and endpoint details.
WMS Target Configuration
| Parameter | Type | Description |
|---|---|---|
base_url | str | The REST API base URL for the Oracle WMS Cloud instance. |
auth_method | Literal | Authentication strategy: oauth2, basic, or api_key. Defaults to oauth2. |
username | str | Authentication username. |
password | SecretStr | Authentication password (handled as sensitive). |
batch_size | int | Number of records per batch write. Default is defined in constants. |
load_method | str | The strategy for writing records (e.g., APPEND_ONLY, UPSERT). |
Sources: src/flext_target_oracle_wms/models.py:40-65, src/flext_target_oracle_wms/_utilities/helpers.py:23-35
Validation Logic
The Validation class and the WmsTargetConfig model perform several checks:
- Presence of required fields:
base_url,username, andpassword. - Verification that the
load_methodis within the set ofVALID_LOAD_METHODS. - Ensuring
batch_sizeis a positive integer.
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:26-36, src/flext_target_oracle_wms/models.py:82-90
Data Transformation
The WMSDataTransformer is responsible for preparing records for the Oracle WMS API. This involves mapping field types and standardizing key names.
Key Transformation Patterns
- Case Sensitivity: Record keys are converted to uppercase (e.g.,
qtybecomesQTY). - Scalar Conversion: The
WMSTypeConverterhandles the translation of Singer types to WMS-compatible values:- Booleans: Converted to strings (
True->"True"). - Objects/Arrays: Serialized into JSON strings.
- Numbers: Cast to
floatorintbased on the presence of a decimal point.
- Booleans: Converted to strings (
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:44-58, src/flext_target_oracle_wms/_utilities/helpers.py:85-105, tests/unit/test_wms_patterns.py:40-80
Payload Transformation Example
# Source Singer RECORD
{
"type": "RECORD",
"stream": "inventory",
"record": {"item_id": "ABC", "qty": 10, "is_active": true}
}
# Transformed Oracle WMS Payload (Conceptual)
{
"ITEM_ID": "ABC",
"QTY": 10,
"IS_ACTIVE": "True"
}
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:100-105, tests/unit/test_wms_patterns.py:83-93
CLI Execution
The FlextTargetOracleWmsCli provides a standard entry point for running the target as a standalone process.
Summary
The Singer Target Implementation for Oracle WMS is a structured ETL component that ensures data received from Singer taps is correctly typed and transformed for Oracle WMS Cloud. By utilizing a modular architecture of Managers and Transformers, it provides a robust and testable pipeline for enterprise data integration. Key features include automatic table name mapping, rigorous configuration validation, and protocol-compliant message handling.