Catalog Management
Relevant source files
The following files were used as context for generating this wiki page:
Catalog Management within the flext-target-oracle-wms project is responsible for maintaining the metadata and structural definitions of data streams received via the Singer protocol. It acts as an in-memory registry that tracks stream schemas, key properties, and mapping relationships between source streams and target Oracle WMS tables. Sources: src/flext_target_oracle_wms/_utilities/client.py:27-29, tests/unit/test_catalog.py:10-12
The system ensures that every incoming RECORD message has a corresponding registered schema before processing. By managing SCHEMA messages, the catalog provides the necessary context for data transformation and validation, ensuring that source data is correctly mapped to the uppercase naming conventions required by Oracle WMS. Sources: src/flext_target_oracle_wms/_utilities/client.py:117-121, src/flext_target_oracle_wms/_utilities/helpers.py:113-115
Catalog Architecture and Components
The catalog management system is primarily implemented through the CatalogManager and supported by mapping utilities that translate Singer-specific schemas into internal catalog entries.
CatalogManager
The CatalogManager is an in-memory storage component that manages SingerCatalogEntry objects. It provides methods to register new streams from SCHEMA messages and retrieve them during record processing.
| Method | Purpose | Returns |
|---|---|---|
add_stream | Validates and registers a new stream schema in the catalog. | p.Result[bool] |
get_stream | Retrieves the stored metadata for a specific stream. | p.Result[SingerCatalogEntry] |
Sources: src/flext_target_oracle_wms/_utilities/client.py:30-65, tests/unit/test_catalog.py:25-35
Schema Mapping Logic
The WMSSchemaMapper works alongside the catalog to normalize stream definitions. A key requirement for Oracle WMS integration is the conversion of stream names to uppercase table names.
The diagram above shows how raw Singer schema messages are transformed into normalized catalog entries. Sources: src/flext_target_oracle_wms/_utilities/helpers.py:133-157
Data Flow: Schema Registration
When the target receives a SCHEMA message, it initiates a registration flow across multiple sub-components to ensure the catalog and table mappings are synchronized.
This sequence demonstrates the coordination between the catalog and table management during the initialization of a stream. Sources: src/flext_target_oracle_wms/_utilities/client.py:126-140, tests/unit/test_sinks.py:65-72
Stream Metadata Storage
Metadata for each stream is stored as a SingerCatalogEntry. This structure maintains the relationship between the tap-provided stream ID and the internal representation used by the target.
Catalog Entry Fields
| Field | Source | Description |
|---|---|---|
stream | schema_msg.stream | The original stream name from the Singer source. |
tap_stream_id | schema_msg.stream | Unique identifier used by the source tap. |
table_name | stream.upper() | The destination table name in Oracle WMS. |
key_properties | schema_msg.key_properties | List of fields used as primary keys for the stream. |
Sources: src/flext_target_oracle_wms/_utilities/helpers.py:151-158, tests/unit/test_catalog.py:59-65, tests/unit/test_wms_patterns.py:108-115
Catalog Validation and Constraints
The catalog enforces several constraints to maintain data integrity during the target lifecycle:
- Schema Presence: A
RECORDmessage for a stream that has not been registered in the catalog will result in a failure. Sources: src/flext_target_oracle_wms/_utilities/client.py:118-121, tests/unit/test_target.py:100-105 - Overwrite Policy: Registering a stream with the same name multiple times overwrites the previous entry, allowing for mid-stream schema updates. Sources: tests/unit/test_catalog.py:37-46, tests/unit/test_workflow.py:73-85
- Validation: The catalog uses
model_validateto ensure that incoming schema messages adhere to the expectedSingerSchemaMessagestructure before they are committed to memory. Sources: src/flext_target_oracle_wms/_utilities/client.py:38-40
Conclusion
Catalog Management is the foundational layer for stream awareness in flext-target-oracle-wms. By centralizing stream metadata and enforcing uppercase naming conventions for Oracle WMS, it provides the deterministic mapping required for the StreamProcessor and WMSDataTransformer to operate correctly. Sources: src/flext_target_oracle_wms/_utilities/client.py:102-110, tests/unit/test_sinks.py:50-55