Metadata Handling
Relevant source files
The following files were used as context for generating this wiki page:
Metadata handling in the flext-dbt-oracle-wms project encompasses the discovery, extraction, and transformation of Oracle WMS entity definitions and record structures into dbt-compatible models. This system serves as the bridge between raw Oracle WMS domain data and structured analytics, ensuring that entity schemas are correctly identified and mapped before the transformation pipeline executes.
The architecture relies on a "Simple API" facade that coordinates between a domain-specific WMS client and a dbt transformer. It handles specific entity metadata for inventory and shipments, while also managing the generation of dbt model definitions (staging and marts) based on discovered source metadata.
Sources: src/flext_dbt_oracle_wms/_simple_api_metadata.py:1-20, src/flext_dbt_oracle_wms/_utilities/client.py:20-40
Metadata Discovery and Extraction
The primary mechanism for metadata handling is the FlextDbtOracleWmsMetadata class, which provides high-level methods to lookup entity definitions and specific record info from Oracle WMS.
Entity Discovery
The system first performs discovery to identify available Oracle WMS entities. This is facilitated by the discover_oracle_wms_entities method, which interacts with the underlying domain client to return a sequence of available entity names.
Sources: src/flext_dbt_oracle_wms/_utilities/client.py:44-51, src/flext_dbt_oracle_wms/_simple_api_metadata.py:34-40
Extraction Logic
Metadata extraction focuses on two primary domains: Inventory and Shipments. The extract_wms_metadata method captures counts and status information for these entities.
| Entity Type | Identifier Fields Used for Metadata |
|---|---|
| Inventory Items | item_id, item_number, id, sku |
| Shipments | shipment_id, tracking_number, id |
Sources: src/flext_dbt_oracle_wms/_simple_api_metadata.py:46-70
The following diagram illustrates the flow of a metadata extraction request through the API facade to the domain client.
A brief explanation: The Metadata Facade first discovers available entities via the Client, then requests specific record metadata (like IDs and SKUs) for inventory and shipments to return a summarized Result object.
Sources: src/flext_dbt_oracle_wms/_simple_api_metadata.py:32-75
dbt Model Metadata Generation
Beyond raw data extraction, the system handles metadata at the architectural level by generating dbt model definitions. This process transforms WMS entity names into staging model objects.
Model Generator
The ModelGenerator class in models.py uses metadata (entity names) to produce DbtModel objects. These objects contain the SQL content, target schemas (e.g., wms_staging), and materialization strategies required for dbt execution.
Sources: src/flext_dbt_oracle_wms/models.py:144-180
Transformation Mapping
When metadata is extracted, it is mapped to specific dbt model types defined in the project configuration:
- Staging: Raw WMS data transformation (materialized as views).
- Marts: Business-ready data models (materialized as tables).
- Metrics: Enterprise KPIs (materialized as views).
Sources: dbt_project.yml:25-55
This sequence shows how discovered entity metadata is passed to the generator to produce the staging layer models dynamically.
Sources: src/flext_dbt_oracle_wms/_simple_api_models.py:30-55
Metadata Validation and Business Rules
Metadata handling includes a validation layer to ensure that extracted records meet the requirements of the dbt models.
Field Validation
The validate_oracle_wms_data method checks extracted metadata against required_fields_per_entity defined in the project settings. If mandatory metadata fields (like item_id for inventory) are missing, the transformation is aborted.
Sources: src/flext_dbt_oracle_wms/_utilities/client.py:141-160
Business Rule Enforcement
The FlextDbtOracleWmsTransformer applies business rules to the metadata. For example, RawItemRecord metadata is validated against a Pydantic model to ensure type safety before being converted into a FlextDbtOracleWmsItemDimension.
Sources: src/flext_dbt_oracle_wms/models.py:75-105
| Component | Metadata Responsibility |
|---|---|
RawItemRecord | Validates raw item metadata (item_id, item_number, item_description). |
DbtModel | Stores dbt-specific metadata like sql_content, dependencies, and wms_entity_type. |
Service | Generates workflow recommendations based on entity metadata volume. |
Sources: src/flext_dbt_oracle_wms/models.py:34-45, src/flext_dbt_oracle_wms/models.py:112-140, src/flext_dbt_oracle_wms/utilities.py:53-75
Technical Implementation Details
Metadata operations are implemented through cooperative inheritance in the Simple API.
class FlextDbtOracleWmsMetadata(FlextDbtOracleWmsBase):
"""Metadata and direct Oracle WMS entity lookup operations."""
def fetch_wms_inventory_info(self, item_id: str) -> p.Result[m.Dict]:
"""Get inventory item data from the Oracle WMS domain client."""
inventory_result = self._extract_entity_records(
"items",
[item_id],
("item_id", "item_number", "id", "sku"),
)
return r[m.Dict].ok(m.Dict.model_validate(inventory_result.value[0]))
Sources: src/flext_dbt_oracle_wms/_simple_api_metadata.py:84-96
In dbt_project.yml, variables are used to handle schema-level metadata, such as the oracle_wms_schema (defaulting to wms_raw) and the list of standard wms_entities (allocation, order_hdr, inventory, etc.).
Sources: dbt_project.yml:64-80
Summary
Metadata handling provides the foundational discovery and validation logic needed to ensure the Oracle WMS to dbt pipeline is both dynamic and robust. It manages everything from individual record lookups to the automated generation of staging model SQL.