Data Models and Typings
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The flext-dbt-oracle-wms project leverages a structured set of data models and type definitions to facilitate the transformation of Oracle Warehouse Management System (WMS) data into DBT-compatible structures. This system acts as a bridge between raw enterprise data and analytical marts, ensuring data integrity through Pydantic-based validation and standardized protocols.
The scope of "Data Models and Typings" covers the domain objects used for record validation, the metadata structures for generating DBT models, and the protocols that define how clients and transformers interact within the pipeline. This architecture ensures that data extracted from Oracle WMS conforms to business rules before being materialized in the database.
Domain Models
Domain models are organized within the FlextDbtOracleWmsModels namespace, which inherits from both base Meltano and Oracle WMS model structures. These models define the schema for various WMS entities such as items and inventory.
WMS Record Models
These models represent the structure of data as it is received from or transformed for the Oracle WMS environment.
| Model Name | Purpose | Key Fields |
|---|---|---|
RawItemRecord | Validates raw input from Oracle WMS. | item_id, item_number, item_description |
FlextDbtOracleWmsItemDimension | Represents an item dimension for WMS analytics. | item_id, item_number, item_description |
DbtModel | Metadata for generating DBT SQL files. | name, dbt_model_type, sql_content, wms_entity_type |
Sources: src/flext_dbt_oracle_wms/models.py:22-132
Model Relationship Diagram
The following diagram illustrates the relationship between raw records, transformed dimensions, and the metadata used for DBT generation.
Transformation Logic
Transformation is handled by the FlextDbtOracleWmsTransformer class, which manages the lifecycle of data from raw dictionary format to typed Pydantic models and finally to DBT-compatible output.
Transformer Flow
- Validation: Raw records are passed to
RawItemRecord.model_validate(). - Model Creation: Validated data is used to instantiate
FlextDbtOracleWmsItemDimension. - Dictionary Export: The
to_dbt_dict()method converts the typed model back into a flat dictionary for DBT loading.
Integration Protocols
Protocols define the expected interface for various components, allowing for decoupled implementations and easier testing. The primary protocols are housed in FlextDbtOracleWmsProtocols.
| Protocol | Description | Key Methods |
|---|---|---|
Client | Public surface for WMS workflows. | discover_oracle_wms_entities, extract_oracle_wms_data, run_full_oracle_wms_to_dbt_pipeline |
Dbt | Interface for DBT execution. | run_dbt_models, test_dbt_models |
WmsIntegration | Low-level data extraction. | extract_wms_inventory_data |
Sources: src/flext_dbt_oracle_wms/protocols.py:18-72
DBT Project Configuration
The data models are materialized according to the configuration defined in dbt_project.yml. This file specifies the schema and materialization strategies for the different layers of the WMS data architecture.
| Layer | Target Schema | Materialization | Tags |
|---|---|---|---|
| Staging | wms_staging | view | staging, oracle_wms |
| Marts | wms_marts | table | marts, oracle_wms, business |
| Metrics | wms_metrics | view | metrics, kpi |
Sources: dbt_project.yml:25-60
Monitoring and Tracking Models
The system uses specific payload structures for tracking the execution of workflows. This is primarily managed through the Service class within the utilities namespace.
# Workflow tracking payload example derived from track_workflow_execution
tracking_payload = {
"workflow_name": "oracle_wms_to_dbt",
"workflow_type": "dbt_oracle_wms",
"entity_names": "items,shipments",
"status": "running",
"tracking_id": "oracle_wms_to_dbt:dbt_oracle_wms"
}
Sources: src/flext_dbt_oracle_wms/utilities.py:83-102
Summary
The Data Models and Typings in flext-dbt-oracle-wms provide a robust framework for enterprise data integration. By using Pydantic for domain models, the project ensures that Oracle WMS data is strictly validated before it enters the DBT pipeline. The separation of concerns through protocols and the clear definition of staging/mart layers in the DBT configuration allow for scalable and maintainable data transformations.