High-Level Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-oracle-wms project provides a structured integration layer for transforming Oracle Warehouse Management System (WMS) data using DBT (Data Build Tool). It acts as an orchestrator that bridges the gap between raw Oracle WMS data extraction and business-ready data modeling within the FLEXT ecosystem. The architecture is designed around a multi-layered approach, utilizing flext-meltano for transformation execution and flext-oracle-wms for direct WMS API communication.
Sources: README.md:9-11, src/flext_dbt_oracle_wms/_utilities/client.py:16-25
System Component Overview
The architecture consists of several interconnected components that handle specific phases of the data lifecycle, from discovery and extraction to validation and final transformation.
Component Interaction Diagram
The following diagram illustrates the relationship between the internal classes and external library dependencies.
The diagram shows how the Public Facade coordinates internal logic while delegating specialized tasks to external FLEXT libraries.
Sources: src/flext_dbt_oracle_wms/_utilities/client.py:20-25, src/flext_dbt_oracle_wms/_simple_api_workflow.py:16-25
Core Architecture Layers
| Layer | Responsibility | Key Classes/Files |
|---|---|---|
| Public Surface | Entry point for CLI and external consumers. | FlextDbtOracleWmsWorkflow, main |
| Orchestration | Manages the end-to-end pipeline execution and state tracking. | FlextDbtOracleWmsClient, FlextDbtOracleWmsWorkflow |
| Domain Logic | Handles WMS-specific data models and business rule validation. | FlextDbtOracleWmsTransformer, FlextDbtOracleWmsModels |
| Infrastructure | Executes DBT runs via Meltano and communicates with WMS APIs. | FlextMeltanoLibraryRunner, oracle_wms_u.OracleWms.Client |
Sources: src/flext_dbt_oracle_wms/_simple_api_workflow.py:27-38, src/flext_dbt_oracle_wms/_utilities/client.py:33-100
Data Pipeline Flow
The project implements a comprehensive "Oracle WMS to DBT" pipeline. This workflow is synchronous and follows a strict sequence of operations to ensure data integrity.
Pipeline Sequence Diagram
This diagram depicts the flow from entity discovery through extraction and validation to the final DBT transformation trigger.
Sources: src/flext_dbt_oracle_wms/_utilities/client.py:59-100, src/flext_dbt_oracle_wms/_simple_api_workflow.py:35-80
Pipeline Execution Logic
- Discovery: Resolves entity names (e.g.,
allocation,inventory,shipment) to be processed. - Extraction: Fetches records from the Oracle WMS source via the domain client.
- Validation: Checks for required fields (e.g.,
item_id,sku) and applies business rules defined inFlextDbtOracleWmsSettings. - Transformation: Maps raw records to internal models (e.g.,
FlextDbtOracleWmsItemDimension) and triggers DBT model execution.
Sources: src/flext_dbt_oracle_wms/_utilities/client.py:108-160, src/flext_dbt_oracle_wms/models.py:53-80
DBT Integration and Modeling
The project utilizes a 5-layer DBT architecture, standardized via flext-meltano. This structure separates raw data from business-ready analytical assets.
Model Layer Hierarchy
| Layer | Materialization | Schema | Purpose |
|---|---|---|---|
| Staging | view | wms_staging | Raw data cleaning and initial casting. |
| Marts | table | wms_marts | Business-ready entities (Operational/Analytical). |
| Metrics | view | wms_metrics | Enterprise KPIs and dashboards. |
| Snapshots | table | wms_snapshots | Change tracking and SCD Type 2 logic. |
| Seeds | table | wms_reference | Static reference and mapping data. |
Sources: dbt_project_migrated.yml:30-58, dbt_project.yml:25-60
Dynamic Model Generation
The ModelGenerator class allows for the programmatic creation of DBT models based on discovered Oracle sources. This facilitates automated scaling as new entities are added to the WMS source.
# Example of generated staging model configuration
# Source: src/flext_dbt_oracle_wms/models.py:186-193
models = [
FlextDbtOracleWmsModels.DbtOracleWms.DbtModel(
name=f"stg_wms_{source}",
dbt_model_type="staging",
wms_entity_type=source,
sql_content=f"select * from {{{{ source('oracle_wms', '{source}') }}}}"
)
for source in oracle_sources
]
Sources: src/flext_dbt_oracle_wms/models.py:177-200
Configuration and Governance
System behavior is governed by FlextDbtOracleWmsSettings, which provides a Single Source of Truth (SSOT) for environment variables, entity requirements, and performance thresholds.
Key Configuration Parameters
- WMS Connection: Defined by
oracle_wms_base_urlandoracle_wms_environment. - Performance:
dbt_threadsandPERFORMANCE_RECOMMENDATION_THRESHOLDcontrol execution concurrency and batching alerts. - Validation:
required_fields_per_entitymaps specific WMS entities to their mandatory schema fields.
Sources: src/flext_dbt_oracle_wms/utilities.py:23-60, src/flext_dbt_oracle_wms/_utilities/client.py:140-150
Conclusion
The flext-dbt-oracle-wms architecture provides a robust, typed, and scalable framework for Oracle WMS data integration. By layering a public API over specialized clients and leveraging standardized DBT patterns, the system ensures consistent data quality and operational visibility throughout the transformation lifecycle.