Workflow Management
Relevant source files
The following files were used as context for generating this wiki page:
The Workflow Management system in flext-dbt-oracle-wms orchestrates the end-to-end data pipeline between Oracle WMS and dbt. It handles the transition from raw entity discovery and extraction to automated dbt model generation and final data transformation. The system is designed around a facade pattern that coordinates domain-backed clients, service helpers, and transformation logic to ensure consistent execution and tracking.
Sources: src/flext_dbt_oracle_wms/_simple_api_workflow.py:1-15, src/flext_dbt_oracle_wms/_utilities/client.py:15-30
Core Architecture and Components
The workflow is managed primarily by the FlextDbtOracleWmsWorkflow class, which integrates multiple layers of the application including metadata discovery, client-side extraction, and dbt execution.
Key Components
| Component | Description |
|---|---|
FlextDbtOracleWmsWorkflow | Orchestrates high-level workflow execution and lifecycle operations. |
FlextDbtOracleWmsClient | Manages low-level interactions with Oracle WMS and Meltano runners. |
DbtOracleWms.Service | Provides tracking, logging, and performance recommendations. |
FlextDbtOracleWmsTransformer | Handles the mapping of raw WMS records into dbt-compatible dimension models. |
ModelGenerator | Dynamically generates dbt staging models based on discovered Oracle sources. |
Sources: src/flext_dbt_oracle_wms/_simple_api_workflow.py:18-20, src/flext_dbt_oracle_wms/_utilities/client.py:18-30, src/flext_dbt_oracle_wms/utilities.py:53-55, src/flext_dbt_oracle_wms/models.py:56-60, src/flext_dbt_oracle_wms/models.py:164-168
Workflow Service Interaction
The following diagram illustrates the relationship between the workflow facade and its supporting services.
Pipeline Orchestration Logic
The primary entry point for execution is run_oracle_wms_to_dbt_workflow. This method coordinates a multi-stage process involving entity resolution, tracking initialization, optional model generation, and data transformation.
Execution Flow
- Resolution: Resolves specific entities (e.g., inventory items or shipments) to be processed.
- Tracking: Initializes a tracking context with a unique
tracking_id. - Model Generation: (Optional) Dynamically generates dbt staging models from WMS sources.
- Pipeline Execution: Depending on flags, either runs the full Meltano/dbt pipeline or performs a metadata-only extraction.
- Logging: Captures the final success or failure state in the tracking service.
Data Extraction and Validation
Data extraction is performed by the FlextDbtOracleWmsClient, which utilizes a real domain client for Oracle WMS. Extracted data must pass validation checks before being passed to the transformation layer.
Validation Requirements
The system enforces business rules and schema integrity:
- Required Fields: Validates records against
required_fields_per_entitydefined in settings. - Business Rules: Uses
FlextDbtOracleWmsTransformerto validate domain-specific rules. - Batch Optimization: The
Servicecomponent provides recommendations if entity counts exceed thePERFORMANCE_RECOMMENDATION_THRESHOLD(default 20).
Sources: src/flext_dbt_oracle_wms/_utilities/client.py:133-157, src/flext_dbt_oracle_wms/utilities.py:44, src/flext_dbt_oracle_wms/utilities.py:80-92
Extraction Sequence
The sequence below shows how the client interacts with the WMS domain and Meltano runner.
Metadata and Model Generation
The system supports a "metadata-first" approach where it can discover available WMS entities and generate the necessary dbt infrastructure before running transformations.
Metadata Extraction Features
The metadata layer retrieves:
- Available Entities: List of all discoverable tables/views in Oracle WMS.
- Inventory Metrics: Counts and details for specific item identifiers.
- Shipment Tracking: Metadata related to shipment IDs and tracking numbers.
Sources: src/flext_dbt_oracle_wms/_simple_api_metadata.py:30-68
Automated dbt Model Generation
The ModelGenerator creates staging models using a standard pattern:
- Materialization: Defaults to
view. - Schema: Targets
wms_staging. - SQL Logic: Generates
select * from {{ source('oracle_wms', 'source_name') }}.
Sources: src/flext_dbt_oracle_wms/models.py:176-200
Conclusion
Workflow Management in this project provides a robust framework for bridging Oracle WMS data with dbt analytics. By combining automated discovery, strict validation, and flexible orchestration via the simple API facade, the system ensures that WMS-to-dbt pipelines are trackable, maintainable, and aligned with enterprise business rules.