Mock Pipelines for Integration Tests

Relevant source files

The following files were used as context for generating this wiki page:

• tests/unit/complete_mock_pipeline.py
• tests/README.md
• src/flext_oracle_wms/README.md
• examples/README.md
• src/README.md
• examples/03_complete_functionality_showcase.py

Introduction

Mock pipelines in the flext-oracle-wms project provide a realistic, high-performance simulation of the end-to-end data flow between Oracle WMS Cloud and downstream data consumers. These pipelines are designed to validate the integration between the TAP (extraction) and TARGET (loading) components, including schema generation, Singer catalog creation, and DBT transformations, without requiring live WMS credentials or network connectivity.

Sources: tests/unit/complete_mock_pipeline.py:1-13, tests/README.md:108-112

The scope of these tests includes simulating complex Oracle WMS data structures for entities such as inventory, orders, and facilities. By using mock data that follows the Oracle WMS Cloud (LGF API v10) standards, the project ensures that code changes are validated against realistic scenarios while maintaining high execution speed and isolation from external dependencies.

Sources: tests/unit/complete_mock_pipeline.py:73-200, src/flext_oracle_wms/README.md:92-98

Pipeline Architecture and Components

The mock pipeline is structured as a multi-stage process that mimics a production data integration lifecycle. It leverages the CompleteMockPipeline class to orchestrate data generation, schema inference, and simulated loading.

Data Flow Overview

The following diagram illustrates the transformation and movement of data through the mock pipeline stages.

The pipeline flows from raw mock data definition through schema validation, extraction, and finally to business-ready transformations.
Sources: tests/unit/complete_mock_pipeline.py:202-243

Key Pipeline Stages

Stage	Class/Method	Description
Data Generation	CompleteMockPipeline.__init__	Initializes 10+ core WMS entities with realistic sample records (IDs, URLs, timestamps).
Schema Discovery	_generate_complete_singer_schemas	Infers JSON Schema types from Python values and field name patterns.
Cataloging	_create_complete_singer_catalog	Produces a Meltano-compatible Singer catalog with metadata and replication methods.
Extraction	_simulate_tap_extraction	Simulates a Singer TAP generating record messages with _sdc metadata.
Loading	_simulate_target_loading	Mocks the target database insertion, tracking columns loaded and timestamps.
Transformation	_simulate_dbt_transformations	Simulates DBT models (Fact/Dimension) using target tables as sources.

Sources: tests/unit/complete_mock_pipeline.py:73-200, tests/unit/complete_mock_pipeline.py:245-280, tests/unit/complete_mock_pipeline.py:400-470

Mock Entity Definitions

The mock pipeline defines realistic data structures for Oracle WMS entities to ensure integration fidelity.

Entity relationships modeled within the mock data to simulate warehouse operations.
Sources: tests/unit/complete_mock_pipeline.py:75-200

Entity Data Attributes

Entities are populated with specific field types to test type inference:

• Company/Facility: Addresses, codes (RAIZEN, SP001), and status.
• Inventory: Quantities (available, allocated), lot numbers, and expiry dates.
• Orders: Order types (OUTBOUND), priorities, and weights.
• Metadata: create_ts, mod_ts, and create_user for testing incremental replication.

Sources: tests/unit/complete_mock_pipeline.py:73-200

Schema Inference Logic

The pipeline implements a strategy pattern for inferring Singer types. It prioritizes field name suffixes before falling back to value type inspection.

Inference Mapping

• ID Fields: Mapped to integer.
• Timestamp Suffixes (_ts): Mapped to string with date-time format.
• Date Suffixes (_date): Mapped to string with date format.
• Numeric Suffixes (_qty, _weight_kg): Mapped to number.
• Code/Number Suffixes (_code, _nbr): Mapped to string.

Sources: tests/unit/complete_mock_pipeline.py:290-330

    def _infer_type_from_field_name(self, field: str) -> t.AttributeMapping | None:
        """Infer type from field name patterns - Template Method Pattern."""
        field_type_mapping: t.MappingKV[str, t.AttributeMapping] = {
            "id": {"type": "integer"},
            "_ts": {"type": ["string", "null"], "format": "date-time"},
            "_date": {"type": ["string", "null"], "format": "date"},
            "_qty": {"type": ["number", "null"]},
        }
        # ... implementation ...

Sources: tests/unit/complete_mock_pipeline.py:304-315

Integration with Singer Protocol

The mock pipeline generates a Singer catalog to validate compatibility with platforms like Meltano.

Replication Method Logic

The pipeline dynamically determines the replication method based on the presence of specific audit fields in the schema properties.

1. INCREMENTAL: Selected if fields like mod_ts, create_ts, or updated_at exist.
2. FULL_TABLE: Default method if no incremental keys are found.

Sources: tests/unit/complete_mock_pipeline.py:365-385

Singer Metadata

The pipeline injects _sdc (Singer Data Capture) metadata into every record to simulate real-world extraction:

• _sdc_extracted_at: The ISO8601 timestamp of extraction.
• _sdc_entity: The source entity name.
• _sdc_sequence: An integer sequence for ordering.
• _sdc_record_hash: A UUID for deduplication.

Sources: tests/unit/complete_mock_pipeline.py:406-412

Transformation Simulation (DBT)

The pipeline simulates a downstream transformation layer by defining Dimension (SCD Type 1/2), Fact, and Aggregate models based on the mock loaded tables.

Model Name	Source Tables	Transformation Type	Business Purpose
dim_item	raw_oracle_wms_item	SCD_TYPE_2	Product hierarchy tracking.
fact_inventory	raw_oracle_wms_inventory	SNAPSHOT	Daily inventory snapshots.
fact_orders	order_hdr, order_dtl	TRANSACTION	Order transaction analysis.
rpt_inventory_dashboard	agg_daily_inventory, dim_facility	REPORTING	Executive dashboarding.

Sources: tests/unit/complete_mock_pipeline.py:440-485

Validation and Reporting

Upon completion, the pipeline generates a comprehensive JSON summary and several artifact files in the complete_pipeline_results directory. These artifacts allow developers to inspect:

• singer_catalog_[timestamp].json: The generated catalog.
• pipeline_summary_[timestamp].json: A high-level report including duration, record counts, and Singer protocol compliance status.

Sources: tests/unit/complete_mock_pipeline.py:512-570

Quality Targets

Integration tests using these pipelines are held to specific quality standards:

• Singer Protocol Compliance: Must be True.
• Meltano Readiness: Must be True.
• Execution Time: Aimed for FAST_EXECUTION.

Sources: tests/unit/complete_mock_pipeline.py:585-595, tests/README.md:148-155

Summary

Mock pipelines serve as a critical component of the flext-oracle-wms testing strategy. By simulating the full lifecycle from raw Oracle WMS entity data to business-ready DBT models, these pipelines provide a fast, reliable, and comprehensive way to validate data integration patterns without the overhead of real API connections.

Sources: tests/unit/complete_mock_pipeline.py:1-13, src/README.md:67-74