End-to-End Pipeline Testing
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
End-to-End (E2E) pipeline testing for the FLEXT Tap Oracle WMS validates the complete data extraction journey from Command Line Interface (CLI) execution to final Singer-compliant data output. These tests focus on verifying that the tap interacts correctly with the Oracle Warehouse Management System (WMS) to perform discovery, schema generation, and data extraction under real-world or simulated production conditions.
The scope of E2E testing encompasses the entire Singer protocol lifecycle, ensuring that the system adheres to schema, record, and state message standards while maintaining performance and data quality. This testing layer sits above Unit Testing and Integration Testing, providing final validation of user workflows and business logic enforcement across component boundaries.
Sources: tests/e2e/README.md:17-21, tests/README.md:17-19, tests/e2e/test_e2e.py:1-6
Testing Architecture and Strategy
The E2E testing framework is designed to validate complete user workflows. However, due to external WMS dependencies, the current implementation relies on a strategic re-enabling plan involving a comprehensive mock environment.
Core Workflows
The testing strategy covers three primary CLI-driven workflows:
- Discovery Workflow: Validates that the tap can connect to the WMS, discover entities (like allocations or inventory), and generate a valid
catalog.json. - Extraction Workflow: Validates the loading of a catalog, execution of authentication, data retrieval with pagination, and generation of Singer messages (SCHEMA, RECORD, STATE).
- Configuration Validation: Ensures the tap handles invalid inputs gracefully, providing clear error messages and correct exit codes.
Sources: tests/e2e/README.md:46-77, tests/integration/test_functional.py:61-65
E2E System Flow
The following diagram illustrates the high-level flow of an E2E extraction test, from CLI invocation to output validation.
This diagram shows the sequential stages of a complete E2E pipeline test, ensuring each component interacts correctly to produce validated output.
Sources: tests/e2e/README.md:58-71, tests/e2e/test_e2e.py:343-370
Mock Environment Infrastructure
To support reliable E2E testing without constant access to a live Oracle WMS instance, a E2EMockWMSEnvironment is utilized. This infrastructure simulates the WMS API server and manages test data sets.
Components of the Mock Environment
| Component | Responsibility |
|---|---|
| MockWMSServer | Simulates WMS API endpoints and HATEOAS pagination. |
| TestDataManager | Manages small, medium, and large datasets for performance and stress testing. |
| ConfigGenerator | Creates various configuration profiles, including valid, invalid, and edge-case settings. |
| OutputValidator | Verifies Singer protocol compliance and JSON schema accuracy of the output. |
Sources: tests/e2e/README.md:105-122, tests/e2e/README.md:126-147
Mock Data Management
Test data is organized into fixtures representing different scales:
- Small Dataset: 100 items, 500 inventory records (Quick validation).
- Medium Dataset: 10,000 items (Performance benchmarking).
- Large Dataset: 1,000,000 records (Stress testing).
Sources: tests/e2e/README.md:130-139
Singer Protocol Compliance Validation
A critical part of E2E testing is ensuring the output matches the Singer specification. The test_complete_singer_protocol_compliance function validates specific requirements across discovered streams.
def test_complete_singer_protocol_compliance(
self,
real_config: FlextTapOracleWmsSettings,
) -> None:
"""E2E: Test complete Singer protocol compliance."""
tap_instance = FlextTapOracleWms(settings=real_config.model_dump(mode="json"))
catalog = self._catalog(tap_instance)
for stream in catalog.streams:
assert isinstance(stream.tap_stream_id, str)
schema = stream.schema_definition
assert schema["type"] == "t.JsonValue"
table_meta = next((meta for meta in stream.metadata if meta.breadcrumb == []), None)
assert "replication-method" in table_meta.metadata
Sources: tests/e2e/test_e2e.py:273-306
Message Sequence Validation
Tests verify that:
- The stream starts with SCHEMA messages.
- RECORD messages correspond to a previously declared schema.
- STATE messages correctly update bookmarks for incremental extraction.
Sources: tests/e2e/README.md:195-214, tests/e2e/test_e2e.py:126-157
Performance and Resource Monitoring
E2E tests include benchmarks to ensure the pipeline meets production requirements. Performance is measured using specific indicators:
- Throughput: Validated at >100 records per second.
- Discovery Time: Must complete in under 300 seconds (5 minutes).
- Memory Usage: Limited to a maximum of 512MB.
- CPU Usage: Monitored to stay below 80% during peak extraction.
Sources: tests/e2e/README.md:223-238, tests/e2e/README.md:255-265, tests/e2e/test_e2e.py:321-331
Error Recovery and Resilience
The E2E pipeline is tested against network failures and configuration errors to ensure graceful degradation.
Network Failure Scenario Flow
This sequence diagram shows the tap's behavior when encountering a simulated network failure during a paginated extraction process.
Sources: tests/e2e/README.md:275-296, tests/integration/README.md:261-274
Summary of Quality Standards
E2E tests are governed by strict quality requirements to ensure the FLEXT Tap Oracle WMS is production-ready.
| Requirement | Description |
|---|---|
| Workflow Coverage | Tests must cover entire user journeys from CLI to output. |
| Reliability | Tests must pass consistently in CI/CD without flakiness. |
| Data Quality | Validation of primary keys, replication keys, and nullable field documentation. |
| Singer Compliance | Full validation of the Singer message format and sequence. |
Sources: tests/e2e/README.md:329-335, tests/e2e/test_e2e.py:209-224
Conclusion
End-to-End Pipeline Testing provides the final verification layer for the FLEXT Tap Oracle WMS, ensuring that all components—authentication, discovery, and extraction—work in harmony to produce high-quality, Singer-compliant data. By utilizing a sophisticated mock infrastructure and rigorous performance benchmarks, the testing suite guarantees that the tap can handle complex Oracle WMS data structures reliably in a production environment.
Sources: tests/e2e/README.md:389-391, tests/e2e/test_e2e.py:405-408