Error Handling & Diagnostics
Relevant source files
The following files were used as context for generating this wiki page:
Error Handling and Diagnostics in the flext-tap-oracle-wms project is a multi-layered system designed to ensure data integrity, system resilience, and clear visibility into extraction failures. It encompasses configuration validation, network resilience, authentication error recovery, and comprehensive diagnostic logging following the FLEXT ecosystem patterns.
The system utilizes a combination of Pydantic models for upfront validation, custom exception hierarchies for runtime errors, and the FLEXT r pattern for consistent result handling. This approach allows the tap to fail loudly during initialization if critical environment variables are missing while providing graceful recovery or detailed error reporting during data extraction phases.
Sources: src/flext_tap_oracle_wms/README.md, tests/unit/README.md, AGENTS.md
Architectural Overview
The diagnostics architecture is integrated into the core Singer SDK components (Tap and Stream) and specialized validation modules.
Error Handling Components
| Component | Responsibility | Relevant Files |
|---|---|---|
| ConfigValidator | Enforces business rules and validates configuration models. | config_validator.py |
| CriticalValidation | Checks environment completeness and critical variable presence. | critical_validation.py |
| Exceptions Hierarchy | Custom error types for specific WMS and Singer failures. | exceptions.py |
| Resilience Logic | Retry mechanisms, backoff strategies, and timeout management. | client.py, settings.py |
| Observability | Integration with flext-observability for monitoring and health checks. | README.md |
Sources: src/flext_tap_oracle_wms/README.md, tests/integration/README.md:95-108
Diagnostic Data Flow
The following diagram illustrates how a request flows through the system and where errors are intercepted and diagnosed:
Validation & Pre-flight Diagnostics
Pre-flight diagnostics focus on ensuring the environment is capable of supporting the extraction process before any API calls are made.
Configuration Validation
The project uses FlextTapOracleWmsSettings (built on Pydantic) to enforce type constraints and required fields. Business rules, such as ensuring an entity is not both included and excluded, are validated via validate_business_rules().
Key validation rules include:
- Duplicate Detection: Ensuring
include_entitiesdoes not contain duplicates. - Date Formats: Validating that
start_dateandend_datefollow ISO8601 formats and that start precedes end. - Numeric Constraints: Ensuring
page_sizeis positive andtimeoutis within acceptable bounds (e.g., < 400 seconds).
Sources: tests/unit/test_config.py:75-79, tests/unit/test_config.py:102-125, tests/unit/test_config.py:136-150
Environment Diagnostics
Diagnostics for the execution environment are handled by CriticalValidation. This checks for the presence of mandatory environment variables like base_url, username, and password before initialization.
Sources: tests/unit/README.md:65-70, tests/integration/test_functional.py:61-68
Runtime Error Management
During execution, the tap handles various runtime failures ranging from transient network issues to permanent API errors.
Network Resilience and Backoff
The client implementation includes resilience patterns to handle unstable connections to the Oracle WMS API.
# Example of configuration fields governing resilience
settings = FlextTapOracleWmsSettings(
timeout=60, # Request timeout in seconds
max_retries=5, # Number of retry attempts
retry_delay=2, # Delay between retries
enable_rate_limiting=True, # Prevent 429 Too Many Requests
max_requests_per_minute=60
)
Sources: tests/unit/test_config.py:38-48, tests/integration/test_wms_connection.py:44-48
Authentication Diagnostics
The authentication system supports multiple methods (Basic, OAuth2) and includes specific error handling for credential failures. In E2E and functional tests, authentication errors are caught and verified to ensure they return meaningful messages rather than generic failures.
Sources: tests/unit/README.md:37-43, tests/e2e/test_e2e.py:270-285
Result Handling (r Pattern)
Following the FLEXT ecosystem standard, methods often return an r object (Result pattern). This prevents unhandled exceptions from crashing the process and forces explicit checking of success or failure states.
# Verification of result pattern in tests
def assert_flext_result_error(result, expected_error_type=None):
assert not result.success, "Expected error result"
assert result.error_message, "Expected error message"
if expected_error_type:
assert expected_error_type in result.error_message
Sources: tests/unit/README.md:214-220, AGENTS.md:16-20
Diagnostic Logging and Observability
The tap utilizes a centralized logging strategy to provide visibility into its internal operations.
Logging Configuration
Logs are generated using flext_core.u.fetch_logger(). Logging levels can be adjusted via configuration (log_level) and include:
- DEBUG: Detailed request/response tracing and pagination details.
- INFO: High-level progress (e.g., "Discovered 5 entities").
- WARNING/ERROR: Failures, retries, and schema mismatches.
Sources: src/flext_tap_oracle_wms/README.md:165-170, meltano.yml:138-150
Error Reporting Matrix
| Error Category | Diagnostic Output | System Behavior |
|---|---|---|
| Config Validation | Pydantic ValidationError with field details | Immediate Termination |
| Authentication | 401 Unauthorized or 403 Forbidden logs | Retry (if applicable) or Terminate |
| Network Timeout | Connection/Timeout error trace | Exponential Backoff & Retry |
| Schema Mismatch | ValueError or Singer Type logs | Log Warning & Skip Record (configurable) |
| Rate Limiting | 429 Too Many Requests status | Wait based on Retry-After header |
Sources: tests/integration/test_functional.py:240-252, tests/unit/test_config.py:102-120, meltano.yml:109-122
Diagnostic Tooling and Testing
The project maintains a rigorous testing suite to ensure error-handling logic functions correctly.
Error Testing Patterns
Tests are designed to simulate failures to verify diagnostic accuracy:
- Mocking Network Failures: Using
unittest.mock.patchto simulaterequests.ConnectionError. - Invalid Configuration: Feeding intentionally broken settings to ensure
ValidationErrortriggers. - API Error Injection: Mocking WMS responses with 500-level errors to test retry logic.
Sources: tests/unit/README.md:139-152, tests/integration/README.md:280-292
Summary
The Error Handling & Diagnostics system in flext-tap-oracle-wms ensures that the integration is both robust and transparent. By enforcing strict configuration validation, implementing intelligent retries for transient failures, and utilizing standardized logging and result patterns, the tap provides developers and operators with the necessary tools to identify and resolve issues quickly. This multi-tiered approach is essential for maintaining production-grade data pipelines against enterprise Oracle WMS environments.
Sources: src/flext_tap_oracle_wms/README.md, tests/README.md