Extractor Rules
Relevant source files
The following files were used as context for generating this wiki page:
Extractor Rules define the governance framework for data extraction from Oracle databases within the FLEXT ecosystem. These rules encompass connection validation, discovery logic, performance constraints, and data type mappings, ensuring compliance with the Singer tap specification (v1.5.0).
The system integrates flext-core typedefs and flext-db-oracle infrastructure to provide a resilient, high-performance bridge between Oracle Enterprise databases and modern data pipelines. Key capabilities include incremental replication, connection pooling, and automated schema inference.
Sources: src/flext_tap_oracle/rules.json:1-25, src/flext_tap_oracle/models.py:53-75
Connection and Configuration Rules
The extractor requires specific parameters to establish a secure and valid connection to an Oracle instance. Configuration is managed via Pydantic v2 models that enforce strict validation constraints.
Connection Validation Logic
Connections must include a host, username, and password. Additionally, at least one of service_name or sid is required to identify the target Oracle instance.
The diagram shows the sequence of validation checks applied to Oracle connection parameters before execution.
Sources: src/flext_tap_oracle/rules.json:45-56, src/flext_tap_oracle/settings.py:52-58
Default Settings and Constraints
The system provides sensible defaults for enterprise workloads while imposing hard constraints on performance-impacting parameters.
| Parameter | Default Value | Constraints |
|---|---|---|
port | 1521 | 1 - 65535 |
batch_size | 1000 | 1 - 50000 |
max_parallel_streams | 4 | 1 - 20 (Must not exceed pool size) |
connection_pool_size | 5 | 1 - 50 |
query_timeout | 300s | 1 - 7200s |
ssl_mode | prefer | disable, allow, prefer, require |
Sources: src/flext_tap_oracle/rules.json:57-113, src/flext_tap_oracle/settings.py:25-45
Stream and Discovery Rules
Discovery logic utilizes flext-db-oracle APIs to introspect the database schema and generate Singer-compliant catalogs.
Table Discovery Process
The FlextTapOracleUtilitiesClientMixin handles the execution of table discovery. It defaults to the USER schema if no specific schema is provided. Discovery excludes system tables by default and focuses on standard user-defined tables.
The diagram illustrates the flow from a CLI command to the retrieval of table names via the specialized Oracle API.
Sources: src/flext_tap_oracle/_utilities/client.py:28-57, src/flext_tap_oracle/tap.py:30-61
Replication Methods
The extractor supports two primary replication methods:
- FULL_TABLE: Replicates the entire table content without tracking state.
- INCREMENTAL: Requires a
replication_key(typically datetime, timestamp, or number) to track progress via bookmarks.
Sources: src/flext_tap_oracle/rules.json:126-132, src/flext_tap_oracle/rules.json:151-159
Data Type Mapping
Oracle internal data types are mapped to Singer-compliant JSON types. This ensures data consistency across the pipeline while preserving precision for critical types like NUMBER.
| Oracle Type | Singer Type | Transformation Logic |
|---|---|---|
VARCHAR2, CLOB, RAW | string | Direct string representation |
NUMBER, BINARY_FLOAT | number | Precision preserved |
DATE, TIMESTAMP | string | Converted to ISO-8601 / RFC3339 |
JSON | t.JsonValue | Native JSON handling |
BLOB, BFILE | string | Stringified representation |
Transformation Flow
The OracleStream class processes raw database results by transforming them based on metadata retrieved from FlextDbOracleMetadataManager.
# Transformation Logic Snippet
if oracle_type_str.upper().startswith(("DATE", "TIMESTAMP")):
if isinstance(value, datetime):
transformed_record[column_name] = value.isoformat()
else:
transformed_record[column_name] = str(value)
Sources: src/flext_tap_oracle/rules.json:185-214, src/flext_tap_oracle/streams.py:175-195
Performance and Resource Management
Performance rules are tuned for high-volume enterprise data extraction, utilizing connection pooling and result-set batching.
Memory Management Recommendations
To balance memory consumption and throughput, the extractor employs specific cursor and fetch settings:
- Cursor Array Size: Defaulted to 100 for batch fetching.
- Fetch Size: Recommended 1000 for standard operations.
- Batch Size Scaling:
- Small Tables: 100
- Medium Tables: 1000
- Large Tables: 5000 (Max recommended 10000)
Sources: src/flext_tap_oracle/rules.json:161-178, src/flext_tap_oracle/settings.py:40-42
Parallelization Constraints
The extractor supports parallel streams but enforces a safety limit of one parallel stream per connection to prevent resource exhaustion on the Oracle server.
Sources: src/flext_tap_oracle/rules.json:168-170
Resilience and Error Handling
Extractor rules define a robust retry and circuit breaker strategy to handle transient network issues and database timeouts.
Retry Strategies
| Error Category | Max Attempts | Backoff Strategy | Base Delay |
|---|---|---|---|
| Connection Errors | 3 | Exponential | 1.0s |
| Query Errors | 2 | Linear | 0.5s |
Circuit Breaker
A circuit breaker is enabled by default to protect the system from repeated failures:
- Failure Threshold: 5 consecutive failures.
- Timeout: 60 seconds before transitioning to half-open state.
- Half-Open Calls: Maximum of 3 calls allowed to test recovery.
Sources: src/flext_tap_oracle/rules.json:226-241
Summary
The Extractor Rules for the Oracle Tap establish a highly structured environment for data integration. By combining strict Pydantic validation for configurations, automated discovery via flext-db-oracle, and rigorous data type mapping, the system ensures that Oracle data is extracted reliably and efficiently in compliance with the Singer protocol.