Home
Relevant source files
The following files were used as context for generating this wiki page:
The flext-tap-oracle project is a modern Singer Tap specifically designed for extracting data from Oracle Databases. It is built using the FLEXT framework, leveraging specialized libraries such as flext-core, flext-db-oracle, and flext-meltano to provide a robust, high-performance ETL integration. The tap follows the Singer specification (v1.5.0), supporting both full table and incremental replication methods while providing deep introspection of Oracle schemas.
Sources: README.md:1-10, pyproject.toml:22-35, src/flext_tap_oracle/rules.json:10-20
Architecture and Core Components
The system is structured as a modular integration that inherits functionality through a parent MRO (Method Resolution Order) chain. It utilizes a facade pattern for public surface access and lazy loading for optimized performance.
Logic Flow and Data Pipeline
The operation flow typically begins with configuration validation, followed by schema discovery or record synchronization.
The diagram above illustrates the high-level data flow from the CLI entry point to the generation of Singer-compliant messages.
Sources: src/flext_tap_oracle/tap.py:202-230, src/flext_tap_oracle/streams.py:23-40, README.md:18-30
Key Components
| Component | Description | Reference |
|---|---|---|
FlextTapOracleCli | Facade for CLI operations (Discover/Sync). | tap.py:164 |
FlextTapOracleSettings | Pydantic-based configuration management. | settings.py:22 |
FlextTapOracleModels | Standardized domain entities and metrics tracking. | models.py:24 |
OracleStream | Main logic for record extraction and type transformation. | streams.py:42 |
FlextDbOracleApi | Layer 2 database abstraction for optimized interaction. | client.py:23 |
Sources: src/flext_tap_oracle/tap.py:164, src/flext_tap_oracle/settings.py:22, src/flext_tap_oracle/models.py:24, src/flext_tap_oracle/streams.py:42, src/flext_tap_oracle/_utilities/client.py:23
Configuration and Rules
Configuration is managed via FlextTapOracleSettings, which utilizes environment variables with the prefix FLEXT_TAP_ORACLE_. Validation rules are strictly defined in rules.json.
Connection Parameters
- Host: Database hostname (Required).
- Port: Default 1521.
- User/Password: Handled as
SecretStrfor security. - Service Name/SID: Identifier for the Oracle instance.
Sources: src/flext_tap_oracle/settings.py:25-50, src/flext_tap_oracle/rules.json:71-85
Replication Rules
The tap supports two primary replication methods:
- FULL_TABLE: Extracts the entire table content.
- INCREMENTAL: Requires a replication key (datetime, timestamp, or number).
Sources: src/flext_tap_oracle/rules.json:161-175
Execution and Command Flow
The CLI supports two main operations: --discover and --sync.
Discovery Process
Discovery involves introspecting the Oracle database to identify tables and generate a Singer catalog.
Synchronization Process
The sync process iterates through selected streams to fetch and transform records.
Data Type Mapping
Oracle-specific types are automatically mapped to Singer-compliant types as defined in the project's rules.
| Oracle Type | Singer Type | Transformation Notes |
|---|---|---|
VARCHAR2, CLOB | string | Converted to standard strings. |
NUMBER | number | Precision is preserved. |
DATE, TIMESTAMP | string | Formatted as RFC3339/ISO8601 strings. |
JSON | t.JsonValue | Native JSON support. |
BLOB, RAW | string | Stringified representation. |
Sources: src/flext_tap_oracle/rules.json:219-245, src/flext_tap_oracle/streams.py:182-210
Performance and Observability
The tap includes built-in metrics collection via OracleTapExecutionStats. It tracks:
- Volume Metrics: Total records and bytes processed.
- Performance Metrics: Records per second, connection time, and query time.
- Quality Metrics: Success rate and error tracking per stream.
Sources: src/flext_tap_oracle/models.py:100-140, src/flext_tap_oracle/rules.json:198-210
Summary
The flext-tap-oracle project provides a high-level, standardized interface for Oracle data extraction. By combining the flext-db-oracle abstraction layer with the Singer protocol, it ensures reliable and performant data pipelines. The system emphasizes security through secret handling and reliability through structured business rule validation and circuit-breaker logic defined in its rule engine.
Sources: src/flext_tap_oracle/rules.json:258-270, src/flext_tap_oracle/settings.py:53-65