Singer Tap Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Singer Tap Integration for Oracle is a modern, zero-boilerplate implementation designed to extract data from Oracle databases following the Singer specification. It leverages the flext-meltano and flext-core abstractions to provide a high-performance, resilient, and enterprise-ready ETL component within the FLEXT ecosystem. Sources: src/flext_tap_oracle/tap.py:1-12, src/flext_tap_oracle/rules.json:1-15
Unlike traditional Singer taps that inherit from singer_sdk.Tap, this integration utilizes a CLI dispatch pattern. It prioritizes direct integration with the flext-db-oracle infrastructure for optimized database operations, including connection pooling, native schema introspection, and efficient batch processing. Sources: src/flext_tap_oracle/api.py:10-25, src/flext_tap_oracle/streams.py:22-35
Architecture and Components
The integration is built on a modular architecture where responsibilities are divided between command runners, stream managers, and configuration models.
Core Architecture Flow
The following diagram illustrates the command-driven architecture of the Oracle Tap, showing how user input is dispatched through CLI commands to execute database operations.
Key Components
| Component | Description | Source |
|---|---|---|
FlextTapOracleCli | Facade for CLI operations, managing discovery and sync dispatch. | src/flext_tap_oracle/tap.py:136 |
FlextTapOracleStreams | Manages stream lifecycle, row estimation, and record transformation. | src/flext_tap_oracle/streams.py:22 |
FlextTapOracleSettings | Pydantic-based runtime configuration with environment variable support. | src/flext_tap_oracle/settings.py:22 |
OracleStream | Represents a single Oracle table/view as a Singer stream. | src/flext_tap_oracle/streams.py:42 |
FlextTapOracleModels | Provides domain entities for execution stats and Singer metadata. | src/flext_tap_oracle/models.py:27 |
Command Execution Logic
The integration implements two primary Singer operations: Discovery and Sync. These are handled by dedicated command classes that encapsulate the business logic and error handling.
Discovery Process
The discovery command introspects the Oracle schema to generate a Singer-compliant catalog. It validates the configuration, connects to the database, and identifies available tables.
Sources: src/flext_tap_oracle/tap.py:32-73
Sync Process
The sync process manages the data extraction. It supports both FULL_TABLE and INCREMENTAL replication methods.
Sources: src/flext_tap_oracle/rules.json:115-122, src/flext_tap_oracle/tap.py:80-130
Stream Management and Type Transformation
The OracleStream class is responsible for converting native Oracle data into Singer-compatible JSON formats. It utilizes FlextDbOracleApi to ensure optimal performance.
Row Estimation and Data Fetching
Rows are estimated using Oracle system views to provide progress metadata. Data fetching uses safe SQL construction with table name validation.
Sources: src/flext_tap_oracle/streams.py:70-98
Type Mapping
Oracle-specific types are mapped to Singer types as follows:
| Oracle Type | Singer Type | Transformation Note |
|---|---|---|
VARCHAR2, CLOB, RAW | string | Converted to UTF-8 strings. |
NUMBER, BINARY_FLOAT | number | Precision preserved. |
DATE, TIMESTAMP | string | Converted to RFC3339 (ISO8601) format. |
JSON | object | Parsed as JsonValue. |
BLOB | string | Stringified representation. |
Sources: src/flext_tap_oracle/rules.json:173-205, src/flext_tap_oracle/streams.py:205-245
Configuration and Settings
Configuration is managed via FlextTapOracleSettings, which inherits from FlextSettingsBase. It uses the environment prefix FLEXT_TAP_ORACLE_. Sources: src/flext_tap_oracle/settings.py:22-30
Connection Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
oracle_host | NonEmptyStr | localhost | Database hostname. |
oracle_port | PortNumber | 1521 | TNS Listener port. |
oracle_service_name | str | XE | Service Name or SID. |
oracle_user | SecretStr | Required | Database username. |
oracle_password | SecretStr | Required | Database password. |
batch_size | int | 1000 | Records per extract batch. |
Sources: src/flext_tap_oracle/settings.py:32-55, src/flext_tap_oracle/rules.json:80-110
Performance and Resilience
The integration includes built-in performance rules and error-handling strategies to manage enterprise workloads.
- Connection Pooling: Uses native Oracle connection pooling via
flext-db-oracle. Sources: src/flext_tap_oracle/rules.json:96-105 - Circuit Breaker: An optional circuit breaker can be enabled to stop execution after a threshold of failures (default: 5 failures, 60s timeout). Sources: src/flext_tap_oracle/rules.json:225-235
- Parallel Streams: Supports up to 4 parallel streams in production environments. Sources: src/flext_tap_oracle/rules.json:160-165
- Retry Strategy: Implements exponential backoff for connection errors (max 3 attempts). Sources: src/flext_tap_oracle/rules.json:215-220
Summary
The Singer Tap Integration for Oracle provides a robust bridge between Oracle databases and the Singer ecosystem. By utilizing the FLEXT command dispatch pattern instead of a standard SDK inheritance model, it achieves deep integration with optimized Oracle drivers and utilities while maintaining full compliance with Singer protocol message types (RECORD, SCHEMA, STATE, ACTIVATEVERSION). Sources: src/flext_tap_oracle/api.py:25-35, src/flext_tap_oracle/rules.json:130-135