Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
The flext-tap-oracle project is a modern Singer Tap designed for Oracle Database, built upon the FLEXT framework. It serves as a specialized data extractor that leverages flext-meltano abstractions and flext-db-oracle for optimized database communication. The architecture is designed to handle enterprise-level Oracle extractions with features such as connection pooling, incremental replication, and schema introspection.
Sources: README.md:9-11, src/flext_tap_oracle/models.py:53-65
Component Hierarchy
The system follows a layered approach, inheriting functionality from the broader FLEXT ecosystem. It utilizes a facade pattern for public exports and relies on lazy loading for performance optimization.
Core Modules
- Tap Facade (
tap.py): Implements the CLI and command patterns forDiscoveryandSyncoperations. - Stream Management (
streams.py): Handles the conversion of Oracle table metadata into Singer-compliant streams and manages data extraction. - Configuration (
settings.py): Manages environment-prefixed settings using Pydantic for validation and business rule enforcement. - Data Models (
models.py): Defines domain entities for execution stats, discovery parameters, and extraction configurations.
The following diagram illustrates the relationship between the major architectural components:
Execution Flow
The tap operates primarily through two modes: Discovery and Sync. These are orchestrated by the FlextTapOracleCli facade.
Discovery Flow
The discovery process inspects the Oracle database to generate a Singer catalog. It uses the FlextDbOracleApi to fetch table metadata and converts it into stream definitions.
Sync Flow
The sync operation performs the actual data extraction. It instantiates OracleStream objects which utilize FlextDbOracleApi.query() to fetch records without direct SQLAlchemy dependencies, ensuring optimal Oracle-specific performance.
Sources: src/flext_tap_oracle/tap.py:84-124, src/flext_tap_oracle/streams.py:89-115
Data Mapping and Transformation
The project defines strict rules for mapping Oracle data types to Singer types within rules.json. Transformation logic in streams.py ensures that complex Oracle types like DATE, TIMESTAMP, CLOB, and BLOB are correctly serialized.
| Oracle Type | Singer/JSON Type | Transformation Logic |
|---|---|---|
| VARCHAR2, CHAR, CLOB | string | Direct string conversion |
| NUMBER, BINARY_FLOAT | number | Numeric validation |
| DATE, TIMESTAMP | string | ISO 8601 formatting |
| BLOB, RAW | string | Stringified representation |
| JSON | t.JsonValue | Native JSON support |
Sources: src/flext_tap_oracle/rules.json:192-225, src/flext_tap_oracle/streams.py:173-205
Configuration and Validation
Configuration is managed via FlextTapOracleSettings, which uses the FLEXT_TAP_ORACLE_ environment prefix.
| Setting | Type | Default | Description |
|---|---|---|---|
oracle_host | NonEmptyStr | "localhost" | Database hostname |
oracle_port | PortNumber | 1521 | Listener port |
oracle_user | SecretStr | - | Auth username |
batch_size | BatchSize | 1000 | Rows per extraction batch |
stream_prefix | str | "" | Prefix for Singer streams |
Sources: src/flext_tap_oracle/settings.py:22-54, src/flext_tap_oracle/rules.json:76-92
Performance and Reliability
The architecture incorporates several "Performance Rules" and "Error Handling Rules" defined in the system metadata:
- Connection Pooling: Uses native Oracle connection pooling via
flext-db-oracle. Sources: src/flext_tap_oracle/rules.json:104-107 - Circuit Breaker: Enabled by default to prevent system exhaustion during database failures (5 failures threshold, 60s timeout). Sources: src/flext_tap_oracle/rules.json:239-244
- Batching: Adjustable
batch_sizeandfetch_size(default 1000) to optimize memory usage. Sources: src/flext_tap_oracle/rules.json:165-170
Summary
flext-tap-oracle provides a robust, enterprise-grade Singer tap by offloading complex database interactions to flext-db-oracle while maintaining strict Singer protocol compliance through flext-meltano. Its modular design allows for scalable data extraction with built-in observability and performance optimizations tailored specifically for Oracle environments.
Sources: src/flext_tap_oracle/models.py:53-85, src/flext_tap_oracle/streams.py:34-45