Internal Data Models
Relevant source files
The following files were used as context for generating this wiki page:
Internal Data Models in flext-tap-oracle provide a structured representation of the Oracle extraction domain, extending the core functionality of both Singer (via flext-meltano) and Oracle database operations (via flext-db-oracle). These models govern how connection settings, stream metadata, execution statistics, and discovery parameters are validated and manipulated throughout the tap's lifecycle. Sources: src/flext_tap_oracle/models.py:22-34, src/flext_tap_oracle/rules.json:17-48
The architecture follows a strictly typed, Pydantic-based approach where domain entities inherit from a common m.Entity base. This ensures that data passing between the Oracle database, the Singer protocol layer, and the CLI remains consistent and compliant with defined business rules. Sources: src/flext_tap_oracle/models.py:34, src/flext_tap_oracle/rules.json:42-47
Core Domain Models
The primary model container, FlextTapOracleModels, consolidates all Oracle tap domain entities. It is organized into a TapOracle namespace to avoid collisions with other projects in the FLEXT ecosystem. Sources: src/flext_tap_oracle/models.py:22-37
Model Hierarchy and Relationships
The following class diagram illustrates the relationship between the core tap models and their parent framework components.
The diagram shows how FlextTapOracleModels acts as a composite of both Meltano (Singer) and Oracle-specific model sets. Sources: src/flext_tap_oracle/models.py:22-24, tests/models.py:18-20
Execution Statistics and Metrics
Performance monitoring is handled by OracleTapExecutionStats, which tracks the volume and velocity of data extraction. It includes logic for calculating success rates and query efficiency. Sources: src/flext_tap_oracle/models.py:75-103
| Field | Type | Description |
|---|---|---|
execution_id | NonEmptyStr | Unique identifier for the tap run. |
total_records | NonNegativeInt | Cumulative count of records extracted. |
duration_seconds | NonNegativeFloat | Wall-clock time for the entire execution. |
oracle_query_time | NonNegativeFloat | Total time spent waiting for Oracle queries. |
failed_streams | MutableSequence[str] | List of stream names that encountered errors. |
Sources: src/flext_tap_oracle/models.py:59-71, src/flext_tap_oracle/models.py:86-111
Configuration and Rules
The tap configuration is managed via FlextTapOracleSettings, which utilizes environment variables with the prefix FLEXT_TAP_ORACLE_. This model handles sensitive data like passwords using SecretStr types. Sources: src/flext_tap_oracle/settings.py:23-45
Connection Parameters Logic
The flow demonstrates the transformation of validated settings into the raw mapping required by underlying Oracle drivers. Sources: src/flext_tap_oracle/settings.py:57-81
Business Rule Constraints
The system enforces specific constraints defined in rules.json and implemented in the Pydantic models:
- Connection Validation: Either
service_nameorsidmust be provided. Sources: src/flext_tap_oracle/rules.json:62-68 - Batch Sizing: Default
batch_sizeis 1000, with a maximum constraint of 50,000 records per batch. Sources: src/flext_tap_oracle/rules.json:80-86 - Parallelism:
max_parallel_streamsmust not exceed theconnection_pool_size. Sources: src/flext_tap_oracle/rules.json:88-92
Stream and Schema Models
The internal representation of data streams relies on OracleStream and metadata components. These models bridge Oracle-specific data types (like CLOB or DATE) to Singer-compatible JSON types. Sources: src/flext_tap_oracle/streams.py:34-56
Data Type Mapping
The following table outlines how Oracle internal types are mapped to Singer protocol types.
| Oracle Type | Singer/JSON Type | Transformation Logic |
|---|---|---|
VARCHAR2, CLOB | string | Direct string representation. |
NUMBER | number | Preserve precision where possible. |
DATE, TIMESTAMP | string | Transformed to ISO 8601 / RFC3339 format. |
JSON | t.JsonValue | Native JSON parsing if supported. |
Sources: src/flext_tap_oracle/rules.json:161-185, src/flext_tap_oracle/streams.py:182-205
Discovery Models
During the discovery phase, the tap_oracle_client_discover_tables function produces a sequence of Table models from FlextDbOracleModels. Sources: src/flext_tap_oracle/_utilities/client.py:26-34
This diagram depicts the transition from raw database metadata to structured internal models during discovery. Sources: src/flext_tap_oracle/_utilities/client.py:35-50
Testing Models
For quality assurance, a separate set of test models exists in TestsFlextTapOracleModels. These use composition to provide mock-friendly versions of the production entities. Sources: tests/models.py:21-36
TestOracleConnection: Simulates a database connection with accessible host/port attributes for assertions. Sources: tests/models.py:44-58TestSingerStream: Defines a test-only stream model including replication methods and selection status. Sources: tests/models.py:65-76TestExtractionConfig: Allows fine-grained control over batch sizes and timeouts during unit and integration tests. Sources: tests/models.py:91-105
Summary
Internal Data Models in flext-tap-oracle serve as the backbone for type-safe database extraction. By leveraging Pydantic entities and a centralized rules engine (rules.json), the project ensures that Oracle-specific complexities—such as connection pooling, query efficiency metrics, and complex type transformations—are handled consistently across the CLI, stream management, and testing suites.