Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The flext-tap-oracle-oic project is a Singer-compliant tap designed to extract data from Oracle Integration Cloud (OIC). It utilizes the FLEXT framework, extending FlextMeltanoAbstractions to provide a typed, scalable, and maintainable integration layer for OIC's Design, Runtime, and Monitoring APIs.
The architecture focuses on high-level abstractions for authentication (OAuth2), adaptive pagination, and consolidated stream management. It enforces a strict separation between domain logic, data models, and the Singer protocol requirements.
Sources: README.md:8-10, src/flext_tap_oracle_oic/tap.py:126-128, src/flext_tap_oracle_oic/models.py:73-77
Core Components
The system is organized into several primary components that handle different aspects of the extraction lifecycle:
- FlextTapOracleOic: The main entry point that manages configuration, discovery, and the overall tap lifecycle.
- FlextOracleOicAuthenticator: Handles OAuth2 client credentials flows to obtain access tokens from Oracle Identity Cloud Service (IDCS).
- FlextTapOracleOicClient: An authenticated HTTP client specialized for OIC's REST endpoints.
- FlextTapOracleOicModels: Defines the schema, data structures, and stream-specific logic.
- FlextTapOracleOicPaginator: Implements adaptive offset-based pagination to handle OIC's response limits efficiently.
System Interaction Flow
The following diagram illustrates the interaction between the tap, the authenticator, and the OIC API during a data extraction request.
Data Stream Architecture
Streams in the OIC tap are built upon the OICBaseStream class, which provides intelligent endpoint discovery and standard Singer replication logic. The project classifies streams into "Core" and "Infrastructure" categories.
Stream Categories and Endpoints
| Category | Base API Path | Included Streams |
|---|---|---|
| Core | /ic/api/integration/v1 | integrations, connections, packages, lookups, libraries |
| Infrastructure | /ic/api/integration/v1 | certificates, adapters |
| Monitoring | /ic/api/monitoring/v1 | executions, metrics |
Sources: src/flext_tap_oracle_oic/constants.py:46-63, src/flext_tap_oracle_oic/_models/streams.py:32-470
Stream Inheritance and MRO
All streams inherit from a common base that centralizes OIC-specific logic such as URL construction based on the OCI region.
Pagination and Performance
The tap employs an adaptive pagination strategy via FlextTapOracleOicPaginator. It calculates the next offset by inspecting OIC response envelopes (e.g., checking hasMore, limit, and offset fields).
Adaptive Sizing Logic
The paginator tracks response times to dynamically adjust the limit parameter. If average response times exceed a threshold (5.0s), the page size is reduced by 20% to prevent timeouts. If responses are fast (<1.0s), the page size is increased by 20% up to the maximum of 1000.
Sources: src/flext_tap_oracle_oic/tap_streams.py:65-83, src/flext_tap_oracle_oic/constants.py:73-77
Configuration and Security
The tap requires specific configuration fields to establish a connection to OIC. These are validated using Pydantic models.
Required Configuration
| Field | Type | Description |
|---|---|---|
oauth_client_id | string | IDCS Client Identifier |
oauth_client_secret | string | IDCS Client Secret (Secret) |
oauth_token_url | string | Full URL to obtain OAuth2 tokens |
oic_url | string | The base instance URL for OIC |
Sources: src/flext_tap_oracle_oic/tap.py:129-152, src/flext_tap_oracle_oic/utilities.py:270-285
Conclusion
The architecture of flext-tap-oracle-oic is built for resilience and scalability within the Oracle Cloud ecosystem. By leveraging the FLEXT framework's abstractions for HTTP communication and the Singer protocol, it provides a robust platform for OIC data extraction. Key features like intelligent URL construction for different OIC API categories (Design vs Runtime) and adaptive pagination ensure efficient operation across varied instance sizes and network conditions.
Sources: src/flext_tap_oracle_oic/models.py:140-160, src/flext_tap_oracle_oic/tap.py:200-215