Stream Registry & Definitions

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_tap_oracle_oic/_models/streams.py
• src/flext_tap_oracle_oic/models.py
• src/flext_tap_oracle_oic/tap.py
• src/flext_tap_oracle_oic/constants.py
• src/flext_tap_oracle_oic/_models/_stream_config.py
• src/flext_tap_oracle_oic/utilities.py

The Stream Registry & Definitions module provides a centralized, typed, and schema-driven architecture for extracting data from Oracle Integration Cloud (OIC). It utilizes the Singer protocol to define how integration metadata, connection configurations, monitoring metrics, and system infrastructure are represented and retrieved from OIC APIs.

This module is built upon the FLEXT framework, employing a consolidated model pattern where all stream definitions are contained within a registry and wired into the main Tap via Method Resolution Order (MRO). This ensures that every stream inherits standard capabilities such as adaptive pagination, OAuth2 authentication, and incremental state management.
Sources: src/flext_tap_oracle_oic/_models/streams.py:1-25, src/flext_tap_oracle_oic/models.py:75-95

Stream Architecture & Base Implementation

All OIC streams derive from OICBaseStream, which provides a professional base implementation for interacting with various OIC API categories (Design, Runtime, Monitoring, B2B, and Process). The architecture separates stream configuration from execution logic, allowing for dynamic endpoint discovery based on the OIC instance region.

The OICBaseStream Model

The OICBaseStream class handles the heavy lifting of API interaction, including:

• Intelligent Endpoint Discovery: Dynamically constructs URLs based on the api_category and instance region.
• Authentication: Integrated OAuth2/IDCS token handling via the parent tap's client.
• Adaptive Pagination: Uses a specific FlextTapOracleOicPaginator for offset-based pagination.
• Parameter Optimization: Automatically handles limit, offset, orderBy, and filtering logic.

Centralized Stream Registry

The ALL_STREAMS registry acts as the single source of truth for all available streams in the tap. Streams are categorized into Core, Infrastructure, Extended, and Monitoring groups, which can be toggled via tap settings.

Stream Classification

Category	Stream Name	Purpose	API Category
Core	integrations	Metadata, endpoints, and connections	core
Core	connections	Adapter configurations and security	core
Core	packages	Deployment and versioning info	core
Core	lookups	Transformation lookup tables	core
Core	libraries	JS/XSLT custom functions	infrastructure
Infrastructure	certificates	SSL/TLS and encryption certs	security
Infrastructure	adapters	Available adapter capabilities	infrastructure
Monitoring	executions	Performance and error stats	monitoring
Monitoring	metrics	System health and usage metrics	monitoring

Sources: src/flext_tap_oracle_oic/_models/streams.py:646-659, src/flext_tap_oracle_oic/constants.py:59-64

Stream Definitions & Schemas

Each stream defines its own Singer-compliant schema using Meltano.SingerPropertiesList. This ensures that the extracted data is strictly typed and documented.

Example: Integrations Stream Definition

The IntegrationsStream extracts complete metadata. It requires the Design API and supports incremental replication via the lastUpdated key.

Field	Type	Description
id	String	Unique Integration ID
name	String	Display name of the integration
status	String	Lifecycle status (e.g., ACTIVATED, DRAFT)
connections	Array (Object)	List of connections used by the integration
lastUpdated	DateTime	Timestamp of the last modification

Sources: src/flext_tap_oracle_oic/_models/streams.py:34-110

Stream Configuration & Filtering

The OicStreamConfiguration model defines how individual streams can be customized through the tap configuration. This includes replication methods, pagination sizes, and sensitive data sanitization.

Configuration Properties

• Replication Method: Supports FULL_TABLE and INCREMENTAL.
• Filtering: Includes status_filter (e.g., only active integrations) and date_range_filter.
• Security: sanitize_sensitive_data (default: True) allows for masking sensitive fields during extraction.
• Pagination: page_size allows tuning performance (max 1000 items per request).

Discovery and Initialization Flow

The tap initializes streams by mapping the ALL_STREAMS registry against user-provided settings. This determines which streams are activated and how their schemas are exposed to the Singer orchestrator.

The Discovery Sequence

1. Tap Initialization: The FlextTapOracleOic class reads the JSON configuration.
2. Stream Mapping: The discover_oic_streams method filters ALL_STREAMS based on flags like include_infrastructure.
3. Schema Generation: For each discovered stream, build_catalog_entry generates a Singer-compliant catalog entry containing the schema and replication metadata.
4. Catalog Output: The final result is a JSON mapping of all valid streams.

Technical Summary

Stream Registry & Definitions form the backbone of the OIC data extraction process. By centralizing definitions in ALL_STREAMS and using a unified base class OICBaseStream, the tap ensures consistent behavior across different API endpoints. The use of Pydantic models for configuration (OicStreamConfiguration) and Singer-based schema definitions provides a robust, type-safe integration for complex enterprise data.