Projects & Integrations Streams
Relevant source files
The following files were used as context for generating this wiki page:
The Projects and Integrations streams are core components of the flext-tap-oracle-oic Singer tap, responsible for extracting resource organization and operational metadata from Oracle Integration Cloud (OIC). These streams utilize the OIC REST API to retrieve detailed information about integration workflows, their lifecycle status, and their grouping within projects.
The Integrations Stream focuses on the primary functional units of OIC, capturing metadata such as versions, patterns, styles, and associated connections. The Projects Stream provides a higher-level organizational view, detailing resource grouping, folder structures, and permissions. Together, they enable comprehensive monitoring and governance of OIC environments.
Sources: src/flext_tap_oracle_oic/_models/streams.py:27-142, src/flext_tap_oracle_oic/_models/streams.py:461-526
Stream Architecture and Implementation
Both streams inherit from OICBaseStream, which provides a standardized framework for interacting with OIC APIs. This base class handles intelligent endpoint discovery, OAuth2 authentication, and adaptive pagination.
Base Stream Logic
The architecture relies on specific API categories and requirements defined in the stream classes. For instance, the Integrations and Projects streams both set requires_design_api: ClassVar[bool] = True, signaling the need for the OIC Design API endpoint.
Sources: src/flext_tap_oracle_oic/models.py:91-200, src/flext_tap_oracle_oic/_models/streams.py:38, src/flext_tap_oracle_oic/_models/streams.py:470
The following diagram illustrates the inheritance and component relationship for these streams:
Integrations Stream Details
The Integrations stream extracts complete metadata including configurations, endpoints, triggers, and execution statistics. It is categorized under the core API category and uses a default expansion for connections and endpoints.
Configuration and Schema
| Attribute | Value | Description |
|---|---|---|
| Name | integrations | Singer stream identifier |
| Path | /integrations | API endpoint relative to base path |
| Primary Key | id | Unique identifier for the record |
| Replication Key | lastUpdated | Used for incremental syncs |
| Default Sort | lastUpdated:desc | OIC API sorting parameter |
| Expand | connections,endpoints | Request additional nested details |
Sources: src/flext_tap_oracle_oic/_models/streams.py:32-41
Data Model Fields
The schema includes critical lifecycle and tracking fields:
- Identity:
id,name,version. - Temporal:
created,lastUpdated. - Organizational:
projectId,folderId. - Lifecycle:
status(e.g., ACTIVATED, DEACTIVATED). - Configuration:
pattern,style,payloadTracking,tracing.
Sources: src/flext_tap_oracle_oic/_models/streams.py:42-142, src/flext_tap_oracle_oic/constants.py:84-93
Projects Stream Details
The Projects stream captures how OIC resources are organized. It provides insights into the folder hierarchy and the density of resources within specific projects.
Configuration and Schema
| Attribute | Value | Description |
|---|---|---|
| Name | projects | Singer stream identifier |
| Path | /projects | API endpoint relative to base path |
| Primary Key | id | Unique identifier for the record |
| Replication Key | lastUpdated | Used for incremental syncs |
| API Category | extended | Differentiates from core resources |
Sources: src/flext_tap_oracle_oic/_models/streams.py:461-471
Key Metadata Fields
- Folders:
foldersarray containing project folder structure. - Resource Counts:
integrationCount,connectionCount. - Access Control:
permissionsarray for security auditing.
Sources: src/flext_tap_oracle_oic/_models/streams.py:472-526
Data Flow: Extraction to Record Yielding
The extraction process follows a sequence from URL parameter construction to record enrichment.
Record Enrichment
Every record yielded by these streams is enriched with tap-specific metadata for auditability:
_tap_extracted_at: The UTC timestamp of extraction._tap_stream_name: The name of the stream that produced the record.
Sources: src/flext_tap_oracle_oic/models.py:269-276
Replication Logic
Replication for both streams is primarily managed via the lastUpdated timestamp. This allows the tap to perform incremental syncs by passing filtering parameters to the OIC API.
URL Parameter Mapping
When a replication value is present in the context, the tap appends query filters to the API request:
if (self.replication_key and context and context.get("starting_replication_value")):
start_date = context["starting_replication_value"]
params[f"{self.replication_key}>="] = start_date
Sources: src/flext_tap_oracle_oic/models.py:228-232
Summary of Integration Statuses
The following statuses are tracked for the Integrations stream as defined in the tap constants:
| Status | Description |
|---|---|
ACTIVATED | Integration is running and processing requests. |
DEACTIVATED | Integration is stopped. |
CONFIGURED | Integration is configured but not yet live. |
DRAFT | Integration is currently being edited. |
ERROR/FAILED | Integration is in a faulty state. |
Sources: src/flext_tap_oracle_oic/constants.py:84-93
The Projects & Integrations streams provide the foundational data required to map out the entire Oracle Integration Cloud environment, supporting both high-level governance and granular operational monitoring.