Lookup Stream
Relevant source files
The following files were used as context for generating this wiki page:
The Lookup Stream is a specialized data extraction component within the flext-tap-oracle-oic project designed to retrieve data transformation lookup tables from Oracle Integration Cloud (OIC). These lookup tables are critical for cross-referencing values and managing transformations across various integrations within the OIC environment.
The stream is implemented as a member of the core OIC stream set and follows the standardized FLEXT namespace patterns, inheriting from OICBaseStream to leverage shared functionality for authentication, pagination, and error handling.
Sources: src/flext_tap_oracle_oic/_models/streams.py:273-281, src/flext_tap_oracle_oic/constants.py:53-59
Stream Architecture and Implementation
The Lookup Stream is defined as the class LookupsStream within the FlextTapOracleOicModelsStreams container. It targets the /lookups API endpoint and uses the name field as its primary identifier for record tracking and state management.
Class Definition and Parameters
The stream implementation defines several key metadata attributes and configuration defaults:
| Attribute | Value | Description |
|---|---|---|
name | lookups | The Singer stream identifier. |
path | /lookups | The relative REST API endpoint path. |
primary_keys | ["name"] | Fields used to uniquely identify records. |
replication_key | lastUpdated | Field used for incremental state tracking. |
api_category | core | High-level categorization of the API endpoint. |
Sources: src/flext_tap_oracle_oic/_models/streams.py:283-290, src/flext_tap_oracle_oic/constants.py:38-41
Data Flow and Request Lifecycle
The stream follows the standard OICBaseStream lifecycle, which involves constructing the API URL based on the OIC region, obtaining OAuth2 credentials, and processing paginated responses.
The flow shows how the stream interacts with the OIC API using offsets and limits derived from OICBaseStream logic.
Sources: src/flext_tap_oracle_oic/models.py:164-220, src/flext_tap_oracle_oic/tap.py:101-125
Data Model and Schema
The stream schema is defined using Singer-compatible JSON types. It captures metadata about the lookup tables, including their status, creation history, and usage metrics.
Fields and Descriptions
| Field | Type | Description |
|---|---|---|
name | String | Unique identifier for the lookup. |
description | String | Textual description of the lookup's purpose. |
type | String | The type of lookup table. |
status | String | Current lifecycle status (e.g., CONFIGURED, ACTIVATED). |
created | DateTime | Timestamp of when the lookup was created. |
lastUpdated | DateTime | Timestamp of the most recent modification. |
valueCount | Integer | The number of entries/rows within the lookup table. |
defaultValue | String | Value returned if no match is found. |
usageCount | Integer | Number of integrations currently using this lookup. |
isReadOnly | Boolean | Indicates if the lookup can be modified. |
Sources: src/flext_tap_oracle_oic/_models/streams.py:291-366, src/flext_tap_oracle_oic/constants.py:102-110
Entity Relationship Representation
The Lookup entities exist within the OIC "core" category and are associated with metadata tracking and potential resource relationships.
The diagram represents the relationship between the Lookup stream data and the broader OIC environment.
Sources: src/flext_tap_oracle_oic/_models/streams.py:283-366, src/flext_tap_oracle_oic/constants.py:186-193
Execution Logic
The LookupsStream relies on specialized utilities for URL construction and response parsing. It specifically expects OIC API responses to be wrapped in an "envelope" structure containing items, totalSize, and hasMore flags.
Pagination and Filtering
The stream utilizes the FlextTapOracleOicPaginator which manages offsets. By default, it requests 100 records per page (max 1000). If incremental sync is enabled, it applies a filter lastUpdated>=[Value] to the request parameters.
# From OICBaseStream.get_url_params
params["limit"] = min(page_size, 1000)
params["offset"] = next_page_token or 0
if self.replication_key and context and context.get("starting_replication_value"):
start_date = context["starting_replication_value"]
params[f"{self.replication_key}>="] = start_date
Sources: src/flext_tap_oracle_oic/models.py:204-215, src/flext_tap_oracle_oic/utilities.py:126-150
Response Normalization
Responses from the /lookups endpoint are processed to detect whether the result is a single record or an enveloped list. The stream enriches each record with internal metadata:
_tap_extracted_at: The UTC timestamp of extraction._tap_stream_name: Hardcoded as "lookups".
Sources: src/flext_tap_oracle_oic/models.py:246-260, src/flext_tap_oracle_oic/utilities.py:84-110
Conclusion
The Lookup Stream provides a robust mechanism for extracting data transformation definitions from Oracle Integration Cloud. By adhering to the OICBaseStream template, it ensures consistent authentication and performance optimization, enabling reliable synchronization of lookup table metadata for downstream analytics or configuration auditing.