Helper Utilities
Relevant source files
The following files were used as context for generating this wiki page:
The Helper Utilities system in flext-tap-oracle-oic provides a unified suite of static methods and internal helper classes designed to handle the complexities of interacting with the Oracle Integration Cloud (OIC) API. It encompasses URL construction, response parsing, state management via bookmarks, and data normalization, ensuring that the Singer tap operates within the expected OIC and FLEXT architectural patterns.
This module primarily resides within the FlextTapOracleOicUtilities class, which extends base utilities from flext-core and flext-meltano to provide domain-specific logic for OIC operations. It serves as a bridge between raw API responses and the structured data required by Singer streams.
Sources: src/flext_tap_oracle_oic/utilities.py:1-25, src/flext_tap_oracle_oic/models.py:72-80
URL and Endpoint Management
The utility system provides robust mechanisms for building and validating OIC API endpoints. This includes intelligent discovery of base URLs based on the OIC region and the specific API category (Design, Monitoring, etc.).
Endpoint Validation Logic
Endpoints are validated to ensure they are non-empty, follow a valid URL format (scheme and netloc), and contain "oic" within the network location to confirm they target an OIC instance.
The diagram shows the sequence of validation checks applied to every OIC endpoint before a request is initiated.
Sources: src/flext_tap_oracle_oic/utilities.py:127-148
API URL Construction
The build_oic_api_url method combines base URLs with resource paths and query parameters, handling leading slashes and query string formatting automatically.
| Parameter | Type | Description |
|---|---|---|
base_url | str | The base OIC instance URL. |
resource_path | str | The specific API path (e.g., /integrations). |
query_params | t.StrMapping | Optional dictionary of query key-value pairs. |
Sources: src/flext_tap_oracle_oic/utilities.py:34-63
Data Parsing and Normalization
OIC responses often include metadata envelopes. Utilities are used to flatten these structures and normalize field names to comply with standard JSON schema identifiers.
Pagination Info Extraction
OIC APIs return pagination details like hasMore, limit, and offset. The utility extract_pagination_info standardizes these into a consistent mapping, providing defaults if fields are missing.
# Example of normalized pagination output
{
"has_more": response.get("hasMore", False),
"limit": response.get("limit", c.DEFAULT_PAGE_SIZE),
"offset": response.get("offset", 0),
"total_count": response.get("count", 0),
"current_page_size": len(items_list),
}
Sources: src/flext_tap_oracle_oic/utilities.py:65-98
Field and Name Sanitization
To ensure compatibility with downstream systems, OIC names are normalized and field names are sanitized.
- Normalize Name: Converts non-alphanumeric characters to underscores and strips leading/trailing underscores.
- Sanitize Field Name: Converts CamelCase to snake_case, removes non-identifiers, and prefixes numeric-starting fields with
field_.
Sources: src/flext_tap_oracle_oic/utilities.py:228-268, src/flext_tap_oracle_oic/constants.py:33-41
State and Bookmark Management
State management is critical for incremental extraction. The utilities provide a standard way to interact with the Singer state object's bookmarks branch.
Bookmark Operations
The system supports getting and setting bookmarks for specific streams to track progress (e.g., replication keys or pagination offsets).
| Operation | Purpose |
|---|---|
get_bookmark | Retrieves a specific key value from a stream's state. |
set_bookmark | Updates a specific key in a stream's state and returns the new state object. |
update_pagination_bookmark | Calculates and sets the next pagination_offset based on the current page size. |
The diagram illustrates the flow of updating a pagination bookmark during a multi-page data extraction process.
Sources: src/flext_tap_oracle_oic/utilities.py:448-477
Model Validation Helpers
A separate set of internal helpers focuses on validating configuration and entity identifiers. These ensure that essential parameters like OIC base URLs, usernames, and passwords meet the project's requirements before the tap attempts to connect.
require_entity_value: Ensures required strings (like IDs or Names) are not empty.validate_optional_port: Verifies that network ports are within the valid range (1-65535).validate_entity_identity_and_port: A composite helper to validate both identity strings and optional port numbers.
Sources: src/flext_tap_oracle_oic/_models/_helpers.py:1-40, src/flext_tap_oracle_oic/models.py:82-126
Summary of Constants
Utilities rely on a centralized set of constants for default behaviors.
| Constant | Value | Description |
|---|---|---|
DEFAULT_PAGE_SIZE | 50 | Default number of records per OIC API page. |
MAX_PAGE_SIZE | 1000 | Maximum allowed records per page for OIC. |
HTTP_UNAUTHORIZED | 401 | Status code used for auth failure detection. |
OIC_API_BASE_PATH | /ic/api/integration/v1 | Root path for core OIC integration APIs. |
Sources: src/flext_tap_oracle_oic/constants.py:43-65
Helper Utilities serve as the operational core of flext-tap-oracle-oic, encapsulating low-level OIC API logic, pagination math, and Singer state management into a reusable, static interface used across all stream implementations.