Configuration & Settings
Relevant source files
The following files were used as context for generating this wiki page:
The Configuration & Settings module for the Oracle Integration Cloud (OIC) Singer Tap provides a unified framework for managing environment-driven configurations, OAuth2 authentication parameters, and stream-specific execution logic. It leverages Pydantic for strict schema validation and type safety, ensuring that the tap operates within the defined constraints of the Oracle OIC API.
This system is structured into three primary domains: Global Tap Settings, Authentication Configuration, and Stream-level Configuration. These settings are typically populated via environment variables prefixed with FLEXT_TAP_ORACLE_OIC_ or passed through a standard Singer config.json file.
Sources: src/flext_tap_oracle_oic/settings.py:1-20, src/flext_tap_oracle_oic/tap.py:168-185
Core Configuration Architecture
The architecture follows a hierarchical model where FlextTapOracleOicSettings acts as the primary orchestrator, inheriting from FlextOracleOicSettings. It manages connection parameters, timeouts, and global flags for metadata inclusion.
Configuration Flow
The following diagram illustrates how raw configuration parameters are processed and validated before being used by the API client and streams.
The system ensures that parameters like oauth_client_id and oauth_audience are present and non-empty before initiating any network requests.
Sources: src/flext_tap_oracle_oic/settings.py:23-45, src/flext_tap_oracle_oic/tap.py:175-200
Global Tap Settings
Global settings define the behavior of the tap across all streams. This includes connectivity thresholds and toggles for specific categories of data.
| Setting | Type | Default | Description |
|---|---|---|---|
base_url | str | DEFAULT_BASE_URL | The root URL for the OIC instance. |
timeout | PositiveInt | 30 | Request timeout in seconds. |
max_retries | NonNegativeInt | 3 | Number of retry attempts for failed requests. |
page_size | PositiveInt | 50 | Number of records to fetch per API call. |
include_extended | bool | False | Toggle to include extended entity metadata. |
include_monitoring | bool | False | Toggle to include monitoring data streams. |
include_logs | bool | False | Toggle to include log data streams. |
Sources: src/flext_tap_oracle_oic/settings.py:28-60, src/flext_tap_oracle_oic/constants.py:38-45
Authentication Configuration
The OicAuthenticationConfig model specifically manages OAuth2/IDCS (Oracle Identity Cloud Service) parameters. It requires HTTPS for both the token URL and the base URL to maintain security standards.
Authentication Parameters
- oauth_client_id: Unique identifier for the OAuth client.
- oauth_client_secret: Secret key (stored as
SecretStrfor security). - oauth_token_url: The IDCS endpoint for token generation.
- oauth_audience: The target resource scope (e.g.,
urn:opc:resource:consumer:all). - token_expiry_buffer: Buffer time in seconds (default
300) to refresh tokens before they expire.
Sources: src/flext_tap_oracle_oic/_models/_auth_config.py:16-65, src/flext_tap_oracle_oic/tap.py:31-70
Stream-Level Configuration
Individual streams can be fine-tuned via the OicStreamConfiguration model. This allows for granular control over replication methods and data sanitization.
Stream Configuration Properties
The OicStreamConfiguration model validates stream settings using Pydantic validators to ensure performance and security.
- replication_method: Supports
FULL_TABLEorINCREMENTAL. Defaults toFULL_TABLE. - replication_key: Required if the method is set to
INCREMENTAL. - sanitize_sensitive_data: Boolean flag (default
True) to enable data masking during extraction. - exclude_test_entities: Boolean flag (default
True) to filter out demo or test integrations from production runs. - status_filter: A sequence of strings (e.g.,
["ACTIVATED", "ERROR"]) to filter entities by lifecycle status.
Sources: src/flext_tap_oracle_oic/_models/_stream_config.py:19-75, src/flext_tap_oracle_oic/constants.py:155-165
Configuration Validation Utilities
The FlextTapOracleOicUtilities class provides static methods to validate and normalize configuration strings.
URL Validation
The utility ensures that the base_url is a valid OIC endpoint by checking for the presence of "oic" in the network location and verifying the URL scheme (HTTP/HTTPS).
@staticmethod
def validate_oic_endpoint(endpoint_url: str) -> p.Result[str]:
"""Validate Oracle OIC endpoint URL."""
if not endpoint_url:
return r[str].fail("OIC endpoint URL cannot be empty")
parsed = urlparse(endpoint_url)
if not parsed.scheme or not parsed.netloc:
return r[str].fail("Invalid URL format")
if "oic" not in parsed.netloc.lower():
return r[str].fail("URL does not appear to be an OIC endpoint")
return r[str].ok(endpoint_url)
Sources: src/flext_tap_oracle_oic/utilities.py:126-144
Field Normalization
To ensure Singer compatibility, OIC field names and integration names are sanitized using regex constants defined in the TapOracleOic constant namespace.
- NORMALIZE_NON_ALNUM_RE: Replaces non-alphanumeric characters with underscores.
- SANITIZE_CAMEL_BOUNDARY_RE: Converts CamelCase boundaries into snake_case.
Sources: src/flext_tap_oracle_oic/utilities.py:230-265, src/flext_tap_oracle_oic/constants.py:30-36
Summary of Constraints
The following constraints are enforced during configuration loading:
- Page Size: Must be between
1and1000. - Timeout: Must be a positive integer.
- HTTPS: Mandatory for authentication endpoints.
- Token Buffer: Minimum of
60seconds required for the expiry buffer.
Sources: src/flext_tap_oracle_oic/settings.py:108-120, src/flext_tap_oracle_oic/_models/_auth_config.py:90-98, src/flext_tap_oracle_oic/_models/_stream_config.py:112-115