Singer Tap Lifecycle
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Singer Tap Lifecycle for flext-tap-oracle-oic defines the sequence of operations for extracting data from Oracle Integration Cloud (OIC). It encompasses configuration loading, authentication, stream discovery, and data extraction using standardized Singer protocol patterns. The system is built upon the FLEXT framework, utilizing FlextMeltanoAbstractions and FlextApi to interface with OIC REST endpoints.
Sources: src/flext_tap_oracle_oic/tap.py:1-40, src/flext_tap_oracle_oic/cli.py:1-25
The lifecycle is triggered through the CLI entry point, supporting standard operations such as --discover for metadata generation and --run for data syncs. It manages complex OIC requirements including OAuth2 token management, adaptive pagination, and incremental state tracking to ensure resilient data pipelines.
Sources: src/flext_tap_oracle_oic/cli.py:44-65, src/flext_tap_oracle_oic/tap.py:317-340
Execution Flow Overview
The execution follows a structured path from environment bootstrap to record extraction.
High-Level Tap Flow
This diagram illustrates the lifecycle from invocation to completion.
1. Initialization and Configuration
The lifecycle begins by loading settings from environment variables (prefixed with FLEXT_TAP_ORACLE_OIC_) or a configuration file. The FlextTapOracleOicSettings class validates the presence of mandatory fields like OAuth credentials and the OIC instance URL.
Sources: src/flext_tap_oracle_oic/settings.py:22-40, src/flext_tap_oracle_oic/tap.py:326-338
| Setting | Type | Description |
|---|---|---|
oauth_client_id | str | OAuth2 client ID for OIC |
oauth_client_secret | SecretStr | Sensitive OAuth2 client secret |
oic_url | str | OIC instance base URL |
include_extended | bool | Toggle for infrastructure metadata streams |
page_size | int | Default record limit per request |
Sources: src/flext_tap_oracle_oic/settings.py:25-60, src/flext_tap_oracle_oic/tap.py:157-173
2. Authentication Protocol
The FlextOracleOicAuthenticator handles the OAuth2 Client Credentials flow. Before any API request is made, the tap requests a Bearer token from the oauth_token_url.
The token is cached and injected into the headers of subsequent requests by the FlextTapOracleOicClient.
Sources: src/flext_tap_oracle_oic/tap.py:133-143
3. Discovery Phase
During discovery (--discover), the tap identifies available streams based on the configuration. By default, it includes core streams (Integrations, Connections, etc.), but can expand to infrastructure streams if include_infrastructure is enabled.
Stream Discovery Components
discover_oic_streams: Instantiates stream classes from the registry.discover_streams: Generates a Singer-compliant JSON catalog.build_catalog_entry: Formats stream schemas and replication keys.
Sources: src/flext_tap_oracle_oic/tap.py:192-243
4. Extraction and Pagination
The extraction lifecycle for each stream involves an adaptive loop managed by the FlextTapOracleOicPaginator.
Adaptive Pagination Logic
The paginator monitors API response times to adjust page sizes dynamically. If the average response time exceeds the SLOW_RESPONSE_THRESHOLD (5.0s), it reduces the page_size by 20%.
Sources: src/flext_tap_oracle_oic/tap_streams.py:22-38, src/flext_tap_oracle_oic/tap_streams.py:76-88
Data Enrichment
Records are parsed from the OIC response envelope (usually under items or data keys). Each record is enriched with metadata before being yielded:
_tap_extracted_at: UTC timestamp of extraction._tap_stream_name: The source stream identifier.
Sources: src/flext_tap_oracle_oic/models.py:317-323, src/flext_tap_oracle_oic/utilities.py:84-118
5. State Management
The lifecycle maintains state to support incremental syncs. The FlextTapOracleOicUtilities provide methods for managing bookmarks within the Singer state object.
update_pagination_bookmark: Persists the current offset to prevent re-fetching data on resumed runs.set_bookmark: Updates replication keys (e.g.,lastUpdated) for incremental streams.
Sources: src/flext_tap_oracle_oic/utilities.py:304-334, src/flext_tap_oracle_oic/utilities.py:375-408
Summary
The Singer Tap Lifecycle for Oracle OIC ensures a reliable bridge between Oracle's enterprise REST APIs and the Singer ecosystem. By leveraging automated OAuth2 handling, dynamic pagination adjustment, and standardized FLEXT models, the tap provides a consistent interface for complex integration metadata and monitoring data extraction.