Monitoring Stream
Relevant source files
The following files were used as context for generating this wiki page:
The Monitoring Stream is a critical component of the flext-tap-oracle-oic project, designed to extract performance, execution, and health data from Oracle Integration Cloud (OIC). It encompasses multiple sub-streams, specifically executions and metrics, which provide visibility into integration flow instances and system utilization.
This system utilizes the OIC Monitoring API to capture detailed metadata about integration runs, including start/end times, durations, and error statuses. By leveraging standardized Singer protocol patterns, it allows for incremental data extraction using timestamps as replication keys, ensuring efficient synchronization of monitoring records for downstream analysis.
Sources: src/flext_tap_oracle_oic/_models/streams.py:610-630, src/flext_tap_oracle_oic/models.py:125-135
Stream Architecture and Logic
The Monitoring Stream is built upon the OICBaseStream class, which provides the foundation for communicating with OIC's specialized monitoring endpoints. Unlike core design-time APIs, monitoring streams target specific paths like /monitoring/v1/integrations and /monitoring/v1/metrics.
Data Flow for Monitoring Extraction
The following diagram illustrates the flow of data from the OIC Monitoring API through the tap's internal processing logic.
The extraction process involves authenticated requests to OIC, followed by normalization of the OIC response envelope into individual records enriched with metadata such as _tap_extracted_at.
Sources: src/flext_tap_oracle_oic/models.py:175-185, src/flext_tap_oracle_oic/models.py:270-285
Data Models
The monitoring system relies on two primary data structures: OICMonitoringRecord for flow instances and OicMetricsRecord for performance telemetry.
Execution Monitoring (executions)
The ExecutionsStream captures data for integration flow instances. It uses instanceId as the primary key and startTime for incremental replication.
Sources: src/flext_tap_oracle_oic/_models/streams.py:610-620
| Field | Type | Description |
|---|---|---|
instanceId | String | Unique execution instance identifier |
integrationName | String | Name of the integration that ran |
status | String | Current execution status (e.g., COMPLETED, FAILED) |
startTime | DateTime | When the integration started |
duration | Integer | Execution duration in milliseconds |
errorCode | String | OIC error code if the flow failed |
| Sources: src/flext_tap_oracle_oic/_models/streams.py:633-660 |
Performance Metrics (metrics)
The MetricsStream extracts high-frequency performance data, including throughput and resource utilization.
Sources: src/flext_tap_oracle_oic/_models/_metrics.py:18-30
| Metric Category | Field | Description |
|---|---|---|
| Performance | throughput_mps | Messages per second processed |
| Resources | cpu_usage_percent | Percentage of CPU consumed |
| Resources | memory_usage_mb | Memory utilization in Megabytes |
| Business | success_count | Number of successful messages |
| Business | error_count | Number of failed messages |
| Sources: src/flext_tap_oracle_oic/_models/_metrics.py:45-85 |
API Implementation Details
The monitoring functionality is mapped to specific OIC API categories and paths.
The url_base property dynamically constructs the endpoint based on the api_category. For monitoring streams, it appends the defined monitoring base path (typically /ic/api/monitoring/v1).
Sources: src/flext_tap_oracle_oic/models.py:145-170, src/flext_tap_oracle_oic/constants.py:48-49
Status Mapping
OIC execution statuses are standardized using the OicJobStatus enumeration to ensure consistent reporting across different integration types.
Sources: src/flext_tap_oracle_oic/constants.py:90-97
| OIC Status | Internal Constant | Description |
|---|---|---|
| COMPLETED | OicJobStatus.COMPLETED | Successful execution |
| FAILED | OicJobStatus.FAILED | Execution resulted in an error |
| RUNNING | OicJobStatus.RUNNING | Flow is currently in progress |
| ABORTED | OicJobStatus.ABORTED | Flow was manually terminated |
| Sources: src/flext_tap_oracle_oic/constants.py:92-97 |
Monitoring Record Logic
The OICMonitoringRecord class includes helper properties to simplify state detection for extracted records.
@property
def is_failed(self) -> bool:
"""Check if execution failed."""
return self.execution_status.lower() in {
c.TapOracleOic.OicJobStatus.FAILED.value.lower(),
"faulted",
c.TapOracleOic.OicJobStatus.ABORTED.value.lower(),
}
Sources: src/flext_tap_oracle_oic/_models/_oic_monitoring.py:53-61
Summary
The Monitoring Stream provides the necessary instrumentation for OIC operational oversight. By separating execution metadata from performance metrics, the tap allows users to monitor both individual integration reliability and overall system health. Through the use of incremental replication keys like startTime and timestamp, the system maintains a high-fidelity history of OIC activity with minimal API overhead.
Sources: src/flext_tap_oracle_oic/_models/streams.py:623, src/flext_tap_oracle_oic/_models/_metrics.py:41-43