Observability & Logging
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Observability and Logging in flext-target-oracle provide a comprehensive framework for monitoring the health, performance, and operational status of data loading pipelines into Oracle databases. The system is designed to provide visibility into connection states, Singer message processing efficiency, and detailed error context for troubleshooting.
This module integrates runtime monitoring, structured logging via the rich and flext-core ecosystems, and health checks suitable for production environments. It ensures that every stage—from database authentication to bulk record insertion—is instrumented with relevant metrics and telemetry.
Sources: src/flext_target_oracle/_utilities/observability.py:1-12, examples/03_production_setup.py:1-25
Core Components
Runtime Observability Utility
The FlextTargetOracleUtilitiesObservability class serves as the central factory for generating structured logs and performance context. It handles specific events such as authentication failures, database unavailability, and schema validation errors.
The diagram shows how the observability utility routes runtime events to specialized handlers that produce either structured logs or performance monitoring contexts.
Sources: src/flext_target_oracle/_utilities/observability.py:17-100
Production Health Monitoring
In production setups, observability is extended through a ProductionTargetManager that tracks processing statistics and provides health check endpoints. This includes monitoring connection pool health and tracking record processing duration.
| Metric / Check | Description | Data Structure |
|---|---|---|
| Health Status | Snapshot of current target health (healthy/unhealthy/degraded). | HealthStatus |
| Processing Stats | In-memory counters for schemas, records, and states processed. | ProcessingStats |
| Connectivity | Active verification of Oracle database reachability. | test_connection() |
| Pool Health | Snapshots of active vs. total connection pool size. | target_oracle_monitor_connection_health |
Sources: examples/03_production_setup.py:45-75, src/flext_target_oracle/_utilities/observability.py:102-114
Logging Architecture
Structured Logging and Context
Logging is implemented using the flext_meltano and flext_core utilities. Logs include correlation IDs and structured context such as stream_name, record_count, and recovery_strategy.
This sequence illustrates the use of context managers to wrap database operations with performance monitoring logs.
Sources: src/flext_target_oracle/_utilities/observability.py:116-133, src/flext_target_oracle/_utilities/loader.py:488-510
Error Factories
The system uses specialized methods to build exceptions that automatically include structured metadata for observability platforms.
- Authentication Errors: Captures the username and service name.
- Connection Errors: Includes connection strings and suggested recovery strategies (e.g.,
retry_with_backoff). - Processing Errors: Logs the ratio of failed records to total records in a batch.
Sources: src/flext_target_oracle/_utilities/observability.py:22-75, src/flext_target_oracle/_utilities/errors.py:65-100
Performance Instrumentation
The loader tracks performance metrics during bulk operations, specifically monitoring the time taken for batch flushes and total records loaded.
@staticmethod
@contextmanager
def target_oracle_monitor_query_performance(
table_name: str,
operation: str = "SELECT",
) -> Generator[t.StrMapping]:
"""Yield a mutable context while timing a query operation."""
context_data = {"table_name": table_name, "operation": operation}
try:
yield context_data
finally:
# Logging of completion logic
pass
Sources: src/flext_target_oracle/_utilities/observability.py:116-133, src/flext_target_oracle/_utilities/loader.py:380-405
Debugging and Configuration
Observability can be toggled and configured via environment variables and settings. Setting FLEXT_DEBUG=true increases the verbosity of the observability logs.
| Configuration Option | Type | Default | Description |
|---|---|---|---|
enable_performance_monitoring | bool | True | Enables query timing and performance logs. |
enable_connection_monitoring | bool | True | Enables connection pool health snapshots. |
FLEXT_LOG_LEVEL | string | INFO | Controls the global logging threshold. |
Sources: src/flext_target_oracle/_utilities/observability.py:135-148, CONTRIBUTING.md:195-200
Conclusion
The Observability & Logging system in flext-target-oracle provides a robust foundation for enterprise data integration. By combining structured error handling, performance monitoring, and production-grade health checks, it allows developers to maintain high reliability and visibility into their Oracle data loading workflows.