Configuration Management
Relevant source files
The following files were used as context for generating this wiki page:
Configuration Management in flext-tap-oracle provides a unified, validated, and type-safe approach to managing settings for Oracle Singer tap operations. It leverages the FLEXT framework's patterns, specifically integrating Pydantic v2 for data validation and environment variable mapping to ensure robust database connections and extraction parameters.
The system is designed to handle sensitive credentials via SecretStr, manage environment-specific rules (e.g., development vs. production), and provide a central authority for business rule validation. This ensures that the tap is correctly configured before initiating expensive database operations or data extraction tasks.
Configuration Architecture
The configuration system is built upon a hierarchical model where FlextTapOracleSettings serves as the primary runtime interface. It inherits from FlextSettingsBase, enabling standard FLEXT features like environment variable prefixes and singleton management.
The architecture separates definition (via rules.json), runtime validation (via FlextTapOracleSettings), and metadata handling.
Sources: src/flext_tap_oracle/settings.py:23-28, src/flext_tap_oracle/rules.json:44-55
Key Configuration Components
1. Settings Model (FlextTapOracleSettings)
The FlextTapOracleSettings class defines the schema for all required and optional parameters. It uses Pydantic's Annotated and Field for rich metadata and defaults.
| Option | Type | Default | Description |
|---|---|---|---|
oracle_host | NonEmptyStr | c.DbOracle.DEFAULT_HOST | Host address of the Oracle DB |
oracle_port | PortNumber | c.DbOracle.DEFAULT_PORT | Listener port for the DB |
oracle_service_name | str | c.DbOracle.DEFAULT_SERVICE_NAME | Service name or SID |
oracle_user | SecretStr | Required | DB username (masked) |
oracle_password | SecretStr | Required | DB password (masked) |
batch_size | BatchSize | 1000 | Records per extraction batch |
stream_prefix | str | "" | Prefix for Singer streams |
environment | str | "production" | Deployment environment name |
Sources: src/flext_tap_oracle/settings.py:32-60
2. Validation and Business Rules
Validation occurs at multiple levels: structural validation via Pydantic and logic-based validation via the validate_business_rules method.
Configuration Rules Schema (rules.json)
The rules.json file acts as the source of truth for the tap's capabilities and validation constraints. It defines specific requirements for connection parameters, Singer protocol compliance, and performance tuning.
Connection Constraints
Constraints defined in rules.json ensure that parameters stay within safe operational limits:
- Port: Must be between 1 and 65535.
- Batch Size: Minimum 1, Maximum 50,000.
- Connection Pool: Minimum 1, Maximum 50.
- Parallel Streams: Must not exceed the connection pool size.
Sources: src/flext_tap_oracle/rules.json:76-135
Environment Specific Rules
The system adjusts its behavior based on the environment setting:
- Production: Strict connection validation, required SSL, and optimized batch sizes.
- Development: Relaxed validation, enabled SQL logging, and detailed metrics.
- Staging: Production-like settings but with detailed metrics enabled.
Sources: src/flext_tap_oracle/rules.json:285-307
Data Flow for Configuration Injection
The configuration is injected into commands like Discover and Sync using a parameter object pattern. The CLI handles the initial loading of JSON files or environment variables, which are then validated into a settings instance.
Security and Secret Handling
The configuration management system prioritizes security for database credentials:
- SecretStr: The
oracle_userandoracle_passwordfields uset.SecretStrto prevent accidental exposure in logs or tracebacks. - Masking: Sensitive parameters are masked in logs as defined in the security rules.
- Resolution: The
_resolve_secretinternal method is used only at the moment of connection to retrieve the plain-text value.
Sources: src/flext_tap_oracle/settings.py:69-73, src/flext_tap_oracle/rules.json:265-272
Performance Configuration Summary
Performance-related settings are managed through specific extraction and connection pool recommendations.
| Feature | Production Default | Recommended Max |
|---|---|---|
max_parallel_streams | 4 | 20 |
connection_pool_size | 10 | 50 |
query_timeout | 300s | 7200s |
fetch_size | 1000 | - |
Sources: src/flext_tap_oracle/rules.json:197-220
Configuration Management in flext-tap-oracle provides the necessary guardrails for enterprise-grade data extraction by combining strict Pydantic validation with flexible, environment-aware rules defined in JSON. This approach ensures that the Oracle Tap is both secure and performant across varying deployment contexts.