Extending the Target
Relevant source files
The following files were used as context for generating this wiki page:
Extending the FLEXT Target Oracle involves implementing new configuration options, enhancing loading functionality, or adding specific database features while adhering to the core design principles of the FLEXT ecosystem. The project follows a "Clean Architecture" approach, emphasizing separation of concerns between message routing, data loading logic, and database-specific operations.
The extension process is governed by strictly typed interfaces and Railway-Oriented Programming (ROP) for error handling. Developers can extend the system by modifying the Target and Loader classes, or by introducing new logic into the supporting utility modules.
Sources: CONTRIBUTING.md:105-131, src/flext_target_oracle/_protocols/base.py:17-150
Core Architecture for Extensions
The target's architecture is built on four primary components that provide extension points for developers:
- Target Class: Manages the Singer message lifecycle and routes data to the loader.
- Loader Class: Contains the core logic for DDL/DML operations and schema transformations.
- Configuration: Pydantic-based settings for validation and feature flags.
- Exceptions: A structured hierarchy for specialized error scenarios.
Component Relationship Diagram
This diagram illustrates how the core components interact when processing a stream, highlighting the boundaries where extensions can be introduced.
The diagram shows the clear separation between the target interface and the loader implementation. Extensions typically occur within the Loader or by adding new fields to Settings.
Sources: CONTRIBUTING.md:112-126, src/flext_target_oracle/_protocols/base.py:112-146
Configuration Extensions
New features are primarily controlled through the FlextOracleTargetSettings class. Adding options here ensures type safety and automatic validation via Pydantic.
Adding New Options
To extend the configuration, developers add fields to the settings model. The project enforces strict typing and uses domain rules for validation.
class FlextOracleTargetSettings(m.BaseModel):
my_new_option: bool = u.Field(
default=False,
description="Enable my new feature",
)
Sources: CONTRIBUTING.md:135-142, examples/01_basic_usage.py:46-65
Implementing New Database Features
The FLEXT Target Oracle maintains a strict rule: SQL must not be generated directly within the target. All database-specific DDL/DML operations must be implemented in the flext-db-oracle library and then consumed via the FlextDbOracleApi.
Loader Extension Workflow
- Interface Definition: Define the new capability in
src/flext_target_oracle/_protocols/base.py. - Implementation: Add the logic to
FlextTargetOracleLoader, utilizingflext-db-oraclefor SQL execution. - Consumption: Call the new loader method from the
FlextTargetOraclemessage processing loop.
| Service Protocol | Responsibility | Key Methods |
|---|---|---|
SchemaService | Table and constraint management | ensure_table_exists |
BatchService | Record queuing and flushing | add_record, flush_all_batches |
RecordService | Data transformation and cleaning | transform_record |
ConnectionService | Session and connectivity | test_connection, get_connection_info |
Sources: CONTRIBUTING.md:148-155, src/flext_target_oracle/_protocols/base.py:110-150
Error Handling Extensions
When extending the target, custom errors should be added to the FlextTargetOracleExceptions hierarchy. This hierarchy inherits from the flext-core base to maintain consistency across the ecosystem.
Exception Flow
The following sequence diagram shows how errors are captured and structured with metadata during a processing failure.
Structured Error Metadata
The FlextTargetOracleErrorMetadata model allows extensions to attach structured context to exceptions, including canonical error codes and correlation IDs.
| Field | Type | Description |
|---|---|---|
code | str | Canonical error code (e.g., from c.ErrorCode) |
context | t.JsonMapping | Structured context such as stream_name or record_count |
correlation_id | str | Cross-service identifier for tracing |
Sources: src/flext_target_oracle/_utilities/errors.py:23-37
Extension Verification (Testing)
Any extension must be accompanied by both unit and integration tests. The project uses pytest with specific markers to distinguish between isolated logic tests and real database operations.
- Unit Tests: Located in
tests/unit/, these should mock all external dependencies, including the Oracle API. - Integration Tests: Located in
tests/integration/, these use a real Oracle container (managed via Docker) to verify DDL/DML execution.
@pytest.mark.integration
@pytest.mark.oracle
def test_new_database_feature(oracle_loader, oracle_engine):
# Verify implementation against a real Oracle instance
pass
Sources: CONTRIBUTING.md:59-86, tests/integration/test_oracle.py:38-42
Summary
Extending the FLEXT Target Oracle requires a disciplined approach to typing and architecture. By leveraging the existing Pydantic settings for configuration and the flext-db-oracle API for database operations, developers can add robust features that remain compatible with the broader Singer and FLEXT ecosystems. All extensions must follow the Railway-Oriented Programming pattern, returning p.Result types to ensure predictable error propagation.
Sources: CONTRIBUTING.md:123-131, src/flext_target_oracle/_protocols/base.py:112-150