Processing Flow & Orchestration

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_target_ldap/api.py
• src/flext_target_ldap/_models/sinks.py
• src/flext_target_ldap/settings.py
• src/flext_target_ldap/models.py
• tests/unit/test_target.py
• tests/unit/test_integration.py

The Processing Flow & Orchestration system in flext-target-ldap manages the ingestion of Singer-formatted data and its transformation into LDAP directory operations. It utilizes an orchestrator-based architecture to coordinate between input stream parsing, configuration validation, and the execution of LDAP actions such as adding, modifying, or deleting entries.

The system is built upon the flext-core framework, providing a structured approach to handling various LDAP object types (Users, Groups, Organizational Units) through specialized sink classes. The orchestration logic ensures that connection management, batch processing, and error handling are handled consistently across different data streams.

Sources: src/flext_target_ldap/api.py:10-30, src/flext_target_ldap/_models/sinks.py:15-40

Architectural Overview

The orchestration layer acts as a facade between the Singer protocol and the underlying LDAP client. The FlextTargetLdap class serves as the entry point, managing the lifecycle of the orchestrator and the Dependency Injection (DI) container.

System Components

Component	Description
FlextTargetLdap	The main facade class that handles CLI execution and orchestrator initialization.
FlextTargetLdapOrchestrator	Coordinates the flow of data and manages settings validation.
FlextTargetLdapSink	Base class for stream-specific logic, handling record processing and batching.
FlextTargetLdapClient	Low-level utility for communicating with the LDAP server.
FlextTargetLdapSettings	Pydantic-based configuration model for validating LDAP connection and operation parameters.

Sources: src/flext_target_ldap/api.py:33-50, src/flext_target_ldap/_models/sinks.py:44-60, src/flext_target_ldap/settings.py:27-45

Core Processing Logic

The flow begins with the run_cli method, which reads Singer messages from standard input. Messages are categorized into SCHEMA, RECORD, or STATE types. The orchestrator routes RECORD messages to the appropriate sink based on the stream name.

The diagram above illustrates the high-level flow from input ingestion to LDAP execution.
Sources: src/flext_target_ldap/api.py:156-200, tests/unit/test_integration.py:84-100

Record Orchestration

When a record is processed, the system follows a strict sequence: DN (Distinguished Name) construction, attribute mapping, and persistence.

Persistence Strategy

The orchestrator supports two primary persistence modes:

1. Add/Modify (Upsert): Attempts to add an entry; if it exists and update_existing_entries is enabled, it performs a modification.
2. Delete: If the record contains a _sdc_deleted_at timestamp, the orchestrator triggers a delete operation on the corresponding DN.

The sequence diagram shows the logic path for handling record updates vs. deletions.
Sources: src/flext_target_ldap/api.py:129-154, src/flext_target_ldap/_models/sinks.py:199-235, tests/unit/test_integration.py:126-145

Specialized Stream Sinks

The orchestration layer uses a factory-like pattern in get_sink_class to select the specialized handler for a given stream. This allows for specific attribute mapping and DN construction logic per entity type.

Sink Implementations

• Users Sink (FlextTargetLdapUsersSink): Maps standard fields like emails to mail and phone_numbers to telephoneNumber. It defaults to inetOrgPerson and person object classes.
• Groups Sink (FlextTargetLdapGroupsSink): Handles groupOfNames entries, mapping the members field to the LDAP member attribute.
• Organizational Units Sink (FlextTargetLdapOrganizationalUnitsSink): Manages organizationalUnit entries, primarily focusing on ou and description attributes.

Sources: src/flext_target_ldap/api.py:75-95, src/flext_target_ldap/_models/sinks.py:270-350, tests/unit/test_target.py:34-45

Data Models and Validation

Orchestration is supported by a robust domain model defined in FlextTargetLdapModels. These models ensure that data is validated before the orchestrator attempts to send it to the LDAP server.

LDAP Entry Model

The Entry model validates business rules such as:

• DN format must contain attribute=value pairs.
• Entries must contain the top object class.
• person entries must have sn (Surname) and cn (Common Name) attributes.

Sources: src/flext_target_ldap/models.py:114-160

Configuration Orchestration

The FlextTargetLdapSettings model handles the normalization of flat environment variables or JSON config files into the nested structure required by the orchestrator.

# From src/flext_target_ldap/settings.py
@u.model_validator(mode="before")
@classmethod
def normalize_flat_settings(cls, data):
    # Logic to move 'host', 'port', 'bind_dn' etc. 
    # into a nested 'connection' object
    ...

Sources: src/flext_target_ldap/settings.py:31-60

Error Handling & Resilience

The orchestrator utilizes a "Fail Loud" principle for structural failures while providing "Singer Safe" exception handling for individual record processing.

1. Validation Gates: The system validates the entire configuration via validate_config() during the setup() phase.
2. Guard Results: Record processing is wrapped in u.guard_result, catching SINGER_SAFE_EXCEPTIONS to prevent a single malformed record from crashing the entire pipeline.
3. Counters: The FlextTargetLdapProcessingResult tracks success_count and error_count throughout the execution lifecycle.

Sources: src/flext_target_ldap/api.py:105-115, src/flext_target_ldap/_models/sinks.py:36-42, src/flext_target_ldap/_models/sinks.py:73-82

Summary

Processing Flow & Orchestration in flext-target-ldap is a multi-stage pipeline that ensures Singer records are safely and accurately reflected in an LDAP directory. By separating stream-specific logic into specialized sinks and centralizing connection and lifecycle management in the FlextTargetLdap facade, the system achieves a balance of flexibility for different LDAP schemas and rigid adherence to the Singer protocol standards. Middle-layer validation via Pydantic models ensures that only schema-compliant data reaches the directory server, while the upsert logic provides resilience against existing directory states.