Direct LDAP Streams

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_tap_ldap/streams.py
• src/flext_tap_ldap/tap.py
• src/flext_tap_ldap/_utilities/client_support.py
• tests/unit/test_tap.py
• tests/unit/test_client_quick.py
• src/flext_tap_ldap/_utilities/_processor.py

Direct LDAP Streams constitute the core data extraction layer of the flext-tap-ldap project. This system is designed to interface directly with LDAP (Lightweight Directory Access Protocol) directory services to extract structured information such as users, groups, organizational units, and schema definitions. It utilizes the Singer protocol standard to facilitate ETL (Extract, Transform, Load) processes within the FLEXT ecosystem.

The architecture relies on a unified streams class that manages specific LDAP directory objects through nested stream implementations. Each stream is responsible for defined LDAP searches, attribute mapping, and result normalization, ensuring high-fidelity data transfer from directory services to downstream consumers.

Sources: src/flext_tap_ldap/streams.py:1-25, src/flext_tap_ldap/tap.py:1-30

Stream Architecture and Hierarchy

The system is organized around the FlextTapLdapStreams class, which serves as a container for various LDAP-specific stream implementations. All direct LDAP streams inherit from LDAPBaseStream, which provides shared logic for connection management, Pydantic-based configuration validation, and search execution.

Stream Class Hierarchy

The following diagram illustrates the relationship between the base stream and specific data implementations.

The hierarchy ensures that connection handling and the LDAP client lifecycle are centralized while allowing individual streams to define their own schemas and search filters.

Sources: src/flext_tap_ldap/streams.py:28-36, src/flext_tap_ldap/streams.py:141-143, src/flext_tap_ldap/streams.py:221-223

Core LDAP Streams

The project implements four primary streams for direct directory extraction. These streams are automatically discovered by the tap during the initialization phase.

Stream Name	Purpose	Search Filter (Default)	Primary Keys
users	Extracts user account entries	(objectClass=inetOrgPerson)	dn
groups	Extracts directory group entries	(objectClass=groupOfNames)	dn
organizational_units	Extracts OU structures	(objectClass=organizationalUnit)	dn
schema	Extracts server schema metadata	(objectClass=schema)	dn

Sources: src/flext_tap_ldap/streams.py:141-326, src/flext_tap_ldap/tap.py:112-120

Data Flow for Search Execution

When a stream's get_records method is invoked, it follows a standardized search execution flow involving client initialization and result normalization.

The search scope is typically set to SUBTREE to ensure comprehensive coverage from the configured base_dn.

Sources: src/flext_tap_ldap/streams.py:107-139, src/flext_tap_ldap/streams.py:188-219

Connection and Client Management

The LDAPBaseStream manages the lifecycle of the FlextTapLdapClient.LDAPClient. The client is created using parameters validated via Pydantic models to ensure strict typing and configuration safety.

Connection Parameters

The system validates the following parameters before establishing a connection:

Parameter	Type	Default	Description
host	String	(Required)	LDAP server hostname or IP.
port	Integer	389	LDAP service port.
use_ssl	Boolean	False	Enables LDAPS encryption.
bind_dn	String	None	Distinguished Name for auth.
bind_password	String	None	Password for the Bind DN.
timeout_seconds	Integer	30	Search timeout.
base_dn	String	None	Starting point for searches.

Sources: src/flext_tap_ldap/streams.py:45-65, src/flext_tap_ldap/tap.py:65-80

Connection Lifecycle

1. Parsing: parse_connection_config validates raw JSON configuration.
2. Instantiation: _create_ldap_client constructs the client with optional SSL and timeout settings.
3. Execution: _search_ldap uses the client to perform authenticated or anonymous searches.

Sources: src/flext_tap_ldap/streams.py:42-105

Specialized Support Features

Oracle OID Support

The system includes specific utilities for Oracle Internet Directory (OID). When oracle_oid_mode is enabled, the client enrichment layer automatically normalizes attributes like orclPassword to userPassword and ensures orclContainer entries are treated as organizationalUnit types.

Sources: src/flext_tap_ldap/_utilities/client_support.py:73-109, tests/unit/test_client_quick.py:175-225

Attribute Normalization

LDAP search results are normalized from raw directory entries into Singer-compatible JSON mappings. This process handles multi-valued attributes by converting them into lists and ensuring single-valued attributes are represented as scalars.

# Normalization Logic (Simplified)
# Sources: [src/flext_tap_ldap/_utilities/client_support.py:40-60]
if len(value) == 1:
    entry_mapping[key_str] = value[0]
else:
    entry_mapping[key_str] = list(value)

Discovery and Execution

The FlextTapLdapTap class orchestrates the discovery of these streams. During discovery, the tap iterates through the available stream classes and generates a Singer Catalog.

Summary

Direct LDAP Streams provide a robust mechanism for extracting organizational data from LDAP servers. By leveraging a common base class (LDAPBaseStream) and specialized client support utilities, the system ensures consistent error handling, configuration validation, and support for vendor-specific directory features like Oracle OID. The integration with the Singer protocol allows these streams to be utilized in standard data pipelines for seamless directory synchronization.

Sources: src/flext_tap_ldap/streams.py:28-40, src/flext_tap_ldap/tap.py:40-60