Singer Tap Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Singer Tap Architecture for flext-tap-ldap provides a standardized framework for extracting data from LDAP Directory Services and LDIF files. It utilizes the Singer protocol to ensure compatibility with various data engineering ecosystems, specifically leveraging FlextMeltanoAbstractions for streamlined integration. The architecture is designed to handle both live LDAP directory searches and offline LDIF (LDAP Data Interchange Format) processing through a unified stream discovery and execution model.
At its core, the system acts as a bridge between LDAP-specific protocols and the Singer-compliant JSON-based stream output. It manages connection configurations, stream schema discovery, and record extraction logic while supporting complex transformations and Oracle OID-specific enrichments.
Sources: src/flext_tap_ldap/tap.py:15-32, src/flext_tap_ldap/streams.py:22-35, src/flext_tap_ldap/ldif_streams.py:23-30
Core Components
The architecture is composed of several specialized modules that handle specific concerns: the Tap Entry Point, Stream Definitions, Client Infrastructure, and Utility Processors.
Tap Entry Point
The FlextTapLdapTap class serves as the primary orchestration layer. It manages the lifecycle of the data extraction process, including configuration validation against a JSON schema, stream discovery, and execution.
The tap supports two primary modes: Discovery mode (for generating catalogs) and Sync mode (for extracting records).
Sources: src/flext_tap_ldap/tap.py:35-50, src/flext_tap_ldap/tap.py:126-140
Stream Management
Streams are divided into two main categories: live directory streams and LDIF-based processing streams. All streams inherit from a base implementation that follows the Singer protocol.
| Stream Type | Description | Key Properties |
|---|---|---|
UsersStream | Extracts user entries (e.g., inetOrgPerson). | dn, modifyTimestamp |
GroupsStream | Extracts group memberships and metadata. | dn, modifyTimestamp |
OrganizationalUnitsStream | Extracts hierarchy data (OU). | dn |
SchemaStream | Extracts LDAP directory schema definitions. | dn |
LdifStream | Processes static LDIF files for migration or analysis. | dn |
Sources: src/flext_tap_ldap/streams.py:168-350, src/flext_tap_ldap/ldif_streams.py:53-65
Data Flow and Execution
The data flow follows a synchronous path from the source (LDAP server or LDIF file) through a normalization layer to the Singer output.
LDAP Connection Logic
The FlextTapLdapClient.LDAPClient acts as a testing convenience wrapper around the internal API. It handles server URI construction, authentication (Bind DN), and timeout management. It also includes specific logic for Oracle OID support, enabling the extraction of orclPassword and mapping it to standard fields.
Sources: src/flext_tap_ldap/client.py:44-100, src/flext_tap_ldap/client.py:125-155
LDIF Processing and Transformation
For file-based sources, the Processor and Transformer utilities handle the extraction. The Transformer allows for attribute mapping, value mapping (e.g., changing "IT" to "Information Technology"), and schema normalization before the data reaches the stream.
Sources: src/flext_tap_ldap/_utilities/_processor.py:465-550
Configuration and Models
The tap configuration is strictly typed and validated using Pydantic models. This ensures that connection parameters such as port, host, and SSL settings meet the required constraints before any network calls are initiated.
Connection Parameters
| Field | Type | Description | Default |
|---|---|---|---|
host | String | LDAP server hostname | (Required) |
port | Integer | LDAP server port | 389 |
use_ssl | Boolean | Whether to use SSL/TLS | False |
page_size | Integer | Size for paged results | 1000 |
timeout_seconds | Integer | Search timeout | 30 |
Sources: src/flext_tap_ldap/tap.py:61-95, src/flext_tap_ldap/models.py:75-105
Error Handling and Validation
The architecture emphasizes a "fail loud" approach when mandatory configuration is missing, while providing ignore_errors flags for large LDIF file processing to prevent a single malformed entry from halting the entire sync. The Validator class uses flext-ldif capabilities to check objectClass constraints and entry integrity during the sync process.
Sources: src/flext_tap_ldap/_utilities/_processor.py:425-455, src/flext_tap_ldap/_utilities/_processor.py:240-260
Conclusion
The flext-tap-ldap architecture provides a robust, typed, and Singer-compliant mechanism for LDAP data integration. By separating the client protocol logic from the stream management and providing specialized processors for LDIF data, it ensures high reliability for both enterprise directory synchronization and legacy data migration tasks.