Meltano & Singer Ecosystem Integration
Relevant source files
The following files were used as context for generating this wiki page:
The flext-tap-ldap project is a Singer-compliant Tap designed for extracting data from LDAP Directory Services and LDIF files. It integrates into the Meltano ecosystem by utilizing the flext-meltano abstraction layer, ensuring compatibility with Singer protocol standards for data extraction, stream discovery, and catalog generation.
The integration provides a unified interface for extracting hierarchical directory data (such as Users, Groups, and OUs) and transforming it into flat Singer RECORD messages. It supports both live LDAP server connections and static LDIF file processing, allowing it to function as a versatile component in modern ETL/ELT pipelines.
Sources: README.md:1-10, src/flext_tap_ldap/tap.py:15-25
Architecture and Core Components
The integration architecture centers around the FlextTapLdapTap class, which inherits from FlextMeltanoAbstractions. This class manages the lifecycle of the tap, including configuration loading, stream discovery, and execution.
Tap Lifecycle and Flow
The tap follows the standard Singer execution flow:
- Initialization: Loading configuration via Pydantic-based settings.
- Discovery: Identifying available LDAP objects and LDIF files to build a Singer Catalog.
- Sync: Iterating through selected streams and emitting
SCHEMAandRECORDmessages.
The diagram above shows the high-level execution flow from the CLI to record emission.
Sources: src/flext_tap_ldap/tap.py:28-40, src/flext_tap_ldap/tap.py:165-210
Stream Discovery and Catalog Management
The discover_streams method is responsible for generating the Singer Catalog. It categorizes streams into two primary types: standard LDAP streams and optional LDIF processing streams.
Standard LDAP Streams
By default, the tap discovers the following streams:
- Users: Filtered by
(objectClass=inetOrgPerson). - Groups: Filtered by
(objectClass=groupOfNames). - Organizational Units: Directory hierarchy nodes.
- Schema: LDAP schema definitions.
LDIF Processing Streams
When enable_ldif_streams is set to True, the tap includes:
ldif_entries: Raw entries extracted from files.ldif_analysis: Statistical summary of LDIF content.
| Component | Description | Key Properties |
|---|---|---|
UsersStream | Extracts user directory entries | ["dn"] |
GroupsStream | Extracts group memberships | ["dn"] |
LdifStream | Processes static LDIF files via flext-ldif | ["dn"] |
LdifAnalysisStream | Generates counts by objectClass/entryType | ["analysis_id"] |
Sources: src/flext_tap_ldap/tap.py:114-145, src/flext_tap_ldap/ldif_streams.py:38-50, src/flext_tap_ldap/ldif_streams.py:186-200
Configuration and Settings
The integration uses FlextTapLdapSettings (subclassing FlextSettings) to manage connection and processing parameters. This ensures that environment variables and JSON configuration files are validated against a strict schema before execution.
# Example Configuration Schema
config_jsonschema = {
"type": "object",
"properties": {
"host": {"type": "string", "description": "LDAP server host"},
"port": {"type": "integer", "default": 389},
"use_ssl": {"type": "boolean", "default": False},
"ldif_files": {"type": "array", "items": {"type": "string"}},
"enable_ldif_streams": {"type": "boolean", "default": False}
}
}
Sources: src/flext_tap_ldap/tap.py:61-105, src/flext_tap_ldap/models.py:75-100
Data Processing and Transformation
The data flow from LDAP/LDIF to Singer involves a transformation layer that flattens nested directory attributes into JSON-compatible formats.
LDIF Processor Logic
The Processor and Transformer classes in the _utilities namespace handle the conversion of complex LDIF structures.
The sequence shows how raw entries are transformed through rules before being formatted as Singer records.
Sources: src/flext_tap_ldap/_utilities/_processor.py:270-300, src/flext_tap_ldap/_utilities/_processor.py:465-500
Singer Record Formatting
Records are emitted as JSON objects containing the dn (Distinguished Name) as the primary identifier, followed by attributes mapped as arrays of strings to maintain LDAP's multi-value nature.
Sources: src/flext_tap_ldap/_utilities/_processor.py:302-315
CLI Integration
The project exposes two main CLI commands: flext-tap-ldap and tap-ldap. These are built using the click library and support standard Singer flags:
--config: Path to the configuration file.--discover: Runs discovery mode and outputs the JSON catalog.--catalog: Path to the catalog file for selected sync.--state: Path to the state file for incremental replication.
Sources: pyproject.toml:90-95, src/flext_tap_ldap/tap.py:230-260
Summary
The Meltano and Singer integration in flext-tap-ldap provides a robust bridge between LDAP directory services and modern data engineering workflows. By leveraging the flext-meltano abstractions, it ensures that LDAP data is extracted with high fidelity, validated via strict schemas, and delivered in a standard format compatible with any Singer-compliant target.