LDIF Streams Implementation
Relevant source files
The following files were used as context for generating this wiki page:
The LDIF Streams implementation in flext-tap-ldap provides a mechanism for ingesting LDAP Data Interchange Format (LDIF) data into a Singer-compatible data pipeline. This module allows the tap to process static LDIF files as an alternative or supplement to live LDAP directory traversal, enabling offline migrations, testing, and data analysis.
The system is built upon the flext-ldif library and follows a clean architecture pattern where stream logic is decoupled from low-level parsing. It supports discovering files from specific paths or directories, validating entry integrity, and transforming attributes through a configurable rules engine.
Sources: src/flext_tap_ldap/ldif_streams.py:1-25, src/flext_tap_ldap/tap.py:15-35
Core Components
The implementation is primarily contained within the FlextTapLdapLdifStreams class, which serves as a container for specialized stream types and utility methods for processing LDIF data.
LdifStream
The LdifStream is responsible for extracting raw entries from LDIF files. It implements the standard Singer stream protocol, yielding records that contain the Distinguished Name (DN), entry classification, object classes, and a map of attributes. It uses simple heuristics to classify entries into types such as user, group, or ou.
Sources: src/flext_tap_ldap/ldif_streams.py:53-125
LdifAnalysisStream
The LdifAnalysisStream provides higher-level metadata about the processed LDIF content. Rather than yielding individual entries, it generates summary records containing counts of total entries, distribution of entry types, and frequency of specific object classes across all processed files.
Sources: src/flext_tap_ldap/ldif_streams.py:165-205
Processing Architecture
The data flow follows a hierarchical discovery and parsing process. The tap checks for specific files defined in ldif_files or performs a recursive search in an ldif_directory using a configurable file pattern.
The diagram shows the logic within get_records where configuration determines the source of the data before it is handed off to the parser.
Sources: src/flext_tap_ldap/ldif_streams.py:85-110
LDIF Utilities and Processing
Low-level processing is handled by the FlextTapLdapUtilitiesProcessorMixin, which provides internal classes for Entry manipulation, File Processing, Validation, and Transformation.
Processor and Validator
The Processor class delegates the actual LDIF parsing to the flext-ldif library. It maintains statistics on processed vs. skipped entries and provides filtering capabilities (e.g., filter_by_objectclass). The Validator ensures that entries meet the required LDAP/LDIF schemas.
| Component | Functionality |
|---|---|
| Processor | Parses files/strings, yields entries, and tracks processing stats. |
| Validator | Performs schema validation using flext-ldif capabilities. |
| Transformer | Applies attribute mappings, value mappings, and removal rules. |
Sources: src/flext_tap_ldap/_utilities/_processor.py:133-250, src/flext_tap_ldap/_utilities/_processor.py:433-460
Entry Transformation Logic
The Transformer component allows for data normalization during the extraction process. Rules can be defined to rename attributes, map specific values (e.g., mapping "IT" to "Information Technology"), or inject default values via schema mappings.
The sequence shows the order of operations when transforming an LDIF entry before it is sent to the target.
Sources: src/flext_tap_ldap/_utilities/_processor.py:495-580, tests/unit/test_ldif_processor.py:65-90
Configuration Options
LDIF streams are enabled and configured through the main tap settings.
| Parameter | Type | Default | Description |
|---|---|---|---|
enable_ldif_streams | boolean | False | Must be true to discover LDIF-related streams. |
ldif_files | array | [] | Explicit list of file paths to process. |
ldif_directory | string | None | Directory path for automated file discovery. |
ldif_file_pattern | string | "*.ldif" | Glob pattern used for directory discovery. |
Sources: src/flext_tap_ldap/tap.py:65-105, src/flext_tap_ldap/ldif_streams.py:130-140
Data Schema
The standard LDIF stream outputs a schema designed to capture the flexible nature of LDAP attributes while providing structure for common fields.
{
"type": "object",
"properties": {
"dn": {"type": "string", "description": "Distinguished Name"},
"entry_type": {"type": "string", "description": "Classification (user, group, ou, other)"},
"object_classes": {"type": "array", "items": {"type": "string"}},
"attributes": {"type": "object", "description": "Key-value map of all entry attributes"}
}
}
Sources: src/flext_tap_ldap/ldif_streams.py:68-83
Summary
The LDIF Streams Implementation provides a robust, library-backed solution for processing LDAP data files. By utilizing the flext-ldif backend, the system ensures high fidelity in parsing while the utility mixins provide the necessary hooks for Singer-specific transformations and validation. This allows flext-tap-ldap to handle complex data extraction scenarios that go beyond simple live directory queries.