LDIF File Processing
Relevant source files
The following files were used as context for generating this wiki page:
LDIF (LDAP Data Interchange Format) file processing in the flext-tap-ldap project provides a robust mechanism for parsing, validating, and transforming LDAP directory data stored in flat files. This system is integrated into the Singer tap framework, allowing LDAP data to be extracted from either a live directory service or exported LDIF files. The core logic leverages the flext-ldif library to handle the complexities of LDIF syntax while providing a high-level API for entry manipulation and stream generation.
The processing pipeline is primarily managed through the FlextTapLdapUtilitiesProcessorMixin, which exposes a namespace for entry objects, file processors, validators, and transformers. This allows the tap to treat LDIF data as a first-class stream source, supporting both bulk data extraction and summary analysis.
Sources: src/flext_tap_ldap/_utilities/_processor.py:1-25, src/flext_tap_ldap/ldif_streams.py:34-40
Core Components
The LDIF processing architecture is built around four primary inner classes within the u.TapLdap namespace.
The Entry Class
The Entry class serves as a testing and manipulation wrapper for LDIF entries. It maintains a mapping of attributes and provides methods for case-insensitive attribute resolution, object class checking, and DN parsing. It delegates canonical validation and parsing to the flext-ldif library backend.
Sources: src/flext_tap_ldap/_utilities/_processor.py:38-164
The Processor Class
The Processor handles the ingestion of LDIF data from files or strings. It supports error handling modes (ignoring or raising errors), entry filtering (by DN pattern or object class), and conversion to Singer-compatible record formats.
Sources: src/flext_tap_ldap/_utilities/_processor.py:166-382
Validator and Transformer
- Validator: Utilizes
flext-ldifto ensure entries adhere to LDAP standards and schema constraints, providing detailed error and warning logs. - Transformer: Applies transformation rules to entries, including attribute mapping, schema normalization, value mapping, and attribute removal/addition.
Sources: src/flext_tap_ldap/_utilities/_processor.py:384-539
Data Flow and Architecture
The following diagram illustrates the flow of data from an LDIF source through the processor and transformer into the Singer tap output.
The system first parses the raw LDIF content. If validation passes, the entry is passed to the transformer where schema and attribute rules are applied before final conversion to a Singer record.
Sources: src/flext_tap_ldap/_utilities/_processor.py:270-330, src/flext_tap_ldap/ldif_streams.py:130-153
LDIF Stream Implementation
LDIF processing is exposed to the Singer tap through specialized stream classes. These streams are enabled via the enable_ldif_streams configuration setting.
| Stream Class | Stream ID | Description |
|---|---|---|
LdifStream | ldif_entries | Extracts full LDAP entries from LDIF files as Singer records. |
LdifAnalysisStream | ldif_analysis | Provides a summary of the LDIF files, including entry counts by type and object class. |
Sources: src/flext_tap_ldap/ldif_streams.py:44-67, src/flext_tap_ldap/ldif_streams.py:168-193
Sequence of Stream Processing
The sequence below shows how the LdifStream interacts with the configuration and the Processor to yield records.
The stream discovers files based on the ldif_directory or ldif_files settings, parses them, and yields records containing the DN, entry type, object classes, and attributes.
Sources: src/flext_tap_ldap/ldif_streams.py:79-100, src/flext_tap_ldap/ldif_streams.py:126-150
Configuration Options
LDIF processing is controlled through specific keys in the tap's settings.
| Setting | Type | Default | Description |
|---|---|---|---|
enable_ldif_streams | boolean | False | Must be true to activate LDIF file processing. |
ldif_files | array[string] | [] | Explicit list of LDIF file paths to process. |
ldif_directory | string | None | Directory path to scan for LDIF files. |
ldif_file_pattern | string | *.ldif | Glob pattern for discovering files in ldif_directory. |
Sources: src/flext_tap_ldap/tap.py:65-115, src/flext_tap_ldap/ldif_streams.py:102-111
Transformation Logic
The Transformer class applies a specific sequence of operations to entries during processing. This is particularly useful for normalizing data from different LDAP implementations (e.g., Oracle OID).
- Schema Mapping: Maps source attributes to target names and provides default values if the source is missing.
- Attribute Mapping: Renames attributes according to a simple key-value map.
- Value Mapping: Replaces specific attribute values (e.g., mapping "IT" to "Information Technology").
- Removal: Drops attributes specified in the
remove_attributeslist. - Addition: Appends new attributes with static or list-based values.
Sources: src/flext_tap_ldap/_utilities/_processor.py:456-539
Transformation Example
# From tests/unit/test_ldif_processor.py:47-65
transformer = u.TapLdap.Transformer(
transformation_rules={
"attribute_mappings": {"CN": "cn", "sn": "surname"},
"attribute_value_mappings": {
"department": {"IT": "Information Technology"},
},
"remove_attributes": ["obsolete"],
"add_attributes": {"status": "active"},
},
)
Sources: tests/unit/test_ldif_processor.py:47-65
Conclusion
LDIF File Processing in flext-tap-ldap provides a comprehensive toolkit for handling LDAP data in transit. By combining robust parsing via flext-ldif with the flexible stream architecture of Singer, the tap enables complex directory migrations and data synchronizations with high reliability and configurable transformation rules.