LDIF Processor Utility

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_tap_ldap/_utilities/_processor.py
• src/flext_tap_ldap/ldif_streams.py
• src/flext_tap_ldap/_utilities/client_support.py
• tests/unit/test_ldif_processor.py
• src/flext_tap_ldap/tap.py
• src/flext_tap_ldap/client.py

The LDIF Processor Utility is a core component of the flext-tap-ldap project, designed to handle the parsing, validation, and transformation of LDAP Data Interchange Format (LDIF) content. It acts as a specialized bridge between the flext-ldif library and the Singer tap infrastructure, providing high-level abstractions for managing directory data stored in flat files or memory.

The utility is primarily exposed through the u.TapLdap namespace via a mixin composition, allowing for seamless integration across the tap's streams and client support layers. Its scope includes file discovery, content analysis, schema mapping, and conversion of LDIF entries into Singer-compatible records for data extraction workflows.

Sources: src/flext_tap_ldap/_utilities/_processor.py:1-25, src/flext_tap_ldap/ldif_streams.py:25-45

Architecture and Core Components

The LDIF Processor Utility is organized into four primary inner classes within the TapLdap namespace. Each class handles a specific stage of the LDIF lifecycle.

Namespace Structure

The utility uses a mixin pattern to inject its functionality into the broader flext-tap-ldap utility set.

The Processor Component

The Processor class serves as the main entry point for ingesting LDIF data. It supports loading from both files and strings, delegating the heavy lifting to the flext-ldif library.

Processing Logic

1. Ingestion: Reads raw LDIF content via parse_file or parse_content.
2. Conversion: Converts raw parsed responses into the utility's internal Entry objects.
3. Filtering: Provides methods to filter entries by attribute existence, DN substrings, or object classes.
4. Transformation: Prepares data for Singer streams via to_singer_format.

Key Processor Methods

Method	Description
load_from_file(file_path)	Loads and parses an LDIF file, updating internal statistics.
parse_content(content)	Iterator that yields Entry objects from a raw LDIF string.
filter_by_objectclass(oc)	Returns a sequence of entries containing the specified object class.
statistics()	Returns a mapping containing total, skipped, and error entry counts.

Sources: src/flext_tap_ldap/_utilities/_processor.py:168-300, src/flext_tap_ldap/ldif_streams.py:75-100

The Entry Model

The Entry class is a testing convenience wrapper for LDAP entries. It maintains attributes as a mapping of keys to sequences of strings, reflecting the multi-valued nature of LDAP attributes.

Attribute Management

The Entry model provides case-insensitive attribute resolution and specific helpers for common LDAP tasks:

• resolve_attribute_values(name): Performs a case-insensitive search for attribute values.
• has_object_class(object_class): Uses LDAP normalization to verify if an entry belongs to a specific class.
• parse_dn(): Breaks the Distinguished Name into its components using the flext-ldif DN parser.

Sources: src/flext_tap_ldap/_utilities/_processor.py:46-120, src/flext_tap_ldap/_utilities/client_support.py:44-65

Validation and Transformation

The Validator and Transformer components provide quality assurance and data normalization.

Validation Workflow

The Validator utilizes the flext-ldif validation engine to check entries against LDAP syntax and schema rules.

Transformation Rules

The Transformer applies a hierarchy of rules to modify entries before they are emitted by the tap:

1. Attribute Mappings: Renames attributes (e.g., CN to cn).
2. Schema Mappings: Normalizes output by mapping source attributes to target keys with optional default values.
3. Value Mappings: Replaces specific attribute values based on a lookup table.
4. Add/Remove: Injects new attributes or prunes obsolete ones.

Sources: src/flext_tap_ldap/_utilities/_processor.py:372-515, tests/unit/test_ldif_processor.py:53-85

Integration with Tap Streams

The LdifStream and LdifAnalysisStream classes in the ldif_streams.py module act as consumers of the processor utility.

Data Flow for LDIF Sync

When the tap is configured with ldif_directory or ldif_files, the following sequence occurs:

Technical Summary

The LDIF Processor Utility provides a robust framework for handling directory data outside of a live LDAP server. By leveraging flext-ldif, it ensures that flat-file data is treated with the same validation and structural integrity as data retrieved via the LDAPClient. Its highly configurable Transformer allows developers to normalize heterogeneous LDIF sources into a consistent schema for downstream data consumers.

Sources: src/flext_tap_ldap/_utilities/_processor.py:28-35, src/flext_tap_ldap/client.py:65-80