Data Processing Utilities
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Data Processing Utilities in flext-tap-ldif provide the core logic for discovering, parsing, and transforming LDIF (LDAP Data Interchange Format) files into Singer-compatible record streams. These utilities are structured as a set of mixin classes unified under the FlextTapLdifUtilities facade, facilitating a clear separation of concerns between file system operations, raw data parsing, and Singer state management.
The system is designed to handle both individual files and directory-based discovery, supporting standard LDAP features such as base64-encoded values, line continuations, and attribute normalization. It integrates deeply with the flext-ldif and flext-meltano libraries to ensure robust extraction and validation of directory data.
Sources: src/flext_tap_ldif/utilities.py:22-34, src/flext_tap_ldif/tap.py:18-24
Core Architecture and Utility Facade
The utility layer uses a Multi-Resource Object (MRO) mixin pattern. The primary entry point is the FlextTapLdifUtilities.TapLdif class, which aggregates four specialized utility modules.
This diagram shows the inheritance structure where the TapLdif namespace provides a unified interface for data processing, state management, file processing, and stream handling.
Sources: src/flext_tap_ldif/utilities.py:22-34
LDIF File Processing
The Processor class handles the lifecycle of an LDIF file, from discovery on the local file system to the generation of raw records.
File Discovery
The system supports discovering files through two primary methods:
- Direct Path: Targeting a specific
file_path. - Directory Globbing: Searching a
directory_pathusing afile_pattern(defaulting to*.ldif).
Discovery includes a safety check against max_file_size_mb to prevent processing excessively large files that might exhaust system memory.
Sources: src/flext_tap_ldif/_utilities/processor.py:28-66
Processing Flow
When a file is processed, the system reads the content and utilizes an underlying LDIF API to parse the raw text into structured entries.
The flow illustrates how raw file data is sequentially converted into validated Singer records.
Sources: src/flext_tap_ldif/_utilities/processor.py:68-121
Data Parsing and Normalization
The LdifDataProcessing utility manages the complexities of the LDIF format, including line continuations and encoding.
Attribute Handling
- Normalization: Attribute names are converted to lowercase and non-alphanumeric characters are replaced with underscores. If an attribute starts with a digit, it is prefixed with
attr_. - Value Decoding: Supports standard colon (
:) for plain text, double colon (::) for Base64 encoded values, and:<for URL references. - Line Continuations: Lines starting with a space are appended to the previous attribute value.
| Feature | Description | Source |
|---|---|---|
| Normalization | Regex-based sanitization of LDAP attributes for JSON compatibility. | data_processing.py:72 |
| Base64 Decode | Automatically decodes :: values using UTF-8. | data_processing.py:98 |
| Multi-values | Converts duplicate attributes into arrays of values. | data_processing.py:27 |
Sources: src/flext_tap_ldif/_utilities/data_processing.py:17-118
Stream and Record Structure
The EntriesStream class implements the Singer stream protocol. It defines the schema for the output data and orchestrates the processor to yield records.
Singer Record Schema
Every yielded record contains the following standard fields:
| Field Name | Type | Description |
|---|---|---|
dn | String | The Distinguished Name of the entry. |
attributes | Object | Map of normalized attribute names to values or lists of values. |
object_class | Array | List of objectClass values for the entry. |
change_type | String | The type of change (defaults to modify). |
source_file | String | Path to the file the entry was extracted from. |
entry_size | Integer | Calculated size of the entry in bytes. |
Sources: src/flext_tap_ldif/_utilities/entries_stream.py:100-116, src/flext_tap_ldif/_utilities/processor.py:109-121
State Management
For incremental synchronization, the StateManagement utility tracks the progress within files.
The state management logic ensures that the tap can resume from the last processed position in a specific file by storing a position (line or entry index) and a last_updated timestamp within the Singer state object.
Sources: src/flext_tap_ldif/_utilities/state_management.py:18-84
Metadata Modeling
The LdifFile model provides a structured way to track the health and statistics of LDIF files during the processing lifecycle. It includes fields for total_lines, entry_count, processing_status, and validation_errors. A computed property ldif_file_summary provides a high-level JSON mapping of the file's processing progress and validity.
Sources: src/flext_tap_ldif/_models/file_metadata.py:14-98
Conclusion
The Data Processing Utilities form the backbone of the flext-tap-ldif integration, abstracting the difficulties of LDIF parsing into a clean, reusable set of tools. By separating file discovery, data normalization, and state tracking, the system provides a robust framework for extracting directory data into modern data warehouses via the Singer protocol.