LDIF Parsing Engine
Relevant source files
The following files were used as context for generating this wiki page:
The LDIF Parsing Engine is a core component of the flext-tap-ldif project designed to extract, validate, and transform LDAP Data Interchange Format (LDIF) data into Singer-compatible records. It leverages the flext-ldif infrastructure to handle complex parsing logic while providing specialized utilities for attribute normalization, line continuation handling, and file discovery.
Sources: src/flext_tap_ldif/_utilities/processor.py:16-18, src/flext_tap_ldif/tap.py:19-21
Core Architecture and Data Flow
The engine operates through a tiered architecture where the Processor class orchestrates file discovery and record generation, delegating specific data transformation tasks to the LdifDataProcessing utility and validation to the LdifEntry model.
Process Flow
- File Discovery: Locates
.ldiffiles based on user-defined directory paths or specific file paths. - Streaming Read: Opens files using specified encodings and processes content into raw entries.
- Normalization: Converts raw LDIF attributes into JSON-safe keys.
- Model Validation: Validates the structure and mandatory fields (like DN) using Pydantic models.
- Singer Mapping: Yields standardized JSON mappings for the Singer stream.
The diagram shows the sequence from stream initialization to the final emission of a Singer record.
Sources: src/flext_tap_ldif/_utilities/entries_stream.py:44-93, src/flext_tap_ldif/_utilities/processor.py:65-98
Components and Logic
LDIF Processor (Processor)
The Processor class acts as the primary controller. It manages the lifecycle of parsing a file, including handling character encoding and managing flext-ldif API calls.
- discover_files: Validates path existence and enforces
max_file_size_mbconstraints. - process_file: Iterates through parsed entries. If
strict_parsingis enabled, it raises exceptions on parse failures; otherwise, it logs errors and continues.
Sources: src/flext_tap_ldif/_utilities/processor.py:22-63, src/flext_tap_ldif/_utilities/entries_stream.py:82-95
Data Processing Utilities (LdifDataProcessing)
This utility class handles the low-level string manipulation required for the LDIF format, such as line continuations and Base64 decoding.
| Function | Responsibility |
|---|---|
build_record_from_lines | Aggregates multi-line LDIF entries and handles line continuation (leading spaces). |
parse_ldif_line | Splits lines into key-value pairs, handling standard values (:), Base64 values (::), and URL references (:<). |
normalize_ldif_attribute_name | RegEx-based normalization of attribute names to ensure JSON compatibility (e.g., lowercasing and replacing non-alphanumeric chars with underscores). |
Sources: src/flext_tap_ldif/_utilities/data_processing.py:16-103, src/flext_tap_ldif/constants.py:32-32
Entry Data Model (LdifEntry)
The engine uses an enforced model to ensure data integrity during the parsing process.
# From src/flext_tap_ldif/_models/entry.py
dn: Annotated[str, u.Field(..., description="Distinguished Name")]
attributes: Annotated[t.MappingKV[str, t.StrSequence], u.Field(...)]
object_classes: Annotated[t.StrSequence, u.Field(...)] = u.Field(default_factory=tuple)
Sources: src/flext_tap_ldif/_models/entry.py:38-51
Configuration and Schema
The parsing engine behavior is controlled via the Tap configuration.
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
encoding | string | utf-8 | File character encoding. |
strict_parsing | boolean | True | Whether to fail the entire process on a parsing error. |
max_file_size_mb | integer | 100 | Maximum size of an LDIF file to process. |
file_pattern | string | *.ldif | Glob pattern for file discovery. |
Sources: src/flext_tap_ldif/tap.py:26-44, src/flext_tap_ldif/constants.py:35-38
Output Schema Fields
The engine maps parsed LDIF data to the following standardized fields in the ldif_entries stream:
| Field | Type | Description |
|---|---|---|
dn | string | Distinguished Name of the entry. |
object_class | array | List of object classes associated with the entry. |
attributes | object | Dictionary of normalized attribute names and their values. |
source_file | string | Path to the source LDIF file. |
line_number | integer | The starting line number in the source file. |
Sources: src/flext_tap_ldif/tap.py:59-71, src/flext_tap_ldif/_utilities/processor.py:91-108
Summary
The LDIF Parsing Engine provides a robust mechanism for ingesting LDIF data. By combining strict schema validation via Pydantic models with flexible string processing utilities, it ensures that LDAP data is accurately represented in the Singer ecosystem while maintaining high performance through streaming file discovery and processing.
Sources: src/flext_tap_ldif/_utilities/processor.py:65-75, src/flext_tap_ldif/_models/entry.py:100-112