Data Conversion and Normalization
Relevant source files
The following files were used as context for generating this wiki page:
Data Conversion and Normalization in the flext-ldap project provides a robust set of utilities to handle the transformation of raw LDAP data into canonical formats. This system ensures that data extracted from various LDAP servers (such as OpenLDAP or Oracle Unified Directory) is unified into a consistent structure, primarily using the m.Ldif.Entry model, which allows for server-agnostic operations and reliable data synchronization.
The module focuses on three primary pillars: normalizing string representations and attribute lists, converting ldap3 search results into LDIF-compatible models, and tracking metadata related to these conversions. This layer acts as the bridge between the raw networking responses from the ldap3 library and the high-level domain models used by the FLEXT framework.
Sources: src/flext_ldap/_utilities/normalization.py, src/flext_ldap/_utilities/conversion.py, src/flext_ldap/adapters/_ldap3/result_extract.py
Normalization Utilities
Normalization logic is encapsulated in FlextLdapUtilitiesNormalization. It provides static and class methods to ensure that LDAP strings, Distinguished Names (DNs), and attribute values follow predictable patterns, especially regarding casing and encoding.
String and Collection Normalization
The system provides methods to normalize individual strings or sequences. These methods are frequently used to sanitize input for filters or to prepare data for comparison.
norm_str(value, case): Normalizes a single string to "lower" or "upper" case.norm_join(values, case): Normalizes a sequence of strings and joins them into a single space-separated string.map_str(values, case, join): Normalizes a collection and optionally joins it with a custom delimiter.
Sources: src/flext_ldap/_utilities/normalization.py:20-70, src/flext_ldap/_utilities/normalization.py:155-180
Attribute Value Conversion
LDAP attributes can return multiple types, including bytes, str, int, or lists thereof. The attr_to_str_list and ldap3_value_to_strings methods convert these into a canonical t.StrSequence (list of strings).
The diagram shows the logic flow for converting heterogeneous LDAP attribute values into a unified string sequence.
Sources: src/flext_ldap/_utilities/normalization.py:72-132, src/flext_ldap/adapters/_ldap3/result_extract.py:117-140
Data Conversion System
The FlextLdapUtilitiesConversion class extends normalization to handle full entry transformations and difference tracking. This is critical for synchronization tasks where the application must decide whether to update an existing entry.
Search Result Transformation
The method search_entry_to_ldif_entry transforms a raw dictionary from an LDAP search into a validated m.Ldif.Entry.
| Step | Action | Description |
|---|---|---|
| 1 | DN Extraction | Extracts and validates the "dn" field; fails if missing. |
| 2 | Attr Filtering | Removes the "dn" field from the attribute payload. |
| 3 | Normalization | Converts all remaining values to string sequences via ldap3_value_to_strings. |
| 4 | Model Creation | Returns a Result-wrapped m.Ldif.Entry. |
Sources: src/flext_ldap/_utilities/conversion.py:32-51, src/flext_ldap/adapters/_ldap3/result_extract.py:23-45
Tracking Differences
During conversion, the system can track changes between the original source data and the converted result using track_conversion_differences.
This sequence illustrates how the utility records changes during the conversion process to provide an audit trail for synchronization.
Sources: src/flext_ldap/_utilities/conversion.py:53-78
Metadata Models and Structures
Conversion and normalization rely on Pydantic models to maintain type safety and validation constraints.
ConversionMetadata
This model stores the state of an entry after it has been processed.
| Field | Type | Description |
|---|---|---|
source_dn | str | The DN as it existed in the source system. |
source_attributes | t.StrSequence | List of original attribute names. |
removed_attributes | t.StrSequence | Attributes dropped during transformation. |
dn_changed | bool | Flag indicating if the entry was moved/renamed. |
attribute_changes | t.StrSequence | Names of attributes whose values changed. |
Sources: src/flext_ldap/_models/ldap.py:164-180, src/flext_ldap/_utilities/conversion.py:18-30
Result Extraction Mixin
The ResultConverterExtractMixin provides logic to extract specific components from mixed input types (LDIF entries, ldap3 entries, or JSON).
extract_dn: Safely retrieves anm.Ldif.DNobject.extract_attributes: Produces a standardizedm.Ldif.Attributesmodel.extract_metadata: Filters and extractsServerMetadatasuch as the detectedserver_type.
Sources: src/flext_ldap/adapters/_ldap3/result_extract.py:18-115
Encoding and Safety
LDAP data often requires special handling for non-ASCII characters or specific transport formats like LDIF.
- Base64 Detection: The
is_base64_encodedmethod checks if a string contains characters above thec.Ldif.ASCII_THRESHOLDor starts with the LDIF base64 indicator (::). - Conditional Safety: The
when_safemethod provides a utility for returning primitive values only if specific conditions are met, preventingNonetype errors in data pipelines. - UTF-8 Handling: Most conversions utilize
UTF-8with "replace" error handling to ensure byte strings are safely converted to Python strings without crashing on invalid sequences.
Sources: src/flext_ldap/_utilities/normalization.py:134-142, src/flext_ldap/_utilities/normalization.py:182-194, src/flext_ldap/_utilities/normalization.py:110-120
Conclusion
The Data Conversion and Normalization module provides the necessary infrastructure to ensure data integrity across the flext-ldap library. By centralizing casing logic, type casting, and difference tracking, the project maintains a "Zero-Debt" baseline where LDAP entries are treated as structured models rather than raw strings, enabling complex features like multi-phase synchronization and server-agnostic attribute comparisons.