Metadata Handling
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Metadata Handling in the flext-ldif library provides a robust framework for preserving original data characteristics, tracking conversions, and maintaining audit trails during LDIF processing. The system is designed to support "Zero Data Loss" by capturing original formatting, case sensitivity, and server-specific extensions that are typically lost during standard LDIF parsing. This metadata is stored within universal containers that accompany domain entities (Entries, ACLs, and Schema items) throughout their lifecycle.
The primary scope of this system includes managing ServerMetadata, which tracks RFC violations, conversion notes, and original strings. It also encompasses utility services for analyzing minimal differences between original and converted data, and mixins that facilitate intelligent metadata updates during server-to-server migrations.
Sources: src/flext_ldif/_models/domain_metadata.py:149-180, src/flext_ldif/_utilities/metadata.py:440-455
Core Metadata Architecture
The metadata architecture is built around a centralized ServerMetadata model. This model acts as a container for various specialized metadata objects that handle specific concerns like validation results, write options, and original formatting details.
Metadata Component Relationships
The following diagram illustrates the relationship between the primary metadata components:
Key Data Structures
| Component | Type | Description |
|---|---|---|
ServerMetadata | DynamicModel | The universal container for all server-specific and conversion metadata. |
ValidationMetadata | FrozenModel | Tracks RFC violations, errors, and warnings found during parsing or validation. |
FormatDetails | FrozenModel | Stores original DN line formatting, spacing, and encoding for round-trip preservation. |
SchemaFormatDetails | FrozenModel | Captures exhaustive details about schema definitions, including field order and quoting styles. |
DynamicMetadata | RootModel | An extensible mapping for arbitrary server-proprietary data and conversion tracking. |
| Sources: src/flext_ldif/_models/domain_metadata.py:46-147 |
Data Preservation and Round-Trip Support
A critical feature of the metadata system is the ability to restore original formatting during write operations. This is achieved by capturing specific details during the parsing phase.
Schema Formatting Analysis
The FlextLdifUtilitiesMetadata class provides methods to dissect raw schema definition strings. It extracts prefix details (e.g., case of attributeTypes), OID spacing, syntax quoting, and the exact order of fields like NAME, DESC, and SYNTAX.
Sources: src/flext_ldif/_utilities/metadata.py:104-143
Original String Storage
The original_strings attribute in ServerMetadata is used to store the "raw" representation of data before any normalization.
- dn_original: The exact DN string as it appeared in the source.
- entry_original_ldif: The complete LDIF block for an entry, including comments.
- schema_attr_uid_original: The full definition string for schema attributes.
Sources: src/flext_ldif/_models/domain_metadata.py:270-280, src/flext_ldif/_utilities/parser.py:65-75
Conversion and Migration Tracking
When entries are migrated between server types (e.g., OID to RFC), the metadata system tracks every transformation to provide an audit trail.
Conversion Metadata Pipeline
The conversion process involves analyzing the source metadata to determine necessary transformations for the target server type.
Tracking Mechanisms
- Attribute Transformations: Stores a detailed record of each attribute name or value change, including the reason for the transformation.
- Boolean Conversions: Tracks how boolean values (e.g.,
1vsTRUE) are mapped between server-specific formats (OID vs RFC). - Conversion History: A sequence of audit records containing timestamps, operations performed, and the differences between source and target states.
Sources: src/flext_ldif/_models/domain_metadata.py:192-205, src/flext_ldif/_models/domain_metadata.py:285-295, src/flext_ldif/services/conversion_metadata.py:35-57
Metadata Management Utilities
The library provides the FlextLdifUtilitiesMetadata class to handle common metadata operations.
Processing Statistics
Entry statistics are tracked via EntryStatistics, which marks whether an entry was filtered, rejected, or modified.
@staticmethod
def _apply_rejection_update(
stats: m.Ldif.EntryStatistics,
rejection_category: str,
reason: str,
) -> m.Ldif.Entry:
"""Apply rejection marking to stats."""
return stats.mark_rejected(rejection_category, reason)
Sources: src/flext_ldif/_utilities/metadata.py:82-87
Minimal Difference Analysis
The system can calculate minimal differences between original and converted strings. This is used for delta tracking and ensuring that only necessary changes are applied. The analyze_minimal_differences function generates a mapping that includes lengths and a list of specific differences.
Sources: src/flext_ldif/_utilities/metadata.py:440-459
Server-Specific Metadata: OID ACLs
The Oracle Internet Directory (OID) implementation demonstrates how metadata is used for complex, proprietary structures. OID ACLs (orclaci) utilize metadata to store:
- OidAclMetadataConfig: Captures OID-specific subject types (e.g.,
groupattr,guidattr), bind modes, and IP filters. - Permissions Mapping: Tracks the conversion of OID permissions (e.g.,
noadd,noselfwrite) into RFC-compliant flags.
Sources: src/flext_ldif/servers/_oid/acl.py:168-195, src/flext_ldif/servers/_oid/acl.py:228-255
Summary of Metadata Fields
| Field | Model | Description |
|---|---|---|
server_type | ServerMetadata | Enum identifying the source/target server (OID, OUD, RFC, etc.). |
rfc_violations | ServerMetadata | List of strings identifying specific RFC non-compliance issues. |
original_attribute_case | ServerMetadata | Map of lowercase names to their original casing (e.g., cn -> CN). |
soft_delete_markers | ServerMetadata | Tracks attributes hidden for the target server but preserved for potential reversal. |
dn_line | FormatDetails | The exact string representation of the DN line. |
| Sources: src/flext_ldif/_models/domain_metadata.py:181-268 |
Conclusion
Metadata Handling is the cornerstone of high-fidelity LDIF processing in the flext-ldif library. By decoupling standard domain data from its original formatting and server-specific context, the system allows for flexible transformations while guaranteeing that the original state can be audited or perfectly reconstructed. This multi-layered approach—combining structured models with dynamic extensions—ensures compatibility across diverse LDAP environments while maintaining strict RFC compliance where required.