Singer Record Model

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_tap_ldif/_models/record.py
• src/flext_tap_ldif/_models/entry.py
• src/flext_tap_ldif/_utilities/data_processing.py
• src/flext_tap_ldif/_utilities/entries_stream.py
• src/flext_tap_ldif/_utilities/processor.py
• src/flext_tap_ldif/tap.py

The Singer Record Model in flext-tap-ldif defines the structure and transformation logic for converting Lightweight Data Interchange Format (LDIF) entries into standardized Singer-compatible records. This model ensures that hierarchical directory data is flattened and typed appropriately for downstream consumption within the Singer ecosystem.

The system utilizes Pydantic-based enforced models to validate directory entries, track source metadata (such as line numbers and file paths), and generate performance metrics during the extraction process.

Sources: src/flext_tap_ldif/_models/record.py, src/flext_tap_ldif/_models/entry.py, src/flext_tap_ldif/tap.py

Record Architecture and Schema

The record model is anchored by the LdifEntry class and the EntriesStream schema definition. A Singer record produced by this tap consists of the Distinguished Name (DN), a collection of attributes, and metadata regarding the source file location.

Core Record Schema

The schema for LDIF entries defines how directory objects are represented in JSON.

Field Name	Type	Description
dn	string	The Distinguished Name of the entry.
attributes	object	A mapping of normalized attribute names to their values.
object_class	array	List of object classes associated with the entry.
change_type	string	The type of LDIF record (e.g., 'entry').
source_file	string	Path to the source LDIF file.
line_number	integer	Starting line number of the entry in the source file.
entry_size	integer	Size of the entry in bytes.

Sources: src/flext_tap_ldif/tap.py:53-65, src/flext_tap_ldif/_utilities/entries_stream.py:83-96

Data Transformation Flow

The transformation from raw LDIF lines to a Singer Record involves parsing, attribute normalization, and model validation.

The Processor utilizes flext-ldif to parse content and then iterates through entries, validating them against the LdifEntry model before yielding them as SingerRecord objects.

Sources: src/flext_tap_ldif/_utilities/processor.py:73-108, src/flext_tap_ldif/_utilities/entries_stream.py:65-68

Entry Validation and Metadata

Directory entries are validated using the LdifEntry model. This model enforces constraints such as requiring a non-empty Distinguished Name (DN) and ensuring that objectClass attributes are correctly synchronized with the object_classes list.

LdifEntry Model Components

• DN Validation: Ensures every entry has a valid identity.
• Attribute Resolution: Provides case-insensitive access to attribute values.
• Metadata Tracking: Captures extracted_at timestamps and source_file origins.

@u.model_validator(mode="after")
def validate_ldif_entry(self) -> Self:
    """Validate LDIF entry structure."""
    if not self.dn:
        msg = "DN cannot be empty"
        raise ValueError(msg)

    # Validate object classes are in attributes
    if self.object_classes and "objectClass" not in self.attributes:
        attributes = dict(self.attributes)
        attributes["objectClass"] = self.object_classes
        self.attributes = MappingProxyType(attributes)

    return self

Sources: src/flext_tap_ldif/_models/entry.py:101-114

Attribute Normalization

Since LDIF attributes can contain characters invalid for JSON keys or specific Singer destinations, the LdifDataProcessing utility performs normalization.

1. Lowercasing: All attribute names are converted to lowercase.
2. Character Substitution: Any non-alphanumeric characters are replaced with underscores (_).
3. Digit Handling: If a normalized attribute starts with a digit, it is prefixed with attr_.

Sources: src/flext_tap_ldif/_utilities/data_processing.py:61-71

Validation Results and Metrics

The LdifValidationResult model tracks the health of the extraction process. It aggregates statistics on total, valid, and invalid entries, providing a success rate and performance metrics (entries per second).

Validation Consistency Rules

The model enforces internal consistency:

• total_entries must exactly equal the sum of valid_entries and invalid_entries.
• An object cannot be marked valid=True if validation_errors are present.

Sources: src/flext_tap_ldif/_models/record.py:15-103

Conclusion

The Singer Record Model provides a robust framework for converting LDIF data into a format suitable for modern data pipelines. By combining strict Pydantic validation with flexible attribute normalization and comprehensive metadata tracking, it ensures high data quality and traceability throughout the extraction lifecycle.