Tap Class Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_tap_ldif/tap.py
• src/flext_tap_ldif/api.py
• src/flext_tap_ldif/constants.py
• src/flext_tap_ldif/models.py
• src/flext_tap_ldif/_utilities/processor.py
• src/flext_tap_ldif/_models/file_metadata.py

The Tap Class Architecture of flext-tap-ldif provides a structured Singer-compliant interface for extracting data from Lightweight Data Interchange Format (LDIF) files. It leverages the FLEXT ecosystem, specifically inheriting from SingerTapBase to integrate with the Meltano and Singer SDK frameworks. The architecture is designed to handle file discovery, schema enforcement, and record streaming with a focus on enterprise-grade ETL (Extract, Transform, Load) requirements.

The core implementation resides in the FlextTapLdif class, which serves as the primary entry point for the Singer tap. It coordinates configuration validation, stream discovery, and the invocation of specialized utilities for LDIF parsing and data processing.

Core Component Structure

The architecture follows a modular pattern where the Tap class acts as a thin facade over domain-specific models and utilities. It uses a Mixin-based approach for constants and models, ensuring strict typing through Pydantic and Singer SDK adapters.

Tap Inheritance and Composition

The FlextTapLdif class inherits from m.Meltano.SingerTapBase, which provides the necessary boilerplate for Singer tap operations. The service layer is managed by FlextTapLdifService, which orchestrates the creation of tap instances and their adaptation to the Singer SDK runtime.

Configuration and Settings

The Tap architecture defines a strict JSON schema for its configuration, ensuring that all input parameters (such as file paths, filters, and encoding) are validated before processing begins.

Configuration Schema

The config_jsonschema defines the parameters used to control the extraction behavior, including file discovery patterns and LDIF-specific filtering logic.

Parameter	Type	Default	Description
file_path	string	N/A	Specific path to an LDIF file.
directory_path	string	N/A	Directory to scan for LDIF files.
file_pattern	string	*.ldif	Pattern for matching files in a directory.
encoding	string	utf-8	Character encoding of the LDIF files.
strict_parsing	boolean	True	Whether to fail on parsing errors.
max_file_size_mb	integer	100	Maximum allowed size per file.

Sources: src/flext_tap_ldif/tap.py:23-41, src/flext_tap_ldif/constants.py:34-37

Stream Discovery and Data Flow

The Tap implements discover_streams() to initialize the data extraction pipelines. The primary stream produced is the ldif_entries stream, which follows a specific schema representing LDIF directory entries.

Record Data Flow

The data flow starts with file discovery in the Processor utility, followed by parsing through flext-ldif, and finally yielding Singer-formatted records.

Entry Schema Definition

The internal method _get_ldif_entries_schema() defines the structure of the data emitted to the Singer target:

• dn: The Distinguished Name of the entry.
• object_class: An array of object classes associated with the entry.
• attributes: A JSON object containing all parsed LDIF attributes.
• source_file: The path to the file from which the entry was extracted.
• line_number: The specific line in the source file where the entry began.

Sources: src/flext_tap_ldif/tap.py:52-68, src/flext_tap_ldif/constants.py:61-71

Internal Utilities and Processing

The Tap relies on the Processor class within the _utilities namespace to handle the heavy lifting of file I/O and parsing.

File Processing Logic

The Processor.process_file method manages the lifecycle of reading an LDIF file. It handles encoding resolution and manages "Singer-safe" exceptions to ensure the tap does not crash unexpectedly unless strict_parsing is enabled.

# From src/flext_tap_ldif/_utilities/processor.py:65-80
def process_file(self, file_path: Path) -> Generator[t.JsonMapping]:
    """Process a single LDIF file and yield records using flext-ldif."""
    FlextTapLdifUtilitiesProcessor.logger.info("Processing LDIF file: %s", str(file_path))
    try:
        yield from self._yield_records(file_path)
    except c.Meltano.SINGER_SAFE_EXCEPTIONS:
        FlextTapLdifUtilitiesProcessor.logger.exception("Failed to process LDIF file: %s", str(file_path))
        if self.settings.get("strict_parsing", True):
            raise

Sources: src/flext_tap_ldif/_utilities/processor.py:65-80

File Metadata Tracking

The architecture includes a metadata model (LdifFile) to track the state and health of files being processed. This includes tracking total lines, entry counts, and validation status.

Field	Type	Description
file_size	NonNegativeInt	Size of the file in bytes.
processing_status	string	Current status (e.g., pending, processed).
last_processed_line	NonNegativeInt	The line number where processing last stopped.
is_valid_ldif	boolean	Boolean flag for format validity.

Sources: src/flext_tap_ldif/_models/file_metadata.py:22-68

Conclusion

The Tap Class Architecture of flext-tap-ldif provides a robust, typed, and framework-aligned method for extracting LDIF data. By separating concerns between the Singer-compliant FlextTapLdif facade, Pydantic-driven models like LdifFile and LdifEntry, and specialized Processor utilities, the system ensures maintainability and scalability for LDAP-related data integration tasks.