File Abstraction Model
Relevant source files
The following files were used as context for generating this wiki page:
The File Abstraction Model in flext-tap-ldif provides a structured framework for representing, discovering, and processing Lightweight Data Interchange Format (LDIF) files within a Singer Tap context. It bridges the gap between raw filesystem resources and the structured data streams required by ETL pipelines, ensuring that file metadata, validation states, and processing progress are tracked accurately.
This model is primarily centered around the LdifFile and LdifStream definitions, which handle the physical properties of files and their logical representation as data sources, respectively. These components work in tandem with the Processor utility to manage the lifecycle of data extraction from discovery to record yielding.
Core Data Structures
The model uses Pydantic-based EnforcedModel classes to define the attributes and constraints of LDIF files and their corresponding streams.
LDIF File Metadata
The LdifFile model represents the physical file and its processing status. It tracks metrics such as line counts, entry counts, and validation errors to provide a comprehensive summary of the file's state.
| Field | Type | Description |
|---|---|---|
file_path | str | Path to the LDIF file on disk. |
file_size | NonNegativeInt | Size of the file in bytes. |
encoding | str | Text encoding (defaults to utf-8). |
total_lines | NonNegativeInt | Total line count in the source file. |
processing_status | str | Status (e.g., "pending"). |
last_processed_line | NonNegativeInt | Last line number successfully read. |
Sources: src/flext_tap_ldif/_models/file_metadata.py:15-68
LDIF Stream Configuration
The LdifStream model defines how a specific file is mapped to a Singer stream. It includes settings for replication methods and filtering.
| Field | Type | Description |
|---|---|---|
stream_name | str | The name of the resulting Singer stream. |
tap_stream_id | str | Unique identifier for the tap stream. |
replication_method | str | Method used (default: FULL_TABLE). |
batch_size | int | Number of records to process per batch. |
filter_object_classes | StrSequence | Optional filters for specific LDAP object classes. |
Sources: src/flext_tap_ldif/_models/file_stream.py:15-55
Component Architecture and Logic
The file abstraction logic is divided between metadata representation, filesystem discovery, and state persistence.
File Processing Flow
The Processor utility handles the discovery of LDIF files within a directory or at a specific path. It enforces size limits and pattern matching before initiating the extraction process.
Entry and Record Modeling
When a file is processed, entries are parsed into LdifEntry models. These models encapsulate the Distinguished Name (DN) and attributes of an LDAP object, providing helper methods to resolve attribute values case-insensitively.
State and Position Management
To support incremental processing and resumes, the model utilizes a state management system that tracks the "position" (usually the last processed line or byte offset) for every file.
State Resolution Logic
The StateManagement utility provides static methods to retrieve and update positions within a Singer state object. The state is structured as a mapping of file paths to their respective processing metadata.
Validation and Metrics
The model incorporates an LdifValidationResult structure to track the integrity of processed files. It maintains counts of valid vs. invalid entries and calculates a success rate and processing performance (entries per second).
- Consistency Check: The model ensures that
valid_entries + invalid_entries == total_entries. - Error Reporting: Detailed error lists and severity levels (e.g., "critical") are captured during the validation phase.
Sources: src/flext_tap_ldif/_models/record.py:15-65
Conclusion
The File Abstraction Model provides a robust, typed interface for handling LDIF data. By separating physical file metadata (LdifFile) from logical stream configuration (LdifStream) and providing specialized utilities for processing and state persistence, the system ensures that LDAP data extraction is both reliable and observable.