Batch Processing Logic
Relevant source files
The following files were used as context for generating this wiki page:
Batch processing in the FLEXT Tap LDIF project facilitates the structured extraction, transformation, and validation of LDAP Data Interchange Format (LDIF) files into Singer-compatible records. The logic encompasses file discovery, parsing of individual entries, attribute normalization, and state-tracked processing to ensure reliability and performance.
The system is designed to handle large LDIF files by yielding records through generators, managing file positions for incremental syncs, and enforcing schema validation via Pydantic models.
Sources: src/flext_tap_ldif/_utilities/processor.py:11-16, src/flext_tap_ldif/_models/file_metadata.py:11-15
File Discovery and Selection
The processing cycle begins with the Processor.discover_files method. This component identifies target files based on provided paths or directory patterns while enforcing size constraints to prevent memory exhaustion.
Discovery Parameters
The discovery logic evaluates the following criteria:
- File Path vs. Directory: Can target a specific file or scan a directory using a glob pattern (defaulting to
*.ldif). - Size Constraints: A configurable
max_file_size_mb(default 100MB) is applied to skip oversized files. - Existence Validation: Files and directories are verified for existence before inclusion.
The diagram above illustrates the decision logic for identifying LDIF files for processing.
Sources: src/flext_tap_ldif/_utilities/processor.py:27-64
Data Extraction and Parsing
Record extraction is handled by Processor.process_file, which utilizes the flext-ldif infrastructure to parse raw file content into structured entries.
Record Yielding Pipeline
- File Reading: Content is read using a configurable encoding (defaulting to UTF-8).
- LDIF Parsing: The
_api.parse_ldifmethod decomposes the content into raw entries. - Schema Mapping: Each entry is mapped to a standard schema including Distinguished Name (DN), Attributes, and metadata.
- Field Extraction:
dn: The entry's unique identifier.attributes: A dictionary of LDIF attributes.objectClass: Specifically extracted for easy filtering.source_file: Path of the origin file for lineage.
Sources: src/flext_tap_ldif/_utilities/processor.py:66-130
Attribute Normalization
During transformation, attribute names are normalized to comply with JSON standards. This includes lowercase conversion, replacing non-alphanumeric characters with underscores, and prefixing names that start with digits.
Sources: src/flext_tap_ldif/_utilities/data_processing.py:84-93
| Normalization Rule | Input Example | Output Example |
|---|---|---|
| Lowercase | GivenName | givenname |
| Special Characters | ms-Exch-User | ms_exch_user |
| Digit Prefix | 2ndAddress | attr_2ndaddress |
Sources: src/flext_tap_ldif/_utilities/data_processing.py:84-93
State Management and Incremental Processing
To support interrupted or incremental syncs, the tap tracks the processing position within files using Singer state.
State Resolution
The StateManagement utility resolves the "position" (typically a line number or record count) for a specific file path from the global state object.
Sources: src/flext_tap_ldif/_utilities/state_management.py:15-30
State Updates
As batches of records are processed, the state is updated with the latest position and a last_updated UTC timestamp.
Sources: src/flext_tap_ldif/_utilities/state_management.py:46-60
The sequence diagram shows how processing positions are retrieved and updated to ensure resume capability.
Sources: src/flext_tap_ldif/_utilities/state_management.py:15-60
Validation and Metrics
Processing logic is reinforced by strict data models that track performance and validity.
Validation Result Model
The LdifValidationResult model provides a summary of the batch operation:
- Success Rate: Calculated as
valid_entries / total_entries. - Severity Tracking: Identifies if any "critical" errors occurred during the batch.
- Performance Metrics: Tracks
validation_timeandentries_per_second.
Sources: src/flext_tap_ldif/_models/record.py:15-104
File Metadata Model
The LdifFile model tracks the physical state of the file being processed, including total_lines, entry_count, and processing_status.
Sources: src/flext_tap_ldif/_models/file_metadata.py:15-80
Summary of Batch Logic
The batch processing system transforms unstructured LDIF data into high-quality Singer records by combining file-system discovery, robust parsing through flext-ldif, and stateful management. By utilizing Pydantic models for validation and performance tracking, the system provides transparent metrics on processing health and data quality.
Sources: src/flext_tap_ldif/_utilities/processor.py:11-130, src/flext_tap_ldif/models.py:18-35