File Metadata Tracking
Relevant source files
The following files were used as context for generating this wiki page:
File Metadata Tracking in the flext-tap-ldif project provides a robust framework for monitoring the state, validity, and processing progress of LDIF (LDAP Data Interchange Format) files. It ensures that the Singer tap can reliably extract data while maintaining high observability into the characteristics of the source files, including size, line counts, and format-specific attributes.
The system relies on structured Pydantic models to enforce data integrity and utilizes state management utilities to persist progress across synchronization cycles. This allows for incremental processing and detailed reporting of processing errors or validation failures encountered during execution.
Data Models for Metadata
The core of metadata tracking is the LdifFile model, which encapsulates both filesystem-level information and LDIF-specific parsing metrics.
The LdifFile Model
The LdifFile model serves as the primary data structure for tracking an individual file's lifecycle. It inherits from m.EnforcedModel, ensuring strict validation of fields such as file path and size.
The model includes computed fields to provide a ldif_file_summary, which calculates processing progress as a percentage based on the last_processed_line relative to total_lines.
Sources: src/flext_tap_ldif/_models/file_metadata.py:12-88, src/flext_tap_ldif/_models/file_metadata.py:91-118
Validation and Metrics
In addition to general metadata, the project tracks validation results and performance metrics via the LdifValidationResult model. This tracks the ratio of valid to invalid entries and records specific validation errors with their associated severity.
| Field | Type | Description |
|---|---|---|
valid | bool | Overall validation result for the file. |
validation_errors | Sequence[Mapping] | Detailed error reports including line numbers and causes. |
total_entries | int | Total number of LDAP entries found in the file. |
valid_entries | int | Count of entries that passed schema/format validation. |
validation_time | float | Time taken to validate the file in seconds. |
Sources: src/flext_tap_ldif/_models/record.py:16-56
Processing and Discovery Logic
Metadata is populated during the discovery and processing phases of the tap's lifecycle.
File Discovery
The Processor utility handles the initial identification of files. It verifies filesystem metadata, such as file existence and size, against configured limits before allowing the file to be queued for processing.
Metadata Extraction during Parsing
As the Processor reads the file content, it updates the metadata counts for entries and change records. The processing logic interacts with the flext-ldif library to parse content and identifies the source line number for every extracted record.
Sources: src/flext_tap_ldif/_utilities/processor.py:90-130
State Management and Persistence
Persistence of metadata is critical for incremental syncs, ensuring the tap does not re-process the entire file if a previous run was interrupted.
File Position Resolution
The StateManagement utility provides methods to resolve and update the "position" (usually the line number) within a specific file state. This state is stored within the standard Singer state mapping under a files key.
State Schema
The internal representation of a file's state includes:
position: The integer representing the progress through the file.last_updated: A UTC ISO-8601 timestamp of when the metadata was last persisted.
Sources: src/flext_tap_ldif/_utilities/state_management.py:53-56
Configuration and Constraints
The behavior of the metadata tracking system is influenced by runtime settings defined in FlextTapLdifSettings.
| Option | Default | Purpose |
|---|---|---|
max_file_size_mb | 100 | Prevents processing files that exceed memory/performance limits. |
strict_parsing | True | If true, processing metadata will include fatal errors if LDIF format is violated. |
encoding | utf-8 | Defines the character set used to read the file and calculate line lengths. |
Sources: src/flext_tap_ldif/settings.py:21-34
Summary
File Metadata Tracking in flext-tap-ldif serves as the operational glue between the raw filesystem and the Singer-compliant data stream. By combining filesystem attributes with LDIF-specific parsing metrics and persistent state management, the system provides developers and operators with clear visibility into the data extraction process and ensures reliability through incremental sync capabilities.