State Management & Bookmarks
Relevant source files
The following files were used as context for generating this wiki page:
State Management within flext-tap-ldif provides the infrastructure for incremental data synchronization from LDIF files. By tracking the processing position and metadata of individual files, the tap can resume extraction from the last successful bookmark rather than re-processing entire datasets.
The system relies on Singer-compatible state objects that map specific file paths to their last known processing offsets and update timestamps. This is critical for handling large directory exports where efficiency and idempotent processing are required.
Sources: src/flext_tap_ldif/_utilities/state_management.py:1-10, src/flext_tap_ldif/_models/file_metadata.py:10-25
State Resolution and Persistence
The core logic for interacting with the Singer state is encapsulated in the StateManagement utility. This class provides methods to resolve and update the position of a tap within a specific LDIF file.
State Structure
The state object is structured as a mapping, where the files key contains nested mappings for each individual LDIF file identified by its path.
| Field | Type | Description |
|---|---|---|
files | JsonMapping | A dictionary where keys are file paths and values are file-specific states. |
position | int | The byte offset or line position within the specific file. |
last_updated | string (ISO 8601) | The UTC timestamp of the last state update. |
Sources: src/flext_tap_ldif/_utilities/state_management.py:12-40
Logic Flow for File Position Update
When a file is processed, the tap updates the state to ensure future runs can skip already-ingested data.
The diagram shows the transformation of a global state object when a specific file's progress is recorded.
Sources: src/flext_tap_ldif/_utilities/state_management.py:53-75
File Metadata and Processing Status
Beyond simple positioning, the tap tracks comprehensive metadata through the LdifFile model. This metadata supports validation and error tracking, ensuring that state transitions are based on valid data.
Metadata Fields
| Field | Description | Source |
|---|---|---|
last_processed_line | Tracks the physical line number last reached in the LDIF file. | file_metadata.py:65 |
processing_status | Status indicator (e.g., "pending"). | file_metadata.py:61 |
entry_count | Number of directory entries successfully extracted. | file_metadata.py:52 |
is_valid_ldif | Boolean flag indicating format validity. | file_metadata.py:73 |
Sources: src/flext_tap_ldif/_models/file_metadata.py:40-75
Processing Logic
The Processor utility utilizes these metadata points to decide whether to yield records. If strict_parsing is enabled in settings, the processor will fail on parsing errors discovered during record yielding.
# From src/flext_tap_ldif/_utilities/processor.py:84-93
try:
yield from self._yield_records(file_path)
except c.Meltano.SINGER_SAFE_EXCEPTIONS:
FlextTapLdifUtilitiesProcessor.logger.exception(
"Failed to process LDIF file: %s",
str(file_path),
)
if self.settings.get("strict_parsing", True):
raise
Sources: src/flext_tap_ldif/_utilities/processor.py:84-93
Incremental Sync Sequence
The following sequence diagram illustrates how the tap interacts with the state management utility during a standard synchronization cycle.
This diagram depicts the retrieval of bookmarks at the start of a sync and the persistence of progress after record extraction.
Sources: src/flext_tap_ldif/_utilities/state_management.py:20-55, src/flext_tap_ldif/_utilities/processor.py:75-85
Validation and Consistency
The state is only as reliable as the data processed. The system uses models like LdifValidationResult to ensure that statistics (total vs. valid entries) remain consistent before state updates are finalized. For instance, the system enforces that valid_entries + invalid_entries must equal the total_entries reported in the processing metadata.
Sources: src/flext_tap_ldif/_models/record.py:85-95
Summary of State Management Implementation
The state management system in flext-tap-ldif acts as a coordination layer between the file-system-based Processor and the Singer protocol requirements. By isolating state resolution into the FlextTapLdifUtilitiesStateManagement class, the tap maintains a clean separation between data extraction logic and persistence tracking.