Singer Specification Compatibility
Relevant source files
The following files were used as context for generating this wiki page:
The flext-tap-ldif project is a Singer-compliant tap designed to extract data from LDIF (LDAP Data Interchange Format) files and output it according to the Singer specification. It leverages the FLEXT ecosystem, specifically flext-meltano and flext-ldif, to provide a typed and structured approach to data extraction. The tap acts as a bridge between flat or hierarchical LDIF files and the standardized JSON-based Singer stream format used in ELT (Extract, Load, Transform) pipelines.
Sources: README.md:1-10, src/flext_tap_ldif/tap.py:1-15, src/flext_tap_ldif/api.py:15-30
Architecture and Core Components
The tap's architecture is built upon the Meltano.SingerTapBase class, which ensures compliance with standard Singer operations such as discovery and record extraction. The system is divided into configuration management, stream discovery, and data processing layers.
Component Overview
| Component | Class/Module | Description |
|---|---|---|
| Tap Engine | FlextTapLdif | The primary entry point that manages configuration and stream discovery. |
| Stream Manager | EntriesStream | Handles the lifecycle of the ldif_entries stream, including schema generation. |
| File Processor | Processor | Orchestrates file discovery and uses flext-ldif for actual parsing. |
| State Handler | StateManagement | Manages incremental sync positions using Singer state JSON. |
Sources: src/flext_tap_ldif/tap.py:18-25, src/flext_tap_ldif/_utilities/entries_stream.py:18-25, src/flext_tap_ldif/_utilities/processor.py:20-25, src/flext_tap_ldif/_utilities/state_management.py:12-15
Data Flow Diagram
The following diagram illustrates the flow from LDIF source files to Singer-compliant record output.
The diagram shows how the Processor discovers files based on user configuration, parses them into intermediate objects, and how EntriesStream converts these into SingerRecord objects for final output.
Sources: src/flext_tap_ldif/_utilities/processor.py:31-70, src/flext_tap_ldif/_utilities/entries_stream.py:38-80
Singer Stream Configuration
The tap implements the ldif_entries stream, which follows the Singer specification for JSON Schema representation and record structure.
Schema Definition
The stream schema is defined dynamically but adheres to a fixed structure for LDIF data.
| Field Name | Type | Description |
|---|---|---|
dn | string | Distinguished Name of the LDAP entry. |
object_class | array | List of LDAP object classes. |
attributes | object | Dictionary containing LDAP attributes and values. |
change_type | string | The LDIF change type (e.g., add, modify). |
source_file | string | Path to the originating file. |
line_number | integer | Starting line number in the source file. |
entry_size | integer | Size of the entry in bytes. |
Sources: src/flext_tap_ldif/tap.py:53-70, src/flext_tap_ldif/_utilities/entries_stream.py:102-118
Replication Methods
The tap supports the following Singer replication methods:
- FULL_TABLE: Extracts all entries from discovered files in every run.
- Incremental (Position-based): While the stream defaults to
FULL_TABLE, the architecture includesStateManagementutilities to resolve and update file positions, enabling incremental syncs based on file-level state.
Sources: src/flext_tap_ldif/_models/file_stream.py:36-40, src/flext_tap_ldif/_utilities/state_management.py:18-35
Configuration and Compatibility
The tap uses a FlextTapLdifSettings class to define the expected configuration, which is compatible with both standard config.json files and environment variables.
Key Configuration Properties
config_jsonschema: ClassVar[t.JsonMapping] = {
"type": "object",
"properties": {
"file_path": {"type": "string"},
"directory_path": {"type": "string"},
"file_pattern": {"type": "string", "default": "*.ldif"},
"encoding": {"type": "string", "default": "utf-8"},
"strict_parsing": {"type": "boolean", "default": True},
"max_file_size_mb": {"type": "integer", "default": 100},
},
}
Sources: src/flext_tap_ldif/tap.py:22-40
Stream Discovery Sequence
The sequence diagram below describes the interaction between the Tap and the Stream components during discovery and data extraction.
State Management
Compatibility with the Singer State specification is handled through the StateManagement utility. This allows the tap to track progress across multiple files, ensuring that subsequent runs can resume from the last processed position.
- resolve_file_state: Retrieves the specific state for a given file path from the global state object.
- update_file_position: Updates the
positionandlast_updatedtimestamp for a file in the state. - State Structure: The state is stored as a JSON mapping with a
fileskey containing individual file paths and their associated metadata.
Sources: src/flext_tap_ldif/_utilities/state_management.py:20-75
Error Handling and Strictness
To ensure robustness in production environments, the tap provides a strict_parsing configuration.
- Strict Mode: If
True, any failure in file discovery or LDIF parsing will raise aRuntimeErroror the original exception, halting the Singer process. - Graceful Mode: If
False, the tap logs errors and skips problematic files or entries, allowing the pipeline to continue with valid data.
Sources: src/flext_tap_ldif/_utilities/entries_stream.py:65-98, src/flext_tap_ldif/_utilities/processor.py:86-88
Conclusion
The flext-tap-ldif implementation provides a highly structured, typed, and FLEXT-integrated approach to the Singer specification. By utilizing the Meltano.SingerTapBase and SingerStreamBase abstractions, it ensures seamless integration with tools like Meltano while providing specialized LDIF processing capabilities such as file discovery, state tracking, and flexible error management.