Home
Relevant source files
The following files were used as context for generating this wiki page:
flext-tap-ldif is a Singer-compliant Tap designed for extracting data from LDAP Data Interchange Format (LDIF) files. It is built using the FLEXT ecosystem, leveraging flext-core, flext-ldif, and flext-meltano to provide a typed, high-performance integration for enterprise ETL workflows. The tap supports both single-file processing and directory-based discovery, with built-in mechanisms for state management and incremental syncs.
Sources: README.md:9-11, pyproject.toml:28-33
System Architecture
The project follows the FLEXT architectural patterns, utilizing a modular structure where core logic is divided into models, utilities, and a main tap facade. It uses lazy loading for exports to optimize performance and strictly adheres to the "Universal Agent Law" for engineering conduct.
High-Level Data Flow
The flow starts with the tap discovering files (single or directory-wide), processing them through an LDIF parser, and yielding Singer-formatted records.
The diagram illustrates the sequential progression from configuration loading to the emission of Singer records and subsequent state updates.
Sources: src/flext_tap_ldif/tap.py:20-55, src/flext_tap_ldif/_utilities/processor.py:27-104
Core Components
Tap Implementation
The FlextTapLdif class serves as the entry point, inheriting from m.Meltano.SingerTapBase. It defines the configuration schema and orchestrates stream discovery.
| Feature | Description |
|---|---|
| Stream Discovery | Returns a list of available streams, primarily ldif_entries. |
| JSON Schema | Defines the structure of LDIF records, including dn, object_class, and attributes. |
| Config Validation | Uses FlextTapLdifSettings for Pydantic-based configuration validation. |
Sources: src/flext_tap_ldif/tap.py:17-69
Processor Utility
The Processor class handles file system interactions and invokes the underlying flext-ldif API for parsing.
- File Discovery: Supports finding files via
file_pathordirectory_pathusing patterns like*.ldif. It includes safety checks for file existence and maximum file size limits (defaulting to 100MB). - File Processing: Reads file content and yields records. It can operate in a "strict" mode where parsing failures trigger exceptions.
Sources: src/flext_tap_ldif/_utilities/processor.py:22-104
Data Models
LDIF File Metadata
The LdifFile model tracks the state and characteristics of files being processed.
The model includes computed fields to provide a ldif_file_summary, which calculates progress percentages based on last_processed_line and total_lines.
Sources: src/flext_tap_ldif/_models/file_metadata.py:12-97
Data Processing Logic
The tap performs several normalization steps to ensure LDIF data is compatible with JSON-based Singer targets.
- Attribute Normalization: Converts LDIF attribute names to lower-case and replaces invalid characters with underscores. If an attribute name starts with a digit, it is prefixed with
attr_. - Base64 Decoding: Automatically detects and decodes attributes using the
::notation in LDIF files. - Line Continuation: Handles LDIF multi-line values where lines starting with a space are appended to the previous attribute value.
Sources: src/flext_tap_ldif/_utilities/data_processing.py:18-106
Configuration Options
The tap is configured via a JSON object or environment variables.
| Property | Type | Default | Description |
|---|---|---|---|
file_path | string | N/A | Specific path to an LDIF file. |
directory_path | string | N/A | Directory to scan for LDIF files. |
file_pattern | string | *.ldif | Glob pattern for file discovery. |
encoding | string | utf-8 | File encoding (via c.DEFAULT_ENCODING). |
strict_parsing | boolean | True | Stop execution on parsing errors. |
max_file_size_mb | integer | 100 | Maximum allowed file size for processing. |
Sources: src/flext_tap_ldif/tap.py:23-42, src/flext_tap_ldif/_utilities/processor.py:33-35
State Management
Incremental syncs are supported through the StateManagement utility, which tracks the position (last processed line/offset) for each file.
Engineering Principles
This repository enforces strict engineering standards via the AGENTS.md directive. Key rules include:
- Zero-Tolerance: Always fix root causes; no dead code or workarounds.
- SSOT (Single Source of Truth): Maintain authoritative sources for identity and configuration.
- Fix-Forward-Only: Coordination between agents is managed through the working tree, forbidding reverts of shared work.
Sources: AGENTS.md:14-72