Developer Onboarding

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• AGENTS.md
• pyproject.toml
• src/flext_tap_ldif/tap.py
• src/flext_tap_ldif/_utilities/processor.py
• src/flext_tap_ldif/_models/file_metadata.py
• src/flext_tap_ldif/_models/file_stream.py
• src/flext_tap_ldif/_utilities/state_management.py

Welcome to the flext-tap-ldif project. This repository contains a Singer Tap designed for extracting data from LDIF (LDAP Data Interchange Format) files. The project is built using the FLEXT ecosystem, emphasizing typed interfaces, lazy-loading exports, and strict engineering standards.

The tap is designed to integrate with the Meltano ecosystem and adheres to the Universal Agent Law for engineering conduct. Developers are expected to follow a "Fix-Forward-Only" approach and ensure that all changes are validated with objective evidence before landing work.

Sources: README.md:1-10, AGENTS.md:10-25, pyproject.toml:25-35

Engineering Standards and Conduct

All developers must adhere to the Universal Agent Law, which provides a strict-total engineering framework. This ensures consistency and quality across multi-agent environments.

• Zero-Tolerance: Fix root causes generically and remove dead code immediately.
• Fix-Forward-Only: Never revert to past states silently; always fix the current state forward.
• SSOT (Single Source of Truth): Fail loud if identity, config, or contracts are absent.
• DI/DIP: Depend on abstractions and inject collaborators; avoid hidden globals.

Sources: AGENTS.md:27-110

Project Architecture

The project follows a modular structure where core logic is divided between models, utilities, and the tap interface. It leverages flext-core, flext-ldif, and flext-meltano as foundational dependencies.

High-Level Component Flow

The following diagram illustrates the flow from LDIF file discovery to record generation:

The tap discovers streams, which then utilize the Processor to locate LDIF files on the filesystem and parse them into Singer-compatible records.

Sources: src/flext_tap_ldif/tap.py:38-51, src/flext_tap_ldif/_utilities/processor.py:25-50

Key Components

Component	Description	File Path
FlextTapLdif	Main Singer tap class implementing stream discovery.	src/flext_tap_ldif/tap.py
Processor	Handles file discovery and LDIF content parsing via flext-ldif.	src/flext_tap_ldif/_utilities/processor.py
LdifFile	Pydantic model for LDIF file metadata and processing status.	src/flext_tap_ldif/_models/file_metadata.py
StateManagement	Utilities for resolving and updating file positions for incremental syncs.	src/flext_tap_ldif/_utilities/state_management.py

Sources: src/flext_tap_ldif/tap.py:19, src/flext_tap_ldif/_utilities/processor.py:15, src/flext_tap_ldif/_models/file_metadata.py:11, src/flext_tap_ldif/_utilities/state_management.py:11

Data Models and Configuration

The tap uses Pydantic-enforced models to validate LDIF file metadata and stream configurations.

Configuration Options

The tap's configuration is defined in FlextTapLdifSettings and exposed via config_jsonschema.

Property	Type	Default	Description
file_path	string	N/A	Specific path to an LDIF file.
directory_path	string	N/A	Directory to search for LDIF files.
file_pattern	string	*.ldif	Pattern for file discovery.
encoding	string	utf-8	File encoding.
strict_parsing	boolean	True	Whether to fail on parsing errors.
max_file_size_mb	integer	100	Maximum size of file to process.

Sources: src/flext_tap_ldif/tap.py:23-44

LDIF File Model

The LdifFile model tracks processing statistics and validation status.

# From src/flext_tap_ldif/_models/file_metadata.py:14-35
class LdifFile(m.EnforcedModel):
    file_path: Annotated[str, u.Field(..., description="Path to LDIF file")]
    file_size: Annotated[t.NonNegativeInt, u.Field(description="File size in bytes")] = 0
    encoding: Annotated[str, u.Field(description="File encoding")] = c.DEFAULT_ENCODING
    total_lines: Annotated[t.NonNegativeInt, u.Field(description="Total lines in file")] = 0
    entry_count: Annotated[t.NonNegativeInt, u.Field(description="Number of entries")] = 0
    processing_status: Annotated[str, u.Field(description="Processing status")] = "pending"

Sources: src/flext_tap_ldif/_models/file_metadata.py:14-45

Development Workflow

Dependency Management

The project uses hatchling as the build backend. Development dependencies include pytest for testing, ruff for linting, and mypy for type checking.

• Python Version: 3.13+
• Key Dependencies: flext-core, flext-ldif, flext-meltano.
• Make Verbs: Use make check, make test, and make gen for standard tasks.

Sources: pyproject.toml:15-30, README.md:38-40

Testing

Testing is structured into unit and integration tests. Protocols and types for testing are located in the tests/ directory and use composition with flext_tests.

Processing Logic

The Processor class handles the conversion of LDIF data into Singer records.

1. Discovery: Scans the provided directory_path for files matching file_pattern.
2. Validation: Ensures files do not exceed max_file_size_mb.
3. Parsing: Reads the file content and uses flext-ldif to extract entries.
4. Transformation: Maps LDIF attributes (e.g., dn, objectClass) to the standard Singer record schema.

Sources: src/flext_tap_ldif/_utilities/processor.py:25-90, src/flext_tap_ldif/_utilities/data_processing.py:15-40

Conclusion

The flext-tap-ldif project provides a robust, typed interface for LDIF data extraction within the Singer ecosystem. By adhering to the FLEXT engineering standards and utilizing the shared model/utility patterns, developers can ensure high reliability and maintainability for LDAP-related data pipelines.