E2E Integration Testing

Relevant source files

The following files were used as context for generating this wiki page:

• tests/e2e/init.py
• tests/e2e/test_integration.py
• tests/conftest.py
• pyproject.toml
• src/flext_tap_ldap/_utilities/_processor.py
• tests/unit/test_ldif_processor.py
• tests/models.py

E2E (End-to-End) Integration Testing for flext-tap-ldap ensures the seamless operation of the Singer tap when interacting with LDAP directory services. The testing suite focuses on validating the CLI surface, discovery mechanisms, synchronization modes, and incremental state management. These tests are designed to execute against mocked or real LDAP environments to verify that data extraction meets the Singer specification.

The integration tests are categorized under specific pytest markers including e2e, integration, and docker, as defined in the project configuration. These tests verify the transition of data from LDAP sources through the flext_tap_ldap.tap.CLI_COMMAND into standard JSON output streams.

Sources: tests/e2e/test_integration.py:1-15, pyproject.toml:355-365

Test Architecture and Components

The integration testing framework utilizes pytest fixtures and the click.testing.CliRunner to simulate command-line interactions with the tap. It mocks the underlying LDAPClient to control directory service responses and verify the tap's logic without requiring a live external LDAP server for every test case.

Key Testing Components

Component	Description	Source
CliRunner	Simulates command-line execution of the tap-ldap entry point.	tests/e2e/test_integration.py:53-56
mock_ldap_config	Provides standard LDAP connection parameters (host, port, base_dn, bind_dn).	tests/e2e/test_integration.py:58-67
sample_catalog	Defines the stream schema and metadata used during sync testing.	tests/e2e/test_integration.py:69-79
sample_state	Represents the replication state (bookmarks) for incremental sync tests.	tests/e2e/test_integration.py:81-84

Integration Flow Diagram

The following diagram illustrates the flow of an integration test execution from the CLI runner through the mocked LDAP client.

Discovery Mode Testing

Discovery mode tests verify that the tap can correctly identify available LDAP streams and their schemas. The tests check for the presence of standard streams like users, groups, organizational_units, and schema, as well as support for custom_streams defined in the configuration.

Custom Stream Discovery

Testing verifies that user-defined streams with specific search_filter and primary_keys are correctly included in the generated catalog. This ensures that the dynamic stream generation logic functions as expected.

settings: dict[str, t.JsonValue] = {
    "ldap_host": "test.ldap.com",
    "base_dn": "dc=test,dc=com",
    "custom_streams": [
        {
            "name": "service_accounts",
            "search_filter": "(&(object_class=account)(uid=svc-*))",
            "primary_keys": ["dn"],
            "schema": {
                "properties": {
                    "dn": {"type": "string"},
                    "uid": {"type": "string"},
                },
            },
        },
    ],
}

Sources: tests/e2e/test_integration.py:168-185, tests/models.py:100-112

Synchronization and Incremental Logic

Sync mode tests validate the full lifecycle of data extraction. This includes verifying that LDAP attributes are correctly mapped to Singer record fields and that incremental syncs apply the correct filters based on stored state bookmarks.

Incremental Sync Validation

The test suite validates that when a state file is provided, the tap includes time-based filters (e.g., modifyTimestamp>=) in the LDAP search request. This prevents full table scans on subsequent runs.

LDIF and File-Based Testing

Beyond direct LDAP protocol testing, the integration suite covers LDIF (LDAP Data Interchange Format) processing. This involves testing the Processor, Validator, and Transformer components that handle flat-file LDAP data.

LDIF Processing Workflow

Tests ensure that the Processor can traverse directories, identify .ldif files, and handle character encoding issues (such as UnicodeDecodeError).

• File Traversal: Verifies that nested directories are searched for matching file patterns.
• Transformation Rules: Validates that attribute_mappings, attribute_value_mappings, and add_attributes are applied during the extraction process.
• Validation: Uses real LDIF validation logic to detect invalid objectClass assignments or structural errors.

Sources: tests/unit/test_ldif_processor.py:27-58, tests/unit/test_ldif_processor.py:60-78, src/flext_tap_ldap/_utilities/_processor.py:350-380

Testing Infrastructure and Environment

The testing environment is configured via pyproject.toml and tests/conftest.py, providing a standardized set of constants and models.

Feature	Description	File Path
Marker Definitions	Defines categories like slow, resilience, and stress for targeted test runs.	pyproject.toml:355-365
Connection Fixtures	Provides ldap_connection_config with default search timeouts and page sizes.	tests/conftest.py:20-31
Data Models	TestLdapEntry and TestLdapSearch entities define the structure of test data.	tests/models.py:114-131
Lazy Loading	The tests/e2e/__init__.py file uses install_lazy_exports to optimize test suite startup.	tests/e2e/init.py:8-23

Sources: tests/conftest.py:20-31, tests/models.py:114-131, pyproject.toml:355-365

Conclusion

E2E Integration Testing in flext-tap-ldap provides a robust verification layer that ensures the tap adheres to both the Singer specification and LDAP protocol requirements. By combining CLI simulation, LDAP mocking, and LDIF transformation testing, the suite guarantees that data flows correctly from complex directory structures into reliable JSON-based data pipelines.