Raw Test Data Bootstrapping

Relevant source files

The following files were used as context for generating this wiki page:

• tests/e2e/sql/01-create-raw-tables.sql
• tests/e2e/sql/02-insert-test-data.sql
• tests/test_macros.sql
• tests/generic/test_data_quality.sql
• src/flext_dbt_ldap/services/client.py

Raw Test Data Bootstrapping is the process of initializing a local or test database environment with a schema and representative data mimicking the output of a Singer-compliant LDAP extraction (e.g., via tap-ldap). This system ensures that dbt models and macros can be validated against a known, static dataset that includes complex LDAP structures such as Distinguished Names (DNs), multi-valued attributes (arrays), and hierarchical organizational units.

The bootstrapping process typically involves three phases: schema creation for the raw_ldap namespace, insertion of mock LDAP entries into relational tables, and the application of dbt macros to validate or transform this data into staging formats.
Sources: tests/e2e/sql/01-create-raw-tables.sql, tests/e2e/sql/02-insert-test-data.sql

Schema Architecture

The bootstrapping logic targets a specific schema named raw_ldap. This schema is designed to store data as if it were extracted by a Singer tap, including standard metadata columns prefixed with _sdc.

Schema Structure and Objects

The following ER diagram illustrates the primary tables created during the bootstrapping phase.

The diagram represents the relational mapping of LDAP objects where DN serves as the primary key.
Sources: tests/e2e/sql/01-create-raw-tables.sql:4-72

Raw Table Definitions

The bootstrapping scripts define three core tables to represent the LDAP directory structure:

Table Name	Primary Purpose	Key Fields
raw_ldap.users	Stores individual identity records.	dn, uid, mail, member_of
raw_ldap.groups	Stores group memberships and descriptions.	dn, cn, member, member_uid
raw_ldap.organizational_units	Represents the directory hierarchy.	dn, ou, business_category

Sources: tests/e2e/sql/01-create-raw-tables.sql:4-72

Data Initialization Logic

Bootstrapping proceeds by inserting specific test cases that exercise different aspects of the dbt transformation logic, such as inactive users, service accounts, and nested organizational units.

Test User Personas

The test data includes diverse user types to verify filtering and transformation rules:

• Active Employees: Standard users like john.doe with complete attributes (mail, department, title).
• Inactive Users: Records like charlie.brown marked as inactive to test status mapping.
• Service Accounts: Accounts like svc-app1 which lack standard person attributes (sn, given_name) but possess specific objectClass values like account.

Sources: tests/e2e/sql/02-insert-test-data.sql:6-57

Group and Membership Mapping

Test groups are populated with arrays of DNs and UIDs to validate normalize_array_field and count_group_members macros.

INSERT INTO raw_ldap.groups (...) VALUES
(
    ...,
    'cn=engineering-team,ou=groups,dc=test,dc=com', 'engineering-team', 
    ARRAY['uid=john.doe,ou=users,dc=test,dc=com', 'uid=bob.wilson,ou=users,dc=test,dc=com'],
    ARRAY['john.doe', 'bob.wilson'],
    ARRAY['groupOfNames', 'top'],
    ...
);

Sources: tests/e2e/sql/02-insert-test-data.sql:68-76

Macro and Quality Validation

Once the raw data is bootstrapped, the project uses dbt macros and generic tests to ensure the data adheres to LDAP standards.

Core Macro Testing

The bootstrapping suite includes specialized tests for LDAP-specific macros that handle string parsing and timestamp conversion.

Macro	Test Focus
parse_dn	Validates extraction of RDN, RDN value, and parent DN.
extract_ou_from_dn	Ensures OU strings are correctly pulled from complex DN paths.
ldap_timestamp_to_timestamp	Converts LDAP generalized time (e.g., 20240101120000Z) to SQL timestamps.
validate_dn_format	Regex-based validation of DN string structures.

Sources: tests/test_macros.sql:6-107

Data Quality Gates

The bootstrapped data is passed through quality gates to ensure referential integrity and schema compliance before being transformed into downstream dimensions.

The flow shows the sequence of data quality checks applied to raw test data.
Sources: tests/generic/test_data_quality.sql:5-52

Implementation in Workflows

The FlextDbtLdapClientMixin provides the programmatic interface for these bootstrapping concepts, specifically mapping raw LDAP entries to the schema structures defined in the SQL scripts.

• Attribute Mapping: The service uses _map_entry_attributes to convert LDAP search results into a flat mapping suitable for SQL insertion.
• Schema Matching: The _matches_schema function determines which table (users, groups, or org_units) a bootstrapped entry belongs to based on its objectClass.
• Preparation: _prepare_ldap_data_for_dbt organizes extracted entries into a dictionary keyed by the target table names defined in the bootstrapping schema.

Sources: src/flext_dbt_ldap/services/client.py:149-195

Summary

Raw Test Data Bootstrapping provides a foundational environment for testing LDAP data transformations without requiring a live directory server. By utilizing SQL scripts to define a Singer-compatible schema and populate it with diverse test cases, the project ensures that dbt macros and models can be rigorously validated for edge cases like missing attributes, complex DN hierarchies, and array-based group memberships.