Search Definitions & Filters
Relevant source files
The following files were used as context for generating this wiki page:
Search Definitions & Filters constitute the core extraction logic for the flext-dbt-ldap project. This module defines the LDAP filters and attribute sets required to identify and retrieve specific entity types—such as users, groups, and memberships—from a directory service for subsequent transformation into a data warehouse via dbt.
The system utilizes standardized LDAP filter strings and constant attribute sequences to ensure consistency across the data pipeline. It supports both full directory extraction and incremental synchronization by dynamically constructing filters based on sync state and timestamps.
Sources: src/flext_dbt_ldap/_constants/search.py:1-29, src/flext_dbt_ldap/services/sync.py:73-86
Core Search Constants
The project centralizes LDAP search definitions within a dedicated constant class. This ensures that the same filters used for extracting raw data are available for validation and modeling phases.
Filter Definitions
Filters are defined as LDAP-compliant strings targeting specific objectClass values.
| Constant | Filter String | Targeted Entity |
|---|---|---|
FILTER_USER | (objectClass=person) | Individual user accounts |
FILTER_GROUP | (objectClass=group) | Security or distribution groups |
FILTER_MEMBERSHIP | `( | (objectClass=person)(objectClass=group))` |
Sources: src/flext_dbt_ldap/_constants/search.py:17-21
Attribute Selection
To optimize performance and ensure schema compatibility, specific attribute sets are requested during the search operation.
- Group Attributes:
cn,description,member,groupType. - Membership Attributes:
cn,member,memberOf,uniqueMember. - User Attributes (Standard):
uid,cn,mail,displayName,department,manager.
Sources: src/flext_dbt_ldap/_constants/search.py:23-26, src/flext_dbt_ldap/services/sync.py:136-143
Search Logic and Data Flow
The search process is managed by service mixins that interface with the flext-ldap API. The FlextDbtLdapClientMixin provides the low-level search execution, while FlextDbtLdapSyncMixin handles high-level workflow coordination.
Search Execution Flow
The following diagram illustrates how a search request is processed from the synchronization service through the LDAP client.
The search uses base_dn, search_filter, and a specific list of attributes to query the directory service. If the operation fails, it returns a failure result instead of throwing an unhandled exception.
Sources: src/flext_dbt_ldap/services/client.py:46-77, src/flext_dbt_ldap/services/sync.py:136-151
Incremental Search Construction
A critical feature of the search module is the ability to perform incremental extractions using the modifyTimestamp attribute. This minimizes data transfer by only retrieving records modified since the last successful sync.
Incremental Filter Logic
When an incremental sync is requested, the system retrieves a stored bookmark (timestamp) and wraps the base filter in a logical AND condition:
def _build_incremental_filter(self, base_filter: str, bookmark: str | None) -> str:
if bookmark is None:
return base_filter
return f"(&{base_filter}(modifyTimestamp>={bookmark}))"
Sources: src/flext_dbt_ldap/services/sync.py:192-195
Sync State Management
The system persists sync state in a JSON file, usually named .flext_dbt_ldap_sync_state.json. This file stores the last successful modifyTimestamp for various sync keys like users or groups.
| Sync Key | Description | Derived From |
|---|---|---|
users | Last sync for user entities | c.DbtLdap.USERS |
groups | Last sync for group entities | "groups" |
Sources: src/flext_dbt_ldap/services/sync.py:82-96, src/flext_dbt_ldap/services/sync.py:214-216
Integration with dbt Pipelines
Search definitions are the entry point for the "Full Pipeline," which orchestrates extraction, validation, and dbt transformation.
The run_full_pipeline method uses the results of the search to feed the validation metrics and subsequently the dbt model execution.
Sources: src/flext_dbt_ldap/services/client.py:86-114, src/flext_dbt_ldap/protocols.py:61-68
Performance and Monitoring
Search operations are monitored through performance metrics and validation checkpoints.
- Validation: The search results are checked against a
min_quality_threshold. Metrics includetotal_entries,valid_dns, and aquality_score. - Performance: The
PerformanceAnalysismodel tracksexecution_timeandrows_processedfor each search and transformation cycle. - Port Defaults: LDAPS connections default to port
636.
Sources: src/flext_dbt_ldap/services/client.py:151-177, src/flext_dbt_ldap/_models/results.py:16-30, src/flext_dbt_ldap/_constants/search.py:28
Summary
The Search Definitions & Filters module provides a robust framework for targeted LDAP data extraction. By combining static filter constants with dynamic incremental logic, it enables efficient data synchronization into the dbt ecosystem. The tight coupling between search filters and project-specific attribute sets ensures that the resulting data warehouse models are populated with all required identity and membership information.