Public API Reference
Relevant source files
The following files were used as context for generating this wiki page:
The flext-dbt-ldap Public API provides a structured interface for managing LDAP data transformations within a dbt (data build tool) ecosystem. It facilitates the extraction of LDAP entries, validation of data quality, and execution of dbt models to synchronize and transform directory data into a structured data warehouse format.
The architecture is built upon the FLEXT ecosystem, extending core protocols from flext-ldap and flext-meltano to ensure consistent data handling and pipeline execution. The API surface is primarily exposed through service facades and mixins that handle stateful client operations and synchronization workflows.
Sources: src/flext_dbt_ldap/init.py:42-79, src/flext_dbt_ldap/protocols.py:15-18
Core Service Architecture
The API uses a mixin-based architecture to provide specialized functionality for LDAP interactions and warehouse synchronization. The main service facades inherit from FlextDbtLdapServiceBase and integrate specialized logic for client communication and data lifecycle management.
The diagram above shows the inheritance structure of the service mixins used to build the public API surface.
Sources: src/flext_dbt_ldap/services/client.py:17-19, src/flext_dbt_ldap/services/sync.py:15-17
Data Pipeline Operations
The API defines a comprehensive pipeline for moving data from an LDAP directory to a dbt-managed data warehouse. This includes extraction, validation, and transformation steps.
Pipeline Flow
The standard operation flow involves extracting raw entries from the LDAP server, validating them against configured quality thresholds, and then triggering dbt transformations.
Key Pipeline Methods
| Method | Description | Return Type |
|---|---|---|
extract_ldap_entries | Fetches entries from LDAP based on filters and attributes. | p.Result[t.SequenceOf[t.Ldap.OperationAttributes]] |
validate_ldap_data | Checks entries for DN validity and required attributes. | p.Result[m.DbtLdap.ValidationMetrics] |
run_full_pipeline | Orchestrates the entire extract-validate-transform flow. | p.Result[m.DbtLdap.DbtLdapPipelineResult] |
transform_with_dbt | Triggers the execution of specific dbt models on the data. | p.Result[m.DbtLdap.DbtRunStatus] |
Sources: src/flext_dbt_ldap/services/client.py:32-116
Synchronization and State Management
The FlextDbtLdapSyncMixin manages the stateful synchronization of specific LDAP entities (Users, Groups, Memberships) to the warehouse, supporting both full and incremental loads.
Incremental Sync Logic
Synchronization state is persisted in a local JSON file (.flext_dbt_ldap_sync_state.json), using modifyTimestamp as the default bookmarking mechanism for incremental updates.
Sync Component Methods
| Method | Focus | Scope |
|---|---|---|
sync_users_to_warehouse | User entities | Incremental Supported |
sync_groups_to_warehouse | Group entities | Incremental Supported |
sync_memberships_to_warehouse | Group-User relations | Full Sync |
run_full_data_warehouse_sync | All components | Orchestration |
Sources: src/flext_dbt_ldap/services/sync.py:53-145
Protocols and Interfaces
The API is governed by a set of strict Python Protocols under FlextDbtLdapProtocols. These define the expected interface for services handling LDAP dbt operations, including modeling, quality checks, and performance monitoring.
Defined Protocols
- Ldap.Dbt: Handles core dbt operations like
compile_dbt_modelsandrun_dbt_models. - Ldap.LdapIntegration: Orchestrates the movement between LDAP and the warehouse.
- Ldap.Modeling: Manages the creation of dimensions (User, Group) and fact tables.
- Ldap.Quality: Handles completeness checks and anomaly detection.
- Ldap.Performance: Focuses on query optimization and monitoring.
Sources: src/flext_dbt_ldap/protocols.py:27-220
Result Models
The API returns structured result objects that provide detailed metrics on the operations performed.
Validation Metrics
Used by the validation service to report on data health.
class ValidationMetrics(m.Value):
total_entries: int # Total LDAP entries evaluated
valid_dns: int # Entries with valid DNs
valid_entries: int # Entries satisfying required attributes
quality_score: float # Score (0.0 to 1.0)
validation_passed: bool # Threshold check result
Sources: src/flext_dbt_ldap/_models/results.py:17-32
Sync and Pipeline Results
| Field | Description |
|---|---|
DbtRunStatus.status | Lifecycle status (e.g., PENDING, COMPLETED). |
DbtRunStatus.models_run | List of dbt models executed. |
SyncResult.overall_success | Boolean indicating if all sync components passed. |
PerformanceAnalysis.execution_time | Total time taken in seconds. |
Sources: src/flext_dbt_ldap/_models/results.py:34-75
Summary
The flext-dbt-ldap Public API Reference defines a robust framework for integrating directory services with modern data stacks. By utilizing a mixin-based service architecture and strict protocols, it ensures that LDAP data is extracted, validated, and transformed into high-quality warehouse models with consistent state management and performance monitoring.