Data Quality Service

Relevant source files

The following files were used as context for generating this wiki page:

• src/flext_dbt_ldif/services/service.py
• tests/unit/_services_parts/data_quality.py
• tests/unit/test_services.py
• tests/unit/test_services_and_api.py
• tests/unit/test_core.py
• src/flext_dbt_ldif/services/unified_service.py (Note: Cited via usage in service.py and tests)

The Data Quality Service is a core component of the flext-dbt-ldif project designed to assess, validate, and report on the integrity of LDAP Data Interchange Format (LDIF) data. It operates as part of an orchestration layer that bridges raw LDIF ingestion with downstream dbt transformations, ensuring that only high-quality data enters the analytical pipeline.

The service provides automated mechanisms to parse LDIF files, calculate quality scores, and verify structural compliance against expected business rules. It is primarily exposed through the FlextDbtLdifServiceMixin.Service class and its unified API surface, allowing for both standalone quality assessments and integrated validation within the complete transformation workflow.

Sources: src/flext_dbt_ldif/services/service.py:16-30, tests/unit/_services_parts/data_quality.py:16-25

Service Architecture and Orchestration

The service is structured as a mixin-based orchestration layer. It utilizes a internal client to handle low-level file parsing and data validation, while the service layer manages the high-level workflow and result formatting.

Component Interaction

The following diagram illustrates the relationship between the service orchestrator and its underlying functional components.

The Service class acts as the central hub, delegating parsing and validation to self.client and model-related tasks to self.model_generator.

Sources: src/flext_dbt_ldif/services/service.py:34-45, tests/unit/test_services.py:40-60

Quality Assessment Workflow

The primary entry point for quality evaluation is run_data_quality_assessment. This workflow performs a two-stage operation: parsing the physical file and then validating the resulting data structures.

Assessment Logic Flow

The assessment ensures that the LDIF file is readable and that its content meets minimum quality thresholds before any dbt models are generated.

1. Parsing: The system reads the LDIF file and converts it into a sequence of JSON mappings (entries).
2. Validation: Each entry is analyzed for structural correctness, focusing on attributes like Distinguished Names (DNs).
3. Scoring: A quality_score (float) is calculated based on the validation results, alongside a validation_status (e.g., "passed").

Sources: src/flext_dbt_ldif/services/service.py:82-104, tests/unit/test_services.py:40-55

Key Methods and Data Structures

The Data Quality Service utilizes specific data models to communicate results across the API surface.

Core Service Methods

Method	Description	Return Type
run_data_quality_assessment(ldif_file)	Executes a focused quality check on a target file.	p.Result[m.DbtLdif.ParseValidationResult]
parse_and_validate_ldif(ldif_file)	Internal implementation that coordinates parsing and validation logic.	p.Result[m.DbtLdif.ParseValidationResult]
validate_ldif_data(entries)	Performs technical validation on a list of already parsed JSON entries.	p.Result[m.DbtLdif.LdifValidationResult]

Sources: src/flext_dbt_ldif/services/service.py:82-104, src/flext_dbt_ldif/services/service.py:157-161

Data Models for Quality Reporting

The service relies on the DbtLdif model namespace to standardize quality metrics.

• ParseValidationResult:
  • entry_count (int): Total number of LDAP entries found in the file.
  • quality_score (float): A value (typically 0.0 to 1.0) representing data health.
  • validation_status (str): The final outcome (e.g., "passed", "failed").
• LdifValidationResult:
  • total_entries (int): Count of entries processed.
  • quality_score (float): The score derived from individual entry checks.

Sources: tests/unit/test_services.py:46-55, src/flext_dbt_ldif/services/service.py:98-104

Analytics and Pattern Analysis

Beyond basic validation, the service supports deeper analysis of entry patterns through the Analytics core component. This includes identifying unique identifiers and detecting duplicates within the LDIF source.

Pattern Analysis Logic

The system analyzes the distribution of Distinguished Names (DNs) to ensure data uniqueness, which is critical for dbt primary key constraints.

Integration with Workflow

Data quality is integrated as a mandatory gate in the run_complete_workflow method. If validation fails, the service halts the pipeline before attempting to generate dbt models or run SQL transformations.

# snippet from src/flext_dbt_ldif/services/service.py
validation = self.client.validate_ldif_data(entries)
if validation.failure:
    return r[m.DbtLdif.WorkflowResult].fail(
        validation.error or "Validation failed",
    )

This prevents the generation of broken staging models that would otherwise fail during the dbt run phase.

Sources: src/flext_dbt_ldif/services/service.py:120-128

Conclusion

The Data Quality Service provides the necessary validation gates to ensure LDIF data integrity within the flext-dbt-ldif ecosystem. by combining file parsing, structural validation, and pattern analysis, it enables developers to maintain high standards for LDAP data before it is materialized in a data warehouse.