Unified Service Integration
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Unified Service Integration system provides a streamlined mechanism for orchestrating LDIF data processing into dbt-ready models. It acts as the core engine for generating lightweight dbt artifacts, including staging and analytics models, based on parsed LDIF entries. This system ensures that raw directory data is transformed into structured SQL models suitable for downstream analytics.
The integration is primarily handled by the FlextDbtLdifUnifiedService, which works in tandem with the broader FlextDbtLdifServiceMixin to provide a complete workflow from raw file parsing to dbt transformation.
Sources: src/flext_dbt_ldif/services/unified_service.py:12-16, src/flext_dbt_ldif/services/service.py:27-41
Core Architecture
The architecture is centered around a multi-tier service model where the UnifiedService focuses on model artifact generation, while the FlextDbtLdifServiceMixin.Service manages high-level orchestration including file I/O and client interactions.
Unified Service Component
The UnifiedService is a specialized component responsible for generating the SQL and metadata for dbt models. It uses defined constants to ensure consistency in naming conventions and materialization strategies across the project.
This diagram illustrates the generation flow where staging models are derived from raw sources, and analytics models are built upon the generated staging models.
Sources: src/flext_dbt_ldif/services/unified_service.py:35-86, tests/unit/test_dbt_models.py:36-70
Service Orchestration
The main service orchestrates the lifecycle of an LDIF processing task. It utilizes the UnifiedService (assigned to self.model_generator) to produce the necessary dbt structures after parsing and validating input data.
Sources: src/flext_dbt_ldif/services/service.py:33-41
Model Generation Logic
Model generation follows a two-step process to maintain dbt best practices, separating raw staging from business-oriented analytics.
Staging Model Generation
Staging models are generated as views. They reference the raw LDIF source and provide the foundation for all subsequent transformations.
| Attribute | Value / Source | Description |
|---|---|---|
| Name | stg_ldif_entries | The standard identifier for staging models. |
| Materialization | view | Configured to be lightweight views by default. |
| Source Reference | {{ source('ldif', 'raw_ldif_entries') }} | Points to the raw data layer. |
Sources: src/flext_dbt_ldif/services/unified_service.py:73-86, dbt_project.yml:33-37
Analytics Model Generation
Analytics models are materialized as tables. They encapsulate business logic and reference the staging models generated in the previous step.
| Attribute | Value / Source | Description |
|---|---|---|
| Name | analytics_ldif_insights | The standard identifier for analytics models. |
| Materialization | table | Persisted as physical tables for performance. |
| Dependencies | ['stg_ldif_entries'] | Explicitly depends on the staging layer. |
Sources: src/flext_dbt_ldif/services/unified_service.py:53-71, dbt_project.yml:45-49
Workflow Integration
The run_complete_workflow method provides a single entry point for the entire integration process.
The sequence shows how the Service coordinates between the Client (for data operations) and the UnifiedService (for metadata/model generation).
Sources: src/flext_dbt_ldif/services/service.py:91-140, tests/unit/test_services.py:126-189
Data Structures
The integration relies on specific Pydantic models to pass data between service layers.
DbtModel Structure
This structure defines the metadata required to write a dbt .sql or .yml file.
| Field | Type | Description |
|---|---|---|
name | str | Name of the dbt model. |
dbt_model_type | str | Type (e.g., 'staging', 'analytics'). |
materialization | str | dbt materialization (table, view, etc.). |
sql_content | str | The actual Jinja/SQL code. |
dependencies | list[str] | List of other models or sources referenced. |
Sources: tests/unit/test_services.py:78-95, tests/unit/test_dbt_models.py:97-107
Implementation Details
The UnifiedService implementation is designed to be idempotent and safe, returning empty lists if no input entries are provided to prevent runtime errors during bulk processing.
# src/flext_dbt_ldif/services/unified_service.py:75-76
def generate_staging_models(
self,
entries: t.SequenceOf[t.JsonMapping],
) -> p.Result[list[m.DbtLdif.DbtModel]]:
"""Generate simple staging models for provided LDIF entries."""
if not entries:
return r[list[m.DbtLdif.DbtModel]].ok([])
Sources: src/flext_dbt_ldif/services/unified_service.py:73-86
Summary
Unified Service Integration provides the foundational logic for converting LDIF data into a functional dbt project structure. By encapsulating the generation of staging and analytics layers within the UnifiedService, the project ensures a consistent and automated path from raw directory exports to high-value analytical models. The system's reliance on structured results and explicit dependency tracking enables robust error handling and clear data lineage.