Core Data Models

The Core Data Models in FLEXT Meltano provide a robust, type-safe foundation for data integration operations. These models leverage Pydantic (via flext_cli.m) to define schemas for project configuration, data source connectivity, stream definitions, and execution results. By centralizing these definitions, the system ensures consistency across the bridge between Go services and Python-based Meltano/Singer operations.

The architecture is categorized into specialized layers: foundation helpers for security and normalization, project-level metadata models, source/target configuration entities, and result-oriented structures for tracking pipeline outcomes. These models enforce strict validation rules, such as mandatory connection configurations and valid stream statuses, while providing automated features like sensitive data masking.

Foundation & Security Helpers

The foundation layer provides utilities for data normalization and security. A critical component is the protection of sensitive configuration data (e.g., passwords, tokens) during serialization.

Sensitive Data Protection

The system automatically identifies and masks sensitive keys within configuration mappings. This is implemented in FlextMeltanoModelsCore.protect_sensitive_config, which scans for keys matching patterns like password, secret, or api_key and replaces their values with [PROTECTED].

Value Normalization

The core models include wrappers for normalizing arbitrary input into validated sequences of strings or booleans, ensuring that data passed through the pipeline conforms to expected types.
Sources: src/flext_meltano/_models/core.py:43-85

Source and Target Configurations

Data integration relies on standardized configurations for Taps (extractors) and Targets (loaders). These models handle connection strings, versioning, and stream-specific settings.

Configuration Entities

The following table summarizes the primary configuration models used for data connectivity:

Sources: src/flext_meltano/_models/sources.py:16-128

Stream Definition

Streams represent the individual data entities (e.g., database tables or API endpoints) being moved. The StreamDefinition model tracks the schema, extraction progress, and operational status.

Project and Transformation Models

Project models manage the high-level orchestration of Meltano environments and DBT (Data Build Tool) transformations.

Pipeline Project Metadata

PipelineProjectModel assesses the maturity of a project based on its environment configuration. It identifies if a project is BASIC, DEVELOPING, or MATURE based on the count of environments and the presence of production markers.
Sources: src/flext_meltano/_models/projects.py:46-111

Transformation Projects (DBT)

Transformation models define the structure of DBT projects, including paths for models, macros, and seeds. They also calculate "Project Structure Complexity" based on the total number of configured paths.

Sources: src/flext_meltano/_models/transformations.py:13-94

Execution and Result Models

Results models provide a structured way to report the outcome of pipeline executions, including success status, logs, and error details.

Pipeline Results

The result structure captures the full context of an operation, facilitating debugging and monitoring.

DBT Manifest Integration

For advanced transformations, the system parses DBT manifest nodes into DbtManifestNode objects, providing typed access to node names, fully qualified names (FQN), and resource types.
Sources: src/flext_meltano/_models/projects.py:14-44

Summary

The Core Data Models establish a strict contract for data operations within the FLEXT Meltano ecosystem. By utilizing Pydantic for validation and implementing specialized logic for maturity assessment, complexity calculation, and sensitive data masking, these models ensure that data pipelines are secure, manageable, and highly predictable. Significance lies in their ability to bridge the gap between high-level orchestration and low-level Singer/Meltano execution details.