Pydantic V2 Models & Settings

Relevant source files

The following files were used as context for generating this wiki page:

• pyproject.toml
• AGENTS.md
• codex.md
• examples/advanced_processing_example.py
• examples/complete_workflow_example.py

Pydantic V2 serves as the foundational data validation and settings management layer for the FLEXT platform. It is used to define strict, type-safe models for data integration, pipeline orchestration, and system configurations. By leveraging Pydantic V2, FLEXT ensures that all untrusted input is parsed once into canonical types, maintaining the project's "Railway Pattern" for robust error handling and functional data flow.

The implementation focuses on high-performance validation, utilizing features like model_validate for option bags and declarative pipeline stages. This approach eliminates manual keyword argument normalization and provides a single source of truth (SSOT) for both runtime execution and automated testing environments.

Sources: pyproject.toml:29, AGENTS.md:§ Meta do FLEXT, codex.md:19-20

Core Model Architecture

Models in FLEXT typically inherit from m.BaseModel (a Pydantic V2 base) and utilize ClassVar[m.ConfigDict] for behavioral configuration. A common pattern involves allowing arbitrary types and extra fields during development or for specific data-heavy integration tasks.

Base Configuration Patterns

The following diagram illustrates the relationship between a processing stage and its data container:

The data flow ensures that every stage of a pipeline receives a validated object rather than raw dictionaries.

Component	Base Class	Config Strategy	Purpose
PipelineStageData	m.BaseModel	extra="allow"	Transports data between processing stages
WorkflowContext	m.BaseModel	arbitrary_types_allowed	Maintains correlation IDs and performance metrics
ProcessingResult	m.BaseModel	Strict Validation	Captures success/failure metrics of an operation

Sources: examples/advanced_processing_example.py:65-85, examples/complete_workflow_example.py:46-54

Settings & Configuration Management

FLEXT enforces a strict policy where config/*.yaml and settings abstractions are the sole sources of truth for business rules and environmental parameters. Static analysis tools like mypy are configured with Pydantic plugins to enforce initialization requirements and type safety for these settings.

Configuration Principles

• Typed SSOT: Tests must read expected values from the same typed SSOT as production code.
• Validation by Construction: Settings changes are validated by ensuring that tests and executable documentation remain valid dynamically rather than using hardcoded values.
• Tooling Integration: pyproject.toml configures pydantic-mypy to warn on untyped fields and required dynamic aliases.

Implementation in Pipelines

Pydantic models are extensively used in parallel processing executors. In the FlextLdifProcessingPipeline, models define the expected shape of both the batch items and the final analysis.

Advanced Processing Patterns

Models such as ProcessingResult and ValidationResult use Pydantic Field (aliased as u.Field) to provide descriptive metadata and default factories. This allows for detailed performance tracking, such as processing_time and items_succeeded counts.

class ProcessingResult(m.BaseModel):
    """Result of processing operation with metrics."""
    operation_id: str = u.Field(description="Unique operation identifier")
    items_processed: int = u.Field(description="Total items processed")
    items_succeeded: int = u.Field(description="Items that succeeded")
    processing_time: float = u.Field(description="Time taken for processing")
    metadata: t.JsonMapping = u.Field(default_factory=dict)

Sources: examples/advanced_processing_example.py:101-118, examples/complete_workflow_example.py:79-99

Workflow Orchestration Models

For enterprise workflows, Pydantic V2 models manage complex state across multiple parallel stages, including validation, analysis, and aggregation.

Workflow Execution Flow

The orchestrator uses WorkflowContext to track correlation IDs and lifecycle metadata, ensuring that as data passes through ThreadPoolExecutor workers, the context remains consistent and validated.

Field	Type	Description
workflow_id	str	Unique identifier for the execution
correlation_id	str	ID for tracking across distributed systems
performance_metrics	dict	Key-value store for stage timing and throughput
stage_results	Sequence	List of WorkflowStageResult models

Sources: examples/complete_workflow_example.py:58-77, examples/complete_workflow_example.py:101-125

Summary

Pydantic V2 is the architectural backbone of FLEXT, providing the mechanism for "Truth with evidence" by forcing all data into typed, validated models. Whether defining a simple data payload or a complex multi-stage workflow result, the project relies on Pydantic's strictness to maintain the "Railway Pattern" and ensure that failures are handled gracefully without ad-hoc error dictionaries or raw exception propagation.

Sources: AGENTS.md:Universal Agent Engineering Core, codex.md:21-22