Type System & Validation
Relevant source files
The following files were used as context for generating this wiki page:
The Type System & Validation framework in flext-tap-ldif provides a robust, schema-enforced architecture for extracting and validating Lightweight Data Interchange Format (LDIF) data. It leverages Pydantic-based models (EnforcedModel) to ensure data integrity during the transformation of LDIF entries into Singer-compatible records. The system governs everything from file-level metadata tracking to individual record validation and performance metrics.
This system is built upon the FLEXT core philosophy of strict typing, using Annotated types and model_validator hooks to enforce business logic at the data layer. It ensures that only valid LDIF structures proceed through the processing pipeline, while providing detailed error reporting for invalid entries.
Core Data Models
The system architecture relies on specialized models to represent the state and validity of the LDIF extraction process. These models are categorized into file metadata tracking and validation result reporting.
LDIF File Metadata
The LdifFile model tracks the physical characteristics and processing state of an LDIF file. It ensures that file paths are valid and that sizes are non-negative through internal validators.
Validation Results
The LdifValidationResult model provides detailed feedback on the quality of the processed data. It maintains counts of valid versus invalid entries and calculates a success rate.
| Field | Type | Description |
|---|---|---|
file_path | str | Path of the validated LDIF file. |
valid | bool | Final status of the validation run. |
validation_errors | SequenceOf[StrMapping] | List of errors with severity and details. |
total_entries | NonNegativeInt | Count of all entries encountered. |
valid_entries | NonNegativeInt | Count of successfully parsed entries. |
validation_time | NonNegativeFloat | Time taken in seconds. |
Sources: src/flext_tap_ldif/_models/record.py:12-45
Validation Logic & Constraints
Validation is performed at multiple stages: during initial model instantiation and during the file processing loop.
Consistency Checks
The system employs model_validator hooks to ensure logical consistency across data fields. For example, a validation result cannot be marked as valid=True if the validation_errors list is populated.
Schema Enforcement
LDIF entries are mapped to a specific internal schema before being yielded as Singer records. This ensures consistency across different source files.
| Constant | Value | Description |
|---|---|---|
DN_FIELD | "dn" | Distinguished Name field. |
ATTRIBUTES_FIELD | "attributes" | LDAP attributes map. |
OBJECT_CLASS_FIELD | "object_class" | List of LDAP object classes. |
SOURCE_FILE_FIELD | "source_file" | Originating file path. |
Sources: src/flext_tap_ldif/constants.py:65-72
Processing & Type Conversion
The Processor class facilitates the transition from raw LDIF text to structured Python objects. It utilizes flext-ldif to parse content and then applies Pydantic models to validate the resulting entries.
Data Flow Sequence
The following diagram illustrates how raw file content is validated and transformed:
Test Protocols and Mock Types
For quality assurance, the project defines specialized test types and protocols that extend the base system to handle mock data and assertions.
Mock Type Definitions
Test-specific types are defined in TestsFlextTapLdifTypes to ensure that mock LDIF data adheres to the same structural requirements as production data.
- MockLdifEntry: A mapping of strings to either single strings or sequences of strings.
- TestLdifValidationResult: A specialized mapping for verifying validation outcomes in test suites.
Sources: tests/typings.py:34-52
Verification Protocols
Runtime-checkable protocols define the interface for test components like data providers and assertions:
- TestLdifAssertion: Requires implementation of
assert_ldif_file_parsedandassert_ldif_entries_valid. - MockLdifFile: Defines standard file operations (
open_file,read_entries) for simulated file system testing.
Sources: tests/protocols.py:44-95
Conclusion
The Type System & Validation module in flext-tap-ldif serves as a rigorous gatekeeper for data integrity. By combining Pydantic models with domain-specific LDIF constants and strict processing logic, it guarantees that LDIF data is accurately extracted and represented. The inclusion of comprehensive test protocols further ensures that the validation logic remains reliable across various LDIF scenarios and edge cases.