Data Analysis & Statistics
Relevant source files
The following files were used as context for generating this wiki page:
The Data Analysis & Statistics module in flext-ldif provides a robust framework for ensuring data quality and operational visibility during LDIF processing. It encompasses validation against RFC 2849 standards, the generation of comprehensive entry statistics, and the analysis of structural differences during server migrations. This system is designed to handle high-volume LDIF data with parallel validation capabilities and detailed error categorization.
The module enables a "Railway Oriented Pipeline" where data is parsed, validated, and analyzed in a continuous flow, ensuring that only compliant entries proceed to transformation or writing stages. It also tracks historical conversion data and minimal differences to support zero-data-loss round-trips.
Sources: examples/03_validation_analysis.py:12-20, README.md:8-10
Validation & Analysis Architecture
The analysis system operates by inspecting Entry objects and their associated metadata. The primary interface for these operations is the LdifClient, which provides high-level methods to validate and analyze sets of entries.
Validation Workflow
The validation process checks entries for RFC compliance and server-specific violations. Results are aggregated into a ValidationResult model, which summarizes the health of the dataset.
The diagram above shows the data flow from raw entries to a comprehensive statistical report.
Sources: examples/03_validation_analysis.py:36-50, examples/07_advanced_processing.py:133-145
Key Analysis Components
| Component | Description | Source |
|---|---|---|
ValidationResult | Summarizes counts of total, valid, and invalid entries and collects error strings. | examples/03_validation_analysis.py:68-74 |
ValidationMetadata | Stores RFC violations, warnings, and server-specific errors within the entry's metadata. | src/flext_ldif/_models/domain_metadata.py:35-59 |
EntryStatistics | Tracks the lifecycle of an entry, including category assignment, filters passed, and rejection reasons. | src/flext_ldif/_utilities/metadata.py:657-679 |
analyze_entries() | Generates a mapping of key metrics such as error rates and entry distributions. | examples/03_validation_analysis.py:82-99 |
Metadata & Statistics Tracking
Every Entry in the system contains a ServerMetadata object that acts as a universal container for analysis data.
Statistical Model Structure
The ServerMetadata model captures multiple dimensions of data:
- RFC Compliance: Tracking specific violations (e.g., "RFC 2849 §2: DN required").
- Processing Stats: Storing
EntryStatisticsfor transformation audits. - Minimal Differences: Detailed tracking of punctuation, case, and spacing changes between original and converted LDIF.
This class diagram illustrates the relationship between domain metadata and the various statistical models.
Sources: src/flext_ldif/_models/domain_metadata.py:145-210, src/flext_ldif/_utilities/metadata.py:61-90
Difference Analysis
The analyze_minimal_differences utility compares original strings with converted outputs. This is critical for zero-data-loss migrations, as it flags even minor formatting changes (like trailing spaces or attribute case variations) that might otherwise be lost during normalization.
Sources: src/flext_ldif/_utilities/metadata.py:534-548
Migration Statistics
During server-to-server migration (e.g., OID to OUD), the system generates specialized statistics to monitor the conversion process.
Migration Configuration Options
The WriteFormatOptions and MigrateOptions classes define how statistics are reported in output files.
| Field | Type | Description |
|---|---|---|
include_statistics_summary | bool | Includes a summary header in the resulting LDIF file. |
statistics_categories | MutableIntMapping | Map of category names (e.g., 'users', 'groups') to entry counts. |
include_removal_statistics | bool | Tracks count of attributes removed during the migration. |
write_rejection_reasons | bool | Annotates the LDIF output with reasons why specific entries were rejected. |
Sources: src/flext_ldif/_models/_settings_migrate.py:230-270
Performance Benchmarking
The test suite defines quality gates and targets for the analysis and processing engine.
- Parsing Speed: Target of >1000 entries per second for standard LDIF files.
- Validation Speed: Target of >2000 entries per second for business rule validation.
- Memory Efficiency: Aim for <1MB of memory usage per 1000 entries processed.
This sequence diagram shows how statistics are updated during a standard validation request.
Sources: tests/README.md:195-205, src/flext_ldif/_utilities/metadata.py:657-660
Conclusion
The Data Analysis & Statistics system in flext-ldif provides more than simple entry counting. By integrating deep metadata tracking, RFC violation detection, and migration-specific statistics into the core Entry models, it allows for high-fidelity LDIF processing. Developers can utilize the analyze_entries and validate_entries APIs to implement automated data quality gates and maintain a comprehensive audit trail of all transformations.
Sources: examples/03_validation_analysis.py:82-99, src/flext_ldif/_models/domain_metadata.py:126-140