Filtering & Categorization
Relevant source files
The following files were used as context for generating this wiki page:
The Filtering & Categorization system in flext-ldif is a core infrastructure service responsible for organizing raw LDIF entries into logical domains and pruning data based on administrative rules. It ensures that LDIF data is partitioned into manageable buckets such as Users, Groups, Hierarchy, and Schema, while applying security and structural filters like Base DN enforcement and forbidden attribute removal.
This system is primarily utilized during migration and transformation pipelines to ensure data integrity and compliance with target server requirements. By utilizing a rule-based engine, it can detect entry types via objectClass markers or DN patterns, providing a structured FlexibleCategories response that tracks successful classifications and rejections.
Sources: src/flext_ldif/services/categorization.py:12-25, src/README.md:100-110
Categorization Engine
The FlextLdifCategorization service orchestrates the classification of entries. It uses a priority-based matching logic to assign entries to one of six canonical categories.
Processing Logic
The categorization process follows a specific order of operations:
- DN Validation: Ensures entries comply with RFC 4514 before categorization.
- Schema Detection: Identifies subschema entries using markers like
cn=subschemasubentry. - Rule Matching: Matches entries against
CategoryRulesusingobjectClassor attribute markers. - Post-Categorization Filtering: Removes forbidden attributes/objectClasses and applies OID whitelists to schema entries.
- Base DN Filtering: Final pass to ensure all entries reside under the specified directory root.
The diagram shows the sequential flow of an entry through the categorization service, including potential rejection points.
Sources: src/flext_ldif/services/categorization.py:167-210, src/flext_ldif/services/categorization.py:339-385
Canonical Categories
Entries are bucketed into specific groups defined in FlextLdifConstants.
| Category | Description | Priority |
|---|---|---|
SCHEMA | LDIF schema definitions (attributetypes, objectclasses). | High |
HIERARCHY | Organizational structure (OUs, Domains, Containers). | 1 |
USERS | Person-related entries (inetOrgPerson, etc.). | 2 |
GROUPS | Group memberships (groupOfNames, uniqueGroup). | 3 |
ACL | Access control information or entries containing ACIs. | 4 |
REJECTED | Entries failing validation or matching no rules. | N/A |
Sources: src/flext_ldif/constants.py:207-220, src/flext_ldif/services/categorization.py:170-185
Filtering Mechanisms
Filtering is applied at multiple stages to transform or exclude data.
Base DN Enforcement
The system applies a hierarchical check (is_under_base) to ensure entries belong to a specific branch. Unlike simple substring matching, this prevents false positives (e.g., dc=example2 will not match a base DN of dc=example).
Sources: tests/integration/test_categorization_real_data.py:91-105, src/flext_ldif/services/categorization.py:112-126
Attribute and ObjectClass Pruning
The service can be configured with forbidden_attributes and forbidden_objectclasses. These are removed from entries after categorization is complete but before the final result is returned. This allows classification to happen based on attributes that may ultimately be illegal in the target system.
Sources: src/flext_ldif/services/categorization.py:221-255
Schema OID Whitelisting
For SCHEMA category entries, the FlextLdifFilters service applies OID-based filtering. It uses WhitelistRules to prune specific OIDs from attributetypes or objectclasses within subschema entries.
Sources: src/flext_ldif/services/categorization.py:321-337, src/flext_ldif/_models/_settings_rules.py:90-110
Rule Configuration Models
Categorization behavior is controlled by the CategoryRules and WhitelistRules Pydantic models.
class CategoryRules(m.Value):
user_dn_patterns: t.Ldif.NormalizedStrFrozenset
group_dn_patterns: t.Ldif.NormalizedStrFrozenset
hierarchy_dn_patterns: t.Ldif.NormalizedStrFrozenset
user_objectclasses: t.Ldif.NormalizedStrFrozenset
group_objectclasses: t.Ldif.NormalizedStrFrozenset
hierarchy_objectclasses: t.Ldif.NormalizedStrFrozenset
acl_attributes: t.Ldif.NormalizedStrFrozenset
Sources: src/flext_ldif/_models/_settings_rules.py:22-54
Rule Priority and Server Defaults
The service resolves rules in the following order:
- Instance Rules: Rules provided directly to the
FlextLdifCategorizationconstructor. - Server Defaults: If no rules match, the service looks up constants from the
server_registrybased on theserver_type(e.g., OID, OUD). - Hierarchy Priority: Certain objectClasses defined by server constants (e.g.,
HIERARCHY_PRIORITY_OBJECTCLASSES) can force an entry into the HIERARCHY category regardless of other attributes.
Sources: src/flext_ldif/services/categorization.py:277-310, src/flext_ldif/constants.py:221-235
Rejection Tracking
A critical feature of the system is the rejection_tracker. It captures every entry that fails to pass through the categorization or filtering layers, associating them with a specific reason.
| Tracker Key | Reason |
|---|---|
INVALID_DN_RFC4514 | Entry DN does not comply with RFC 4514 syntax. |
BASE_DN_FILTER | Entry is not hierarchically under the configured base DN. |
CATEGORIZATION_REJECTED | Entry did not match any defined category markers. |
Sources: src/flext_ldif/services/categorization.py:27-33, src/flext_ldif/services/categorization.py:339-365
Metadata and Statistics
The system integrates with FlextLdifUtilitiesMetadata to mark entries with their processing results.
mark_filtered: Records whether an entry passed a specific filter (e.g., Base DN).mark_rejected: Attaches a rejection reason to the entry metadata.category_assigned: Stores the final category within theEntryStatisticsmodel.
Sources: src/flext_ldif/_utilities/metadata.py:465-495, src/flext_ldif/services/categorization.py:175-185
Filtering and Categorization serve as the primary traffic controller for LDIF processing, ensuring that data is valid, correctly classified, and stripped of unwanted elements before reaching downstream migration or writing services.