Migration Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Migration Engine is a specialized service within the FLEXT-LDIF library designed to facilitate server-to-server LDIF data migration. It handles the transformation of entries between different LDAP server formats (such as OID, OUD, and OpenLDAP) while maintaining data integrity, schema compliance, and specific attribute mappings like boolean values and Access Control Lists (ACLs).
The engine operates as an orchestration layer that combines parsing, entry transformation via processing pipelines, and serialized writing. It supports high-level operations ranging from single-file conversions to structured directory-wide migrations that categorize output into organized phases (e.g., schema, hierarchy, users, and groups).
Sources: src/flext_ldif/services/migration.py:1-12, src/flext_ldif/_models/_settings_migrate.py:27-38, examples/04_server_migration.py:165-175
Architecture and Components
The Migration Engine is built around the FlextLdifMigrationPipeline class, which manages the lifecycle of a migration task. It utilizes specialized services for different stages of the migration process.
Core Pipeline Flow
The migration process follows a linear sequence: detection/normalization of server types, parsing of source LDIF, entry-level transformation, and finally, writing the migrated data to the target destination.
The diagram shows the standard flow of data from source discovery through transformation to final output.
Sources: src/flext_ldif/services/migration.py:117-152, examples/04_server_migration.py:177-200
Key Classes and Data Structures
| Class/Model | Responsibility |
|---|---|
FlextLdifMigrationPipeline | Orchestrates the migration of files or directories. Handles server type coercion and directory walking. |
MigrateOptions | Consolidates migration parameters including mode selection (Structured, Categorized, Simple) and filtering rules. |
MigrationPipelineResult | Encapsulates the outcome of a migration, including the migrated entries, output file paths, and processing statistics. |
WriteFormatOptions | Controls the serialization format, including line folding, attribute sorting, and metadata commenting. |
Sources: src/flext_ldif/services/migration.py:15-18, src/flext_ldif/_models/_settings_migrate.py:27-45, src/flext_ldif/_models/_settings_migrate.py:101-115
Migration Modes
The engine supports three distinct modes of operation to accommodate different migration complexity requirements.
Structured Migration
Structured migration produces an organized set of up to 6 LDIF files. This mode is typically used for complex enterprise migrations where entries must be applied in a specific order (e.g., schema before data).
- 00-schema: Schema definitions.
- 01-hierarchy: Base DNs and organizational units.
- 02-users: User entries.
- 03-groups: Group memberships.
- 04-acls: Access control lists (converted to target format).
- 06-rejected: Entries that failed validation or filters.
Sources: src/flext_ldif/_models/_settings_migrate.py:33-35, examples/demo_structured_migration.py:11-20
Categorized and Simple Modes
- Categorized Mode: Allows users to define custom
categorization_rulesand map specific entry categories to user-defined filenames. - Simple Mode: The default mode which processes source LDIF into a single output file (defaults to
migrated.ldif).
Sources: src/flext_ldif/_models/_settings_migrate.py:36-38, src/flext_ldif/_models/_settings_migrate.py:82-88
Transformation Logic
The engine performs deep transformation of entries based on the source and target server types.
Server-to-Server Mapping
Specific rules are applied when moving between servers like OID and RFC-compliant targets. This includes:
- Boolean Conversion: OID uses "1"/"0" for boolean flags like
orclIsEnabled, while RFC/OUD expects "TRUE"/"FALSE". - ACL Renaming: Mapping attributes such as
orclaci(OID) to standardaci(RFC/OUD). - Schema DN Normalization: Converting server-specific schema DNs (e.g.,
cn=subschemasubentry) to standard RFC schema DNs (cn=schema).
Sources: tests/unit/test_migration_pipeline_servers.py:38-70, tests/unit/test_migration_pipeline_servers.py:89-110, tests/unit/test_migration_pipeline_servers.py:133-145
Data Filtering
Entries are filtered based on the base_dn configuration. This prevents "false positive" substring matches (e.g., ensuring dc=example2 is not included when the base DN is dc=example).
This diagram illustrates the hierarchical DN validation used to prevent incorrect inclusion of entries during migration.
Sources: tests/integration/test_categorization_real_data.py:100-118, src/flext_ldif/services/migration.py:110-115
Configuration Options
Migration behavior is controlled via MigrateOptions and the FlextLdifMigrationPipeline attributes.
| Parameter | Type | Default | Description |
|---|---|---|---|
source_server_type | ServerTypes | RFC | The format of the input data. |
target_server_type | ServerTypes | RFC | The desired format for the output data. |
base_dn | str | None | Filter entries to only those under this hierarchical path. |
input_dir | Path | None | Directory to scan for *.ldif files. |
output_dir | Path | None | Directory where migrated files will be written. |
sort_entries_hierarchically | bool | False | Sorts entries by DN depth then alphabetically. |
Sources: src/flext_ldif/services/migration.py:20-80, src/flext_ldif/_models/_settings_migrate.py:46-95
Implementation Example: File Migration Core
The _migrate_file_core method demonstrates the internal sequence of the engine:
def _migrate_file_core(self, input_file: Path, output_file: Path | None) -> p.Result[m.Ldif.MigrationPipelineResult]:
# 1. Read source
read = u.Cli.files_read_text(input_file)
# 2. Parse into entries
parse_result = FlextLdifParser().parse_string(read.value, server_type=self.source_server_type)
entries_list = list(parse_result.value.entries)
# 3. Transform entries via Pipeline
migrate_result = self.migrate_entries(entries_list)
# 4. Write to target destination
return self._write_migrated_file(input_file, output_file, entries_list, migrate_result.value)
Sources: src/flext_ldif/services/migration.py:165-195
Conclusion
The Migration Engine provides a robust framework for LDAP data transition. By automating server-specific transformations and providing structured output categorization, it reduces the manual effort and risk associated with migrating complex directory structures between heterogeneous server environments.