Schema Mapping & Transformations
Relevant source files
The following files were used as context for generating this wiki page:
Schema Mapping & Transformations define the declarative contract between a source system and a destination schema. This system determines which source resources become specific destination models, which fields are carried over, how objects are uniquely identified, and which records are skipped or modified during synchronization.
The schema_mapping block in the config.yml file serves as the primary configuration point. The generator module uses these entries to render DiffSync model classes, while adapters read the mapping entries at load time to process incoming data records.
Sources: dev/knowledge/schema-mapping.md:5-10, dev/knowledge/sync-architecture.md:38-48
Core Mapping Structure
Each entry in the schema_mapping list is a SchemaMappingModel that defines the relationship between a source resource and a destination node.
| Component | Description |
|---|---|
name | The destination model/kind. Must match a node in the destination schema. |
mapping | The source resource path (e.g., a REST collection or SDK endpoint). |
identifiers | The natural key fields that uniquely identify the object across systems. |
filters | Conditions used to drop records before they enter the data store. |
transforms | Ordered Jinja2 expressions used to rewrite fields after filtering. |
fields | Specific field-by-field mapping logic. |
Sources: dev/knowledge/schema-mapping.md:27-37
Data Processing Flow
When an adapter loads data, it follows a strict sequence: fetching records, applying filters, executing transforms, and converting the result into a DiffSync-compatible shape.
The diagram shows the lifecycle of a source record as it is processed by the adapter's model_loader and the DiffSyncModelMixin.
Sources: dev/knowledge/schema-mapping.md:39-43, dev/guides/adding-an-adapter.md:54-62
Field Mapping Logic
Fields are defined as SchemaMappingField objects. The sync engine evaluates four distinct levers for each field in a specific order:
- Static: Assigns a literal value, completely ignoring the source object.
- Mapping (Direct): Reads a value from the source record using dot-notation paths (e.g.,
status.value). - Mapping + Reference: Treats the source value as a foreign key. The adapter resolves this to the
unique_idof another mapped model. - Name: Defines the final destination field name.
Sources: dev/knowledge/schema-mapping.md:39-50
Reference Handling & Write Order
References allow the destination to rebuild relationships. Because a model must exist before it can be referenced, the write order is critical. The system auto-computes this order based on the reference edges in the schema_mapping.
Sources: dev/knowledge/schema-mapping.md:52-56
Filtering & Transformations
Filtering Operations
Filters drop records before they are loaded into the store. A record is only kept if it passes every defined filter.
| Category | Operations |
|---|---|
| Equality | ==, != |
| Ordering | >, <, >=, <= (coerced to int) |
| Membership | in, not in, contains, not contains |
| Presence | is_empty, is_not_empty |
| Pattern | regex |
| Network | is_ip_within |
Sources: dev/knowledge/schema-mapping.md:57-73
Jinja2 Transformations
Transforms rewrite fields using a Jinja2 NativeEnvironment. This ensures that results maintain native Python types (like list or int) instead of being forced into strings. The entire record is available as template context.
transforms:
- field: bundle_number
expression: |-
{% set suffix = name[(name.rstrip('0123456789'))|length:] %}
{{ suffix | int if suffix else None }}
Sources: dev/knowledge/schema-mapping.md:75-88, examples/netbox_to_infrahub/config.yml:320-324
Automated Model Generation
The infrahub-sync/generator module translates config.yml mappings into Python code. It maps schema attribute types to Python type annotations for the generated DiffSync models.
| Schema Kind | Python Type |
|---|---|
| Text, String, TextArea, DateTime | str |
| Number, Integer | int |
| Boolean, Checkbox | bool |
| List | list[Any] |
Sources: infrahub_sync/generator/init.py:12-23
Identifier & Attribute Resolution
The generator determines which fields serve as identifiers and which as attributes by inspecting both the destination schema and the user-provided config.yml. If identifiers are not explicitly defined in the mapping, the generator defaults to attributes marked as unique in the schema that are also present in the mapping.
Sources: infrahub_sync/generator/init.py:60-75, infrahub_sync/generator/init.py:78-95
Generator Logic Flow
The generator relies on a set of internal functions and Jinja2 filters to build the model infrastructure.
The diagram illustrates how the generator orchestrates configuration data and schema definitions to produce the final model code.
Sources: infrahub_sync/generator/init.py:146-168
Implementation in Adapters
Adapters must reuse the centralized loading helpers to ensure consistency across different systems. The model_loader function in an adapter should always call model.filter_records and model.transform_records through the DiffSyncModelMixin.
Sources: dev/guidelines/writing-an-adapter.md:16-25, dev/guides/adding-an-adapter.md:54-62
Summary
Schema Mapping & Transformations provide a robust, declarative way to handle data migration and synchronization. By decoupling the connector logic (Adapter) from the data transformation logic (Mapping), the system allows for complex migrations between disparate data models without requiring custom transformation code for every integration.