Incremental Sync & State Cache
Relevant source files
The following files were used as context for generating this wiki page:
Incremental sync reduces data extraction overhead by only processing objects that changed since the last successful run. This system utilizes a state cache to store diff plans, resource snapshots, and cursor watermarks, allowing the engine to reuse previously computed data and maintain synchronization efficiency.
Sources: dev/knowledge/incremental-and-cache.md:4-7, dev/knowledge/sync-architecture.md:28-35
Cursor Tiers and Extraction Strategies
The system categorizes model extraction capability into Cursor Tiers. Each tier represents a different level of filtering strength supported by the source adapter.
Tier Classification
Adapters declare the supported strategy for each model via the cursor_tier_for(model_name) method.
| Tier | Value | Meaning |
|---|---|---|
NONE | 0 | Source cannot filter by change; requires full extraction. |
ID | 1 | Source exposes stable IDs; changes are detected by comparing ID sets. |
TIMESTAMP | 2 | Source filters by modification time; extracts only "changed-since" records. |
Sources: dev/knowledge/incremental-and-cache.md:9-18, infrahub_sync/cache/cursors.py:8-16
Implementation Flow
When a tier higher than NONE is selected, the adapter must implement specific methods to handle the incremental data flow.
The diagram shows how the system decides between full extraction and incremental extraction based on the declared tier.
Sources: dev/knowledge/incremental-and-cache.md:20-28
State Cache and Diff Plans
The State Cache persists synchronization metadata and data snapshots to disk, typically under .infrahub-sync-cache/<sync_name>/.
Components of the Cache
- Diff Plan: A serialized Parquet file (
plan.parquet) containing the computed create, update, and delete actions. - Snapshots: Parquet sidecar files representing snapshots of the source and destination at the time of the run.
- Cursor State: The saved value (timestamp or ID watermark) from the previous run, managed by
persist_cursors_for_run().
Sources: dev/knowledge/incremental-and-cache.md:38-47, examples/netbox_to_infrahub/config.yml:8-12
Plan Execution Flow
The Potenda engine can separate the computation of a diff from its application using the cache.
This flow illustrates the separation between plan generation and plan application using persisted Parquet files.
Sources: dev/knowledge/incremental-and-cache.md:38-43, dev/knowledge/sync-architecture.md:28-35
Resync Cadence and Guardrails
To prevent data drift and accidental mass deletions, the system implements periodic resyncs and row-count monitoring.
Full Resync Cadence
Incremental drift is managed by forcing a full extract every $N$ runs. This is configured via the full_resync_every parameter in config.yml.
Sources: dev/knowledge/incremental-and-cache.md:30-36
Row-Count Guardrails
The check_rowcount_guardrail() function compares current destination row counts against a persisted baseline. If the count drops below a defined threshold, the run fails to prevent accidental data loss. Users can override this with the --allow-rowcount-drop flag.
Sources: dev/knowledge/incremental-and-cache.md:50-54, infrahub_sync/cache/guardrails.py
Configuration Example
The following YAML snippet demonstrates how to configure incremental settings in config.yml.
name: from-netbox
# Incremental behavior configuration
incremental:
full_resync_every: 20 # Force full resync every 20 runs
# Cache directory can be relocated via environment variable:
# export INFRAHUB_SYNC_CACHE_DIR="/shared/volume/sync-cache"
Sources: dev/knowledge/incremental-and-cache.md:34-36, dev/knowledge/incremental-and-cache.md:46-47, examples/netbox_to_infrahub/config.yml:18-20
Summary
The Incremental Sync and State Cache system optimizes synchronization by tracking changes via cursors and persisting execution plans in Parquet format. By implementing tiers (None, ID, Timestamp) and applying safety guardrails, the system ensures both performance for large datasets and protection against data corruption.
Sources: dev/knowledge/incremental-and-cache.md, dev/knowledge/sync-architecture.md