Inventory Refresh & Persister Engine
Relevant source files
The following files were used as context for generating this wiki page:
- app/models/manageiq/providers/base_manager/refresher.rb
- app/models/manageiq/providers/inventory/persister/builder.rb
- app/models/manageiq/providers/inventory/persister/builder/shared.rb
- app/models/ems_refresh/save_inventory.rb
- spec/models/manageiq/providers/inventory/persister/builder_spec.rb
- spec/models/manageiq/providers/base_manager/refresher_spec.rb
The Inventory Refresh & Persister Engine is the core mechanism within ManageIQ for synchronizing external provider state with the internal database. It manages the lifecycle of resource discovery, data parsing, and efficient database persistence. The engine supports both full and targeted refreshes, utilizing a specialized "Persister" to abstract the complexities of database interactions and data mapping.
Sources: app/models/manageiq/providers/base_manager/refresher.rb:64-88, app/models/manageiq/providers/inventory/persister/builder.rb:68-75
Core Architecture & Workflow
The refresh process is divided into four primary stages: collection, parsing, saving, and post-processing. The BaseManager::Refresher coordinates these stages, while the Persister acts as a container and builder for InventoryCollection objects, which represent sets of related database records (e.g., VMs, Hosts, Disks).
Refresh Lifecycle Flow
The following diagram illustrates the high-level workflow of the inventory refresh process:
The engine determines whether to perform a full manager refresh or a targeted refresh based on the number of targets and manager capabilities.
Sources: app/models/manageiq/providers/base_manager/refresher.rb:41-94, app/models/manageiq/providers/base_manager/refresher.rb:201-224
The Refresher Components
The Refresher class is responsible for managing the execution of refresh tasks across multiple Management Systems (EMS).
| Component | Responsibility |
|---|---|
preprocess_targets | Merges targets and determines if a full refresh is required based on full_refresh_threshold. |
collect_inventory_for_targets | Interacts with the provider to retrieve raw resource data. |
parse_targeted_inventory | Transforms raw provider data into InventoryCollection objects stored within a Persister. |
save_inventory | Triggers the persist! method on the Persister to commit changes to the database. |
Sources: app/models/manageiq/providers/base_manager/refresher.rb:96-160, spec/models/manageiq/providers/base_manager/refresher_spec.rb:23-55
Persister Builder System
The Persister::Builder is a DSL-driven component used to define how inventory data should be handled, mapped, and saved. It allows for declarative definition of model classes, strategies, and default values.
Builder Configuration Options
Builders utilize a variety of properties to control persistence behavior:
- model_class: Automatically derived or explicitly set class for the database records.
- manager_ref: The unique keys used to identify records (e.g.,
:ems_ref,:guid). - saver_strategy: Defines how data is saved, such as
:defaultor:concurrent_safe_batch. - strategy: Defines lookup methods, such as
:local_db_find_referencesor:local_db_find_missing_references.
Sources: app/models/manageiq/providers/inventory/persister/builder.rb:78-105, spec/models/manageiq/providers/inventory/persister/builder_spec.rb:45-80
Shared Builder Definitions
Common resource types like VMs and Templates share standard configurations defined in the Shared concern:
The INVENTORY_RECONNECT_BLOCK is a critical component that handles reconnecting archived or disconnected entities when they reappear in the inventory.
Sources: app/models/manageiq/providers/inventory/persister/builder/shared.rb:5-50, app/models/manageiq/providers/inventory/persister/builder/shared.rb:108-132
Data Persistence & Saving
The persistence layer handles complex operations like genealogy linking, Hardware/OS updates, and tag mapping.
Specialized Save Logic
For specific models, the engine uses custom save logic to ensure data integrity, particularly when dealing with SmartState (drift) data. For example, operating_systems persistence logic checks for drift_states to avoid overwriting high-fidelity scan data with lower-fidelity provider data.
Sources: app/models/manageiq/providers/inventory/persister/builder/shared.rb:134-155, app/models/ems_refresh/save_inventory.rb:227-236
Inventory Synchronization Workflow
The saving process involves identifying new, updated, and disconnected records:
Conclusion
The Inventory Refresh & Persister Engine provides a standardized, scalable framework for managing provider data. By separating collection, parsing, and persistence, and providing a powerful Builder DSL, it allows provider-specific plugins to implement complex synchronization logic while benefiting from optimized database operations and robust error handling.