Inventory Fetch Performance & Projections
Relevant source files
The following files were used as context for generating this wiki page:
The Dynamic Inventory Fetch Performance system optimizes the retrieval of infrastructure data from Infrahub. It shifts the resource cost from being proportional to the total database size to being proportional only to the data requested by the user.
This module addresses performance defects where related nodes were fetched individually or entire node types were loaded unnecessarily. It implements bounded, batched "warming" passes and attribute projections to minimize HTTP round-trips and data transfer volume.
Sources: specs/001-inventory-fetch-performance/plan.md:12-25, specs/001-inventory-fetch-performance/spec.md:65-75
System Architecture
The architecture relies on three primary pillars: query projection, batched peer warming, and request metrics.
The diagram shows the flow from user definition to optimized data retrieval, highlighting how projections filter queries and the warmer batches secondary requests.
Sources: specs/001-inventory-fetch-performance/plan.md:95-108, specs/001-inventory-fetch-performance/tasks.md:120-135
Query Projection
The NodeProjection class translates user-requested attributes into narrowed GraphQL queries. It identifies which attributes are "roots" (first-level segments) and calculates an exclude set to prevent the SDK from fetching unnecessary data.
Key Logic and Rules
- Narrowing Strategy: The system calculates the complement of requested roots against the schema. It passes this as an
excludeargument because the SDK does not support narrowing viaincludealone. - Identity Preservation: Identifiers like
id,hfid, anddisplay_labelare always included in the query. - Default Behavior: If the user provides no attribute selection, the system preserves the historical wide query to ensure backward compatibility.
Sources: plugins/module_utils/projection.py:35-85, specs/001-inventory-fetch-performance/data-model.md:38-55
Projection Data Model
| Attribute | Description |
|---|---|
attrs | The full selection of dotted paths (e.g., site.name). |
roots | The first path segment of each selection. |
include | Valid roots found in the schema, including high-cardinality relationships. |
exclude | Roots missing from the schema and hierarchical pseudo-fields. |
narrowed | Boolean flag indicating if a user selection was provided. |
Sources: specs/001-inventory-fetch-performance/data-model.md:40-50
Batched Peer Warming
The PeerWarmer and RefillLedger components eliminate the "N+1" query problem where related nodes were previously fetched one at a time.
PeerWarmer Logic
The PeerWarmer collects all related node IDs referenced by the primary host nodes. It groups these IDs by node type and fetches them in chunks defined by the client's pagination_size.
Sources: plugins/module_utils/peers.py:30-65, specs/001-inventory-fetch-performance/plan.md:17-25
This sequence illustrates how the system deduplicates related nodes and fetches them in bulk rather than per host.
Sources: plugins/module_utils/peers.py:90-140, specs/001-inventory-fetch-performance/tasks.md:160-180
RefillLedger Rules
- Value Presence: Attributes with values
False,0, or""(empty string) are treated as present. OnlyNonetriggers a refill request. - Bounded Requests: The number of queries for related nodes is determined by the number of distinct related-node types and pages of results, not by the host count.
Sources: specs/001-inventory-fetch-performance/spec.md:210-220, specs/001-inventory-fetch-performance/data-model.md:60-75
Performance Metrics & Reporting
The RequestCounter class implements the SDK's Recorder protocol to track HTTP round-trips. This allows the system to report its own cost at raised verbosity levels.
Metrics Components
- HTTP Tracking: Every HTTP response increments the counter, including schema lookups and paginated results.
- Verbosity Levels:
- Default: The system remains silent.
- -v (INFO): The system reports the total request count, total nodes loaded beyond hosts, and the number of batches.
- -vvv (DEBUG): The system provides a per-kind breakdown of requests.
Sources: plugins/module_utils/metrics.py:15-45, specs/001-inventory-fetch-performance/tasks.md:235-255
Measured Improvements
| Metric | Previous State | Current Optimized State |
|---|---|---|
| Wall-clock time (650 hosts) | 102.71s | 3.23s |
| HTTP Requests | 700 | 15 |
| Data Transfer | 3758 KB | 1555 KB |
| Sources: specs/001-inventory-fetch-performance/plan.md:32-35 |
Error Handling and Resilience
The system prioritizes partial success over total failure.
- Schema Failures: If a specific node type schema cannot be resolved, the system warns the user but continues fetching other valid types.
- Missing Peers: If a batch of related nodes fails to load, the affected host attributes are left empty, and a warning is emitted.
- Empty Results: A run that finds zero hosts but encounters no errors completes successfully with an empty inventory. If all requested types failed, the system raises a
RuntimeError.
Sources: specs/001-inventory-fetch-performance/spec.md:120-145, specs/001-inventory-fetch-performance/tasks.md:210-230
Implementation Summary
The fetch performance improvements ensure that dynamic inventories remain scalable as the Infrahub database grows. By combining NodeProjection to limit the initial payload with PeerWarmer to batch relationship resolution, the collection achieves significant reductions in both latency and server load without altering the final inventory output.
Sources: specs/001-inventory-fetch-performance/plan.md:12-15, specs/001-inventory-fetch-performance/spec.md:200-205