fix(store): keep one copy of each node per branch

BE
4d ago
+468-22

Why

A process that reuses one client, such as a worker or a scheduled sync, used more memory on every query. Each time a query returned a node that was already in the client store, the store added another copy and kept the previous one. Lookups returned only the newest copy, and no method freed the earlier ones. client.store.count() counted the copies, not the nodes.

Goal: the store keeps one copy of each node per branch, so its memory use stays bounded by the number of distinct nodes.

Non-goals:

  • Merging the data of the earlier copy into the new one, so that fields the second query did not request stay available. That is #347 and the behaviour reported in #413. This PR keeps the current rule: the copy from the latest query wins.
  • Public remove() or clear() methods on the store. These would be new public API and belong in a separate change to develop.
  • Removing a node from the store when it is deleted with node.delete(). It changes what RelatedNode.peer and peer-based HFIDs return after a delete, and needs to handle nodes deleted by a cascade, so it is a separate change.

Closes #1506

What changed

Behavioural changes:

  • When the store receives a node whose id it already holds on that branch, it replaces the previous copy and removes that copy's ID, HFID and key entries. Keys set with store.set(node, key=...) move to the new copy.
  • When the same node object is stored again after its HFID changed, for example after a rename and save(), the previous HFID no longer finds it.
  • client.all() and client.filters() store a related node, for example from prefetch_relationships, include or only, through a private store method. A related node replaces a copy that was also stored as a related node, so peers stay current. It does not replace a node returned by a query for its own kind or stored by save(), because a related node built from relationship data can hold fewer fields and would otherwise replace that node for lookups by HFID and key. The check and the write happen under the branch lock.
  • A node stored before it had an id, then stored again after it got one, keeps one entry, and its previous HFID entries are removed.
  • client.store.count() returns the number of distinct nodes.
  • When the new copy has no HFID, for example from a query with only that did not request it, a lookup by the earlier HFID returns the new copy. Before, it returned the earlier copy. The store guide described the earlier behaviour and is updated.
  • A lookup on a branch the store does not hold returns "not found" without adding an empty entry for that branch.

Implementation notes:

  • The store indexed nodes by _internal_id, which the SDK generates again for every node object, so two objects for the same Infrahub node were stored under two keys.
  • The decision about which copy to keep is in one method, NodeStoreBranch._resolve_existing(). It returns the new copy today. Merging, as #347 asks, would mean changing only that method.
  • _index() and _unindex() add and remove a node's entries. The store records the keys and HFID entries it wrote for each node and removes exactly those, because a node's HFID can change after it is stored, for example after a local rename. _unindex() only removes entries that still point to the copy being removed.
  • A lookup by HFID whose entry points to a node no longer in the store raises NodeNotFoundError, or returns None with raise_when_missing=False.
  • Each branch store has a re-entrant lock around set(), get() and count(), because the sync client's batch saves nodes from several threads and set() now reads and changes several dictionaries in turn. It is re-entrant because computing an HFID inside set() can read the same store again. There is no test for the lock: a threaded test passed with and without it.
  • No new public methods.

What stayed the same:

  • Lookups by id and HFID return the copy from the latest query, except in the HFID case above. A lookup by key now does too; before, a key kept pointing to the copy it was set with.
  • Nodes without an id, such as nodes not yet saved, are stored and found as before.
  • The same id on two branches is still two entries.

How to review

  1. infrahub_sdk/client.py (the two places that store related nodes) and infrahub_sdk/store.py: NodeStoreBranch.set(), _set_related(), _store(), _resolve_existing(), _index(), _unindex(), _get_by_hfid(), and NodeStoreBase._get() / _set() / _set_related().
  2. Tests:
    • tests/unit/sdk/test_client.py::test_method_all_twice_keeps_one_store_entry_per_node fetches the same node twice through client.all(). It checks the store holds one entry with the values from the second response, and does not check which object the store keeps, so it remains valid if the copies are merged later. test_method_all_with_prefetch_keeps_stored_node_over_peer_copy checks that a related node from prefetch_relationships does not replace a node already stored. test_store.py::test_node_store_set_related_replaces_only_a_related_copy checks that a related copy replaces an earlier related copy but not a node stored with set().
    • tests/unit/sdk/test_store.py covers replacement (keys, old and new HFID), one and two new copies without HFID, a local rename before a new query, a rename of the same object stored again, and the unknown-branch lookup.
  3. docs/docs/python-sdk/guides/store.mdx and changelog/1506.fixed.md.

How to test

uv run pytest tests/unit/sdk/test_store.py tests/unit/sdk/test_store_branch.py tests/unit/sdk/test_client.py
uv run invoke lint-code

Results on this branch:

  • Each new test fails on v1.24.0 and passes with the fix.
  • invoke lint-code passes.
  • Unit suite: 3147 passed, 8 failed. The 8 are in tests/unit/pytest_plugin/test_plugin.py and fail the same way on v1.24.0: on the machine used, the infrahub-testcontainers pytest plugin raises RuntimeError: invalid CPU frequency data at start-up.

Impact & rollout

  • Backward compatibility: no API change. Code that relied on client.store.count() counting copies, or on finding an earlier copy by its _internal_id, sees the new behaviour.
  • Infrahub: Infrahub calls the client store. Its CI has not yet been run against this branch.

Checklist

  • Tests added/updated
  • Changelog entry added (changelog/1506.fixed.md)
  • External docs updated (docs/docs/python-sdk/guides/store.mdx)
  • Internal .md docs updated: not needed, no file under dev/ or AGENTS.md describes the store
  • Infrahub CI run against this branch

๐Ÿค– Generated with Claude Code