fix(store): keep one copy of each node per branch
ย
BE
4d ago+468+468+012345678901234567890123456789-22-22-01234567890123456789
Why
A process that reuses one client, such as a worker or a scheduled sync, used more memory on every query. Each time a query returned a node that was already in the client store, the store added another copy and kept the previous one. Lookups returned only the newest copy, and no method freed the earlier ones. client.store.count() counted the copies, not the nodes.
Goal: the store keeps one copy of each node per branch, so its memory use stays bounded by the number of distinct nodes.
Non-goals:
- Merging the data of the earlier copy into the new one, so that fields the second query did not request stay available. That is #347 and the behaviour reported in #413. This PR keeps the current rule: the copy from the latest query wins.
- Public
remove()orclear()methods on the store. These would be new public API and belong in a separate change todevelop. - Removing a node from the store when it is deleted with
node.delete(). It changes whatRelatedNode.peerand peer-based HFIDs return after a delete, and needs to handle nodes deleted by a cascade, so it is a separate change.
Closes #1506
What changed
Behavioural changes:
- When the store receives a node whose
idit already holds on that branch, it replaces the previous copy and removes that copy's ID, HFID and key entries. Keys set withstore.set(node, key=...)move to the new copy. - When the same node object is stored again after its HFID changed, for example after a rename and
save(), the previous HFID no longer finds it. client.all()andclient.filters()store a related node, for example fromprefetch_relationships,includeoronly, through a private store method. A related node replaces a copy that was also stored as a related node, so peers stay current. It does not replace a node returned by a query for its own kind or stored bysave(), because a related node built from relationship data can hold fewer fields and would otherwise replace that node for lookups by HFID and key. The check and the write happen under the branch lock.- A node stored before it had an
id, then stored again after it got one, keeps one entry, and its previous HFID entries are removed. client.store.count()returns the number of distinct nodes.- When the new copy has no HFID, for example from a query with
onlythat did not request it, a lookup by the earlier HFID returns the new copy. Before, it returned the earlier copy. The store guide described the earlier behaviour and is updated. - A lookup on a branch the store does not hold returns "not found" without adding an empty entry for that branch.
Implementation notes:
- The store indexed nodes by
_internal_id, which the SDK generates again for every node object, so two objects for the same Infrahub node were stored under two keys. - The decision about which copy to keep is in one method,
NodeStoreBranch._resolve_existing(). It returns the new copy today. Merging, as #347 asks, would mean changing only that method. _index()and_unindex()add and remove a node's entries. The store records the keys and HFID entries it wrote for each node and removes exactly those, because a node's HFID can change after it is stored, for example after a local rename._unindex()only removes entries that still point to the copy being removed.- A lookup by HFID whose entry points to a node no longer in the store raises
NodeNotFoundError, or returnsNonewithraise_when_missing=False. - Each branch store has a re-entrant lock around
set(),get()andcount(), because the sync client's batch saves nodes from several threads andset()now reads and changes several dictionaries in turn. It is re-entrant because computing an HFID insideset()can read the same store again. There is no test for the lock: a threaded test passed with and without it. - No new public methods.
What stayed the same:
- Lookups by
idand HFID return the copy from the latest query, except in the HFID case above. A lookup by key now does too; before, a key kept pointing to the copy it was set with. - Nodes without an
id, such as nodes not yet saved, are stored and found as before. - The same
idon two branches is still two entries.
How to review
infrahub_sdk/client.py(the two places that store related nodes) andinfrahub_sdk/store.py:NodeStoreBranch.set(),_set_related(),_store(),_resolve_existing(),_index(),_unindex(),_get_by_hfid(), andNodeStoreBase._get()/_set()/_set_related().- Tests:
tests/unit/sdk/test_client.py::test_method_all_twice_keeps_one_store_entry_per_nodefetches the same node twice throughclient.all(). It checks the store holds one entry with the values from the second response, and does not check which object the store keeps, so it remains valid if the copies are merged later.test_method_all_with_prefetch_keeps_stored_node_over_peer_copychecks that a related node fromprefetch_relationshipsdoes not replace a node already stored.test_store.py::test_node_store_set_related_replaces_only_a_related_copychecks that a related copy replaces an earlier related copy but not a node stored withset().tests/unit/sdk/test_store.pycovers replacement (keys, old and new HFID), one and two new copies without HFID, a local rename before a new query, a rename of the same object stored again, and the unknown-branch lookup.
docs/docs/python-sdk/guides/store.mdxandchangelog/1506.fixed.md.
How to test
uv run pytest tests/unit/sdk/test_store.py tests/unit/sdk/test_store_branch.py tests/unit/sdk/test_client.py
uv run invoke lint-code
Results on this branch:
- Each new test fails on
v1.24.0and passes with the fix. invoke lint-codepasses.- Unit suite: 3147 passed, 8 failed. The 8 are in
tests/unit/pytest_plugin/test_plugin.pyand fail the same way onv1.24.0: on the machine used, theinfrahub-testcontainerspytest plugin raisesRuntimeError: invalid CPU frequency dataat start-up.
Impact & rollout
- Backward compatibility: no API change. Code that relied on
client.store.count()counting copies, or on finding an earlier copy by its_internal_id, sees the new behaviour. - Infrahub: Infrahub calls the client store. Its CI has not yet been run against this branch.
Checklist
- Tests added/updated
- Changelog entry added (
changelog/1506.fixed.md) - External docs updated (
docs/docs/python-sdk/guides/store.mdx) - Internal .md docs updated: not needed, no file under
dev/orAGENTS.mddescribes the store - Infrahub CI run against this branch
๐ค Generated with Claude Code