fix(pools): take the pool locks of a create from a template before its transaction
Human-written summary
This deadlock bug was uncovered during performance testing for the latest Number Pool and IP Pool-related changes on stable. If multiple requests to create objects using templates came in concurrently and used the same Number Pools, then the requests could deadlock and time out. The fix is to get all the locks (Number Pools included) that are required for a given create before beginning the actual create.
The code changes are mostly refactoring to get all the Number Pool locks (schema or user, scoped or not) for a given create with templates. Ideally, most of this logic would move to its own component and not be in create.py, but this is a problem for a different PR
tried a new thing where Claude leaves (maybe too many) inline comments to help the reviewer (human or not). not sure if this is good or bad
Why
A team creates objects from a template whose component templates take numbers from a number pool. At the same time, another request takes a number from the same pool, in the division of one of the components. Both requests stop responding until the client gives up.
The two requests took the same two locks in opposite order, and neither wait has a time limit:
| Step | Create from the template | Direct create in the component's division |
|---|---|---|
| 1 | Takes the lock of its own division, before its transaction | Takes the lock of its division, before its transaction |
| 2 | Saves a node with a number from the pool, which locks the number pool vertex in Neo4j until the commit | Opens its transaction and picks a number |
| 3 | Next component: waits for the lock of the component's division | Node create query: waits for the Neo4j lock on the pool vertex |
Goal: a create from a template takes every pool lock it needs before it opens its transaction.
Non-goals: moving the template code out of create.py, and the cases listed under "Not covered" below.
Closes: no GitHub issue is filed for this bug yet. Related: IFC-3547 describes a wait cycle that also needs a waiting range change; this pull request has no test for it.
What changed
Behavioral changes:
create_node()reads the template's components before it takes the mutation's locks, and adds these locks to the mutation's lock names:- the division lock of each number pool that the template or a component template names, with or without an allocation scope
- the division lock of each
NumberPoolattribute of a component - the lock of each IP address pool and IP prefix pool that the template or a component template names
- The preview that
create_node()builds to compute lock names now has the id of the node it creates.create_node()generates the id before it builds the preview, unless the payload setsid. Before, the two had different generated ids, sorelationship_count.<identifier>.<node id>locks named an object that never existed. This applies to every create, not only creates from a template. An id thatcreate_node()generates is not checked against the database; an id the payload sets still is. - A validation error of a component that draws from a pool, or holds one that does, is now raised before the object's own checks. Other components are validated after them, as before.
Implementation notes:
- To read a component's division,
create_node()builds an unsaved preview of each component that draws from a pool, and of each component above one. A template whose components draw from no pool builds no preview. - Each
TemplateComponentholds the id of the component it describes, generated when the component templates are read, so the preview and the created component share it. Node.new_from_fields()takes the generated id as agenerated_idparameter, used when the fields hold noid.Node.new()does not take it.- Components are built through a new helper,
_new_component_node(), which callsnew_from_fields()to pass the component's id. Before, they were built withNode.new(). handle_template_relationships()is removed;_do_create_node()callscreate_components()directly.lock_utils.py:get_pool_lock_names()andip_pool_lock_name()are extracted fromget_lock_names_on_object_mutation(), which returns the same names as before for the same node.
Not covered:
- A pool fills a field that another pool's scope names. The create takes the lock of an empty division before the transaction and the lock of the real division inside it.
- An IP pool whose address or prefix kind has a
NumberPoolattribute. The IP node is created inside the transaction, where it takes that number pool's lock. A direct create withfrom_poolon a relationship does the same. - A caller that passes a database with a transaction already open.
How to review
backend/infrahub/core/node/create.py:create_node(), thenget_template_pool_lock_names()and its helpers.backend/infrahub/core/node/__init__.py: thegenerated_idparameter ofnew_from_fields(). It is deliberately not a parameter ofNode.new(), which takes field values as keyword arguments: the IP poolGetResourcemutations spread theirdatainto it, so agenerated_idkey there would set an id that is not checked.test_get_resource_data_cannot_set_the_id_of_the_address_without_a_checkcovers this.backend/tests/component/core/node/test_template_pool_lock_order.py: the test matrix.
Parts that deserve extra scrutiny:
- The component templates are now read before the locks and outside the transaction, as the template already is. A component template that another request changes while the create waits for a lock is applied as it was read.
- The preview repeats work the create does again for each pooled component.
How to test
uv run pytest backend/tests/component/core/node/test_template_pool_lock_order.py
uv run pytest backend/tests/component/core/resource_manager/test_number_pool_scope_allocation.py
uv run pytest backend/tests/component/core/resource_manager/test_ipaddress_pool.py
test_template_pool_lock_order.pyholds 16 tests. 13 parametrized cases cover the object, a component and a nested component against template-named number pools (with and without a scope),NumberPoolattributes, IP address pools and IP prefix pools. Each case holds one pool lock from another task and asserts that the create waits for it with no transaction open. Three more tests cover a payload that sets theid, a division that holds the object created, and the shared id.test_a_template_create_and_a_create_in_the_division_of_its_second_component_both_completeruns the two requests against each other. Without the fix, both are still blocked after 30 seconds.test_get_resource_data_cannot_set_the_id_of_the_address_without_a_checkintest_ipaddress_pool.pyasserts that agenerated_idkey in thedataof an IP address pool allocation does not set the id of the new address.- On the code before this pull request, 11 of the first 14 tests fail. The three that pass pin behavior that was already correct.
- Local run: 2,102 tests passed in the node, resource manager, template, create and upsert mutation, and unit core suites.
Not tested: the scenario on a full stack with Redis and several API server workers, the wait cycle for IP pools (the tests assert only the lock order), and releases before 1.12.0.
Impact & rollout
- Backward compatibility: no API or schema change.
- Performance: a component in a scoped number pool costs one more read of the pool before the locks. In a local measurement taken before the second commit, a template with five components in one scoped pool ran ten more queries, and a template that names no pool ran none.
- Deployment notes: safe to deploy.
Checklist
- Tests added/updated
- Changelog entry added (
uv run towncrier create ...) - External docs updated (if user-facing or ops-facing change): not applicable
- Internal .md docs updated (internal knowledge and AI code tools knowledge)
- I have reviewed AI generated content
🤖 Generated with Claude Code