fix(pools): take the pool locks of a create from a template before its transaction

AJ
2d ago
+1014-68

Human-written summary

This deadlock bug was uncovered during performance testing for the latest Number Pool and IP Pool-related changes on stable. If multiple requests to create objects using templates came in concurrently and used the same Number Pools, then the requests could deadlock and time out. The fix is to get all the locks (Number Pools included) that are required for a given create before beginning the actual create.

The code changes are mostly refactoring to get all the Number Pool locks (schema or user, scoped or not) for a given create with templates. Ideally, most of this logic would move to its own component and not be in create.py, but this is a problem for a different PR

tried a new thing where Claude leaves (maybe too many) inline comments to help the reviewer (human or not). not sure if this is good or bad

Why

A team creates objects from a template whose component templates take numbers from a number pool. At the same time, another request takes a number from the same pool, in the division of one of the components. Both requests stop responding until the client gives up.

The two requests took the same two locks in opposite order, and neither wait has a time limit:

StepCreate from the templateDirect create in the component's division
1Takes the lock of its own division, before its transactionTakes the lock of its division, before its transaction
2Saves a node with a number from the pool, which locks the number pool vertex in Neo4j until the commitOpens its transaction and picks a number
3Next component: waits for the lock of the component's divisionNode create query: waits for the Neo4j lock on the pool vertex

Goal: a create from a template takes every pool lock it needs before it opens its transaction.

Non-goals: moving the template code out of create.py, and the cases listed under "Not covered" below.

Closes: no GitHub issue is filed for this bug yet. Related: IFC-3547 describes a wait cycle that also needs a waiting range change; this pull request has no test for it.

What changed

Behavioral changes:

  • create_node() reads the template's components before it takes the mutation's locks, and adds these locks to the mutation's lock names:
    • the division lock of each number pool that the template or a component template names, with or without an allocation scope
    • the division lock of each NumberPool attribute of a component
    • the lock of each IP address pool and IP prefix pool that the template or a component template names
  • The preview that create_node() builds to compute lock names now has the id of the node it creates. create_node() generates the id before it builds the preview, unless the payload sets id. Before, the two had different generated ids, so relationship_count.<identifier>.<node id> locks named an object that never existed. This applies to every create, not only creates from a template. An id that create_node() generates is not checked against the database; an id the payload sets still is.
  • A validation error of a component that draws from a pool, or holds one that does, is now raised before the object's own checks. Other components are validated after them, as before.

Implementation notes:

  • To read a component's division, create_node() builds an unsaved preview of each component that draws from a pool, and of each component above one. A template whose components draw from no pool builds no preview.
  • Each TemplateComponent holds the id of the component it describes, generated when the component templates are read, so the preview and the created component share it.
  • Node.new_from_fields() takes the generated id as a generated_id parameter, used when the fields hold no id. Node.new() does not take it.
  • Components are built through a new helper, _new_component_node(), which calls new_from_fields() to pass the component's id. Before, they were built with Node.new().
  • handle_template_relationships() is removed; _do_create_node() calls create_components() directly.
  • lock_utils.py: get_pool_lock_names() and ip_pool_lock_name() are extracted from get_lock_names_on_object_mutation(), which returns the same names as before for the same node.

Not covered:

  • A pool fills a field that another pool's scope names. The create takes the lock of an empty division before the transaction and the lock of the real division inside it.
  • An IP pool whose address or prefix kind has a NumberPool attribute. The IP node is created inside the transaction, where it takes that number pool's lock. A direct create with from_pool on a relationship does the same.
  • A caller that passes a database with a transaction already open.

How to review

  1. backend/infrahub/core/node/create.py: create_node(), then get_template_pool_lock_names() and its helpers.
  2. backend/infrahub/core/node/__init__.py: the generated_id parameter of new_from_fields(). It is deliberately not a parameter of Node.new(), which takes field values as keyword arguments: the IP pool GetResource mutations spread their data into it, so a generated_id key there would set an id that is not checked. test_get_resource_data_cannot_set_the_id_of_the_address_without_a_check covers this.
  3. backend/tests/component/core/node/test_template_pool_lock_order.py: the test matrix.

Parts that deserve extra scrutiny:

  • The component templates are now read before the locks and outside the transaction, as the template already is. A component template that another request changes while the create waits for a lock is applied as it was read.
  • The preview repeats work the create does again for each pooled component.

How to test

uv run pytest backend/tests/component/core/node/test_template_pool_lock_order.py
uv run pytest backend/tests/component/core/resource_manager/test_number_pool_scope_allocation.py
uv run pytest backend/tests/component/core/resource_manager/test_ipaddress_pool.py
  • test_template_pool_lock_order.py holds 16 tests. 13 parametrized cases cover the object, a component and a nested component against template-named number pools (with and without a scope), NumberPool attributes, IP address pools and IP prefix pools. Each case holds one pool lock from another task and asserts that the create waits for it with no transaction open. Three more tests cover a payload that sets the id, a division that holds the object created, and the shared id.
  • test_a_template_create_and_a_create_in_the_division_of_its_second_component_both_complete runs the two requests against each other. Without the fix, both are still blocked after 30 seconds.
  • test_get_resource_data_cannot_set_the_id_of_the_address_without_a_check in test_ipaddress_pool.py asserts that a generated_id key in the data of an IP address pool allocation does not set the id of the new address.
  • On the code before this pull request, 11 of the first 14 tests fail. The three that pass pin behavior that was already correct.
  • Local run: 2,102 tests passed in the node, resource manager, template, create and upsert mutation, and unit core suites.

Not tested: the scenario on a full stack with Redis and several API server workers, the wait cycle for IP pools (the tests assert only the lock order), and releases before 1.12.0.

Impact & rollout

  • Backward compatibility: no API or schema change.
  • Performance: a component in a scoped number pool costs one more read of the pool before the locks. In a local measurement taken before the second commit, a template with five components in one scoped pool ran ten more queries, and a template that names no pool ran none.
  • Deployment notes: safe to deploy.

Checklist

  • Tests added/updated
  • Changelog entry added (uv run towncrier create ...)
  • External docs updated (if user-facing or ops-facing change): not applicable
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content

🤖 Generated with Claude Code