PR #480 - preloopdev/preloop

Stack: 1/6, splitting #479 into reviewable pieces. Base: main. Merge in order.

Summary

The review fixes left open on #378. #378 merged without them: on main today a max-parallel matrix stalls once its first slots are taken, and nested reusable callers can fail to fold. In the lab harness (benchmarks/control-load, 2 nodes, Postgres 18) the original #378 finishes 60 of 603 jobs on the capped workload; with these fixes the capped workload completes all its legs.

These are the urgent correctness fixes. They are deliberately small and do not change the promotion machinery.

What changed

Max-parallel stall: a completion that frees a cohort slot now admits a parked leg. Matrix legs never name their siblings in job_needs, so the dependents-only trigger never fired. The settle asks whether the settled leg's cohort has a blocked leg with zero remaining needs and fewer active legs than its cap (Sweep::cohort_has_admissible_leg); a true answer runs the shared promotion sweep. That sweep loads the run graph, so while a capped cohort has a parked leg and a free slot every completion pays that load. #5 in this stack removes it.
Cohort waiter retry: a waiter re-parked at the cohort cap is retried when capacity frees on another gate (FIFO; only groups with no live holder; it keeps its wait_id).
Nested reusable callers: try_fold_reusable_caller continues past an incomplete caller instead of returning. Nested callers share inner job ids, so an incomplete outer caller hid a complete inner one.
Non-terminal status guard: complete_job/settle_job reject Queued/Pending/InProgress with BadRequest before taking the run lock or writing anything.
Tests (PG): cohort admission, dependent of a capped base waits for every leg, gate-waiter retry, nested-caller fold, non-terminal completion guard. The first commit adds the tests and is red by design for the five scenarios; the fix commits turn them green one by one.
Docs: internal/settle-redesign.md (steps 0-11, capped-cohort caveat, tests) and the CHANGELOG entry, which moves from the released 0.34.0 section (it never reached main) to Unreleased.

Not included

Later PRs in the stack: deferred reusable callers (#2), removal of the dead in-memory completion path (#3), row-based enqueue and NeedsView groundwork (#4), job-scoped max-parallel admission (#5), dependent and job-waiter promotion over rows (#6).

Test plan

On this PR's tip, against a disposable Postgres 17 (max_connections=2000, PRELOOP_PG_WRITERS=3 PRELOOP_PG_READERS=3, PRELOOP_TEST_POSTGRES_URL set):

cargo fmt --all --check: clean
cargo clippy --workspace --all-targets -- -D warnings: clean
cargo test --locked -p preloop-runner-server --lib -- control::: 343 passed, 0 failed, 0 skipping: lines (the shared suite covers lite as well)

just test-ci and the local engine CI run were not run on this PR's tip: the laptop engine's runner pool does not advertise the preloop-cpane label that ci.yml requires, so jobs fail at enqueue. The checks on this PR are the first full CI run for these commits.

Stack

#480 fix(pg): max-parallel stall, cohort waiter retry, nested-caller fold, non-terminal guard
#481 fix(pg): deferred reusable callers
#482 refactor(pg): remove the in-memory completion path
#483 refactor(pg): row-based enqueue target, needs view, settle hand-off (no behaviour change)
#484 perf(control/pg): max-parallel admission over rows
#485 perf(control/pg): dependents and job waiters over rows

Each PR's base is the previous PR's branch, so each shows only its own diff. Supersedes #479 (closed).

Summary by CodeRabbit

Bug Fixes
- Matrix-cohort jobs parked at the parallelism limit can now proceed when a slot becomes available, while retaining their queue order.
- Completing a job now refreshes eligible dependents, including those of other jobs completed as part of the same update.
- Fail-fast completion cancels all non-terminal sibling jobs.
- Attempts to complete a job with a non-terminal status are rejected without changing its status or queue state.