PR #730 - preloopdev/preloop

Limits

What	Default	Knob
One run's whole snapshot capture (prebuilt merge / local workspace / remote checkout)	90 minutes	`SNAPSHOT_CAPTURE_TIMEOUT` (constant)
Post-deadline cleanup budget (enumeration + blocking-copy wait + removals)	30 s total, 10 s for blocking copies	constants `SNAPSHOT_CAPTURE_CLEANUP_GRACE` / `SNAPSHOT_CAPTURE_CLEANUP_WAIT`
Shared snapshot object cache at `<state_dir>/snapshot-object-cache`	100 GiB; `0` disables	`PRELOOP_SNAPSHOT_OBJECT_CACHE_MAX_BYTES`

Aggregate capture timeout

Each submit-time snapshot branch in submit_run_inner_with_webhook_delivery_unreserved is wrapped in one tokio::time::timeout(SNAPSHOT_CAPTURE_TIMEOUT, ..).
90 minutes is a backstop above a 30-minute transfer plus two 15-minute work commands and headroom for pack/fsck/publish, so a large repository never trips it but a wedged filesystem or dead lock holder still ends.
On expiry the complete cleanup is bounded by a 30 s grace:
- staging cleanup waits at most 10 s for blocking copies spawned by the capture (LFS staging copy, cross-device publish copy). spawn_blocking work cannot be cancelled; if it does not drain, the tree is left instead of racing a write.
- enumeration and recursive removals run inside the same 30 s bound; if the filesystem is wedged enough to trip the capture deadline, the error still returns after the grace and the run-scoped leftovers (snapshots/.staging/<run_id>-*, snapshots/<run_id>) are identifiable for the startup sweep.
- the unaccepted run-local repository is removed when the timeout lands after publication (for example while snapshot_repo_timing runs). Removal is gated on staging ownership being certain: a still-running blocking copy keeps both staging and final trees. Shared checkout-cache repositories and other runs' snapshots are never touched.
- the submission fails with an operator-facing ApiError naming the deadline and likely causes; no jobs are queued.
The remote-checkout branch keeps its normal "degrade to a direct forge checkout" behaviour for ordinary fetch errors, but a timeout is fatal: queueing jobs against a capture that never finished would silently change what is tested.
Dropping the capture future kills its Git children through the deadline runners. Whole process group, not just the direct child: kill_on_drop alone only kills the direct child, leaving git-remote-*/index-pack descendants running against the cache. All bounded async runners (run_command, run_command_capped, run_command_with_stdin, run_command_status) now hold the ProcessGroup guard introduced for run_command_status_in_group in #693, so a drop SIGKILLs the group. The new regressions fail when that guard's drop is disabled and pass with it.
The capture future is boxed (CaptureFuture) so the already-huge submit state machine does not grow past the debug-build stack limit — github::tests::webhook_replay_reuses_existing_run overflowed without it.

Shared object-cache cap

prune_snapshot_object_cache(state_dir, max_bytes, sweep):

Entries are snapshot-object-cache/<key>.git plus the persisted <key>.last-head and <key>.index sidecars. The counted total includes both sidecars, and eviction subtracts only the bytes actually removed, so a partial removal is reported honestly. An index that alone pushes an entry over the cap forces eviction (regression covers this).
Eviction is not an unlocked check-then-delete. Every candidate is retired under the same per-entry cache lock ensure_object_cache refreshes under (try_acquire_cache_lock; busy entries are skipped, not waited for), and only after, while that lock is held:
- no capture pin (<key>.pin-*) exists. ensure_object_cache creates a CachePin before releasing the refresh lock; the pin lives through create_workspace_snapshot_inner until the snapshot's objects/info/alternates file is written.
- no alternates in snapshots/*/objects/info/alternates or snapshots/.staging/*/objects/info/alternates (published and in-progress layouts; nested repository.git included) reference the entry. Real paths are compared; the scan is repeated under the lock because the pre-scan may predate a publication whose pin has since been dropped.
Eviction is fail closed: an unreadable snapshots/ or staging directory, an alternates file that exists but cannot be read, an unreadable alternates line, or a partial pin listing aborts eviction rather than producing an incomplete "unreferenced" set. Only a proven absent layout (ENOENT) counts as no reference.
Single-flight per state directory, owned by AppState::snapshot_object_cache_sweep (SnapshotObjectCacheSweep): the permit is acquired before spawning, so a task aborted before its first poll cannot wedge the guard, and it clears on completion/error/panic/drop. Startup and post-capture requests share it; a request while one runs is dropped, not queued. Detached spawn_blocking size walks hold their own ownership count, so a cancelled sweep cannot be replaced while a walk is still running. Two engines (or test states) have independent guards.
Pins older than the aggregate capture deadline plus the staging grace can only be crash leftovers and are removed under the lock; a leaked pin otherwise only ever delayed eviction.
Runs at startup (next to sweep_workspace_snapshots in bootstrap.rs) and best-effort after each successful create_workspace_snapshot, outside the cache lock.
Documented caveat: git clone --local hardlinks objects, so directory_bytes and reclaimed bytes are a soft ceiling, not guaranteed physical reclaim.

Per-namespace permits — not implemented

No namespace/tenant id is reachable at submit_run_inner:

preloop_gha_protocol::WorkflowSubmission has no namespace field;
submit_run_inner takes only (&Arc<SharedState>, WorkflowSubmission);
RunRecord has no namespace field;
the run row and allocate_run_number are hard-coded to control::types::DEFAULT_NAMESPACE.

Follow-up plumbing required: carry a namespace id through WorkflowSubmission → RunRecord → SubmitRun, then add Arc<Mutex<HashMap<NamespaceId, Arc<Semaphore>>>> to AppState with PRELOOP_SNAPSHOT_CONCURRENCY_PER_NAMESPACE (default 4) acquired only around the capture. Not invented here.

Review dispositions

Thread	Finding	Disposition
r4330680970	Unreadable alternates treated as unreferenced	Fixed earlier in `7ae710b2`; `snapshot_object_cache_references` now returns `Option` and aborts on any non-ENOENT error; regressions `unreadable_alternates_abort_eviction` + `unreadable_snapshot_enumeration_aborts_eviction` retained.
r4330680981	Timeout after publish leaves `snapshots/<run_id>`	Fixed: `discard_unaccepted_snapshot_run` removes the unaccepted run-local repository, gated on staging ownership certainty; regression `timeout_after_publication_removes_only_the_unaccepted_run_repository` also proves another run's snapshot and the shared checkout cache survive and the backing cache is reclaimed.
r4330680995	Sidecars excluded from quota	Fixed: sidecar sizes counted and only actually-removed bytes subtracted; regression `sidecar_index_counts_toward_the_ceiling`.
r4330681005	Overlapping post-capture sweeps	Fixed: per-state-dir single-flight acquired before spawn, detached walks counted; regression `object_cache_sweeps_coalesce_and_resume`. Same fix covers the duplicate r4331650180.
r4330681017	Unbounded cleanup after deadline	Fixed: complete cleanup bounded by 30 s (10 s blocking-copy wait); regression `cleanup_after_timeout_is_bounded`.
r4331650151	Changelog describes lock release inaccurately	Fixed: changelog now says the refresh lock is released when `ensure_object_cache` returns, the cache pin bridges to alternates, staging cleanup waits for blocking copies, and the process group is killed on cancellation.
r4331650180	Duplicate: overlapping sweeps	Same fix as r4330681005.

Tests

snapshots::snapshot_limit_tests (17 tests; temp dirs, no git binaries except the cancellation regression):

oldest-first eviction and cap stop; referenced (published/staging) survival; fresh lock; capture pin; 0 disable; under-cap no-op;
fail-closed unreadable alternates and unreadable snapshot enumeration;
blocking-copy wait before staging removal;
bounded cleanup under an injected stall (error returns inside the grace, staged leftovers remain identifiable);
timeout after publication removes only the unaccepted run repository, leaves another run and checkout-cache, and unpins the backing cache for a later sweep;
uncertain staging ownership keeps the published repository;
oversized .index alone forces eviction and is removed;
sweeps coalesce across startup/post-capture scheduling and resume after completion or after a detached walk releases ownership; a second state directory is independent;
outer capture timeout over a shell-wrapper fake Git: descendant group killed, staging removed, cache lock reusable.

preloop-observability dropped-future regressions: command_runner_takes_the_group_with_a_dropped_future, stdin_command_runner_takes_the_group_with_a_dropped_future, status_command_runner_takes_the_group_with_a_dropped_future (plus the pre-existing group-runner test). All fail when ProcessGroup::drop is disabled.

Manual feature smoke (throwaway, deleted after the run; real workspace + fake git delegating to the real binary, hanging on the final repository's rev-list after publish):

SMOKE timeout error: snapshot capture for run 99999999-9999-4999-8999-999999999999 exceeded the 2s engine deadline and was aborted; no jobs were queued. ...
SMOKE staging entries after timeout: []
SMOKE published repository remains: false
SMOKE locks/pins after timeout: []
SMOKE fake rev-list descendant pid 25694 gone: true

Run:

cargo fmt --all --check
cargo clippy --locked -p preloop-observability -p preloop-runner-server --all-targets -- -D warnings
cargo test --locked -p preloop-observability --lib
cargo test --locked -p preloop-runner-server --lib