fix(cli): warn when queued jobs can never start (offline or runners control plane)

Merged
BN
2mo ago
+335-10

preloop run streamed '[preloop] still waiting: N job(s) queued' forever when no runner was registered, or failed with a bare connection error when serve was down.

  • Add GET /api/v1/runners (native bearer): registered runners with labels; documented in the native OpenAPI surface.
  • cmd_run now wraps the submit and event-stream sends with a 'cannot reach control plane at {url}; is preloop serve running?' context, and on the first backpressure tick checks runner registration: zero runners gets an explicit warning that queued jobs will never start unless one appears.

What & why

Protocol surface

  • I confirm this change touches the runner protocol interface: YES / NO

Required gates

  • just test-ci passes locally (fmt-check + clippy -D + full test suite + runner-watch conformance)
  • If protocol/wire shapes changed: the change is validated against the official runner (golden replay via runner-watch, or a live capture showing the official bytes), not only unit tests
  • If the touched subsystem has property tests (concurrency_properties, scheduling, matrix expansion, …): they are extended for the changed contract and pass (PROPTEST_CASES=256 cargo test -p preloop-runner-server)
  • New wire fields/events are additive and serde-defaulted where the official runner would not send them
  • No secrets, credentials, or internal/deployment-specific paths are introduced (captures with live tokens must be redacted or excluded)

Verification performed

Checklist

  • Tests added/updated for any new observable contract
  • Docs updated (docs/, CONTRIBUTING.md) where behavior changed
  • Changelog-worthy user-facing change described in the PR body

Summary by cubic

Warns when queued runs can never start due to an offline control plane or no claimable runners. Previously preloop run streamed “still waiting” forever or failed with a bare connection error.

  • CLI: wraps run submission and event-stream sends with a “cannot reach control plane at {url}” context; on backpressure ticks queries GET /api/v1/runners?run_id= and prints a one-time warning when queued > 0 and claimable == 0; keeps rechecking while any claimable runner exists; falls back on older servers to count (including count == 0 triggers the dead‑pool warning).
  • Server: adds GET /api/v1/runners (native bearer). With run_id, returns queued and claimable computed with the scheduler’s job_matches_runner_capabilities and runner group constraints; documents the endpoint in OpenAPI; exposes capabilities_of for consistent matching; tests cover listing, auth gating, and run‑scoped claimability.

Summary by CodeRabbit

  • New Features

    • Added an authenticated runner-listing endpoint showing registered runners, IDs, labels, and availability.
    • Added optional run-specific capacity details, including queued jobs and runners able to claim them.
    • CLI now displays warnings when queued jobs have no available matching runner capacity.

  • Bug Fixes

    • Improved error messages when run submissions or event-stream requests cannot reach the control plane.

  • Tests

    • Expanded coverage for runner listings, authentication, filtering, and capacity reporting.