[rollout] feat: add agent_started_at and OpenSandbox request timeout env

Merged
AL
2d ago
+603-29

Summary

New Features

  • Rollout results now report when the rollout's agent started (agent_started_at). A caller can tell a rollout that failed during environment setup from one that failed after its agent ran, and retry only the former without re-running a graded attempt.
  • Managed OpenSandbox environments take their client request timeout from OPENSANDBOX_REQUEST_TIMEOUT_SEC, so a sandbox create that the server holds until the sandbox is running (for example while a node scales up) isn't abandoned by Harbor's default client timeout.

Improvements

  • Every terminal response after the agent started carries the timestamp, including the lease-expiry, crash and cancellation responses the server synthesizes.
  • A long-polling result request returns as soon as the agent starts, instead of waiting out the poll timeout.

Changelog

Added

  • RolloutResultResponse.agent_started_at: datetime | None: UTC, None until the agent starts; set on in-progress, final, cancelled, lease-expired and crashed results.
  • ExecutionBackend.reports_agent_start (class attribute, default False). A backend that records agent start itself sets it and calls RolloutContext.mark_agent_started (the first call wins); for any other backend the server records the time when execute() begins.
  • HarborBackend records agent start from Harbor's TrialEvent.AGENT_START, which fires after environment start, healthcheck and agent setup. A trial that fails during setup reports None; a failure inside the hook is logged and never fails the trial.
  • LocalBackend records agent start when a concurrency slot frees up, so a rollout that expires while queued behind max_concurrent reports None.
  • OPENSANDBOX_REQUEST_TIMEOUT_SEC sets the default request_timeout_sec for built-in OpenSandbox environments. It must be a positive whole number of seconds; anything else is ignored, and an explicit request_timeout_sec kwarg still wins.

Changed

  • docs/rollout-sdk.md and docs/architecture.md document the new field, the backend flag and the variable.

How to Test

  • uv run pytest tests/unit/rollout/test_result_registry.py tests/unit/rollout/test_local_backend.py tests/unit/rollout/test_server_admission.py tests/unit/rollout/test_harbor_backend.py tests/unit/rollout/test_harbor_contract.py tests/unit/rollout/test_harbor_source_images.py
  • test_harbor_contract.py runs real Harbor trials: the stamp lands between trial start and end, and a trial whose environment start fails leaves it None. It passes on Harbor 0.20.0, 0.23.0 and 0.24.0.

Checklist

  • PR title follows [module] type: description format
  • ruff check . and ruff format --check . pass
  • ty check --error-on-warning osmosis_ai/ passes; uv run --no-editable pyright --verifytypes osmosis_ai --ignoreexternal has no unexpected errors (only the Harbor harness_agent baseline)
  • pytest passes (new tests added)
  • Public API changes are documented
  • No secrets or credentials included