Improve real-world workflow conformance

Merged
BN
2mo ago
+3368-320

Summary

  • fix packed-golden ownership/setuid repair, hosted-image parity, passwordless sudo, and restart-time runner-pool recovery
  • correctly resolve workflow/job environment expressions and validate partial-clone snapshot caches
  • match results/cache URL shape and support both JSON and protobuf Twirp cache clients, including sccache's ghac backend
  • terminalize unhostable reusable-workflow callees instead of wedging concurrency groups
  • preserve diagnostics across worker crashes by reconciling active steps and falling back to uploaded step logs
  • make SmolVM force-delete retry transient nonempty-directory races
  • fix docker create output parsing so container:/services: jobs stop feeding docker's stderr to docker start
  • preserve expression-valued step inputs across job-message persist/restore (server restart no longer empties cache key: templates)
  • stage composite inner remote actions on demand, propagate composite inner-step failures, and resolve nested with: values against inner-step outputs
  • keep the hosted toolcache writable by the runner user
  • document both five-repository conformance campaigns and remaining environmental limitations

Mechanism parity with the official runner (v2.336.0)

A follow-up pass diffed every fix against the official source (DockerCommandManager.cs, CompositeActionHandler.cs, ActionManager.cs, JobExtension.cs, ResultsHttpClient.cs, runner-images configure scripts) and aligned the mechanisms, not just outcomes:

  • docker create ID: parse stdout only (first line), matching outputStrings.FirstOrDefault(); stderr is logged but never parsed.
  • Composite inner failures: keep running later steps and merge the result (MergeTaskResults), like CompositeActionHandler; condition-evaluation errors break like the official.
  • Toolcache: root:root, chmod -R 777, plus AGENT_TOOLSDIRECTORY/RUNNER_TOOL_CACHE in /etc/environment — matching configure-environment.sh.
  • Env evaluation: an unevaluable ${{ }} fails the step at execution (AssertString) and fails the job at build time — never shipped as a literal.
  • Job completion: CompleteJob.stepResults is authoritative (TaskResult strings/numbers applied per step); orphaned in-progress steps are reconciled only after.
  • Action staging: nested remote actions inside composite manifests are pre-downloaded recursively at prepare time (PrepareActionsRecursiveAsync), and on-demand staging resolves through the server's runnerresolve endpoint before the api.github.com fallback.

Campaign evidence

Campaign 1 (moby/moby, microsoft/TypeScript, astral-sh/ruff, neovim/neovim, nodejs/node): real checkout, toolchain installation, compilation, caches, reusable workflows, matrices, changed-file gates, and concurrency paths. Node's local jobs passed setup, protobuf-backed sccache, and full make build-ci before failing during Test without uploading final output; the resulting diagnostic gap is fixed, the unknown assertion is not claimed.

Campaign 2 (mastodon/mastodon, prometheus/prometheus, home-assistant/core, apache/kafka, vercel/next.js) targeted untested surfaces:

  • Service containers verified end to end: postgres:14-alpine and redis:7-alpine created, health-checked, started, and reachable from steps on their mapped ports.
  • Composite nested actions (ruby/setup-ruby, actions/setup-node, actions/setup-java) staged on demand and executed; inner-step failures no longer swallowed.
  • Dynamic matrices from job outputs (fromJSON(needs.*.outputs.versions)), reusable workflow_call with secrets inheritance and dual-branch checkout, and Rails assets:precompile all pass.
  • Remaining documented findings: changed-file actions fetch the snapshot base commit without auth after checkout cleanup (HA), amd64 job containers still need the Rosetta/binfmt path (prometheus), Gradle OOM under pool contention (kafka), and next.js's 31k-file workspace snapshot never completes a synchronous submit.

Verification

  • env -u PRELOOP_SYSTEM_TOKEN just test-ci
  • just dogfood with the official runner: job_succeeded=1
  • regression tests for the container-ID stdout parse, expression round-trip, composite failure merging, nested with: output resolution, authoritative stepResults, and the toolcache image parity

Summary by CodeRabbit

  • New Features

    • Added support for nested remote and composite actions, improved conditions, cleanup behavior, and workflow environment handling.
    • Added JSON and Protocol Buffers support for cache requests.
    • Added workflow step-result reporting, snapshot timing metrics, repository statistics, and run-timing diagnostics.
    • Added stricter expression and action-path validation.
    • Added compatibility with newer supported SmolVM releases and pinned runtime libraries.

  • Bug Fixes

    • Improved filesystem ownership, setuid handling, execution timeouts, queue recovery, concurrency handling, and orphan cleanup.
    • Improved workspace-cache recovery, run-log fallback, and action downloads.
    • Same-version updates now detect and repair changed installations.