Gate the test matrix and retry setup-uv's flaky manifest fetch
Two paired CI fixes: make the required all-green check actually gate the test matrix, and stop a known setup-uv flake from becoming merge-blocking once it does.
Motivation and Context
The gate. The test matrix job in shared.yml has had job-level continue-on-error: true since #708 (May 2025). That flag makes the checks reusable workflow report success to all-green no matter what the matrix does, so the only required status check passes even when every test leg fails. Concrete example: in run 28935462867 the test (3.10, locked, ubuntu-latest) job failed, yet the run concluded success and all-green passed. Scanning the last 200 CI runs (8 days): 14 green runs had all 20 test legs failing. Failures are still visible as red ✗ on the PR checks list — which is why nothing broken has actually landed — but the gate itself is decorative, and RELEASE.md had to document the manual workaround ("a green workflow run does not mean the tests passed").
The flag made sense in #708 as insulation for the then-new, then-flaky Windows matrix (the old hang issues), but it silences every leg, not just Windows, and the Windows instability it guarded against was fixed (most recently #3070). Removing it needs a companion: job-level continue-on-error was also suppressing fail-fast cancellation, so fail-fast: false is added to keep today's run-all-legs behavior (identical compute; without it, the first failing leg would cancel the other 19 and destroy per-leg signal).
The flake. With the gate re-armed, a known setup-uv flake would become merge-blocking: the action fetches its version manifest from raw.githubusercontent.com in a single request with a hardcoded 5-second timeout and no retry (astral-sh/setup-uv#869; the timeout was introduced by astral-sh/setup-uv#883, first shipped in v8.2.0 — the version pinned here). The fetch happens even with an exact version: pin because the download URL only exists in the manifest, and it is unfixed through v8.3.2 (the fetch code is byte-identical), so bumping the pin doesn't help. It hit this repo 3 times in the last ~19 days (1, 2, 3) — roughly 0.06% of jobs, which across the 23 gating setup-uv executions per run would flake-block a run every couple of days.
Each setup-uv step in shared.yml therefore becomes a pair: the first attempt has step-level continue-on-error: true, and an identical retry step runs on steps.setup-uv.outcome == 'failure' (outcome is the pre-continue-on-error result; failure() would never fire here). The retry step has no continue-on-error, so deterministic errors — bad version, real checksum mismatch — fail both attempts and the job goes red; nothing is masked. Same pattern as e.g. apache/shenyu's setup-java retry. The steps are duplicated inline on purpose; a composite action felt like more machinery than a temporary workaround deserves — these steps should be deleted once upstream adds a retry.
Notes:
- A flake absorbed by the retry no longer shows a red ✗ on the PR — look for a non-skipped "Install uv (retry)" step in the run. The failed first attempt also leaves a harmless "Error retrieving cache key from state." warning from its post step; expected, not a cache bug.
- The setup-uv steps in
conformance.yml,deploy-docs.yml,docs-preview.yml, andpublish-pypi.ymlstay single-shot on purpose: they don't feedall-green, no flakes were observed there (300 conformance runs scanned), and the release/preview workflows are better left churn-free here. The pattern is trivially copyable if one of them starts flaking. - Side effect of the re-armed gate:
release-buildinpublish-pypi.ymlhasneeds: [checks], so releases become genuinely test-gated too, and the RELEASE.md caveat about checking individual jobs is now stale (removed here).
How Has This Been Tested?
- actionlint, zizmor, and the repo pre-commit hooks pass with no findings (zizmor output identical to the current file).
- The retry wiring (
continue-on-error+outcomesemantics) was exercised withact: retry fires on first-attempt failure, is skipped on success, and a double failure fails the job. - Live validation on this PR (temporary commits, since dropped): one commit pointed the first setup-uv attempt at an unreachable
manifest-fileURL to force the flake — the retry fired on all 20 legs and the jobs stayed green; one commit added a deliberately failing test —all-greenwent red, confirming the re-armed gate blocks.
Breaking Changes
None for SDK users. For contributors: PRs with failing tests now fail all-green instead of showing a green run with red legs.
Types of changes
- Bug fix (non-breaking change which fixes an issue)
Checklist
- I have read the MCP Documentation
- My code follows the repository's style guidelines
- New and existing tests pass locally
- I have added appropriate error handling
- I have added or updated documentation as needed
Additional context
The upstream root cause is fixable in setup-uv itself (a retry in its fetch wrapper, or skipping the manifest for exact version pins); happy to take that to astral-sh/setup-uv#869 separately. When upstream ships a retry, the duplicated steps here should be deleted — the inline comments say so.