PR #3043 - modelcontextprotocol/python-sdk

De-flakes the conformance CI entirely on the python-sdk side: unexpected suite failures are re-verified solo before failing the job, the interpreter spawn storm that causes the timing flakes is reduced at the source, results are uploaded as artifacts whenever they matter, and the one server-side race in the everything-server is fixed.

Motivation and Context

sse-retry fails the client conformance job on ~4–5% of runs (including pushes to main, e.g. https://github.com/modelcontextprotocol/python-sdk/actions/runs/28443055485), always the same way: the harness runs all ~40 scenarios' client subprocesses concurrently on a 2-vCPU runner, and the resulting contention pushes the scenario's wall-clock reconnect measurement into its warning band. The client is spec-compliant — the overshoot is scheduler latency, not SDK behavior (idle the measured delay is ~503ms against a 700ms gate; with the suite pinned to 2 cores it reaches ~670ms+).

Three changes, all local to this repo:

run-client.sh — solo re-verification, not blind retries. On suite failure, each scenario listed under "Unexpected failures (not in baseline)" is re-run alone on the then-quiet runner (once by default; CONFORMANCE_SOLO_ATTEMPTS overrides). Solo, a compliant client passes the timing check deterministically, while anything genuinely broken fails solo too — so this is flake detection, not flake hiding. Guard rails so it cannot green-wash anything else: stale-baseline errors are never retried; an empty or unparseable failure list passes the original exit code through; extracted names are validated before use; a rescued run writes a FLAKE_RESCUED marker into the results dir so the rescue is visible and its evidence is uploaded. The deliberate trade-off: failures that only reproduce under 40-way parallel load are excused — the suite asserts spec compliance, not behavior under contention.
Spawn-storm reduction. uv sync --compile-bytecode (plus pre-compiling the editable sources) stops ~40 fresh interpreters racing to byte-compile the same modules during the measurement window, and the client command execs .venv/bin/python directly instead of paying uv run's lockfile re-check in every concurrent spawn. This shrinks the contention that causes the flake in the first place.
related_request_id in the everything-server's test_sampling. Without it, the sampling request routes to the standalone GET stream and is silently dropped if the harness client hasn't finished opening that stream (~2ms race), hanging tools-call-sampling for the full 60s client timeout — the server-side flake observed on main.

Both jobs now save --output-dir results and upload them when the job fails or a flake was rescued, so every future incident carries its measured numbers (today the suite summary prints only counts for warning-level findings).

Related: the upstream scenario fix is proposed in modelcontextprotocol/conformance#375; this PR makes python-sdk CI robust regardless of when/how that lands, and the wrapper simply never fires once the upstream fix is pinned.

How Has This Been Tested?

The wrapper was driven end-to-end against a locally-packed harness tarball identical in behavior to the CI pin, with real and simulated clients:

healthy suite → pass-through, exit 0 (421 checks, baseline clean)
one-shot-flaky client → suite red, solo rerun passes → exit 0, FLAKE_RESCUED marker written, solo results saved
persistently-broken client → both solo attempts fail → exit 1, solo evidence saved, no marker
stale baseline entry co-occurring with a flake → exit 1 without any reruns ("not retrying")
infra-shaped failure (no unexpected-failures section) → original exit code passed through
CONFORMANCE_PKG unset → clean guard message; equals-form flags filtered; shellcheck clean

The everything-server fix was verified both directions over streamable HTTP with no GET stream open: on the base code the sampling request is dropped and the call times out; with the fix it arrives on the tools/call POST stream and completes (also passes the harness's tools-call-sampling scenario 1/1).

Breaking Changes

None. CI-only plus one example-server fix.

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist

I have read the MCP Documentation
My code follows the repository's style guidelines
New and existing tests pass locally
I have added appropriate error handling
I have added or updated documentation as needed (n/a — CI internals)

Additional context

The solo reruns intentionally drop --expected-failures: a scenario in the unexpected list is by definition not baselined, so the solo verdict needs no baseline context (and single-scenario semantics are a strict superset — warnings and client errors still fail). Solo results land in <output-dir>-solo next to the suite's, so uploaded artifacts are self-contained for both the rescued and the still-fails cases.

_{AI Disclaimer}