prerelease-test skill: fold in the 0.10.1 campaign's process lessons

MA
19h ago
+443-47

All Submissions:

  • Have you followed the guidelines stated in CONTRIBUTING.md file?
  • Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change

  • New feature (non-breaking change which adds functionality) — tooling only: .claude/skills/prerelease-test/, no framework or package source.

Changes To Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you successfully ran tests with your changes locally?

Description

Updates the prerelease-test skill with what the 0.10.1a1 pre-release pass taught, so the next pass starts from a working setup instead of rediscovering it. Process only: no campaign results, findings or artifacts are included (those live on the testing/prerelease branch).

Workflow

  • Branch layout: the skill now points at the long-lived testing/prerelease branch: REGISTRY.md, reusable fixtures/, a runs/<date>-v<version>/ directory per pass, and a close-out step into history/.
  • Deliverables: adds CAMPAIGN_STATE.md and a filled AGENT_BRIEF.md, plus size limits for run bulk (about 2 MB per cluster, 10 MB per pass). The previous archive grew to 320 MB.
  • Picking the branch: when several r/pre-* branches share a date, test the one whose head carries the release tag.

New and changed scripts

  • scripts/bootstrap_envs.sh (new) builds the PyPI-only venvs (new, prev, ctrl-X, driver, new-ent, prev-ent) and prints every reflex package each one resolved. It fixes a baseline problem seen this pass:
    • With --prerelease=allow, the previous release's >= sibling floors resolved to the new alphas, so the "0.10.0 baseline" ran 0.10.1a1 code, moment and lucide among them. A final prev now resolves without pre-releases.
    • Installs are wheel-only, because uv reuses a cached sdist build for later installs of the same version.
  • scripts/check_wheel_sources.py (new) compares every source file changed between two tags with the installed copy. It replaces an ad-hoc shell loop.
  • scripts/drive_app.py now reports a page that rendered no text as an anomaly. A slow first dev load used to come back as "clean" with an empty body.

Checks added, each one surfaced a real finding this pass

  • Sibling floors of the reflex wheel versus a stock pip install -U reflex.
  • Wheels built from sdists losing .pyi stubs (isolated build env plus unpinned hook dependencies), with a throwaway uv cache.
  • Parity of security changes across enterprise surfaces, e.g. X-Forwarded-For over the REST API versus the websocket.
  • Standing coverage rows for:
    • frontend structural sharing with components that mutate state objects (AG Grid edits, recharts freezing);
    • import-time side effects under forked workers;
    • legacy form payload diffs;
    • network and security headers;
    • the hosting CLI against a fake control plane.

Agent brief

  • Start servers in their own process group and stop the group. pkill -f kills its own shell.
  • Check the port is free before starting a server; a stale server answered a test this pass.
  • --no-config and --only-binary installs.
  • Prod single-port setup, Redis, and websocket header injection.
  • More known-benign noise.
  • The structured final report the orchestrator parses.

Orchestration: a new "Surviving interruptions" section, written after this pass hit two usage limits and a spend limit. It covers periodic commits of the run directory, resuming agents with SendMessage instead of respawning them, and what to do when the configured model is unavailable.

Test plan

  • check_wheel_sources.py against the published 0.10.1a1 install: 77 identical, and only the 3 unpublished workspace packages reported as not installed. Against a tampered copy of reflex/app.py: DIFFERS, exit 1.
  • bootstrap_envs.sh NEW=0.10.1a1 PREV=0.10.0 in a fresh scratch dir: prev resolved every sibling to its final (core/code/moment 0.10.0, lucide 1.1.0, hosting-cli 0.2.0, build-sdk 0.1.0). The old script gave code/moment/lucide/hosting-cli alphas.
  • drive_app.py against a static page with an empty #root: EMPTY PAGE reported. A page with text: no empty-page report.
  • ruff check and ruff format are clean on the changed scripts; bash -n passes on the bootstrap script.

Generated by Claude Code