chore: add agent-run dashboard flow checks and Magellan explorer

Draft
AD
4d ago
+1415-0

DES-432

Summary

Two agent-driven tools for catching dashboard UX drift. Neither blocks merges.

Flow checks (e2e/flows/) protect known flows:

  • Each flow is a plain-English story.md (Given / When / Then, with every Then proven by an /rpc/ call, not by text on screen) next to a Playwright flow.spec.ts that a Claude Code agent compiled from it.
  • mise run test:flows replays the scripts. It needs no LLM, so it is cheap and repeatable.
  • mise run flows:heal <flow> gives the story, the failure and the branch diff to claude -p (e2e/flows/HEAL.md). It returns one verdict as JSON:
    • healed: the UI moved, so the script is rewritten.
    • broken: the flow can't be finished; you get the failing step and a screenshot.
    • changed: the PR changes the flow on purpose, so the story needs an edit.
  • .github/workflows/flow-checks.yml boots the stack the same way wake-stack-e2e does, replays the scripts, heals only the failing flows, and keeps one sticky PR comment updated.
  • Three starter flows:
    • server-from-source: fully repeatable, a good per-PR check.
    • assistant-create: uses the LLM builder.
    • catalog-connect: uses the live registry and an upstream server.

Magellan (e2e/magellan/) is a nightly explorer for problems no story covers. It runs against production, in a dedicated integration org, as a bot user:

  • mise run magellan:explore gives the agent the dashboard files merged to main recently and lets it explore those pages, then roam.
  • A finding needs hard evidence (console error, failed RPC, dead end, race, or a control with no accessible name) and must reproduce twice. A feature that is off, or a permission the bot lacks, is not a finding unless the page gives no explanation.
  • mise run magellan:file turns findings into Linear Triage issues. Screenshots go to Linear's own storage. Each issue gets a stable fingerprint, so a bug seen on several nights stays one issue, and a draft story.md that can become a flow check once the bug is fixed. It is a dry run unless --file is passed.
  • Login: WorkOS login can't be scripted, so the bot reuses a captured gram_session, which is valid while it is used at least once every 72 hours. A daily keepalive job calls auth.info to keep it alive and fails with re-capture steps when the session is gone. Setup is in e2e/magellan/README.md.
  • Agent rules for a shared org: it touches only data named Magellan <time>, deletes that data before it finishes, never changes org settings, and never sends anything outside the product.
  • Public repo: the workflow uploads no artifacts, and CI logs show only finding fingerprints. The session state file sits outside the run directory and is deleted on exit.
  • Secrets and variables live in a magellan GitHub environment. Variables: MAGELLAN_SITE_URL, MAGELLAN_ORG. Secrets: MAGELLAN_SESSION, ANTHROPIC_API_KEY, MAGELLAN_LINEAR_API_KEY.
  • The local stack still works for trying out prompt changes.

Motivation

Changes can quietly break whether a user can finish a flow, and nothing catches it today. Running an agent as the per-PR gate would be slow, costly and too willing to say "pass". Here the agent writes and repairs scripts, the scripts do the gating, and the open-ended exploring runs nightly instead.

Findings from the spike:

  • Heal runs take 6–16 minutes each. That is why only failing flows get healed.
  • Flows that depend on an LLM or a live upstream server are better suited to the nightly run than to every PR.
  • Magellan's first local run reported findings that turned out to come from the local stack: feature gates off, and state left behind by earlier runs. That is why Magellan now targets production.

Neither workflow has run in CI yet. Magellan also needs a bot user and its session captured once by hand before its first run.

Gutternote review steps

No user-visible change: this PR adds tooling only.