fix(mobile): fail an action agent-device refuses for an unknown screen origin as an action failure

GR
4h ago
+44-11
  • agent-device 0.21.37 (callstack/agent-device#4439) refuses an iOS ref action with COMMAND_FAILED, details.reason: window-origin-unknown, dispatched: no when its capture can't place the window on screen. It happens on a full-screen iPhone app right after a large web view loads. While that screen stays up the runner reads the tree through its private AX fallback, which never gets the viewport back (measured over 150 s).
  • The engine reported the refusal as ENGINE_FAILURE, an infrastructure error (exit 3), although nothing was sent and the device is fine. It now maps to NOT_ACTIONABLE (ACTION_FAILED, exit 1) in staleOr, next to the covered-target refusals from #1686. A refusal with dispatched: unknown stays ENGINE_FAILURE.
  • agent-device's hint names its CLI (Take a fresh snapshot … or use screenshot …). The message carries the test's move instead: Nothing was sent. For a tap, tap the point instead with screen.tapAt(). A point tap isn't aimed from the capture, so agent-device takes it; other actions have no point form.
  • No retry: re-observing returns the same capture state for as long as the screen stays heavy, so a retry only moves the failure to the deadline.
  • The refused reasons and their hints are one map, NOT_ACTIONABLE_REASONS. undefined keeps agent-device's hint, and a string replaces its CLI advice.
  • No docs change: callstack/agent-device#4573 (merged) makes agent-device accept a full-screen root, so the web view case goes away at the next agent-device bump, and the error hint names the move.

Issue

Fixes #1821

Verified

Ran it locally: yes

  • Repro: gronxb/e2e-repro, a UIKit app with a WKWebView that loads 3,000 links. The test reloads it and taps a native button, three rounds. iPhone 17 Pro Max simulator, iOS 26.5, agent-device 0.21.37.
  • e2e run on main's published packages and on this branch's packed e2e, @e2e-dev/core, @e2e-dev/mobile:
main (@e2e-dev/mobile@0.12.7)this branch
✗ locator.tap getByTestId("reload-web") — ENGINE_FAILURE, exit 3✗ locator.tap getByTestId("reload-web") — ACTION_FAILED, exit 1
hint: Take a fresh snapshot after the accessibility channel recovers, or use screenshot to choose screen coordinates.hint: Nothing was sent. For a tap, tap the point instead with screen.tapAt().
  • The documented move works: the same three rounds with screen.tapAt({ x: 220, y: 98 }) pass on 0.21.37.
  • errors.test.ts: the new case fails on main (ENGINE_FAILURE) and passes here.