fix(ios): make back the verified left-edge swipe

Merged
TH
7h ago
+1296-676

Summary

On a header-less screen with its own Back button (React Native native-stack, headerShown: false), iOS back refused: it guessed at buttons and found none. Now iOS back (with or without --in-app or --system) is the left-edge back swipe, verified by a change of the app's screen. An agent presses a visible control by its ref itself.

How the swipe is sent

  • Simulator: one edge-masked HID touch, the right-edge preset; parity is pinned in contracts/fixtures/back-swipe-geometry.json. An XCTest drag starting 20 pt in popped 2 in 6; the edge touch popped 6 in 6.
  • Physical device: a hold-free synthesized fling, 200 ms, from 2 pt inside the leading edge to 0.9 of the width, safe on a tree XCTest cannot serialize. There is no XCTest drag any more. XCTest's press-and-drag holds at the edge for at least 0.1 s and then moves at 500 px/s, and it missed 2 in 10 on an iPhone 17 Pro. The fling popped 10 in 10 there (evidence).
  • Preparation: both swipes are resolved before the sample taken just before the swipe, and every tree and window read happens then, so the gap holds only the contact.

How the change is verified

  • Comparison: each sample is a 48×96 grayscale grid. Its top 7 rows are always left out: the portrait status bar and Dynamic Island, whether shown or hidden. The status bar is never read. A change means more than 1% of the remaining cells moved by more than 24 of 256 levels.
  • Settle: time-based, at most 8 samples 0.25 s apart. The screen must be still for a contact allowance (0.2 s on a simulator, 0.4 s on a device) plus 0.25 s plus the slowest capture seen while settling, so slow captures settle longer.
  • Credit: a change counts only from the just-before sample, and only when the gap from it to the after sample did not outrun the still span. All three times are taken when a capture completed. Otherwise the result is back_swipe_unverified.
  • Log: every exit logs AGENT_DEVICE_RUNNER_BACK_SWIPE_VERIFICATION with outcome, code, changed, stableMs, justBeforeToAfterMs, gestureMs, maxCaptureMs, settleSamples and settled.

Reasons and removals

  • Reasons: back_swipe_no_change (sent, screen unchanged), back_swipe_unverified (sent, unknown outcome), back_swipe_not_dispatched (dispatched: no).
  • visionOS: refused by the daemon at the back capability fact with UNSUPPORTED_OPERATION. The runner's visionOS arm answers only a direct request, with BACK_SWIPE_NOT_DISPATCHED.
  • Deleted: the button lookup, the blind tap, backSystem (recorded in RUNNER_PROTOCOL.md), and the device XCTest drag.
  • Moved: the long-press, sequence and planned-gesture dispatch disclosure moved to #4474. This PR adds no envelope field.
  • Rebased onto #4474 as one commit. PreparedTouch(notSent:) is an exhaustive switch; #4474's .notDispatched maps to the same pre-touch refusal as .unsupported (BACK_SWIPE_NOT_DISPATCHED for back), so nothing traps at run time.

Validation

Physical-iPhone verification was skipped by maintainer decision. This PR merges on simulator evidence.

Simulator validation at 1a1d0ea, rebased onto #4474, on an iPhone 17e simulator with iOS 27:

  • Runner XCTests: NavigationTests, SynthesizedInteractionTests and ScrollDragExecutionTests ran 74 tests, with 0 failures and 1 environment skip. They cover:
    • swipe, no change, still-changing and slow-toggle screens;
    • hidden status bar, lit panel, rejected-query route;
    • slow host → performed, and 0.4 s captures with a 0.2 s contact → performed;
    • a 1.5 s contact → BACK_SWIPE_UNVERIFIED;
    • the device fling on readable and unserializable trees (no window query in the gap, no XCTest press);
    • #4474's synthesized-interaction tests.
  • Builds: the runner builds for iOS simulator, iOS device, macOS, tvOS and visionOS.
  • Replays: 01-settings.ad and 02-deep-navigation.ad pass (back 1.65 s each), including under host CPU load at an earlier head. 01-physical-lifecycle.ad passes when run on the simulator.
  • CI: the iOS 26.2 simulator smoke replays.

Earlier hardware data: iPhone 17 Pro, iOS 27.0.1, before the fling became the product path (evidence). The synthesized 200 ms fling popped 10 of 10. Verification outcomes were not measured on hardware at the final head.

Known risk: on heavy-tree screens, a physical-iPhone back may report BACK_SWIPE_UNVERIFIED even though the screen popped. That happens when the gap from the just-before sample to the after sample exceeds the device still span; one device capture took about 0.4 s. The runner log line AGENT_DEVICE_RUNNER_BACK_SWIPE_VERIFICATION records stableMs, justBeforeToAfterMs, gestureMs and maxCaptureMs, so the device still span can be sized from real numbers later, without weakening the gap check.

Scenario (simulator, CLI = MCP)base 799e9c0head
Header screen→ Home→ Home
Header-less Productrefused→ Products
Rootrefused after blind tapback_swipe_no_change
back ∥ back / back ∥ pressrefused→ Home / → Products

Risk: an app with a left-edge drawer may open the drawer instead.

🤖 Generated with Claude Code