feat(agent): hold modifier keys on tap, double tap, and right click

Merged
ZI
6d ago
+317-38

The agent's tap, double_tap, and right_click tools take an optional modifiers list when the target's engine declares tapModifiers, in both agent.act and the MCP server. An agent can now Shift-click or Ctrl-click the way the test API's tap({ modifiers }) can since #626.

{ tool: 'tap', args: { target: 'n42', modifiers: ['Shift'] } }
  • ctx.target.tapModifiers tells executors whether the target can hold keys for a click. The grammar tools read it, so agent.act, MCP, and custom executors use the same flag. Engines without it get the same schema as before.
  • ctx.actions.tap, doubleTap, and secondaryTap take { modifiers }. Bad values fail with INVALID_ARGUMENT. An engine that doesn't declare tapModifiers refuses them with UNSUPPORTED_CAPABILITY.
  • Held keys are recorded with the action (tap button "Save" with Shift held) and replayed from the cache. The trace reader rejects an entry whose modifiers are malformed or sit on a verb that can't hold them.
  • Docs: reference/agent.mdx, executors.mdx, reference/mcp.mdx, skills/e2e/references/mcp.md. Changeset: e2e patch.

Closes #953

Verified

Ran it locally: yes, on the testbed /controls page with a temporary click-log button that records event.shiftKey and the other modifier flags.

MCP, built server, driven with an MCP client:

>> tap {"target":"n6062","modifiers":["Shift"]}            Tapped #n6062.
>> tap {"target":"n6062"}                                  Tapped #n6062.
>> double_tap {"target":"n6062","modifiers":["Alt"]}       Double-tapped #n6062.
>> right_click {"target":"n6062","modifiers":["ControlOrMeta"]}  Right-clicked #n6062.
>> tap {"target":"n6062","modifiers":["Bogus"]}   INVALID_ARGUMENT: call tap: modifiers.0: Invalid option ...
>> hover {"target":"n6062","modifiers":["Shift"]} INVALID_ARGUMENT: call hover: Unrecognized key: "modifiers"
status "Click log" text="click[shiftKey] click[] click[altKey] click[altKey] dblclick[altKey] contextmenu[metaKey]"

agent.act('click the "Click log target" button once while holding Shift') with toHaveText('click[shiftKey]'), live and then from the cache:

✓ the agent shift-clicks 5.52s ai 9.6k tokens · 2 model calls   Cache 1 missed
✓ the agent shift-clicks 331ms                                  Cache 1 replayed

The recorded entry: { "name": "tap", "summary": "tap button \"Click log target\" with Shift held", "modifiers": ["Shift"] }.

Unit: agent-tap-modifiers.test.ts (7 tests, including trace rejection), full e2e unit suite 2663 passed, decision 157 passed. pnpm typecheck and oxlint are clean.