wally browser-use: a Rust browser agent that stops before payment

AM
1d ago
+14908-6

wally browser-use: a browser agent that does a web task in Chrome and stops before any payment. A decision model chooses each step from the controls on the page. Code executes it and decides what may never be clicked or typed.

It needs only the wally binary and Chrome: no Python, no uv. The loop is Jev Ultrafast's, ported to Rust in src/browser/. This PR includes #231 (the Rust port) and #232 (more sites).

How it works

  1. snapshot.js reads the page in one call: visible text and every control. That covers open shadow roots, same-origin iframes, inner scroll areas, native date inputs and button-shaped divs.
  2. One /systemone request asks the model for the next operation (click, type, select, scroll, done) and its target. The request type is generated from the pinned contract (createSystemOne).
  3. The guards run, then the action, then the page is read again. A decision made on a page that has changed since is thrown away.

Safety (in code, not the model)

  • Never pressed: pay and place-order buttons at any length, and commit phrases ("confirm booking", "book now", …) on any element. Single words like "buy" that also appear in product titles are skipped only on long links. Submitting a form that holds card, OTP or password fields is refused in any language.
  • Never typed into: card, CVV, OTP, UPI PIN or password fields (a "Card PIN code" is a credential, not an address). No expiry-style date fills. Text goes only to a plain field, checked just before typing.
  • Never repeated: a commit press, or the same typed text, on a page that hasn't changed.
  • Out of reach: cross-origin frames, which is where card widgets live.
  • Stop reason: every guard stop prints what it refused.

How it's tested

Offline, no network or key. The unit tests, plus 49 integration tests that run the real binary in headless Chrome against a local site and a fake model that always reaches for the riskiest control. 14 probes must end with a guard refusal and no paid or typed_card event.

cargo test --release --lib -- browser::
cargo test --release --test test_wally_browser          # all
cargo test --release --test test_wally_browser probe    # the 14 safety probes

Live with jev-latest on the Experiential gateway. Keep the key in your own shell:

export WALLY_BROWSER_API_BASE=https://api.experientiallabs.ai/v1
export WALLY_BROWSER_API_KEY=<your Experiential key>
wally browser-use -m jev-latest --text-model deepseek-v4.1-flash --start-url <site> "<task>"
SiteTaskResult
WikipediaOpen an articleDone
Google FlightsOne-way, 2 adults, a date, nonstopDone (jev sometimes loops; a rerun finishes)
Amazon.inSort by price, add to cartDone
Amazon.inBuy nowRefused "Buy Now"
FlipkartAdd a bottle to the cartDone
FlipkartBuy nowStops with nothing bought

CI

All jobs, including windows and windows-arm64, passed on b5e576d.

Not covered yet

  • Cross-origin iframes (on purpose), file uploads, drag-and-drop, canvas UIs and closed shadow roots.
  • Password logins: sign in yourself in the Wally Chrome profile, or use --chrome attach.
  • Pages with more than 64 controls get split into several questions, which costs accuracy and speed.

Summary by cubic

Adds wally browser-use, an eve-driven browser agent that completes a web task in Chrome and stops before any payment, and raises wally decisions' choice limit from 26 to 255 options. Decision models stay refused on chat surfaces so they never route to text generation.

New Features

  • The agent runs as a native Rust loop over one CDP WebSocket to Chrome's loopback DevTools endpoint, with no per-step subprocess; the page image goes with each step and the results tab stays open when the run ends.
  • A dedicated Chrome on port 9334 reuses a profile that keeps cookies and logins between runs, or the agent attaches to a Chrome the person already runs with remote debugging. Chrome survives interrupts and the run ends at the session-key deadline.
  • Hard stops live in code: payment controls (including charge), price-only buttons, and card/CVV/OTP/UPI fields are never acted on or typed into, while postal PIN codes stay fillable as address fields. Book/confirm/reserve verbs need the person's yes, coordinate clicks are refused, dialogs are dismissed, and bot checks hand the window to the person.
  • Guards treat page text as untrusted, normalize before matching, re-read the live element, check values as well as fields, track each value's source, and confirm every action with a kept profile or in attach mode; targets inside a chunk eve refused are rejected.
  • The session key reaches the agent through the environment, decision and text calls retry through the gateway, and a signed-in session is refreshed ahead of expiry.
  • Round-2 through round-6 adversarial probes are pinned as tests and verified against real Stripe and Braintree fixtures.

Summary by CodeRabbit

  • New Features
    • Added a browser-use command that can navigate websites and complete tasks from natural-language goals. Configure the starting page, browser profile or Chrome connection, and step limit.
    • Added a local browser inspector for viewing agent activity and issuing commands.
    • Browser actions verify that page controls are current and usable, and may stop before sensitive payment or personal-data actions.
    • The browser remains open when a task finishes.