Improve browser research reliability and bound context growth

Draft
GR
2mo ago
+95-293

Summary

  • replace the generic GPT prompt and long browser skill with a compact browser-research contract
  • keep full browser output on disk while limiting model-visible tool output to 8 KiB / 400 lines
  • use medium response verbosity for GPT-5.5 and make final-answer delivery explicit
  • cap individual browser snippets at two minutes and tell the agent to switch strategy after three no-evidence calls

This stays pure TypeScript/JavaScript. It does not add Python, a notebook runtime, a new browser API, or task-specific benchmark logic.

Evaluation

Internal Bench Hard, all 106 tasks, GPT-5.5, medium reasoning:

VersionPassesAgent costObserved trace cost
Original BrowserCode baseline81/106$176.92$236.50
Immediate pre-guardrail recipe86/106$152.76$196.66
This branch87/106$132.26$181.02

Against the immediate predecessor, this is +1 pass with 13.4% lower agent cost, 18.6% fewer cached input tokens, 7.9% fewer uncached input tokens, and 6.2% fewer output tokens. Browser calls increased 9.9%, so the saving comes from leaner model context rather than less browsing.

The 81/106 comparison spans a larger target evolution and should be read as a product-level comparison, not an isolated causal A/B. The 86 -> 87 comparison used the same eval-platform commit and isolates the final evidence-budget commit.

Verification

  • browser-execute tests: 7 passed, 6 skipped
  • browser skill tests: 2 passed
  • provider transform tests: 354 passed
  • filtered package typechecks: 17/17 passed

Summary by cubic

Bounds browser research behavior and model context to improve reliability and cost. Previously we used a generic prompt and surfaced full tool output; now we use a compact browser-research contract, persist the CDP session, cap per-call time at 2 minutes, and limit model-visible tool output to 8 KiB/400 lines while saving the full output to disk.

  • browser_execute now returns compact context with full output saved. Tool metadata includes truncated and outputPath when truncation occurs. Screenshots still auto-attach.
  • Max per-call timeout reduced from 10 minutes to 2 minutes.
  • Skill doc rewritten to emphasize short deterministic snippets, event-before-action, evidence discipline, reuse via await import(...?t=...), and switching strategy after three no-evidence calls.
  • Prompt specialized for browser research and requires final responses to start with "FINAL ANSWER:"; stresses exhaustive, evidence-backed delivery.
  • gpt-5.5 models use medium textVerbosity; tests cover the new default.

Review/Rollout

  • Required: If any flows relied on >2 minute snippets, split them into smaller calls.
  • Required: If anything parsed full console output from the tool result, read the saved file at metadata.outputPath or use focused Grep/Read instead.
  • Update expectations for gpt-5.5 verbosity if tests asserted low.
  • Expected impact: +1 pass on internal bench (87/106) with ~13% lower agent cost; savings come from tighter context rather than fewer browser calls.