Add goal-oriented Agent Sandboxes guides

DM
2d ago
+1965-140

Goal-oriented guides for Agent Sandboxes, stacked on #2658. Bill added UI guides (workbench, profiles, inspect) and hardening on this branch. The base is Samuel's agent-sandboxes branch, so this diff shows only the new work.

New pages

  • Build with Agent Sandboxes:
    • answer questions that need dozens of tool calls;
    • work with large tool results;
    • give an agent a whole API (typed module plus pre-run type check);
    • pause for human approval and resume later;
    • keep work across conversation turns.
  • Connect your agent: Pydantic AI (CodeMode(monty_pool_factory=logfire.agent_sandboxes), pydantic/pydantic-ai#11179) and any agent framework (a run_code tool).
  • What sandbox code can do: the Python subset, the 50 ms and 20 MiB limits, storage limits, and a "Coming soon" list for pricing, regions, and self-hosting.
  • Overview: an Experimental marker, and a table comparing one-tool-call-per-turn, container sandboxes, and Agent Sandboxes.

Changes to #2658's pages

  • Correct key-creation path (Agent Sandboxes → Setup → Create execution key), and the real permission label.
  • Fixed admonition indentation.
  • Raised the agent-sandboxes extra to pydantic-monty-client>=1.3.0, to pick up the 1.3.0 security fix and ExternalModule.

Measured, not asserted. The claims come from real-model runs with Pydantic AI on staging-eu (3 runs each, strict grading):

ScenarioModelPlain tool callsAgent Sandboxes
Availability from 1,680 rows of request statsgpt-6.1-sol1/3 correct, 111k input tokens3/3, 4.7k
samegpt-6-luna0/3, 180k3/3 (but it returned raw rows, so no token saving)
Deploy/error correlation (2,160 rows)gpt-6.1-sol3/3, 70.5k3/3, 4.0k
same, 288 rowsgpt-6.1-sol3/3, 10.1k3/3, 22.6k (no saving at this size)
Refund 37 orders with approvalgpt-6.1-sol37 approval prompts, 2 model requests (~26.4k tokens) after approval1 approval prompt, 0 model calls after approval

The approval guide's own code (with Bill's plan-matching hardening) ran 5/5 with gpt-6.1-sol. That needed two additions:

  • tell the model it is not called again after approval;
  • type-check before running.

How it was checked

  • Every code block was run. Outputs on the pages are real.
  • Hosted runs used the GCP test deployment and staging-eu. Pages that test deployment couldn't serve were run against a local Monty relay, or locally with production limits.
  • Approval and Pydantic AI pages were also run end to end with a real model.
  • Two independent review passes: adversarial accuracy and newcomer persona.

Before this ships

  • The Pydantic AI page needs a pydantic-ai-harness release that includes #11179.

  • Recheck on staging/prod for:

    • parallel host calls under asyncio.gather: they were nearly serialized on the test deployment;
    • type-check memory: about 5 MiB is charged when a script imports asyncio.

    Both are reported to the Monty team.

  • sessions.md has two claims I could not verify on hosted: resume after the 1-hour connection limit, and how restored limits interact with checkout() options.

Drafted with Claude (Opus 5.5), with my review and direction.