Vision & Screenshots

Relevant source files

The following files were used as context for generating this wiki page:

• browser_use/browser/context.py
• browser_use/agent/service.py
• browser_use/agent/prompts.py
• browser_use/dom/buildDomTree.js
• browser_use/agent/message_manager/service.py
• browser_use/agent/views.py

Vision and Screenshots represent the multimodal sensory layer of the browser-use framework. This system enables AI agents to "see" the web by capturing real-time visual states, processing them into structured data, and overlaying interactive metadata for better reasoning and validation. By combining raw pixel data (screenshots) with DOM-derived visual context (bounding boxes and labels), the system bridges the gap between traditional text-based automation and advanced visual LLM capabilities.

The primary purpose of this module is to provide agents with a visual ground truth that supplements the DOM tree. It handles everything from the initial capture of base64-encoded screenshots to the generation of complex visual artifacts like diagnostic GIFs that document the agent's decision-making process over time.

Screenshot Acquisition and Processing

The core of the vision system resides in the BrowserContext, which interfaces with Playwright to capture the browser's current viewport or the full page. Screenshots are converted into base64-encoded strings and stored within the BrowserState object.

Capture Mechanism

Screenshots are taken using the take_screenshot method, which allows for either viewport-only or full-page captures. These images are essential for multimodal models to verify the layout, handle overlapping elements, and identify visual cues that may not be explicitly captured in the DOM text.

Sources: browser_use/browser/context.py:448-463, browser_use/agent/prompts.py:64-70

The diagram above shows the lifecycle of a screenshot from the initial agent step to its inclusion in the message history.

Visual Context and Highlighting

To assist multimodal models in mapping textual descriptions to visual elements, the system employs a sophisticated highlighting mechanism. This is achieved via the buildDomTree.js script, which injects a highlight container into the browser's document.

Highlighting Logic

Interactive elements identified by the DOM service are assigned a highlightIndex. The system then creates visual overlays (rectangles and labels) directly on the web page. These overlays use a rotating color palette to ensure distinctiveness.

• Highlight Overlay: A div with a border and 10% opacity background color.
• Labels: Small, colored badges typically placed at the top-right of the element, containing the highlightIndex.
• Coordinates: The system calculates both viewportCoordinates (relative to the viewport) and pageCoordinates (relative to the entire document) to ensure accurate placement even when the page is scrolled.

Sources: browser_use/dom/buildDomTree.js:34-123, browser_use/dom/buildDomTree.js:462-520

Vision Prompting

When vision is enabled, the AgentMessagePrompt formats the human message as a list containing both the text description of the state and the image URL (data URI). This allows models like GPT-4o to "look" at the labeled image and correlate it with the interactive elements list.

Sources: browser_use/agent/prompts.py:165-177, browser_use/agent/message_manager/service.py:146-160

Visual History and GIF Generation

A significant feature of the Vision module is the ability to generate a chronological GIF of the agent's actions. This serves as a diagnostic tool for developers to review what the agent "saw" at each step.

History Reconstruction

The Agent class maintains an AgentHistoryList. Each AgentHistory item contains a BrowserStateHistory object which stores the screenshot from that specific step.

GIF Overlay Architecture

The create_history_gif function processes these screenshots and applies several overlays:

1. Task Frame: An initial frame displaying the ultimate task in a large, centered font.
2. Step Overlays: Each subsequent frame includes the step number (bottom left) and the "Next Goal" (centered bottom).
3. Logo: An optional project logo in the corner.

Sources: browser_use/agent/service.py:734-802, browser_use/agent/views.py:77-88

Parameter	Description	Source
duration	Time (ms) each frame is displayed in the GIF.	Agent.service:736
show_goals	Whether to overlay the agent's next goal on the frame.	Agent.service:737
viewport_expansion	Pixels added to the viewport for element visibility.	BrowserContext.py:91
image_tokens	The token cost assigned to an image in the LLM context.	MessageManager.py:44

Validation through Vision

The system includes an automated validation step where the LLM acts as a "validator." This validator uses the latest screenshot to determine if the task was completed successfully or if the visual state contradicts the agent's internal reasoning.

Implementation Details

Browser Configuration for Vision

The BrowserContextConfig allows fine-tuning how visual state is handled, including resolution and element highlighting.

# Configuration snippet from browser_use/browser/context.py:42-105
@dataclass
class BrowserContextConfig:
    browser_window_size: BrowserContextWindowSize = field(
        default_factory=lambda: {'width': 1280, 'height': 1100}
    )
    highlight_elements: bool = True
    viewport_expansion: int = 500  # Expand what the "vision" sees

Cleanup

To prevent diagnostic overlays from interfering with subsequent screenshots or clicks, the system includes a remove_highlights method. This script removes the playwright-highlight-container and strips the browser-user-highlight-id attributes from DOM elements.

Sources: browser_use/browser/context.py:465-492

Summary

Vision and Screenshots provide the essential visual feedback loop for browser-use agents. By combining base64 image capture with precise DOM-aligned highlighting, the project enables multimodal models to operate with spatial awareness. The integration of visual validation and GIF generation ensures that the agent's actions are not only performant but also transparent and auditable by humans.