Evaluation & Benchmarking
Relevant source files
The following files were used as context for generating this wiki page:
Evaluation and Benchmarking in the browser-use framework encompasses the systematic testing of AI agents against standardized datasets, the validation of agent outputs through specialized LLM validators, and the analysis of history and performance metrics. The system is designed to measure how accurately agents can navigate complex web structures and complete tasks across diverse websites.
The benchmarking suite utilizes datasets like Mind2Web to provide a ground truth for browser interactions. This involves loading processed test cases, executing them via the Agent class, and validating the state transitions and final outputs to ensure the model's performance meets the required technical standards.
Sources: tests/test_mind2web.py:1-20, browser_use/agent/service.py:1-50
Dataset-Based Benchmarking
The project supports benchmarking using the Mind2Web dataset, which consists of real-world web navigation tasks. The benchmarking flow involves loading JSON-formatted test cases and running them through the agent to evaluate success rates.
Benchmarking Workflow
- Loading Data: Test cases are loaded from
mind2web_data/processed.json. - Session Setup: Playwright browsers are initialized in headless mode for efficient batch processing.
- Task Execution: The
Agentis instantiated with a specificconfirmed_taskfrom the dataset. - Integrity Checks: The system verifies that each test case contains necessary fields like
website,confirmed_task, andaction_reprs.
The diagram shows the sequence of operations for running a benchmark suite against the Mind2Web dataset.
Sources: tests/test_mind2web.py:27-55, tests/test_mind2web.py:90-125
Automated Output Validation
The Agent includes a _validate_output mechanism that uses an LLM-based validator to verify if the task was completed according to user requirements. This prevents the agent from falsely reporting completion when elements are missing or the state is incorrect.
Validation Logic
The validator acts as a separate "judge" that analyzes the final browser state, screenshots, and action results. It returns a structured ValidationResult containing:
is_valid: A boolean indicating if the output is correct.reason: A detailed explanation for the validity decision.
| Component | Description |
|---|---|
| System Message | Instructs the LLM to act as a validator for browser interactions. |
| AgentMessagePrompt | Provides the current browser state, screenshot, and result context to the validator. |
| ValidationResult | Pydantic model used to parse the structured response from the validator LLM. |
Sources: browser_use/agent/service.py:488-530
History Analysis and Performance Metrics
The evaluation system relies on AgentHistoryList to track and analyze agent behavior over time. This history is used to measure success rates, identify common failure points, and debug token usage.
Performance Tracking Features
- Action Tracking: Extracts all actions performed by the model to analyze sequencing.
- Error Analysis: Aggregates all errors encountered during a run to identify systemic issues.
- Screenshot Logs: Captures visual state at every step for manual verification or GIF generation.
- Token Monitoring: Monitors input tokens to prevent exceeding context limits, specifically handling overflow by cutting history when necessary.
This diagram illustrates the relationship between the history tracking classes used for post-run evaluation.
Sources: browser_use/agent/views.py:100-200, browser_use/agent/message_manager/tests.py:116-130
DOM Extraction Evaluation
A critical part of benchmarking is evaluating the efficiency of DOM extraction. The system tests different viewport_expansion settings to find the balance between providing enough context to the LLM and minimizing token consumption.
Extraction Metrics
Testing is performed across various websites (e.g., Google, Amazon, GitHub) to measure:
- Element Count: Total interactive elements identified.
- Token Count: Size of the element tree when converted to a string for LLM input.
- Visibility Accuracy: Ensuring only top-level, interactive elements are processed.
Sources: browser_use/dom/tests/extraction_test.py:15-50
Telemetry and Failure Handling
Evaluation data is also captured via the ProductTelemetry service, which logs specific events during the agent's lifecycle.
| Event Type | Data Captured |
|---|---|
| AgentRunTelemetryEvent | Model name, task description, vision settings, and version. |
| AgentStepTelemetryEvent | Action details, step number, and consecutive failure counts. |
| AgentEndTelemetryEvent | Final success status, total steps, and list of all errors. |
The agent maintains a consecutive_failures counter. If it reaches the max_failures threshold (defaulting to 3), the run is terminated and evaluated as a failure.
Sources: browser_use/agent/service.py:115-135, browser_use/agent/service.py:270-300, browser_use/agent/service.py:475-480
Conclusion
Evaluation & Benchmarking in browser-use provides a robust framework for assessing agent performance. By combining dataset testing (Mind2Web), LLM-driven output validation, and detailed telemetry/history analysis, the system ensures that browser agents are both reliable and efficient. These tools allow developers to benchmark different LLMs and configuration settings (like DOM expansion) against complex, real-world web scenarios.
Sources: browser_use/agent/service.py:435-450, tests/test_mind2web.py:1-15