Computer Use & Web Browsing
Relevant source files
The following files were used as context for generating this wiki page:
The Computer Use and Web Browsing module enables the Hermes agent to interact with external environments through a structured set of tools. This system supports navigating web applications, interacting with DOM elements, capturing visual data, and executing terminal commands to perform complex technical tasks.
The module provides a bridge between the agent's reasoning capabilities and the practical execution of actions in a browser or terminal. It allows for systematic exploratory QA, data extraction, and automated workflow execution.
Core Toolset Architecture
The browsing architecture relies on a suite of specific tools that manage navigation, interaction, and observation. These tools are often supervised by core modules such as browser_supervisor.py and browser_camofox.py to ensure stable and stealthy interactions.
Browser Interaction Tools
The agent uses several tools to manipulate and observe the web browser:
browser_navigate: Changes the current browser URL to a target location.browser_snapshot: Retrieves a text-based snapshot of the Document Object Model (DOM) for accessibility and structure analysis.browser_click: Interacts with specific elements identified by references (e.g.,@eN) or text.browser_type: Enters text into input fields.browser_vision: Captures screenshots and applies AI analysis to identify visual issues or interactive elements via theannotate=trueflag.browser_console: Monitors JavaScript execution and captures console errors.
Sources: skills/software-development/dogfood/SKILL.md:14-22, skills/software-development/dogfood/SKILL.md:121-131
Execution Flow Diagram
The following diagram illustrates how the agent plans and executes a browsing session.
The flow shows a systematic path from planning to evidence collection for web-based tasks. Sources: skills/software-development/dogfood/SKILL.md:33-100
Systematic Web Application QA (Dogfooding)
A primary use case for the browsing module is systematic Quality Assurance (QA). The agent follows a 5-phase workflow to identify bugs, visual regressions, and functional failures.
QA Workflow Phases
| Phase | Action Description |
|---|---|
| Phase 1: Plan | Create output directories and define sitemaps for landing pages and user flows. |
| Phase 2: Explore | Use browser_snapshot and browser_vision to understand page structure and visual layout. |
| Phase 3: Evidence | Record reproductions steps, expected vs actual behavior, and console errors. |
| Phase 4: Categorize | De-duplicate issues and assign severity levels (Critical, High, Medium, Low). |
| Phase 5: Report | Generate a Markdown report with inline screenshot references using the MEDIA:<path> tag. |
Sources: skills/software-development/dogfood/SKILL.md:37-119
Technical Execution & Terminal Interaction
Beyond browsing, the module utilizes terminal primitives for OS-level interactions. This includes repository management and automated file manipulation.
Environment Primitives
The agent uses the following system tools to support browsing and development tasks:
terminal: Invokes shell scripts and third-party CLIs (e.g.,git,gh).read_file/write_file: Manages local data and logs.web_extract: Converts a URL directly into a Markdown summary for quick knowledge ingestion.web_search: Searches the live web to find relevant URLs for the browser tools.
Sources: agent/learn_prompt.py:85-91, optional-skills/software-development/code-wiki/SKILL.md:31-35
Interaction Sequence
This sequence shows how the agent coordinates between browsing and code analysis.
The diagram represents the coordination between web browsing and local environment management. Sources: optional-skills/software-development/code-wiki/SKILL.md:47-65, agent/learn_prompt.py:126-140
Tool Parameter Reference
The following table summarizes parameters for the primary browser tools used in the module.
| Tool | Key Parameters | Return Value |
|---|---|---|
browser_vision | question (str), annotate (bool) | Analysis text + screenshot path |
browser_click | ref (string like "@e1"), text (str) | Success/Failure status |
browser_type | ref (str), text (str) | Text confirmation |
browser_console | clear (bool) | List of JS errors and logs |
web_extract | url (str) | Markdown formatted content |
Sources: skills/software-development/dogfood/SKILL.md:121-131, agent/learn_prompt.py:85-91
Conclusion
The Computer Use and Web Browsing module integrates visual analysis, DOM interaction, and terminal execution. By combining browser_supervisor.py and browser_camofox.py with high-level browsing skills, the agent performs complex web interactions and system tasks autonomously while maintaining structured logs and visual evidence.