Browser Automation & Chrome Extension
Relevant source files
The following files were used as context for generating this wiki page:
Browser automation within Clawdbot is a sophisticated system designed to allow AI agents to navigate, interact with, and extract data from the web. It integrates directly with the agent's toolbelt, providing a dedicated environment for executing web-based tasks ranging from simple content fetching to complex multi-step interactions.
The system relies on a combination of core tools (like browser and web_fetch), specialized skills, and sandboxed execution environments to ensure that agents can perform automation safely and effectively. This capability is central to the project's goal of enabling agents to act as "personal assistants" with full web navigation capabilities.
Sources: src/agents/system-prompt.ts:139, src/agents/system-prompt.ts:241
System Architecture
The browser automation system is structured as a set of tools and configuration layers that bridge the gap between the agent's logic and the actual web browser.
Component Overview
- Browser Tool: The primary interface for controlling the dedicated browser instance.
- Web Fetch/Search Tools: Specialized tools for content extraction and discovery.
- Sandbox Environment: A Docker-based runtime that isolates browser execution and prevents unauthorized host access.
- Canvas Integration: A visual layer that allows agents to present and snapshot HTML content on connected nodes.
The following diagram illustrates the relationship between these components:
Browser Configuration and Policy
Browser automation is subject to strict security and configuration policies, managed via the sandboxInfo and global configuration.
Sandbox Security Policies
When running in a sandboxed environment, browser control is limited by specific allowlists. This prevents the agent from accessing internal or sensitive URLs unless explicitly permitted.
| Policy Attribute | Description |
|---|---|
hostBrowserAllowed | Boolean flag indicating if the host's browser can be controlled. |
allowedControlUrls | A list of specific URLs the agent is allowed to visit. |
allowedControlHosts | A list of domain hosts permitted for browser automation. |
allowedControlPorts | Specific ports (e.g., 80, 443) allowed for web connections. |
browserControlUrl | The specific URL used to interface with the browser automation service. |
Sources: src/agents/system-prompt.ts:311-325
UI Configuration
The project provides a configuration form for managing browser settings, categorized under "Browser" in the metadata.
export const SECTION_META = {
// ...
browser: { label: "Browser", description: "Browser automation settings" },
// ...
};
Sources: ui/src/ui/views/config-form.render.ts:98
Core Automation Tools
Clawdbot exposes several high-level tools that agents use to perform automation.
The Browser Tool
The browser tool is the main driver for interactive web tasks. It allows for:
- Navigation to URLs.
- Interaction with DOM elements.
- Executing JavaScript within the page context.
- Managing browser sessions within the sandbox.
Sources: src/agents/system-prompt.ts:153, skills/coding-agent/SKILL.md:27
Content Extraction (web_fetch and Firecrawl)
For simple data retrieval, web_fetch is used to extract readable content. The system supports multiple extraction modes, including integration with external services like Firecrawl.
Canvas Presentation Layer
The canvas tool provides a way for agents to present HTML content visually on connected nodes (Mac, iOS, Android). This is often used to show the results of browser automation or to provide a dedicated UI for the agent's current task.
Canvas Actions
| Action | Purpose |
|---|---|
present | Shows the canvas with a target URL. |
navigate | Changes the current URL of the canvas. |
snapshot | Captures a screenshot of the current canvas state. |
eval | Executes JavaScript in the canvas context. |
Sources: skills/canvas/SKILL.md:32-40
Architecture of Canvas Host
The Canvas Host is an HTTP server running on port 18793 that serves content from a root directory (default ~/clawd/canvas/).
Sources: skills/canvas/SKILL.md:12-25
Implementation Details
The system is built on modern Node.js technologies, utilizing Playwright for browser control and Readability for content parsing.
Key Dependencies
- playwright-core: The engine behind the
browsertool for low-level automation. - @mozilla/readability: Used by
web_fetchto clean up HTML into readable text. - express/hono: Power the internal bridge and canvas host servers.
- chokidar: Used for live-reloading canvas content when files change.
Sources: package.json:148-164
Environment and Runtime
The runtime line in the system prompt indicates the browser's status and the agent's capabilities (e.g., capabilities=inlinebuttons). Automation is often performed using the exec tool to run shell-based commands that interact with the browser environment.
Sources: src/agents/system-prompt.ts:480, skills/coding-agent/SKILL.md:120
Summary
Browser Automation & Chrome Extension support in Clawdbot provides a unified interface for agents to interact with the web. By combining the browser tool for interactive sessions, web_fetch for data extraction, and the canvas system for visual presentation, the project enables a wide range of autonomous web-based workflows within a secure, sandboxed environment.