Adding Custom Actions
Relevant source files
The following files were used as context for generating this wiki page:
The web-ui project provides a robust framework for extending the capabilities of AI agents by adding custom actions. This system allows developers to define specialized browser interactions or utility functions that the LLM can invoke during a task. Custom actions are primarily managed through a registry system within a controller, which bridges the gap between the agent's decision-making logic and actual browser execution.
By implementing custom actions, developers can enable features such as clipboard manipulation, specialized content extraction, or complex multi-step navigation patterns that are not part of the standard browser-use library. This extensibility is central to the project's "Custom Agent" architecture, allowing it to adapt to specific research or automation requirements.
Sources: src/controller/custom_controller.py:1-45, src/agent/custom_agent.py:80-120
Action Registry Architecture
Custom actions are registered using the CustomController class, which inherits from the base Controller. The registration process uses a decorator-based approach to add new functions to the Registry. Once registered, these actions become available as part of the dynamic ActionModel used by the agent to generate valid JSON responses.
The Registration Flow
The following diagram illustrates how a custom action is defined and made available to the agent:
The registry transforms Python functions into schema-validated actions that the LLM can understand and utilize.
Sources: src/controller/custom_controller.py:31-45, src/agent/custom_agent.py:122-127
Implementation Methods
There are two primary ways to add custom actions within the project: through the global CustomController or inline within specialized modules like deep_research.
1. Using CustomController
The CustomController class is the standard location for persistent custom actions. It typically registers actions during initialization.
| Action Name | Description | Parameter | Return Type |
|---|---|---|---|
copy_to_clipboard | Copies provided text to the system clipboard. | text: str | ActionResult |
paste_from_clipboard | Pastes text from the clipboard into the current browser page. | browser: BrowserContext | ActionResult |
Sources: src/controller/custom_controller.py:31-45
2. Inline Action Registration
In modules like deep_research.py, actions can be registered dynamically to a controller instance. For example, the extract_content action is added specifically for research tasks to convert page HTML into pure Markdown.
controller = CustomController()
@controller.registry.action(
'Extract page content to get the pure markdown.',
)
async def extract_content(browser: BrowserContext):
page = await browser.get_current_page()
url = page.url
# Implementation details using jina reader or extractor...
return ActionResult(extracted_content=msg)
Sources: src/utils/deep_research.py:72-92
Data Models and Schemas
Custom actions rely on Pydantic models to ensure the LLM provides valid arguments. The CustomAgent utilizes a factory method type_with_custom_actions to inject these registered actions into the CustomAgentOutput model.
Agent Output Structure
The agent's output is structured to include both the reasoning state and the sequence of actions to be performed.
| Component | Class | Description |
|---|---|---|
| Current State | CustomAgentBrain | Contains evaluation, memory, progress, plans, and thoughts. |
| Actions | list[ActionModel] | A sequence of actions defined by the controller's registry. |
Sources: src/agent/custom_views.py:23-52, src/agent/custom_agent.py:122-127
Action Execution Flow
When an agent decides to use a custom action, the request follows a specific sequence through the custom agent architecture.
This flow ensures that the agent maintains a record of the result of every custom action executed.
Sources: src/agent/custom_agent.py:246-285, src/agent/custom_prompts.py:21-65
Integration with the UI
The WebUI allows users to toggle between the original (org) agent and the custom agent. Selecting the "custom" agent type in the settings tab triggers the use of CustomAgent and CustomController, thereby enabling the custom actions defined in the codebase.
- File Path:
webui.py - Logic: The
run_custom_agentfunction initializes theCustomControllerand passes it to theCustomAgentconstructor.
Sources: webui.py:417-458, src/utils/default_config_settings.py:11-34
Summary
Adding custom actions is a standardized process of registering Python functions with the CustomController. This process automatically updates the agent's schema, allowing the LLM to use new capabilities like clipboard interaction or specialized Markdown extraction. The seamless integration between the registry, the Pydantic data models, and the CustomAgent execution loop provides a highly extensible environment for browser automation.