Controller & Action Registry
Relevant source files
The following files were used as context for generating this wiki page:
The Controller & Action Registry system is a core architectural component that defines how AI agents interact with the web browser. It acts as a bridge between the agent's decision-making logic and the execution of specific browser commands. By registering actions within a central registry, the system enables the Large Language Model (LLM) to discover available capabilities, such as clicking elements, inputting text, or extracting page content.
The system is designed for extensibility, allowing developers to define custom actions that go beyond standard browser interactions. This modularity ensures that specialized agents, like the CustomAgent or the Deep Research module, can utilize a tailored set of tools to fulfill complex user instructions effectively.
Sources: src/controller/custom_controller.py, src/agent/custom_agent.py:108-121, src/utils/deep_research.py:75-96
Architecture and Components
The registry system relies on a hierarchical structure where the Controller manages a Registry of executable functions. These functions are mapped to ActionModel definitions, which the LLM uses to structure its output.
CustomController
The CustomController extends the base Controller class to provide specialized functionality. It is responsible for initializing the registry and registering custom actions that involve operating system-level interactions, such as clipboard management.
Key methods include:
_register_custom_actions(): A private method that uses the@self.registry.actiondecorator to bind Python functions to the agent's available toolset.
Sources: src/controller/custom_controller.py:27-44
Action Registry in Deep Research
The registry is also utilized dynamically within specific utilities. For instance, the deep_research module registers a unique extract_content action to the controller's registry at runtime. This action specifically uses the MainContentExtractor to transform web pages into pure Markdown format, facilitating better information gathering.
Sources: src/utils/deep_research.py:75-96
Data Flow and Execution
The flow of an action begins with the LLM selecting a tool from the registry and ends with the browser context executing the command.
The diagram shows the lifecycle of an action from generation to execution and feedback.
Sources: src/agent/custom_agent.py:245-280, src/agent/custom_prompts.py:17-48
Multi-Action Execution
The system supports executing multiple actions in a single step. The CustomAgent calls controller.multi_act, which processes a list of ActionModel objects sequentially. If an action fails or the page state changes unexpectedly, the controller manages the resulting errors and updates the agent's memory accordingly.
Sources: src/agent/custom_agent.py:348-369
Registry Components Summary
| Component | Description | Source File |
|---|---|---|
Registry | Stores the mapping between action names and their implementation functions. | src/controller/custom_controller.py |
ActionModel | A dynamic Pydantic model created from the registry to define the JSON schema for LLM calls. | src/agent/custom_agent.py:124-128 |
ActionResult | Encapsulates the outcome of an action, including extracted content or error messages. | src/agent/custom_controller.py:12-25 |
CustomAgentOutput | The top-level data structure for LLM responses, containing the agent's "brain" (thought process) and a list of actions. | src/agent/custom_views.py:36-54 |
Action Definitions
Actions are registered with descriptive strings that the MessageManager uses to generate prompts for the LLM.
Custom Action Examples
The following table highlights specific custom actions registered in the project:
| Action Name | Purpose | Implementation Details |
|---|---|---|
extract_content | Gets pure Markdown from the page. | Uses MainContentExtractor and jina.ai reader. |
copy_to_clipboard | Copies text to the system clipboard. | Uses the pyperclip library. |
paste_from_clipboard | Pastes text into the active browser field. | Retrieves text via pyperclip and types via Playwright keyboard. |
Sources: src/controller/custom_controller.py:35-44, src/utils/deep_research.py:75-96
Action Sequencing
The system enforces strict rules for action sequencing to maintain stability:
- Actions are executed in the order they appear in the JSON list.
- If the page changes significantly, the sequence is interrupted to provide the agent with a fresh state.
- The
max_actions_per_stepconfiguration limits how many actions an agent can propose at once to prevent hallucinations.
Sources: src/agent/custom_prompts.py:84-93, webui.py:465-472
Integration with Agent State
The Controller interaction is tightly coupled with the CustomAgentStepInfo. When an action is executed, the ActionResult is processed to update the agent's memory and task progress.
The sequence diagram illustrates how the controller mediates between the Agent's intent and the Browser's execution.
Sources: src/agent/custom_agent.py:340-380, src/agent/custom_message_manager.py:83-97
Summary
The Controller & Action Registry provides a robust framework for managing AI-driven browser interactions. By centralizing action definitions and providing a standardized execution path through CustomController, the system ensures that agents can operate with high precision while remaining easily extensible for new types of research and web automation tasks.