Proprietary Model Providers
Relevant source files
The following files were used as context for generating this wiki page:
Proprietary Model Providers in the browser-use library refer to the commercial Large Language Model (LLM) services integrated via LangChain to power browser automation agents. These providers include OpenAI, Anthropic, Google (Gemini), and Amazon (Bedrock), which serve as the "brain" of the agent by processing browser state, reasoning through tasks, and generating structured JSON actions for execution.
The library relies on the standard langchain_core.language_models.BaseChatModel interface, ensuring that agents can seamlessly switch between different proprietary models by providing the appropriate LangChain implementation during initialization.
Architecture and Integration
The integration of proprietary models is handled through a standardized interaction flow where the MessageManager prepares state-aware prompts and the proprietary model returns structured actions.
Provider Integration Table
| Provider | LangChain Package | Example Model | Authentication |
|---|---|---|---|
| OpenAI | langchain-openai | gpt-4o, gpt-4o-mini | OPENAI_API_KEY |
| Azure OpenAI | langchain-openai | gpt-4o | AZURE_OPENAI_KEY, AZURE_OPENAI_ENDPOINT |
| Anthropic | langchain-anthropic | claude-3-5-sonnet | API Key via Environment |
langchain-google-genai | gemini-2.0-flash-exp | GEMINI_API_KEY | |
| AWS Bedrock | langchain-aws | bedrock_claude | AWS Credentials |
Sources: pyproject.toml:23-28, README.md:43-52, examples/use-cases/find_and_apply_to_jobs.py:171-176, examples/browser/using_cdp.py:46-46
Data Flow Diagram
This diagram illustrates how the Agent coordinates with proprietary LLMs through the MessageManager and Controller.
The MessageManager is responsible for token management, ensuring that the combined history of system prompts, task instructions, and browser state does not exceed the proprietary model's max_input_tokens (default 128,000).
Sources: browser_use/agent/message_manager/service.py:34-50, README.md:43-52
Model Configuration and Implementation
Proprietary models are instantiated as LangChain chat objects and passed to the Agent class.
OpenAI and Azure OpenAI
The library heavily uses OpenAI's GPT-4o for its vision and function-calling capabilities. Azure OpenAI is also supported, requiring additional configuration such as api_version and azure_endpoint.
# Standard OpenAI
model = ChatOpenAI(model='gpt-4o')
# Azure OpenAI
model = AzureChatOpenAI(
model='gpt-4o',
api_version='2024-10-21',
azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT', ''),
api_key=SecretStr(os.getenv('AZURE_OPENAI_KEY', ''))
)
Sources: examples/features/custom_output.py:41-41, examples/use-cases/find_and_apply_to_jobs.py:171-176
Google Gemini
Google's Gemini models are used for browser automation tasks, particularly in environments utilizing Chrome DevTools Protocol (CDP).
model = ChatGoogleGenerativeAI(model='gemini-2.0-flash-exp', api_key=SecretStr(api_key))
Sources: examples/browser/using_cdp.py:46-46
Handling Model Outputs
Proprietary models are expected to return a valid JSON format matching the AgentOutput schema. The MessageManager provides utilities to handle models that do not support native function calling by converting messages and extracting JSON from raw string content.
JSON Extraction and Validation
For models that return JSON wrapped in markdown code blocks, the MessageManager uses a cleanup process:
- Checks for
```delimiters. - Extracts content between code blocks.
- Removes language identifiers (e.g.,
json). - Parses the resulting string via
json.loads.
Sources: browser_use/agent/message_manager/service.py:317-331
Sequence Diagram: Model Request Lifecycle
Token and Memory Management
Proprietary models have strict input limits and costs. The MessageManager provides a mechanism to trim the interaction history to fit within a specified budget.
- Image Tokens: Visual context (screenshots) consumes a significant amount of tokens (default 800 per image).
- Token Estimation: Uses a rough estimate of 3 characters per token if a specific tokenizer is not available.
- Message Trimming: If the total token count exceeds
max_input_tokens, theMessageManagerfirst removes images and then truncates the oldest parts of the most recent state message.
Sources: browser_use/agent/message_manager/service.py:34-45, browser_use/agent/message_manager/service.py:246-288
Summary
Proprietary Model Providers are the primary engine for decision-making in browser-use. By integrating with the LangChain ecosystem, the library supports major industry models like GPT-4o, Claude 3.5 Sonnet, and Gemini. The system is designed to handle the specificities of these providers, including structured output requirements, vision capabilities, and token-based pricing/limitations through a robust MessageManager service.