Local & Open Source Models

Relevant source files

The following files were used as context for generating this wiki page:

• tests/test_models.py
• browser_use/agent/message_manager/service.py
• pyproject.toml
• browser_use/agent/prompts.py
• README.md

The browser-use library provides a flexible architecture that supports both proprietary and open-source Large Language Models (LLMs). This compatibility is achieved through integration with the LangChain ecosystem, allowing users to deploy local models via providers like Ollama or access open-source models via specialized API endpoints such as DeepSeek. These models are integrated into the Agent workflow to perform browser automation tasks, including navigation, element interaction, and data extraction.

Sources: README.md:37-60, tests/test_models.py:50-86

Architecture and Integration

The project leverages langchain and its provider-specific packages to interface with various models. Local and open-source models are treated as standard BaseChatModel objects within the Agent and MessageManager classes. This abstraction ensures that the logic for browser control remains consistent regardless of the underlying model's origin.

Supported Local & Open Source Providers

The following table summarizes the primary local and open-source model integrations identified in the project configuration and testing suites.

Provider	LangChain Package	Example Models	Notes
Ollama	langchain-ollama	qwen2.5:latest	Requires local Ollama service.
DeepSeek	langchain-openai	deepseek-chat, deepseek-reasoner	Uses OpenAI-compatible API base URL.
Google	langchain-google-genai	gemini-1.5-pro, gemini-2.0-flash	Open-weights/accessible models.

Sources: pyproject.toml:22-26, tests/test_models.py:53-86

Model Communication Flow

The following diagram illustrates how the MessageManager prepares requests for different model types, including handling specific constraints for models like DeepSeek.

The MessageManager provides utility functions to ensure compatibility with models that may not support native tool calling or specific message sequences.

• merge_successive_messages: Specifically handles constraints for models like deepseek-reasoner which do not allow multiple consecutive human messages. Sources: browser_use/agent/message_manager/service.py:321-338
• convert_messages_for_non_function_calling_models: Converts ToolMessage and complex AIMessage structures into plain text for models lacking functional tool-calling capabilities. Sources: browser_use/agent/message_manager/service.py:300-319
• extract_json_from_model_output: A robust parser that can handle JSON wrapped in markdown code blocks, frequently seen in open-source model outputs. Sources: browser_use/agent/message_manager/service.py:340-355

Local Model Deployment (Ollama)

For fully local execution, the system supports Ollama. The integration utilizes ChatOllama with a specified context window to handle the large DOM trees generated during browser sessions.

Configuration for Local Models

To use a local model like qwen2.5, the Ollama service must be running. The testing suite validates this by attempting to ping the local Ollama API at http://127.0.0.1:11434/.

Sources: tests/test_models.py:57-60, tests/test_models.py:108-116

# Example: Configuring a local model
ChatOllama(
    model='qwen2.5:latest',
    num_ctx=128000,
)

Sources: tests/test_models.py:57-60

Vision Capabilities and Constraints

Open-source models vary in their support for vision. The Agent provides a use_vision flag that can be disabled for text-only models like deepseek-chat.

• Vision-Enabled: Models like gpt-4o or gemini-1.5-pro receive screenshots as base64-encoded strings within a HumanMessage. Sources: browser_use/agent/prompts.py:202-212
• Text-Only: Models like deepseek-chat or deepseek-reasoner are configured with use_vision=False in the testing suite. Sources: tests/test_models.py:98-105

Prompt Engineering for Open Source Models

System prompts are designed to enforce a strict JSON response format, which is critical for open-source models that may be more prone to hallucinating or deviating from expected output structures.

Testing and Validation

The project includes a dedicated model testing suite (tests/test_models.py) to verify that local and open-source models can correctly perform standard actions like search_google or go_to_url.

• Action Verification: The suite checks if the model successfully identifies and emits actions such as search_google, go_to_url, or open_tab.
• Error Handling: It specifically tracks if models reach the max_failures threshold or produce validation errors.

Sources: tests/test_models.py:118-144

Summary

Local and open-source model support in browser-use is built on a modular system of message transformation and robust JSON extraction. By leveraging LangChain's diverse providers, the library enables developers to choose between high-performance proprietary models and privacy-focused local deployments like Ollama, while the MessageManager handles the underlying differences in message handling and functional capabilities.