DOM Processing & Extraction

Relevant source files

The following files were used as context for generating this wiki page:

• browser_use/dom/service.py
• browser_use/dom/buildDomTree.js
• browser_use/dom/history_tree_processor/service.py
• browser_use/dom/views.py
• browser_use/browser/context.py

DOM Processing & Extraction is a core system within the browser-use library designed to convert complex web pages into a structured, simplified format optimized for AI agents. It bridges the gap between raw browser state and the LLM by filtering for interactive elements, calculating spatial coordinates, and generating unique identifiers for element interaction.

The system utilizes a hybrid approach, executing JavaScript within the browser to build a raw node map, which is then parsed into a Python-based tree structure. This process includes visibility checks, interactivity filtering, and shadow DOM/iframe traversal to ensure the agent has a complete but concise view of the page state.

Sources: browser_use/dom/service.py, browser_use/dom/buildDomTree.js

System Architecture & Data Flow

The DOM extraction process follows a strictly defined pipeline starting from the BrowserContext and ending with a DOMState object containing a tree of nodes and a selector map.

Extraction Pipeline

1. JavaScript Execution: The DomService reads buildDomTree.js and evaluates it within the Playwright page context.
2. Node Traversal: The script traverses the document body, including recursive entry into IFrames and Shadow Roots.
3. Filtering: Elements are filtered based on visibility (isElementVisible), interactivity (isInteractiveElement), and whether they are the top element at their coordinates (isTopElement).
4. Serialization: The result is returned as a flat DOM_HASH_MAP to avoid circular reference issues during transport between JS and Python environments.
5. Tree Reconstruction: DomService reconstructs the tree hierarchy in Python using the returned hash map.

Sources: browser_use/dom/service.py:34-78, browser_use/dom/buildDomTree.js:464-601

Core Components

DomService

The primary orchestrator for DOM operations. It handles the interface between Python and the browser's JavaScript engine.

• get_clickable_elements(): The main entry point that triggers the JS extraction and returns a DOMState.
• _parse_node(): Converts raw dictionary data from JS into DOMElementNode or DOMTextNode objects.

Sources: browser_use/dom/service.py:22-140

buildDomTree.js

A comprehensive script injected into the browser to perform low-level DOM analysis. Key logic includes:

• Interactivity Detection: Identifies clickable elements by checking tag names (a, button, input), ARIA roles, event listeners (click, mousedown), and specific CSS properties like cursor: pointer.
• Coordinate Calculation: Computes both viewportCoordinates (relative to the visible area) and pageCoordinates (relative to the entire document).
• Visual Highlighting: Optionally creates a playwright-highlight-container to draw colored bounding boxes and index labels over interactive elements for visual verification.

Sources: browser_use/dom/buildDomTree.js:14-350

Data Models

The DOM state is represented by several specialized data classes.

Class	Description	Key Fields
DOMElementNode	Represents an HTML element.	tag_name, xpath, attributes, highlight_index, is_interactive
DOMTextNode	Represents raw text content.	text, is_visible
DOMState	Encapsulates the full extracted state.	element_tree, selector_map
CoordinateSet	Stores spatial data.	top_left, bottom_right, center, width, height

Sources: browser_use/dom/views.py:18-70, browser_use/dom/history_tree_processor/view.py

Element Selection & Identification

XPath & CSS Selectors

The system generates robust selectors to ensure the agent can interact with elements reliably.

• XPath Generation: getXPathTree in JS builds a hierarchical path, stopping at boundaries like Shadow Roots or IFrames to maintain context.
• Enhanced CSS Selectors: The BrowserContext provides logic to convert XPaths to CSS selectors and augment them with stable attributes (ID, name, data-testid) for better resilience.

Sources: browser_use/dom/buildDomTree.js:141-175, browser_use/browser/context.py:649-743

Selector Map

The SelectorMap is a dictionary mapping integer highlight_index values to DOMElementNode objects. This allows the LLM to refer to an element by a simple number (e.g., [5]) while the system retrieves the full metadata and locator required for the action.

Sources: browser_use/dom/service.py:117-124, browser_use/dom/views.py:165

History Tree Processing

To maintain consistency across page reloads or navigations, the HistoryTreeProcessor hashes DOM elements to track them over time.

@staticmethod
def _hash_dom_element(dom_element: DOMElementNode) -> HashedDomElement:
    parent_branch_path = HistoryTreeProcessor._get_parent_branch_path(dom_element)
    branch_path_hash = HistoryTreeProcessor._parent_branch_path_hash(parent_branch_path)
    attributes_hash = HistoryTreeProcessor._attributes_hash(dom_element.attributes)
    xpath_hash = HistoryTreeProcessor._xpath_hash(dom_element.xpath)
    return HashedDomElement(branch_path_hash, attributes_hash, xpath_hash)

Sources: browser_use/dom/history_tree_processor/service.py:61-68

This hashing mechanism allows the system to compare elements in the current DOM against previous states, enabling the agent to recognize if it is looking at the same button or input field even if some dynamic attributes have changed.

Sources: browser_use/dom/history_tree_processor/service.py:14-52

Extraction Configuration

The behavior of the DOM extraction is controlled via BrowserContextConfig.

Option	Type	Default	Description
highlight_elements	bool	True	Whether to draw visual boxes on the page.
viewport_expansion	int	500	Pixels outside the viewport to include in extraction.
include_dynamic_attributes	bool	True	Whether to use dynamic data attributes in selectors.

Sources: browser_use/browser/context.py:75-135

The DOM Processing & Extraction module provides the structural foundation for browser automation by transforming a chaotic web environment into a predictable, indexed, and spatial-aware model for AI consumption.