Deep Research Module
Relevant source files
The following files were used as context for generating this wiki page:
The Deep Research Module is a specialized component of the Browser Use Web UI designed for exhaustive, iterative information gathering and professional report generation. Unlike standard single-task agents, this module employs a multi-stage pipeline that includes high-level research planning, automated web browsing for data extraction, systematic information recording, and final synthesis into publication-ready Markdown reports.
The module leverages CustomAgent instances to navigate the web, using specialized tools like the Jina Reader for markdown extraction and sophisticated system prompts that guide the AI through "Deep Thinking" phases for both searching and recording information.
Sources: src/utils/deep_research.py:46-218, webui.py:408-417
Architecture and Core Logic
The module operates as a stateful loop that alternates between planning, executing browser-based searches, and recording findings. It uses a directory-based storage system to maintain temporary artifacts, query results, and the final consolidated dataset.
Workflow Execution Flow
The following diagram illustrates the iterative nature of the research process, from initial task input to the final report generation.
The workflow begins with a search_system_prompt that defines the agent as a "Deep Researcher" capable of devising a research plan and generating up to a maximum number of specific search queries per iteration.
Sources: src/utils/deep_research.py:53-157
Components and Roles
| Component | Description | File Reference |
|---|---|---|
| Search Planner | An LLM-driven stage that generates JSON objects containing a high-level research plan and a list of queries. | src/utils/deep_research.py:126-150 |
| CustomAgent | Orchestrates the browser interactions, handling multi-tab management and action execution. | src/agent/custom_agent.py:44-118 |
| Content Extractor | Uses MainContentExtractor and Jina Reader (r.jina.ai) to convert web pages into clean Markdown. | src/utils/deep_research.py:84-100 |
| Info Recorder | Summarizes findings while maintaining source URLs, titles, and "thinking" metadata to avoid redundancy. | src/utils/deep_research.py:160-205 |
| Report Writer | A final synthesis stage that produces a polished Markdown document with citations and tables. | src/utils/deep_research.py:236-285 |
Functional Implementation
Iterative Search Loop
The deep_research function manages the main loop. In each iteration, it updates search messages with previous results and history queries to prevent duplication and ensure depth.
Sources: src/utils/deep_research.py:214-255
Automated Execution Capabilities
The module includes specific instructions for handling complex content:
- Link Depth: Agents are instructed to click relevant links to "go deeper" rather than staying on search engine results pages.
- PDF Handling: When a PDF is encountered, the agent is mandated to use the
extract_contenttool to parse the document into markdown rather than simply viewing it. - Browser Persistence: If
use_own_browseris enabled, the system restricts the iteration to one query at a time to maintain context within a single Chrome instance.
Sources: src/utils/deep_research.py:64-79, src/utils/deep_research.py:245-249
Data Management and Storage
The module maintains a rigorous data structure to ensure traceability and auditability of the research process.
Directory Structure
Research artifacts are saved in a timestamped or UUID-based directory:
./tmp/deep_research/{task_id}/query_results/: Stores raw.mdfiles for every query executed, named as{iteration}-{query_index}.md.record_infos.json: A consolidated list of all verified information, including URLs, titles, summaries, and AI "thinking" reasoning.final_report.md: The finished publication-ready document.
Sources: src/utils/deep_research.py:54-56, src/utils/deep_research.py:258-265, src/utils/deep_research.py:302-305
Final Report Synthesis
The generate_final_report function uses a writer_system_prompt that enforces specific professional standards:
- Citation Format: Uses bracketed sequential numbers (e.g., [1], [2]) corresponding to a reference list.
- Data Presentation: Encourages the use of Markdown tables for numerical data and comparisons.
- Output Restriction: Strictly forbids preambles or meta-commentary, ensuring the output is "publication-ready."
Sources: src/utils/deep_research.py:288-330
Configuration and Integration
The module is integrated into the Gradio Web UI via the run_deep_search function, which initializes the LLM and manages the asynchronous execution of the research task.
UI Configuration Parameters
| Parameter | Default | Description |
|---|---|---|
max_search_iterations | 3 (UI) / 10 (Code) | The maximum number of research loops to perform. |
max_query_num | 1 (UI) / 3 (Code) | Max search queries generated by the planner per loop. |
use_vision | False | Enables visual processing for the browsing agents. |
headless | True | Runs the research browser without a visible GUI. |
Sources: webui.py:539-550, webui.py:734-741, src/utils/deep_research.py:210
Conclusion
The Deep Research Module provides a robust framework for high-autonomy information gathering. By combining iterative planning with a specialized recording phase that captures AI "reasoning" for every piece of data, it ensures that the final generated reports are not just summaries, but structured analyses grounded in verifiable sources and specific extracted evidence.