On-Device Web Search

Relevant source files

The following files were used as context for generating this wiki page:

• packages/conduit_ddgs/VENDORED.md
• packages/conduit_ddgs/pubspec.yaml
• packages/conduit_ddgs/test/fixtures/duckduckgo_results.html
• packages/conduit_ddgs/test/fixtures/brave_results.html
• README.md
• packages/conduit_core/test/fixtures/chat_blobs/05_web_search_status_sources.json
• packages/conduit_ddgs/test/fixtures/mojeek_results.html

On-device web search allows Conduit to perform internet searches directly from the mobile client. This system enables large language models (LLMs) to retrieve real-time information without relying on an external search service or backend relay. The functionality is primarily implemented in the conduit_ddgs package, which provides pure-Dart scrapers for multiple search engines.

Sources: README.md:123-125, packages/conduit_ddgs/pubspec.yaml:2-5

Core Architecture

The system utilizes a series of scrapers derived from the ddgs package to extract search results from HTML responses. It supports several search engines, including DuckDuckGo, Brave, Bing, Mojeek, and Wikipedia. The implementation prioritizes privacy and efficiency by performing requests via an http.Client owned by the caller.

The diagram below shows the high-level data flow when an LLM initiates a web search:

Supported Search Engines

The system attempts engines in a fixed order and stops at the first one that returns valid results. This reduces latency and minimizes the footprint on individual search providers.

Engine	Method	Implementation Details
DuckDuckGo	HTML Scraper	Extracts from html.duckduckgo.com. Unwraps redirected links.
Brave	HTML Scraper	Parses search.brave.com SERP layout.
Bing	HTML Scraper	Extracts organic results while skipping ads.
Mojeek	HTML Scraper	Provides fallback search capability.
Wikipedia	HTML Scraper	Scrapes information specifically from the Wikipedia domain.

Sources: packages/conduit_ddgs/VENDORED.md:22-25, packages/conduit_ddgs/test/fixtures/duckduckgo_results.html

Search Logic and Handling

The search process includes several safety and performance optimizations:

• Engine Cooldown: If an engine blocks a request (e.g., via Captcha or rate-limit), the system cools down that engine instead of retrying it immediately on subsequent queries.
• Link Unwrapping: Scrapers extract the final destination URL from engine-specific redirectors (e.g., DuckDuckGo's //duckduckgo.com/l/?uddg=).
• Deduplication: When multiple engines return the same URL, the system keeps the first position but uses the version with the longest snippet.
• Request Cancellation: Searches use AbortableRequests, allowing the application to cancel a search mid-flight if the user stops the generation.

Sources: packages/conduit_ddgs/VENDORED.md:32-41

Result Structure

Search results are converted into a typed WebSearchResult object. This structure is used to inform the assistant's response and provide citations.

{
  "action": "web_search",
  "items": [
    {
      "title": "Flutter release notes",
      "link": "https://docs.flutter.dev/release/release-notes",
      "snippet": "Release notes for the latest stable Flutter release."
    }
  ],
  "done": true
}

Sources: packages/conduit_core/test/fixtures/chat_blobs/05_web_search_status_sources.json:52-64

Data Extraction Process

The extraction logic relies on specific CSS selectors within the HTML fixtures of each engine. For example, DuckDuckGo results are parsed from the .serp__results container, while Brave uses specific svelte- classes for result content.

Maintenance and Testing

The scrapers are maintained using a "live smoke" tool that queries real engines to detect markup changes. Fixtures are updated when an engine's HTML structure changes significantly, causing the parser to return empty results.

• Tool: dart run tool/live_smoke.dart [query]
• Fixtures: Located in test/fixtures/. These files are stripped of scripts and styles before being used for unit testing the parsers.

Sources: packages/conduit_ddgs/VENDORED.md:46-55

The on-device web search module ensures Conduit remains functional as a standalone client by providing the necessary tools for real-time information retrieval without external dependencies.