Web Search & Scraping

Relevant source files

The following files were used as context for generating this wiki page:

• packages/coding-agent/src/web/search/index.ts
• packages/coding-agent/src/web/scrapers/index.ts
• packages/coding-agent/src/web/scrapers/go-pkg.ts
• packages/coding-agent/test/tools/web-scrapers/wikipedia.test.ts
• packages/coding-agent/test/tools/browser-readable.test.ts
• packages/coding-agent/src/prompts/system/web-search.md

Web Search & Scraping in Oh My Pi is a retrieval system designed to provide agents with real-time information from the internet. It combines broad search capabilities with specialized scrapers to convert web content into a structured, LLM-friendly Markdown format. This system ensures that agents have access to up-to-date documentation, external APIs, and general knowledge without relying solely on their static training data.

The module is divided into search orchestration, which interfaces with external providers like Exa, Brave, or Jina, and specialized scrapers that handle high-value technical domains such as Go package documentation, Wikipedia, and generic article-style pages.
Sources: packages/coding-agent/DEVELOPMENT.md, packages/coding-agent/src/prompts/system/web-search.md:1-5

Architecture Overview

The architecture follows a modular provider-and-handler pattern. Retrieval requests are routed through a central search index, while content extraction is delegated to specialized scrapers based on URL patterns.

The diagram shows how a high-level agent request flows through the search orchestration layer into specialized extraction logic.
Sources: packages/coding-agent/src/web/search/index.ts, packages/coding-agent/src/web/scrapers/index.ts

Specialized Scrapers

The project implements specialized handlers for domains where generic HTML-to-Markdown conversion is insufficient. These handlers often use secondary APIs or proxy services to retrieve clean data.

Go Package Documentation (pkg.go.dev)

The Go scraper targets Go module information. It extracts metadata including module paths, versions, licenses, and package synopses. It leverages the proxy.golang.org service to verify module versions and publication times.

Component	Responsibility
modulePath	Extracted module identifier from URL.
GoModuleInfo	Interface for JSON data from the Go proxy (Version, Time).
handleGoPkg	Primary handler for pkg.go.dev URLs.
Export Index	Extracts a list of identifiers (limited to first 50).

Sources: packages/coding-agent/src/web/scrapers/go-pkg.ts:7-20, packages/coding-agent/src/web/scrapers/go-pkg.ts:145-160

Wikipedia Integration

The Wikipedia handler is optimized for encyclopedic content. It filters out sections that are less useful for AI context, such as "References" and "External links," while handling internationalized domains and special characters in titles (e.g., "C++").
Sources: packages/coding-agent/test/tools/web-scrapers/wikipedia.test.ts:16-30, packages/coding-agent/test/tools/web-scrapers/wikipedia.test.ts:40-50

Generic Web Scraping & Readability

For pages without specialized handlers, the system uses a readability-focused extraction method. This is designed to strip boilerplate navigation and advertisements, focusing on the primary <article> or <main> content.

The sequence demonstrates the conversion process for standard web pages into readable text/markdown.
Sources: packages/coding-agent/test/tools/browser-readable.test.ts:4-15, packages/coding-agent/src/web/scrapers/go-pkg.ts:125-130

Search Orchestration

The system prioritizes accuracy and recency when performing web searches. It is governed by a specific set of priorities for synthesis:

1. Primary Sources: Prefers official documentation and specifications over blog summaries.
2. Transparency: Distinguishes between confirmed facts and inferences.
3. Data Inclusion: Specifically extracts version numbers, dates, and code snippets.

Common Providers

The search infrastructure supports multiple backends registered in the BUNDLED_PI_REGISTRY. These include:

• Exa: For semantic and neural search.
• Kagi: For privacy-focused search.
• Perplexity: For LLM-enhanced search results.
• Brave/DuckDuckGo: For standard web indexing.

Sources: packages/coding-agent/src/prompts/system/web-search.md:7-25, packages/coding-agent/src/extensibility/plugins/legacy-pi-bundled-registry.ts:430-450

Content Processing Logic

When content is fetched, it undergoes several transformation steps to ensure it is suitable for an LLM's context window:

• Truncation: Large lists (like Go exports or imports) are elided to prevent token overflow.
• Basic HTML to Markdown: Converts HTML structure (headings, lists, code blocks) to simple Markdown.
• Status Reporting: Results include metadata such as fetchedAt, method used, and finalUrl (handling redirects).

// Example of structured result building in scrapers
return buildResult(content, {
    url,
    finalUrl: pageResult.finalUrl,
    method: "go-pkg",
    fetchedAt: new Date().toISOString(),
    notes,
});

Sources: packages/coding-agent/src/web/scrapers/go-pkg.ts:240-246, packages/coding-agent/src/web/scrapers/go-pkg.ts:175-180

Conclusion

The Web Search & Scraping module provides a robust bridge between the oh-my-pi agent and the live web. By combining targeted scrapers for technical documentation (like Go packages) with generic readability tools, the system ensures high-quality context retrieval. This architectural approach allows the project to scale to new documentation sources simply by registering additional SpecialHandler functions in the scraper registry.