Web Scraper Feature
Relevant source files
The following files were used as context for generating this wiki page:
The Web Scraper, also known as DocScraper, is an agentic web scraping tool designed to intelligently collect raw text from online sources like articles and blog posts. Its primary purpose is to build structured datasets that can be used as inputs for DocETL pipelines. The feature is powered by an AI agent that can search the web, execute Python code in a sandboxed environment to scrape content, and iteratively discover new links to build a comprehensive dataset based on a user's natural language query.
The DocScraper is a standalone feature within the DocWrangler frontend, accessible via the /scraper route. It provides an interactive chat-based interface where the user guides the agent, and can monitor its progress, view the code it executes, see the links it finds, and inspect the resulting dataset in real-time.
Sources: website/src/app/scraper/layout.tsx:4-6, website/src/app/scraper/page.tsx:324-335, website/src/app/api/scraper/route.ts:747-755
Architecture Overview
The DocScraper feature is composed of a frontend client, a backend API, and several external services for search and code execution. The architecture is designed for interactive, stateful, and secure web scraping sessions.
The diagram below illustrates the high-level architecture of the DocScraper feature.
Frontend Interface
The user interface for DocScraper is a multi-panel dashboard built with React and Next.js, providing a comprehensive view of the scraping process.
Sources: website/src/app/scraper/page.tsx
UI Layout
The screen is divided into four main resizable panels:
- Agent Chat: An interactive chat where the user provides the initial query and can communicate with the scraping agent. It displays the agent's thought process and tool usage.
- Found Links: A list of URLs discovered by the agent through web searches. Each entry includes the title, URL, and a snippet.
- Code Execution: Shows the Python code the agent executes in the sandbox, along with its status (running, completed, error) and any output or errors.
- Dataset: A live table view of the scraped data. It displays the collected rows and columns and provides options to download the dataset as JSON or CSV.
Sources: website/src/app/scraper/page.tsx:320-597
User Interaction Flow
The scraping process is initiated and managed through the UI as follows:
- Initiation: The user provides a natural language query (e.g., "travel blog posts about Japan") and an optional schema (e.g., "url, scraped_text, author").
- Session Start: Upon clicking "Start Scraping," the frontend generates a unique
sessionId(e.g.,20250130-uuid) and sends the initial request to the/api/scraperbackend. - Monitoring: The user monitors the agent's progress through the chat panel. Tool invocations for searching and code execution are displayed in real-time.
- Live Data: The "Found Links," "Code Execution," and "Dataset" panels are updated incrementally as the agent works, using data extracted from the streaming chat messages.
- Completion & Download: Once the agent has collected enough data or the user stops the process, the final dataset can be downloaded.
The frontend uses the @ai-sdk/react useChat hook to manage the streaming conversation with the backend.
Sources: website/src/app/scraper/page.tsx:64-70, 252-273
Backend API
The core logic of the scraping agent resides in the Next.js API route handler at /api/scraper/route.ts. It orchestrates the AI model, tool execution, and state management for each scraping session.
Sources: website/src/app/api/scraper/route.ts
API Endpoint: POST /api/scraper
This endpoint is the main entry point for all communication with the scraping agent.
| Parameter | Type | Description |
|---|---|---|
messages | array | The history of chat messages between the user and the agent. |
schema | string | (Optional) The desired schema for the dataset, provided by the user. |
sessionId | string | A unique identifier for the scraping session, used for state management. |
The API responds with a streaming data response (DataStreamResponse) containing the agent's text and tool-related data.
Sources: website/src/app/api/scraper/route.ts:680-686
Agent System Prompt
The agent's behavior is guided by a detailed system prompt that defines its mission, tools, and workflow.
Key Instructions:
- Mission: To deliver a clean, structured dataset at
/data/{sessionId}.json. The focus is on scraping raw text, not summarizing. - Cost-Effective Workflow: The agent is heavily instructed to prioritize scraping and link extraction over expensive new searches.
- Use
search_internetonce to get seed URLs. - Immediately use
execute_codeto scrape all found URLs in parallel. - Extract new links from scraped pages using
BeautifulSoup. - Continue scraping newly discovered links with
execute_code. - Only perform another search if new angles are needed.
- Use
- Tool Usage: The agent must use tools for all external actions, such as searching the web or executing code.
- Dataset Handling: It must follow a "load -> append -> write" pattern to avoid overwriting data and must log dataset statistics after each execution.
Sources: website/src/app/api/scraper/route.ts:747-849
Agent Tools
The agent has access to a set of tools to perform its tasks. These tools are defined using ai-sdk and zod for schema validation.
| Tool Name | Description | Key Parameters |
|---|---|---|
search_internet | Discovers new sources using the Tavily API. It is budget-constrained to encourage efficient use. | query, mode |
execute_code | Executes Python code in a secure Modal sandbox. This is the primary tool for scraping, parsing, and data manipulation. | code |
read_json | Reads the current state of the dataset from the persistent Modal volume, allowing the agent to inspect its progress. | filepath |
Sources: website/src/app/api/scraper/route.ts:325-327, 599-601, 654-656
The following sequence diagram shows a typical interaction flow involving the agent and its tools.
Sandboxed Code Execution with Modal
To securely execute arbitrary scraping code, the backend integrates with Modal Labs. This provides an on-demand, sandboxed Python environment with persistent storage.
Sources: website/src/app/api/scraper/route.ts:20-101
Execution Flow
The executeModalSandbox function manages the entire lifecycle of a code execution task.
This diagram shows the steps involved in executing a piece of Python code sent by the AI agent.
Persistent Storage
State between code executions is maintained using a Modal Volume.
- A volume named
scraper-datais created. - This volume is mounted into each sandbox at the
/datadirectory. - The agent is instructed to save its collected data to a session-specific file:
/data/{sessionId}.json. - Because the volume persists even after the sandbox is terminated, subsequent
execute_codecalls within the same session can access and append to the same dataset file, enabling iterative data collection. - The
readModalFilefunction provides a way to read the content of this file from the volume.
Sources: website/src/app/api/scraper/route.ts:26-29, 52-55, 110, 180-186
Session and State Management
Each scraping task is managed within a session, identified by a unique sessionId.
- Session ID: A unique ID (e.g.,
20250130-abc-123) is generated by the frontend when a new scraping task begins. This ID is passed in every API request. - Data Isolation: The
sessionIdis used as the filename for the dataset stored in the Modal volume (/data/{sessionId}.json), ensuring that data from different sessions is kept separate. - Usage Tracking: The backend uses in-memory
Mapobjects (searchCounts,tavilyCredits) keyed by thesessionIdto track and enforce usage limits for the Tavily search API. This is a simple implementation that resets on server restart.
Sources: website/src/app/scraper/page.tsx:256-260, website/src/app/api/scraper/route.ts:278-296, 839-842