Interactive AI Assistants
Relevant source files
The following files were used as context for generating this wiki page:The DocWrangler frontend incorporates several interactive AI-powered assistants to aid users in building, refining, and understanding data processing pipelines. These assistants leverage large language models (LLMs) to provide contextual help, generate code, and improve prompts, streamlining the development workflow.
The assistants are implemented as distinct React components that communicate with backend API endpoints. This architecture allows for specialized AI interactions tailored to specific user tasks, such as general guidance, prompt engineering, and full pipeline generation from natural language. The system is designed to be flexible, allowing users to optionally provide their own OpenAI API keys for privacy and usage control.
Core Architecture
The AI assistant features follow a client-server architecture. Frontend React components capture user input and relevant application state, which are then sent to dedicated Next.js API routes for processing by an LLM.
This diagram illustrates the general interaction flow between the user, the DocWrangler UI, and the backend AI services.
API Headers
Communication between the frontend and the backend API routes is managed using custom HTTP headers to pass context and configuration.
| Header | Description | Example | Source File |
|---|---|---|---|
x-use-openai | A boolean flag ("true") indicating that the user's personal OpenAI key should be used instead of the service's default. | true | website/src/components/AIChatPanel.tsx:77-80 |
x-openai-key | The user's personal OpenAI API key, if provided. | sk-... | website/src/components/AIChatPanel.tsx:81-83 |
x-namespace | A unique identifier for the user's workspace to keep data and sessions separate. | 20240925-abc-123 | website/src/components/AIChatPanel.tsx:85 |
x-source | Identifies the origin of the AI request, allowing the backend to tailor the system prompt or logic. | ai_chat, prompt_improvement | website/src/components/AIChatPanel.tsx:100 |
Sources: website/src/components/AIChatPanel.tsx:75-88, website/src/components/PromptImprovementDialog.tsx:501-504, website/src/app/api/chat/route.ts:25-26
AI Assistants
DocWrangler provides several specialized AI assistants, each with a dedicated UI component and backend logic.
1. General AI Chat Panel
The AIChatPanel is a draggable and resizable chat window that provides general assistance for building and refining DocETL pipelines. It acts as a knowledgeable expert on the DocETL DSL and data analysis best practices.
Sources: website/src/components/AIChatPanel.tsx
Functionality
- Context-Awareness: Before each user message is sent, the chat panel serializes the current pipeline state (
serializeState()) and includes it in a detailed system prompt. This gives the AI full context of the user's work. - System Prompt: A comprehensive system prompt instructs the AI on its role as the "DocWrangler assistant," its capabilities, best practices, and specific rules for providing guidance.
- Personal API Key: Users can toggle a switch to use their personal OpenAI API key, which is then passed to the backend via the
x-use-openaiandx-openai-keyheaders. - Analysis Goal: Users can set a high-level goal for their pipeline, which is included in the context sent to the assistant to receive more relevant suggestions.
The system prompt is dynamically constructed to include the current pipeline state.
Sources: website/src/components/AIChatPanel.tsx:165-241
Data Flow
The following diagram shows the sequence of events when a user sends a message in the AI Chat Panel.
2. Prompt Improvement Assistant
The PromptImprovementDialog is a specialized tool for iteratively refining the prompt of a specific pipeline operation. It uses an AI assistant to analyze a prompt and suggest changes, allowing the user to provide feedback for further revisions.
Sources: website/src/components/PromptImprovementDialog.tsx
Functionality
- Revision History: The assistant maintains a tree of prompt revisions. Users can provide feedback on a suggestion, which creates a new branch in the revision history, or they can navigate back to previous versions.
- Structured Output: The AI is instructed to wrap its suggestions in specific XML-like tags (
<prompt>,<comparison_prompt>,<resolution_prompt>,<schema>). The frontend then parses these tags to display a diff view or update the prompt. - Contextual Analysis: The assistant receives the full pipeline state, the selected operation's details, and any bookmarked user notes related to that operation to provide highly relevant suggestions.
- Direct Editing and Diff View: Users can switch between a diff view showing the changes and a direct editing mode to manually tweak the AI's suggestion.
The core logic for extracting the revised prompt from the AI's response is handled by the extractTagContent function.
Sources: website/src/components/PromptImprovementDialog.tsx:77-82, 203-209
Workflow
This flowchart outlines the process of improving a prompt using the assistant.
3. Natural Language Pipeline Generator
The NaturalLanguagePipelineDialog enables users to generate a complete DocETL pipeline YAML configuration from a natural language description and a sample dataset (JSON or CSV).
Sources: website/src/components/NaturalLanguagePipelineDialog.tsx
Functionality
- Two-Step Process: The UI guides the user through a two-step process: 1) Upload a dataset and describe the goal, and 2) Review the generated pipeline YAML and apply it to the workspace.
- Data Sampling: A sample of the uploaded data is extracted and sent to the LLM to provide context about the data's structure and content.
- Structured Generation: The AI is prompted to return a pipeline name in
<name>tags and the full YAML configuration in<pipeline>tags. - Placeholder Injection: The generated YAML uses placeholders like
DATASET_PATH_PLACEHOLDER, which are replaced with the actual file paths when the user applies the pipeline.
The prompt sent to the /api/generate endpoint is constructed to include detailed instructions, the data sample, and the user's goal.
Sources: website/src/components/NaturalLanguagePipelineDialog.tsx:363-426
Data Flow
4. Web Scraping Agent (DocScraper)
The DocScraper is an autonomous agent designed for iterative web scraping, accessible via the /api/scraper endpoint. It uses a set of tools and a cost-aware workflow to collect data based on a user's request.
Sources: website/src/app/api/scraper/route.ts
Functionality
- Tool-Based Operation: The agent uses a predefined set of tools:
search_internet: Uses the Tavily API to find initial seed URLs.execute_code: Runs Python code in a secure Modal sandbox to scrape URLs, parse content, and discover more links.read_json: Inspects the state of the collected dataset.
- Cost-Effective Workflow: The agent is heavily optimized for cost. It is instructed to use the cheap
search_internettool sparingly and to rely primarily onexecute_codeto scrape pages and find new links from the content (e.g., pagination, sitemaps). - Budgeting: The agent operates under a strict budget for Tavily API calls (
MAX_TAVILY_CREDITSandMAX_SEARCHES_PER_SESSION) to prevent excessive costs. - Persistent State: Scraped data is saved to a Modal volume, ensuring that data persists across multiple
execute_codecalls within the same session.
The agent's core logic and workflow are defined in its system prompt.
// Source: website/src/app/api/scraper/route.ts:553-652
const systemPrompt = `You are DocScraper, an iterative web scraping agent focused on cost-effective data collection.
MISSION:
- Deliver a clean, structured dataset saved at /data/${sessionId}.json that satisfies the user's request.
- Your job is to SCRAPE RAW TEXT from articles, not to summarize or transform it.
COST-EFFECTIVE WORKFLOW:
1. Use search_internet to get initial seed URLs (costs 1 credit, returns up to 20 URLs)
2. **IMMEDIATELY use execute_code to scrape ALL URLs returned by Tavily in parallel** - don't wait, scrape them right away!
3. In the same code execution: extract additional links from the scraped pages using BeautifulSoup
4. Continue with execute_code to scrape discovered URLs - no additional Tavily calls needed
...
CRITICAL: After EVERY search_internet call, your VERY NEXT action must be execute_code to scrape those URLs!
...
`;
Sources: website/src/app/api/scraper/route.ts:553-652
Conclusion
The interactive AI assistants in DocWrangler represent a significant enhancement to the user experience. By integrating LLMs directly into the development environment, these features lower the barrier to entry for creating complex data processing pipelines. They provide intelligent, context-aware support that ranges from general guidance and fine-grained prompt optimization to the fully automated generation of entire pipelines, making the DocETL platform more accessible and powerful.
Sources: website/src/app/playground/page.tsx:400-418, README.md:13-15