DocWrangler UI Playground

Relevant source files

The following files were used as context for generating this wiki page:

• website/src/app/playground/page.tsx
• website/src/components/PipelineGui.tsx
• website/src/components/DatasetView.tsx
• website/src/components/PromptImprovementDialog.tsx
• website/src/components/NaturalLanguagePipelineDialog.tsx
• website/src/components/AIChatPanel.tsx
• README.md
• website/src/components/FileExplorer.tsx
• website/src/components/Output.tsx
• website/src/components/BookmarksPanel.tsx

DocWrangler is the interactive UI playground for the DocETL system. It is designed to help users iteratively develop, test, and refine complex document processing pipelines in a visual, real-time environment. The playground provides a comprehensive interface for managing data files, constructing pipeline operations, inspecting outputs, and leveraging AI-powered assistants to improve prompts and generate entire pipeline configurations from natural language.

The primary goal of DocWrangler is to facilitate rapid experimentation. Users can build pipelines step-by-step, run individual operations to see immediate results, and export the finalized pipeline configuration as a YAML file for use in production environments. The entire UI is built as a single-page application using Next.js and React.

Sources: README.md:20-26, website/src/app/playground/page.tsx:501-518

Architecture and Core Components

The DocWrangler UI is structured around a main application component, CodeEditorPipelineApp, which orchestrates a set of resizable panels. The overall layout is managed using ResizablePanelGroup components, allowing the user to adjust the workspace to their needs. State is managed globally through React Contexts, primarily PipelineProvider and BookmarkProvider, ensuring that all components have access to the current pipeline state, files, and user bookmarks.

Sources: website/src/app/playground/page.tsx:238-241, 712-715

Component Layout

The following diagram illustrates the high-level component structure of the DocWrangler playground interface.

Key UI Panels

The main workspace is composed of several key panels, each with a distinct function. The visibility of these panels can be toggled from the top bar.

Panel	Component	Description	Source File
File Explorer	FileExplorer	Manages project files, including uploading datasets and pipeline YAMLs, and selecting the active file.	website/src/components/FileExplorer.tsx
Bookmarks	BookmarksPanel	Displays user-created notes and feedback, which can be associated with specific pipeline operations.	website/src/components/BookmarksPanel.tsx
Pipeline GUI	PipelineGUI	The central canvas for visually building the pipeline by adding, configuring, and reordering operations.	website/src/components/PipelineGui.tsx
Output	Output	Shows the results, cost, and logs from running a pipeline or a single operation.	website/src/components/Output.tsx
Dataset View	DatasetView	Displays the content of the currently selected dataset (JSON or CSV) in a tabular format.	website/src/components/DatasetView.tsx

Sources: website/src/app/playground/page.tsx:255-278, 720-807

State Management

Global application state is managed using React's Context API, which avoids prop-drilling and provides a centralized state container.

PipelineContext

The PipelineProvider is the primary state manager for the playground. It wraps the main application and provides the pipeline state and dispatcher functions to all child components.

Key context values include:

• operations: An array of Operation objects representing the pipeline.
• files, currentFile: State for managing user-uploaded files.
• cost: The calculated cost of executed LLM operations.
• namespace: A unique identifier for the user's workspace to persist state in localStorage.
• saveProgress(): A function to save the current pipeline state to browser storage.
• clearPipelineState(): A function to reset the workspace.
• serializeState(): A function that creates a string representation of the current pipeline state, used by AI assistant features.

Sources: website/src/app/playground/page.tsx:43-46, 289-307

AI-Assisted Features

DocWrangler integrates several AI-powered features to assist with pipeline development. These features typically interact with a backend API endpoint (/api/chat or /api/generate) and use the current pipeline state as context.

Sources: website/src/components/AIChatPanel.tsx:80-89, website/src/components/PromptImprovementDialog.tsx:501-510, website/src/components/NaturalLanguagePipelineDialog.tsx:460-467

Natural Language Pipeline Generator

This feature allows users to generate an entire DocETL pipeline from a natural language description. It is implemented in the NaturalLanguagePipelineDialog component.

The process is as follows:

1. Upload & Describe: The user uploads a JSON or CSV dataset and provides a text prompt describing their goal.
2. Generation: The component sends the user's prompt, along with sample data from the uploaded file and a comprehensive set of instructions (/llms-full.txt), to the /api/generate endpoint.
3. Review & Apply: The API returns a generated pipeline name and YAML configuration. The user can review the YAML and, if satisfied, apply it to the workspace. Applying the pipeline uses the useRestorePipeline hook to parse the YAML and update the application state.

The following diagram shows the data flow for this feature.

Prompt Improvement Assistant

The PromptImprovementDialog provides an interactive, AI-driven way to refine prompts for LLM-based operations.

Key Features:

• Two-Step Process: First, the user selects an operation to improve. Then, the assistant analyzes the prompt and suggests changes.
• Context-Aware: The assistant receives the full pipeline state via serializeState() to make informed suggestions.
• Revision History: It maintains a tree of prompt revisions, allowing users to navigate different versions and provide iterative feedback.
• Diff View: Shows the differences between the original prompt and the AI-suggested version.
• Schema Suggestions: The AI can suggest changes to the output schema by including a <schema> tag in its response.
• Resolve Operation Support: It correctly handles resolve operations, which have separate comparison_prompt and resolution_prompt.

The interaction flow is depicted below:

AI Chat Panel

The AIChatPanel is a general-purpose assistant that can answer questions about the user's pipeline. It appears as a floating, resizable window.

• Contextual Awareness: Like the prompt improver, it sends the serialized pipeline state to the /api/chat endpoint with each message, allowing it to answer questions like "Am I doing this right?" or "Help me with jinja2 templating" in the context of the user's current work.
• Goal Setting: Users can define a high-level goal for their analysis, which is included in the system prompt to guide the assistant's suggestions.

Sources: website/src/components/AIChatPanel.tsx:142-238

Conclusion

The DocWrangler UI Playground is a powerful and feature-rich integrated development environment for DocETL. It combines a visual pipeline builder with real-time feedback and a suite of AI-powered assistants. This design significantly lowers the barrier to entry for creating complex data processing workflows, enabling both novice and expert users to iteratively develop, debug, and optimize pipelines for document analysis tasks. Its modular architecture, based on React and Next.js, provides a flexible and extensible platform for future development.

Sources: README.md:20-26, website/src/app/playground/page.tsx