Backend API Endpoints
Relevant source files
The following files were used as context for generating this wiki page:The DocETL backend is composed of two distinct services that work together to power the DocWrangler UI and execute data processing pipelines. The first is a Python-based server for core pipeline functionality, and the second is a Next.js-based backend for UI-centric AI features. This dual-backend architecture separates the core data processing engine from the interactive, AI-assisted features of the web interface.
The primary backend is a Python server responsible for executing DocETL pipelines, managing the user's workspace filesystem, and handling data conversions. The secondary backend consists of Next.js API Routes that provide higher-level, AI-driven functionalities such as natural language pipeline generation, an iterative web scraper, and a chat-based assistant. Configuration for these two backends is managed separately in the root .env file for the Python server and website/.env.local for the Next.js API routes.
Sources: README.md, website/README.md
System Architecture
The diagram below illustrates the high-level architecture, showing the interaction between the user's browser (DocWrangler UI), the Next.js API routes, and the Python DocETL server.
Next.js API Routes
These endpoints are part of the Next.js application and are primarily used to support the interactive features of the DocWrangler UI. They are defined under website/src/app/api/.
Sources: website/README.md
Web Scraper Agent (/api/scraper)
This endpoint provides an iterative, AI-powered web scraping agent. It uses a combination of an LLM, a search API (Tavily), and a sandboxed code execution environment (Modal) to discover and scrape web content based on a user's request.
Key Features:
- Stateful Sessions: Uses a
sessionIdto manage the state of a scraping job, allowing for iterative refinement. - Sandboxed Execution: Executes Python scraping code in a secure Modal sandbox, which has common scraping libraries like
requests,beautifulsoup4, andplaywrightpre-installed. - Cost Management: Implements a credit system to manage calls to the Tavily search API, with a hard limit of
MAX_SEARCHES_PER_SESSIONandMAX_TAVILY_CREDITSto control costs. - Persistent Storage: Scraped data is saved to a Modal volume, ensuring data persists between scraping iterations. The data for a session is stored at
/data/{sessionId}.json. - Tool-Using Agent: The underlying LLM agent can use several tools:
search_internet: To find initial URLs using Tavily.execute_code: To run Python code for scraping URLs and discovering new links.read_json: To inspect the currently collected dataset.
Sources: website/src/app/api/scraper/route.ts:1-666
The following sequence diagram shows the workflow for a scraping task.
Sources: website/src/app/api/scraper/route.ts
Natural Language Pipeline Generation (/api/generate)
This endpoint generates a DocETL pipeline YAML file from a natural language description provided by the user. It is used by the "Generate Pipeline from Natural Language" feature in the UI.
Flow:
- The UI collects a user prompt, an optional uploaded dataset (JSON/CSV), and the current default model.
- A sample of the dataset is extracted and formatted into a detailed prompt.
- The prompt, which includes instructions from
/llms-full.txt, is sent to the/api/generateendpoint. - The endpoint uses an LLM (specified as
gpt-5) to generate a response containing the pipeline name and the full YAML configuration. - The UI parses the name and YAML from the response and allows the user to apply it to their workspace.
Sources: website/src/components/NaturalLanguagePipelineDialog.tsx:254-367, public/llms.txt
AI Chat Assistant (/api/chat)
This is a versatile endpoint that powers multiple AI-assisted features in the DocWrangler UI. The behavior of the endpoint is determined by the x-source HTTP header.
x-source Header Value | Feature | Description |
|---|---|---|
ai_chat | AI Chat Panel | Provides general assistance for building and refining pipelines. It receives the serialized pipeline state as context. |
prompt_improvement | Improve Prompt Dialog | Analyzes an existing operation's prompt and suggests improvements based on best practices and user feedback. |
The endpoint can also use a personal OpenAI API key if the x-use-openai header is set to true and the x-openai-key header is provided.
Sources: website/src/components/AIChatPanel.tsx:55-70, website/src/components/PromptImprovementDialog.tsx:492-506
Tutorial Dataset Downloader (/api/downloadTutorialDataset)
This endpoint acts as a server-side proxy to download tutorial datasets from Google Drive. It takes a Google Drive fileId as a query parameter, downloads the file, and streams it back to the client. This is used to load example pipelines and their associated data into the user's workspace without running into CORS issues.
Sources: website/src/components/TutorialsDialog.tsx:206-214
Python (FastAPI) Backend Endpoints
The core DocETL engine is a Python server built with FastAPI. It is responsible for the direct execution and management of pipelines and files. These endpoints are not documented in the provided source files, but their existence and purpose can be inferred from the project structure and README.md. The following is a high-level summary based on typical FastAPI patterns and file names.
Inferred Endpoints
| Method(s) | Path | Router File | Description |
|---|---|---|---|
WebSocket | /run | routes/pipeline.py | Executes a pipeline and streams status and results back to the client over a WebSocket connection. |
GET | /status/{run_id} | routes/pipeline.py | Checks the status of an ongoing or completed pipeline run. |
GET | /results/{run_id} | routes/pipeline.py | Fetches the final results of a completed pipeline run. |
POST | /convert | routes/convert.py | Converts data between different formats. |
GET | / | routes/filesystem.py | Lists files and directories in the user's workspace (namespace). |
GET | /read | routes/filesystem.py | Reads the content of a specific file. |
POST | /write | routes/filesystem.py | Writes or updates a file in the workspace. |
DELETE | /delete | routes/filesystem.py | Deletes a file from the workspace. |
This table is an inferred summary based on file names and common application structure. The exact implementation details are not present in the provided source files.
Summary
The DocETL backend is a sophisticated, dual-component system. The Python/FastAPI server provides the robust, low-level engine for pipeline execution and file management, forming the core of the docetl package. Complementing this is a set of Next.js API routes that deliver a rich, AI-enhanced user experience in the DocWrangler UI, handling complex tasks like agent-based web scraping and natural language pipeline generation. This separation of concerns allows the core engine to remain focused and efficient while the UI-facing services can iterate quickly on user-facing features.