Welcome to DocETL
Relevant source files
The following files were used as context for generating this wiki page:DocETL is a system for creating and executing data processing pipelines, particularly suited for complex document processing tasks powered by Large Language Models (LLMs). It provides a declarative YAML interface for defining complex data operations on unstructured datasets. The project is composed of two primary components: an interactive UI for development and a Python package for production use.
The system is designed to be extensible, supporting various LLM providers through litellm, custom data parsers, and a rich set of operators. It also includes advanced features like an agentic reasoning optimizer (MOAR) and AI-assisted pipeline generation to streamline development and enhance performance.
Sources: README.md:10-13, website/public/llms.txt:3-5
Core Components
DocETL is split into two main parts that serve different stages of the development lifecycle: the DocWrangler UI for interactive development and the Python package for production execution.
Sources: README.md:18-19
DocWrangler: The Interactive UI Playground
DocWrangler is the web-based frontend for DocETL, designed for iterative pipeline development. It is a Next.js application located in the website/ directory of the repository.
Key Features:
- Iterative Development: Experiment with prompts and see results in real-time.
- Visual Pipeline Building: Construct pipelines step-by-step in a graphical interface.
- Pipeline Export: Export the final pipeline configuration as a YAML file for production use.
- AI-Assisted Features: Includes tools to generate pipelines from natural language descriptions and improve existing prompts.
The playground is hosted at docetl.org/playground but can also be run locally.
Sources: README.md:23-31, website/README.md:1, website/src/app/playground/page.tsx
Python Package
The docetl Python package is intended for running finalized pipelines in a production environment. It can be used from the command line or integrated into other Python applications.
Key Features:
- CLI Interface: Execute pipelines directly from the terminal using the
docetlcommand. - Programmatic Execution: Run pipelines from within Python code.
- Extensibility: Defines core operations and parsers as entry points, allowing for custom extensions.
Installation is done via pip:
pip install docetl
Sources: README.md:33-39, pyproject.toml:55
System Architecture
DocETL operates on a client-server architecture, with the DocWrangler frontend communicating with a Python backend that executes the pipeline logic.
This diagram illustrates the interaction between the Next.js frontend (DocWrangler), the Python backend which runs the DocETL engine, and external LLM services.
Sources: README.md:44, 104, pyproject.toml:40, website/src/app/api/scraper/route.ts:25
Environment Configuration
The system uses two separate environment files to manage settings for the frontend and backend.
| File | Location | Purpose | Example Variables |
|---|---|---|---|
.env | Project Root | Configures the backend Python server that executes pipelines. | OPENAI_API_KEY, BACKEND_HOST, BACKEND_PORT |
.env.local | website/ | Configures frontend TypeScript features like the AI chat assistant and prompt improvement. | OPENAI_API_KEY, MODEL_NAME, NEXT_PUBLIC_BACKEND_HOST |
Sources: README.md:46-49, 104-107
Pipeline Concepts
DocETL pipelines are defined in a declarative YAML format. A pipeline consists of datasets, operations, and a specification of the execution flow.
Sources: website/public/llms.txt:5
Data Loading
Input data for a pipeline must be in JSON or CSV format. The data source is configured in the datasets section of the pipeline YAML.
- JSON: A list of objects, where each object is a document.
- CSV: A file where the first row contains headers, and each subsequent row is a document.
Example configuration:
datasets:
documents:
type: file
path: "data.json" # or "data.csv"
Sources: website/public/llms.txt:30-59
Operators
Operators are the fundamental building blocks of a DocETL pipeline. They define the processing steps that transform the data. The system includes a variety of core, auxiliary, and code-based operators.
The available operators are defined as entry points in the pyproject.toml file.
| Operator Type | Name | Description |
|---|---|---|
| LLM-Powered | map | Applies a prompt to each document individually. |
| LLM-Powered | reduce | Aggregates results from multiple documents into a single output. |
| LLM-Powered | filter | Keeps or discards documents based on a boolean condition from a prompt. |
| LLM-Powered | resolve | A specialized operator for complex reasoning tasks. |
| Auxiliary | split | Breaks a document field into smaller chunks. |
| Auxiliary | gather | Merges chunks back into a single document structure. |
| Auxiliary | unnest | Flattens a list within a document into multiple documents. |
| Auxiliary | sample | Selects a subset of documents. |
| Code-Based | code_map | Applies a Python function to each document. |
| Code-Based | code_filter | Filters documents using a Python function that returns a boolean. |
This table shows a selection of the core operators available in DocETL.
Sources: pyproject.toml:56-71, website/public/llms.txt:16-27, mkdocs.yml:23-40
The following diagram shows a conceptual flow of a simple pipeline.
This diagram illustrates how a dataset flows through a sequence of map, filter, and reduce operations to produce a final output.
Sources: website/public/demos/epstein_email_pipeline.yaml:220-226
Advanced Features
DocETL includes several advanced features designed to accelerate development and optimize pipeline execution.
Natural Language Pipeline Generation
DocWrangler provides a feature to generate a complete DocETL pipeline from a natural language description. The user provides a dataset and describes their goal, and an AI agent constructs the corresponding YAML configuration.
The process flow is as follows:
- The user uploads a dataset (JSON/CSV) and writes a prompt describing the desired outcome.
- The UI sends a request to the
/api/generateendpoint. The request payload contains a detailed prompt including system instructions, a sample of the user's data, and the user's goal. - The backend LLM processes this prompt and generates a response containing the pipeline name and the full YAML configuration.
- The UI displays the generated YAML for the user to review and apply to their workspace.
This sequence diagram shows the interaction for generating a pipeline from natural language.
Sources: website/src/components/NaturalLanguagePipelineDialog.tsx:21-24, 280-343
MOAR: Reasoning Optimizer
DocETL includes a MCTS-based Optimizer for Agentic Reasoning (MOAR), which uses a graph-search algorithm to automatically rewrite and optimize pipelines. It applies "directives"—pre-defined transformation patterns—to improve cost or accuracy.
Example Directives:
doc_summarization: Adds amapoperation at the beginning of a pipeline to summarize long documents, reducing token count for downstream operations.cascade_filtering: Injects a series of cheapercode_filterorgpt-5-nanofilters before an expensivefilteroperation to reduce the number of documents the main filter needs to process.doc_chunking_topk: Replaces amaporfilteron a long document with asplit -> topk -> reducepattern to find and process only the most relevant chunks.
Sources: docetl/reasoning_optimizer/AGENT.md:4, docetl/reasoning_optimizer/directives/doc_summarization.py:18-26, docetl/reasoning_optimizer/directives/cascade_filtering.py:18-23, docetl/reasoning_optimizer/directives/doc_chunking_topk.py:21-29
DocScraper: Web Scraping Agent
The project contains an experimental web scraping agent named DocScraper. This agent can be invoked via an API endpoint to collect data from the internet. It uses a set of tools to perform its tasks, demonstrating the framework's extensibility.
Tools:
search_internet: Uses the Tavily API to find URLs based on a query.execute_code: Runs Python code in a sandboxed environment (via Modal Labs) to scrape web pages, extract links, and parse content (including PDFs).read_json: Reads the state of the collected dataset from the sandbox volume.
This feature is designed for cost-effective data collection by prioritizing scraping and link extraction within the execute_code tool over repeated, expensive calls to the search API.
Sources: website/src/app/api/scraper/route.ts:468, 220-430
Project Dependencies & Structure
The project's dependencies and structure are defined in pyproject.toml.
Key Dependencies
| Dependency | Purpose |
|---|---|
litellm | Provides a unified interface to various LLM providers (OpenAI, Azure, etc.). |
pydantic | Used for data validation and defining schemas for operator configurations and outputs. |
fastapi | The web framework used for the backend server. |
pandas | Used for data manipulation, particularly with the Pandas integration. |
modal | Powers the sandboxed execution environment for the DocScraper agent. |
typer | Used to build the command-line interface. |
Sources: pyproject.toml:13-54
Entry Points
DocETL uses Python's entry points mechanism to register its core components, making the system modular and extensible.
docetl.operation: Registers all available pipeline operators (e.g.,map,reduce,code_filter).docetl.parser: Registers functions for parsing different data formats (e.g.,xlsx_to_string,paddleocr_pdf_to_string).
This design allows developers to add custom operators or parsers to the framework without modifying the core codebase.
Sources: pyproject.toml:56-81
Conclusion
DocETL is a comprehensive framework for building, executing, and optimizing LLM-powered document processing pipelines. It combines an interactive development environment (DocWrangler) with a robust Python backend, offering a flexible and powerful tool for developers. Its architecture supports a wide range of use cases, from simple data extraction to complex, multi-step reasoning tasks, while advanced features like the MOAR optimizer and AI-assisted tooling help streamline the development process and improve pipeline efficiency.