Getting Started

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• Makefile
• pyproject.toml
• website/README.md
• website/src/app/playground/page.tsx
• website/src/components/NaturalLanguagePipelineDialog.tsx
• website/src/components/PromptImprovementDialog.tsx

DocETL is a tool for creating and executing data processing pipelines, particularly for complex document processing tasks. It offers two primary methods of use: an interactive UI playground called DocWrangler for development and a Python package for production environments. This guide covers the initial setup for both approaches.

There are two main ways to get started with DocETL. For iterative development and prompt engineering, the DocWrangler UI Playground is recommended. For integrating DocETL into production workflows, the Python package is the intended method.

Sources: README.md:9-13

1. Using the Python Package (Production)

This method is for using DocETL as a library within a Python project.

Sources: README.md:43-44

Prerequisites

• Python 3.10 or later
• An API key for your chosen LLM (e.g., OpenAI)

Sources: README.md:46-47, pyproject.toml:5

Installation

Install the package using pip:

pip install docetl

This command installs the core dependencies required to run DocETL pipelines.

Sources: README.md:49-51

Core Dependencies

The pip install command will install several core packages, including:

• litellm: For interacting with various LLM providers.
• typer: For the command-line interface.
• fastapi & uvicorn: For the server component (as an optional dependency).
• pandas: For data manipulation.
• modal: For sandboxed code execution.

Sources: pyproject.toml:13-33, pyproject.toml:40-46

Configuration

Create a .env file in your project's root directory to store your API keys. This file is used by the backend Python server that executes the pipelines.

# .env
OPENAI_API_KEY=your_api_key_here

This is the minimum required configuration for operations that utilize LLMs.

Sources: README.md:53-56

2. Using DocWrangler (UI Playground)

DocWrangler is the interactive UI for developing, testing, and visualizing DocETL pipelines. It consists of a Next.js frontend and a Python backend.

Sources: README.md:20-25, website/README.md:1-3

Architecture Overview

DocWrangler operates with a separate frontend and backend, each with its own environment configuration. The frontend is a Next.js application, while the backend is a Python server that executes the actual DocETL pipelines.

This diagram illustrates the high-level architecture and configuration flow.

Environment Configuration

Proper setup requires two separate environment files. It is critical to distinguish between them.

Root .env File

This file configures the backend Python server. It is placed in the project's root directory.

Variable	Description	Example Value
OPENAI_API_KEY	API key for the LLM used in pipeline execution.	your_api_key_here
BACKEND_ALLOW_ORIGINS	Allowed origins for CORS.	http://localhost:3000
BACKEND_HOST	Host for the backend server.	localhost
BACKEND_PORT	Port for the backend server.	8000
FRONTEND_DOCKER_COMPOSE_PORT	Host port mapping for the frontend container.	3031
BACKEND_DOCKER_COMPOSE_PORT	Host port mapping for the backend container.	8081

Sources: README.md:90-106

website/.env.local File

This file configures the frontend Next.js application (DocWrangler UI). It is placed inside the website/ directory and is used for UI-specific features like the AI chat assistant and the "Improve Prompt" functionality.

Variable	Description	Example Value
OPENAI_API_KEY	API key for UI assistant features.	sk-xxx
MODEL_NAME	The model used by the UI assistant.	gpt-4o-mini
NEXT_PUBLIC_BACKEND_HOST	Host of the backend server, accessible by the frontend.	localhost
NEXT_PUBLIC_BACKEND_PORT	Port of the backend server, accessible by the frontend.	8000

Sources: website/README.md:8-16, README.md:108-115

Option A: Docker Setup (Recommended)

This is the quickest way to get DocWrangler running locally.

1. Create Environment Files: Create the root .env and website/.env.local files as described above.

2. Run Docker: Execute the following command from the project root.

   make docker
   

   This command builds the Docker images and starts the frontend and backend services. The UI will be accessible at http://localhost:3000.

The flow of the make docker command is as follows:

To stop and remove the containers and associated volume, use:

make docker-clean

Sources: README.md:125-127

AWS Bedrock Integration

To use AWS Bedrock models, configure your AWS credentials and run the Docker command with AWS_PROFILE and AWS_REGION variables. Bedrock models are prefixed with bedrock.

AWS_PROFILE=your-profile AWS_REGION=your-region make docker

Sources: README.md:131-149

Option B: Manual Setup

This method is for developers who want to modify the source code.

1. Clone Repository:

   git clone https://github.com/ucbepic/docetl.git
   cd docetl
   

   Sources: README.md:153-156

2. Create Environment Files: Create the root .env and website/.env.local files as described in the sections above.
   Sources: README.md:158-183

3. Install Dependencies: Run the following make commands to install Python and UI dependencies.

   # Install Python dependencies using uv
   make install
   
   # Install UI (Node.js) dependencies
   make install-ui
   

   Sources: README.md:185-188

4. Start Development Server:

   make run-ui-dev
   

   This starts the Next.js development server.
   Sources: README.md:197-199

5. Access the UI: Visit http://localhost:3000/playground in your browser.
   Sources: README.md:201-202, website/src/app/playground/page.tsx

Makefile Command Reference

The Makefile provides several useful commands for setup and development.

Command	Description
docker	Builds and runs the application using Docker Compose.
docker-clean	Stops and removes Docker containers and volumes.
install	Installs Python dependencies using uv.
install-ui	Installs frontend (Node.js) dependencies.
run-ui-dev	Starts the frontend development server.
tests-basic	Runs the basic test suite to verify the setup.

Sources: Makefile