Local Development Setup

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• pyproject.toml
• website/package.json
• website/README.md
• mkdocs.yml
• website/src/app/api/scraper/route.ts
• website/src/components/AIChatPanel.tsx

The DocETL project consists of two primary components: a Python backend for executing data processing pipelines and a Next.js frontend, called DocWrangler, which provides an interactive UI for pipeline development. Setting up a local development environment allows for contributing to the project, running it without an internet connection (after initial setup), and testing changes across both the backend and frontend.

This guide covers the architecture, environment configuration, and step-by-step instructions for setting up the full DocETL stack on a local machine using either Docker or a manual process.

Sources: README.md:10-14, website/README.md:1-2

System Architecture

The local development environment runs a full-stack application comprising a frontend web server and a backend API server.

This diagram shows the local development architecture. The DocWrangler frontend (Next.js) runs on port 3000 and communicates with the DocETL backend (Python/FastAPI) on port 8000. The backend, in turn, makes requests to external LLM services.

Sources: README.md:100-103, website/README.md:16-18

Prerequisites

Before starting, ensure you have the following installed:

• Python: Version 3.10 or later.
• Node.js: Required for the frontend (inferred from npm/yarn commands).
• Docker: Required for the Docker-based setup.
• make: Used for command shortcuts.
• uv: The Python package installer used by the project. It can be installed via a curl command if not present.

Sources: pyproject.toml:5, README.md:63, README.md:154-155

Environment Configuration

DocETL uses two separate environment files for the backend and frontend. This separation isolates keys for pipeline execution from keys used for UI-only features.

This diagram illustrates the separation of concerns between the two environment files.

Sources: README.md:58-61, website/README.md:5-9

Backend Configuration (.env)

Create a .env file in the project's root directory. This file configures the Python server that executes pipelines.

# .env

# Used by DocETL pipeline execution engine
OPENAI_API_KEY=your_api_key_here

# BACKEND configuration
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True

# FRONTEND configuration (for Docker)
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000

# Host port mapping for docker-compose
FRONTEND_DOCKER_COMPOSE_PORT=3031
BACKEND_DOCKER_COMPOSE_PORT=8081

# Supported text file encodings
TEXT_FILE_ENCODINGS=utf-8,latin1,cp1252,iso-8859-1

Sources: README.md:98-118

Variable	Description	Default/Example
OPENAI_API_KEY	API key for the LLM provider used in pipeline operations.	your_api_key_here
BACKEND_ALLOW_ORIGINS	Comma-separated list of allowed origins for CORS.	http://localhost:3000,...
BACKEND_HOST	Host for the backend server.	localhost
BACKEND_PORT	Port for the backend server.	8000
FRONTEND_DOCKER_COMPOSE_PORT	Host port mapped to the frontend container.	3031
BACKEND_DOCKER_COMPOSE_PORT	Host port mapped to the backend container.	8081
TEXT_FILE_ENCODINGS	Supported text file encodings.	utf-8,latin1,...

Frontend Configuration (website/.env.local)

Create a .env.local file inside the website/ directory. This file configures UI-specific features like the AI chatbot and prompt improvement assistant.

# website/.env.local

# Used by TypeScript features: improve prompt, chatbot, etc.
OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

# Backend location for the frontend to connect to
NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000
NEXT_PUBLIC_HOSTED_DOCWRANGLER=false

Sources: website/README.md:10-18, README.md:121-128

Variable	Description	Default/Example
OPENAI_API_KEY	API key for UI assistant features.	sk-xxx
MODEL_NAME	The model used by the UI assistant features.	gpt-4o-mini
NEXT_PUBLIC_BACKEND_HOST	Host of the backend server, for the frontend to connect.	localhost
NEXT_PUBLIC_BACKEND_PORT	Port of the backend server.	8000

Setup Methods

You can set up the local environment using Docker for a quick start or by manually installing dependencies for more control over the development process.

Option A: Docker Setup (Recommended)

This method uses Docker Compose to build and run the frontend and backend services in containers.

This flowchart shows the simplified process for getting started with Docker.

1. Create Environment Files: Create the .env and website/.env.local files as described in the configuration section.
2. Run Docker: Execute the following command from the root directory.

   make docker
   

   This command builds the necessary images and starts the services. The UI will be accessible at http://localhost:3000.
3. Cleanup: To stop and remove the containers and the associated Docker volume, run:

   make docker-clean
   

AWS Bedrock Integration

To use AWS Bedrock models, first configure your AWS credentials, then run the make command with your profile and region. Bedrock models are prefixed with bedrock.

aws configure
AWS_PROFILE=your-profile AWS_REGION=your-region make docker

Sources: README.md:93-149

Option B: Manual Setup

This method is for developers who want to work on the source code directly without Docker.

This flowchart outlines the manual installation steps for both backend and frontend.

1. Clone the repository:

   git clone https://github.com/ucbepic/docetl.git
   cd docetl
   

2. Set up environment files: Create .env in the root and website/.env.local in the website directory as previously described.
3. Install dependencies:

   # Install Python dependencies with uv and setup pre-commit hooks
   make install
   
   # Install UI (Node.js) dependencies
   make install-ui
   

   Alternatively, you can use uv directly for Python dependencies:

   uv sync --all-groups --all-extras
   

4. Start the development server:

   make run-ui-dev
   

5. Access the UI: Visit http://localhost:3000/playground in your browser.

Sources: README.md:151-187

Dependency Management

Backend (Python)

Backend dependencies are defined in pyproject.toml and managed using uv.

Key Dependencies:

• fastapi, uvicorn: For the web server.
• litellm: For interacting with various LLM providers.
• pandas, numpy, scikit-learn: For data manipulation and analysis.
• typer: For the command-line interface.
• boto3: For AWS integration (e.g., Bedrock).

The project also defines optional dependency groups:

• parsing: Includes libraries like pymupdf and python-docx for handling different file formats.
• server: Includes fastapi and uvicorn for running the server component.
• dev: Includes tools for development and testing like pytest, ruff, and mkdocs.

Sources: pyproject.toml:9-80

Frontend (TypeScript/Next.js)

Frontend dependencies are defined in website/package.json and managed with npm, yarn, or pnpm.

Key Dependencies:

• next: The React framework.
• react, react-dom: The UI library.
• ai: Vercel AI SDK for building chat interfaces.
• tailwindcss: For styling.
• @radix-ui/*: For accessible UI components.
• lucide-react: For icons.
• framer-motion: For animations.

Sources: website/package.json:10-80

Running Tests

To verify that your setup is correct and that the core functionalities are working, you can run the basic test suite.

make tests-basic

This command runs a set of tests that may incur a small cost (estimated < $0.01 with OpenAI) as it makes live calls to the LLM API.

Sources: README.md:191-193