Configuration
Relevant source files
The following files were used as context for generating this wiki page:Configuration in the docetl project is managed through a dual-system approach, separating the settings for the backend Python server from the frontend Next.js application (DocWrangler). This separation allows for distinct configurations for pipeline execution and user interface features. The primary methods of configuration are environment variables loaded from .env files and declarative YAML files that define the structure and logic of data processing pipelines.
The backend relies on a root .env file for pipeline execution parameters, such as API keys for LLM providers and server settings. The frontend uses a website/.env.local file to manage UI-specific features, like the AI chat assistant and the "Improve Prompt" functionality. This architecture ensures that production pipeline execution is decoupled from the interactive development environment's settings.
Environment Variables
The project uses two distinct environment files to manage settings for its backend and frontend components.
This diagram shows the two main configuration files and the system components they control.
Sources: README.md:41-45, website/README.md:5-10
Backend Configuration (.env)
The backend configuration is managed by a .env file located in the project's root directory. This file is crucial for the Python server that executes DocETL pipelines.
Sources: README.md:42-45, README.md:112-128
Example .env file:
# Used by DocETL pipeline execution engine
OPENAI_API_KEY=your_api_key_here
# BACKEND configuration
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True
# FRONTEND configuration
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000
# Host port mapping for docker-compose
FRONTEND_DOCKER_COMPOSE_PORT=3031
BACKEND_DOCKER_COMPOSE_PORT=8081
# Supported text file encodings
TEXT_FILE_ENCODINGS=utf-8,latin1,cp1252,iso-8859-1
Sources: README.md:112-128
The following table details the key variables for the backend:
| Variable | Description | Example |
|---|---|---|
OPENAI_API_KEY | API key for the LLM provider (e.g., OpenAI) used by the pipeline execution engine. | sk-xxx... |
BACKEND_ALLOW_ORIGINS | Comma-separated list of allowed origins for CORS. | http://localhost:3000 |
BACKEND_HOST | Host for the backend server. | localhost |
BACKEND_PORT | Port for the backend server. | 8000 |
BACKEND_RELOAD | Enables auto-reloading for the development server. | True |
TEXT_FILE_ENCODINGS | Supported text file encodings. | utf-8,latin1 |
AWS_PROFILE | AWS CLI profile for Bedrock integration. Defaults to 'default'. | your-profile |
AWS_REGION | AWS region for Bedrock integration. Defaults to 'us-west-2'. | us-east-1 |
Sources: README.md:112-128, README.md:164-165
Frontend Configuration (website/.env.local)
The frontend configuration is handled by a .env.local file within the website/ directory. These settings control UI-specific features in DocWrangler that interact with LLMs, such as the AI chat assistant and the prompt improvement dialog.
Sources: website/README.md:3-10, README.md:130-137
Example website/.env.local file:
# Used by TypeScript features: improve prompt, chatbot, etc.
OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini # Model used by the UI assistant
NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000
NEXT_PUBLIC_HOSTED_DOCWRANGLER=false
Sources: README.md:130-137, website/README.md:12-19
The following table details the key variables for the frontend:
| Variable | Description | Example |
|---|---|---|
OPENAI_API_KEY | API key used by UI features like the chatbot and prompt improver. | sk-xxx... |
OPENAI_API_BASE | The base URL for the OpenAI-compatible API. | https://api.openai.com/v1 |
MODEL_NAME | The specific LLM model used by the UI assistant features. | gpt-4o-mini |
NEXT_PUBLIC_BACKEND_HOST | The hostname of the backend server, accessible from the user's browser. | localhost |
NEXT_PUBLIC_BACKEND_PORT | The port of the backend server. | 8000 |
NEXT_PUBLIC_HOSTED_DOCWRANGLER | A flag to indicate if the UI is running in a hosted environment. | false |
Sources: website/README.md:12-19, README.md:130-137
The frontend allows users to toggle the use of a personal OpenAI key for features like the AI Chat Panel and Prompt Improvement dialog. This is managed via a Switch component in the UI.
Sources: website/src/components/AIChatPanel.tsx:30-38, website/src/components/PromptImprovementDialog.tsx:550-561
Pipeline Configuration (YAML)
The core of docetl's functionality is defined in YAML files. These files declaratively specify the entire data processing pipeline, from data sources to the sequence of operations and output format.
Sources: website/public/llms.txt:5-8, website/src/components/PresidentialDebateDemo.tsx:21-160
A typical pipeline YAML has the following top-level structure:
| Key | Description |
|---|---|
datasets | Defines the input data sources for the pipeline. |
default_model | The default LLM to be used for all LLM-based operations unless overridden. |
operations | A list of all processing steps (e.g., map, reduce, filter). |
pipeline | Defines the execution flow, specifying the sequence of operations. |
system_prompt | A global prompt describing the dataset and the desired LLM "persona". |
Sources: website/src/components/PresidentialDebateDemo.tsx:21-160, website/public/llms.txt:202-207, website/src/components/NaturalLanguagePipelineDialog.tsx:321-326
The diagram below illustrates the structure of a typical pipeline configuration file.
LLM Provider Configuration
docetl uses litellm to support various LLM providers. The primary configuration is done via the OPENAI_API_KEY environment variable, but the system is extensible to other providers like AWS Bedrock. Bedrock models are specified by prefixing the model name with bedrock.
Sources: pyproject.toml:18, README.md:154-171
The diagram below shows the interaction flow when a UI feature makes a request that involves an LLM.
This sequence shows how a UI component like the AI Chat Panel uses its configuration to communicate with an LLM provider via a dedicated API route.
Sources: website/src/components/AIChatPanel.tsx:100-136, website/src/components/PromptImprovementDialog.tsx:300-320
Summary
The configuration system in docetl is designed to be flexible and robust, clearly separating backend pipeline execution from frontend UI enhancements. Backend settings in the root .env file control core processing and data handling, including integration with various LLM providers like OpenAI and AWS Bedrock. Frontend settings in website/.env.local are dedicated to interactive features within the DocWrangler UI, providing a separate context for development and experimentation. At the heart of the system are the YAML pipeline files, which provide a powerful, declarative way to define complex document processing workflows. This layered configuration approach supports both production use via the Python package and iterative development in the interactive playground.
Sources: README.md:38-45, website/README.md:5-10