Deployment with Docker

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• website/README.md
• pyproject.toml
• website/src/app/playground/page.tsx
• website/src/components/NaturalLanguagePipelineDialog.tsx

The docetl project, including the DocWrangler interactive UI, can be deployed locally using Docker. This is the recommended method for a quick start, as it encapsulates the frontend and backend services into managed containers. The deployment is orchestrated using Docker Compose, simplified through make commands.

This setup provides a complete environment for pipeline development and execution, with a Next.js frontend for the user interface and a Python backend for running DocETL pipelines.

Sources: README.md:28-34, README.md:65-69

Architecture Overview

The Docker deployment consists of two main services: a frontend service for the DocWrangler UI and a backend service for the DocETL pipeline execution engine. These services are configured via separate environment files but run within the same Docker network, allowing them to communicate. A Docker volume is used for data persistence.

The following diagram illustrates the high-level architecture of the Docker deployment.

This diagram shows the user interacting with the frontend container, which in turn makes API calls to the backend container to execute pipelines. Both containers are configured by local .env files.

Sources: README.md:58-63, README.md:92-95, website/README.md:5-10

Prerequisites

Before deploying with Docker, you must have Docker and Docker Compose installed on your system. You will also need an OpenAI API key (or a key for your preferred LLM) for features that rely on Large Language Models.

Sources: README.md:43-44, README.md:67-69

Configuration

Configuration is managed through two separate environment files. It is crucial to create both files before running the Docker containers.

Sources: README.md:58-63

Backend Configuration (.env)

Create a file named .env in the project's root directory. This file configures the Python backend server that executes DocETL pipelines.

Example .env file:

OPENAI_API_KEY=your_api_key_here  # Used by DocETL pipeline execution engine
# BACKEND configuration
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True

# FRONTEND configuration
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000

# Host port mapping for docker-compose (if not set, defaults are used in docker-compose.yml)
FRONTEND_DOCKER_COMPOSE_PORT=3031
BACKEND_DOCKER_COMPOSE_PORT=8081

# Supported text file encodings
TEXT_FILE_ENCODINGS=utf-8,latin1,cp1252,iso-8859-1

Sources: README.md:72-90

The table below details the variables for the backend configuration.

Variable	Description
OPENAI_API_KEY	API key for the LLM provider used by the pipeline execution engine.
BACKEND_ALLOW_ORIGINS	A comma-separated list of origins allowed to access the backend API.
BACKEND_HOST	The host on which the backend server runs inside the container.
BACKEND_PORT	The port on which the backend server listens inside the container.
BACKEND_RELOAD	Enables or disables hot-reloading for the backend server.
FRONTEND_HOST	The host for the frontend server inside the container.
FRONTEND_PORT	The port for the frontend server inside the container.
FRONTEND_DOCKER_COMPOSE_PORT	Maps the container's frontend port to a host port.
BACKEND_DOCKER_COMPOSE_PORT	Maps the container's backend port to a host port.
TEXT_FILE_ENCODINGS	Comma-separated list of supported text file encodings.

Sources: README.md:72-90

Frontend Configuration (website/.env.local)

Create a file named .env.local inside the website/ directory. This file configures the Next.js frontend, specifically for UI-based features like the "improve prompt" and chatbot functionalities.

Example website/.env.local file:

OPENAI_API_KEY=sk-xxx  # Used by TypeScript features: improve prompt, chatbot, etc.
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini  # Model used by the UI assistant

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000
NEXT_PUBLIC_HOSTED_DOCWRANGLER=false

Sources: README.md:93-101, website/README.md:14-22

The table below details the variables for the frontend configuration.

Variable	Description
OPENAI_API_KEY	API key used by frontend TypeScript features (e.g., chatbot).
OPENAI_API_BASE	The base URL for the OpenAI API.
MODEL_NAME	The specific LLM model used by the UI assistant features.
NEXT_PUBLIC_BACKEND_HOST	The hostname the frontend uses to connect to the backend.
NEXT_PUBLIC_BACKEND_PORT	The port the frontend uses to connect to the backend.
NEXT_PUBLIC_HOSTED_DOCWRANGLER	A flag indicating if the app is running in a hosted environment.

Sources: README.md:93-101, website/README.md:14-22

Running the Application

Once the configuration files are in place, you can start the application using a make command.

make docker

This command performs the following actions:

1. Creates a Docker volume for persistent data.
2. Builds the docetl Docker image, which includes installing Python dependencies from pyproject.toml.
3. Starts the frontend and backend containers using Docker Compose.

The DocWrangler UI will be accessible at http://localhost:3000.

The following diagram shows the sequence of events when make docker is executed.

AWS Bedrock Integration

The Docker deployment supports integration with AWS Bedrock. To use this feature, you must first configure your AWS credentials.

1. Configure AWS CLI:

   aws configure
   

2. Test Credentials:

   make test-aws
   

3. Run with AWS Profile:
   You can run the Docker containers with AWS support using make or docker compose directly.

   Using make:

   AWS_PROFILE=your-profile AWS_REGION=your-region make docker
   

   Using docker compose:

   AWS_PROFILE=your-profile AWS_REGION=your-region docker compose --profile aws up
   

The following environment variables are used for Bedrock integration:

• AWS_PROFILE: Your AWS CLI profile name (defaults to 'default').
• AWS_REGION: The AWS region for Bedrock (defaults to 'us-west-2').

Bedrock models should be prefixed with bedrock in your pipeline configurations.

Sources: README.md:117-140

Management and Cleanup

To stop the containers and remove all associated Docker resources, including the persistent data volume, run the following command:

make docker-clean

This command is destructive and will delete the Docker volume, resulting in the loss of any data stored there.

Sources: README.md:113-115

Summary

The Docker-based deployment provides a streamlined and isolated environment for running the DocWrangler UI and the DocETL backend. By using docker-compose and simple make commands, developers can quickly set up a local instance for creating, testing, and executing complex document processing pipelines. The separation of frontend and backend configurations allows for flexibility, and the built-in support for AWS Bedrock extends its capabilities to cloud-based LLM services.