LLM Integration

Relevant source files

The following files were used as context for generating this wiki page: - [README.md](https://github.com/odra28/docetl/blob/HEAD/README.md) - [website/public/llms.txt](https://github.com/odra28/docetl/blob/HEAD/website/public/llms.txt) - [pyproject.toml](https://github.com/odra28/docetl/blob/HEAD/pyproject.toml) - [docetl/optimizers/join_optimizer.py](https://github.com/odra28/docetl/blob/HEAD/docetl/optimizers/join_optimizer.py) - [docetl/reasoning_optimizer/AGENT.md](https://github.com/odra28/docetl/blob/HEAD/docetl/reasoning_optimizer/AGENT.md) - [website/src/components/AIChatPanel.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/AIChatPanel.tsx) - [website/src/app/api/scraper/route.ts](https://github.com/odra28/docetl/blob/HEAD/website/src/app/api/scraper/route.ts) - [docetl/reasoning_optimizer/directives/deterministic_doc_compression.py](https://github.com/odra28/docetl/blob/HEAD/docetl/reasoning_optimizer/directives/deterministic_doc_compression.py) - [website/src/components/NaturalLanguagePipelineDialog.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/NaturalLanguagePipelineDialog.tsx)

DocETL is a system fundamentally designed for creating and executing LLM-powered data processing pipelines for complex document processing tasks. It integrates Large Language Models (LLMs) at multiple levels, from core data transformation operators to agentic pipeline optimization and interactive developer assistance in the UI.

The system provides a low-code, declarative YAML interface for defining these pipelines, which are executed by a Python backend. The integration is designed to be flexible, supporting a wide range of LLM providers through the litellm library, and enabling sophisticated, AI-driven analysis and transformation of unstructured data.

Sources: website/public/llms.txt:3-5, README.md:12-15, pyproject.toml:18

Core Architecture

The LLM integration in DocETL is architected around a Python backend for pipeline execution and a Next.js frontend (DocWrangler) for interactive development. LLM functionalities are present in both, but serve different purposes.

Sources: README.md:17-20, 52-55

LLM Provider Abstraction

DocETL uses the litellm library to abstract away the specifics of different LLM providers. This allows users to switch between models from providers like OpenAI, Anthropic, Google, and AWS Bedrock by changing configuration values, without altering the pipeline logic. This dependency is declared in the project's pyproject.toml file.

Sources: pyproject.toml:18, website/public/llms.txt:29, README.md:120-122

The diagram below shows how litellm acts as a bridge between DocETL's operations and various LLM APIs.

Configuration

API keys and other LLM-related settings are managed through environment variables. The system uses two primary configuration files:

1. .env (root directory): For the Python backend that executes DocETL pipelines. It holds keys for models used in operations like map and resolve.
2. website/.env.local (in website directory): For the Next.js frontend (DocWrangler). It holds keys for UI-based features like the AI Chat Panel and the "Improve Prompt" functionality.

Sources: README.md:46-49, 61-64

The key environment variables for LLM integration are:

Variable	File	Description
OPENAI_API_KEY	.env	API key for the LLM provider used by the backend pipeline execution engine.
OPENAI_API_KEY	website/.env.local	API key used by frontend TypeScript features like the chatbot and prompt improver.
MODEL_NAME	website/.env.local	The model used by the DocWrangler UI assistant.
AWS_PROFILE	.env	The AWS CLI profile to use for AWS Bedrock integration.
AWS_REGION	.env	The AWS region for Bedrock.

Sources: README.md:47, README.md:62, README.md:124-125

LLM-Powered Components

LLMs are integrated into several key components of the DocETL ecosystem, from data transformation to pipeline optimization and user assistance.

Core Operators

Several core operators in DocETL are LLM-powered, allowing for complex, AI-driven data transformations defined in the pipeline YAML.

Operator	Description
map	Applies an LLM prompt to each document individually.
resolve	Deduplicates entities by comparing pairs of items with an LLM.
reduce	Aggregates a group of documents into a single output using an LLM.
parallel_map	A variant of the map operation for parallel processing.
filter	Uses an LLM to decide whether to keep or discard a document.
equijoin	Performs a join between two datasets using an LLM-based comparison prompt.

Sources: website/public/llms.txt:21-26, pyproject.toml:34-40

Agentic Optimization

DocETL includes a "Reasoning Optimizer" that uses an LLM agent to automatically rewrite and optimize pipelines. This system employs a graph-search algorithm where the agent applies "directives" to transform the pipeline structure.

Key features of the Reasoning Optimizer:

• Directives: Pre-defined transformation rules like deterministic_doc_compression, gleaning (validation loops), and chaining (breaking complex ops).
• Instantiate Schemas: Pydantic models that define the structured output an LLM agent must produce to apply a directive.
• Agentic Workflow: The agent analyzes the pipeline, proposes a directive, generates a configuration (the instantiate schema), and applies the transformation.

Sources: docetl/reasoning_optimizer/AGENT.md:4-6, docetl/reasoning_optimizer/directives/deterministic_doc_compression.py:19-25, docetl/reasoning_optimizer/directives/README.md:6-10

The following diagram illustrates the agent's workflow for creating a new directive.

LLM-Assisted Prompt Synthesis

The JoinOptimizer is a concrete example of how DocETL uses an LLM to automate parts of the pipeline authoring process. It can synthesize comparison_prompt and resolution_prompt for resolve operations.

The process involves:

1. Analyzing the preceding map prompt to understand the data's context.
2. Checking for semantic duplicates in the data using an LLM.
3. Generating a tailored comparison prompt for entity resolution.
4. Generating a resolution prompt for merging duplicate entries.

Sources: docetl/optimizers/join_optimizer.py:48-50, docetl/optimizers/join_optimizer.py:101-104, docetl/optimizers/join_optimizer.py:237-240, docetl/optimizers/join_optimizer.py:291-295

This sequence diagram shows how the JoinOptimizer synthesizes a comparison prompt.

Frontend LLM Features

The DocWrangler UI integrates LLMs to assist developers directly in the IDE.

AI Chat Panel

A chat interface powered by ai/react allows users to ask questions about their pipeline. The panel serializes the current pipeline state and sends it as context to the LLM, enabling the assistant to provide relevant guidance.

Sources: website/src/components/AIChatPanel.tsx:180-230

// website/src/components/AIChatPanel.tsx:81
const {
    messages,
    setMessages,
    input,
    handleInputChange,
    handleSubmit,
    isLoading,
    error: chatError,
  } = useChat({
    api: "/api/chat",
    initialMessages: [],
    id: "persistent-chat",
    headers: {
      ...chatHeaders,
      "x-source": "ai_chat",
    },
    // ...
  });

Natural Language Pipeline Generation

Users can describe their desired outcome in natural language, and the system uses an LLM to generate a complete DocETL pipeline YAML. This feature uses a detailed system prompt that includes the DocETL specification, best practices, and the user's data schema.

Sources: website/src/components/NaturalLanguagePipelineDialog.tsx:210-260

// website/src/components/NaturalLanguagePipelineDialog.tsx:220
const promptText = `You are a DocETL pipeline generator...

Follow these DocETL guidelines:
${instructions}

I need to create a DocETL pipeline for the following data:

${sampleData}

The task I need a pipeline for is: ${userPrompt}

...
<name>
[The pipeline name you've created]
</name>

<pipeline>
[The complete YAML configuration]
</pipeline>`;

Web Scraping Agent

The application includes a sophisticated web scraping agent that uses an LLM with tools (search_internet, execute_code) to iteratively discover and scrape data from the web. This agent operates within a sandboxed Python environment provided by Modal.

Sources: website/src/app/api/scraper/route.ts:300-350

Summary

DocETL's integration of LLMs is comprehensive and multi-faceted. LLMs are not just a feature but a core component of the system's architecture. They power data transformations within pipelines, drive an autonomous agent for optimizing those pipelines, assist in the automatic generation of prompts, and provide interactive assistance to developers in the DocWrangler UI. This deep integration allows DocETL to effectively tackle complex, unstructured data processing tasks that would be difficult or impossible with traditional ETL tools.

Sources: README.md:12-15, website/public/llms.txt:3-5, docetl/reasoning_optimizer/AGENT.md:4-6