System Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• website/README.md
• website/src/app/api/scraper/route.ts
• website/src/components/NaturalLanguagePipelineDialog.tsx
• website/src/components/AIChatPanel.tsx
• website/src/components/PromptImprovementDialog.tsx
• docetl/moar/search_utils.py
• mkdocs.yml
• website/src/app/showcase/ai-rfi-response-analysis/page.tsx
• website/src/components/PresidentialDebateDemo.tsx

The DocETL project is a comprehensive system for creating, executing, and optimizing complex document processing pipelines, with a strong emphasis on leveraging Large Language Models (LLMs). The architecture is split into two primary components: a user-facing interactive web application called DocWrangler and a powerful Python backend engine that handles pipeline execution. This dual-component design allows for both iterative development in a user-friendly environment and robust, scalable execution in production.

The system is designed to be highly modular, integrating various services for LLM operations, sandboxed code execution, and data persistence. The frontend provides a rich user experience with several AI-assisted features, while the backend focuses on the core logic of running declarative YAML-based pipelines. The entire system can be run locally using Docker, which orchestrates the frontend and backend services.

Core Components

The system is composed of a frontend application and a backend execution engine, each with distinct responsibilities and configurations.

Component	Technology	Description	Source Files
DocWrangler (Frontend)	Next.js, TypeScript	An interactive UI playground for iteratively developing and testing DocETL pipelines. It allows users to experiment with prompts, build pipelines step-by-step, and export configurations.	website/README.md, README.md:27-31
DocETL Engine (Backend)	Python	A Python package and server responsible for running production pipelines from the command line or programmatically. It parses YAML configurations and executes the defined operations.	README.md:40-42
AI Services (Frontend)	Next.js API Routes	A set of backend-for-frontend APIs that power UI-based LLM interactions like the AI Chatbot, Prompt Improvement, and Natural Language Pipeline Generation.	website/README.md:5-8, website/src/components/AIChatPanel.tsx, website/src/components/PromptImprovementDialog.tsx
Web Scraper (Backend Service)	Modal	A sandboxed Python environment for executing web scraping code. It is invoked via a Next.js API route and uses a persistent volume for data storage.	website/src/app/api/scraper/route.ts:16-25

High-Level System Diagram

This diagram illustrates the main components of the DocETL system and their interactions.

Frontend Architecture (DocWrangler)

The frontend, named DocWrangler, is a Next.js application written in TypeScript. It serves as an interactive development environment (IDE) for building and testing DocETL pipelines.

Sources: website/README.md:1, README.md:27-28

Key Features & Components

• Interactive Pipeline Builder: Allows users to construct pipelines visually or through code.
• AI-Assisted Features: DocWrangler includes several components that leverage LLMs to aid development:
  • AIChatPanel.tsx: A chat interface that provides contextual help for building pipelines. It sends the current pipeline state to an LLM for analysis and suggestions.
  • PromptImprovementDialog.tsx: An AI-powered tool to analyze and suggest improvements for prompts within pipeline operations. It supports iterative feedback and revision history.
  • NaturalLanguagePipelineDialog.tsx: Enables users to generate a full DocETL pipeline YAML configuration from a natural language description and a sample dataset.
• Showcase Pages: The application includes several demonstration pages (e.g., ai-rfi-response-analysis, lease-contract-red-flags) that show practical applications of DocETL pipelines with interactive data explorers.

Sources: website/src/components/AIChatPanel.tsx, website/src/components/PromptImprovementDialog.tsx, website/src/components/NaturalLanguagePipelineDialog.tsx, website/src/app/showcase/page.tsx:17-80

Natural Language to Pipeline Flow

The "Generate Pipeline from Natural Language" feature demonstrates a typical frontend-to-backend interaction flow. The user provides a dataset and a prompt, which the frontend sends to a Next.js API route (/api/generate) to generate a complete YAML pipeline.

This sequence diagram shows the flow for generating a pipeline from natural language.

Backend Architecture

The backend consists of multiple parts: the core Python engine for pipeline execution, Next.js API routes that serve the frontend, and external services for specialized tasks like web scraping.

DocETL Python Engine

The core of the project is the docetl Python package, which can be run from the command line or used as a library. It is responsible for:

• Parsing and validating pipeline YAML files.
• Executing a sequence of operations (map, reduce, resolve, etc.).
• Interacting with LLM providers (like OpenAI, Anthropic, Gemini via LiteLLM) to perform AI-powered operations.
• Optimizing pipelines for cost or accuracy using the MOAR (MCTS-based Optimizer and Rewriter) agent.

Sources: README.md:40-46, mkdocs.yml:21-40, docetl/moar/search_utils.py

A typical pipeline configuration is defined in YAML, as shown in this example for analyzing presidential debates.

datasets:
  debates:
    type: file
    path: "data.json"

default_model: gpt-4o-mini

operations:
  - name: extract_themes_and_viewpoints
    type: map
    prompt: |
      Analyze the following debate transcript...
  - name: unnest_themes
    type: unnest
    unnest_key: themes
  - name: summarize_theme_evolution
    type: reduce
    reduce_key: theme
    prompt: |
      Analyze the following viewpoints on the theme...

pipeline:
  steps:
    - name: debate_analysis
      input: debates
      operations:
        - extract_themes_and_viewpoints
        - unnest_themes
        - summarize_theme_evolution

Sources: website/src/components/PresidentialDebateDemo.tsx:20-120

Web Scraping Service

The web scraping functionality is implemented using Modal, a serverless platform for running Python code. This isolates the scraping logic from the main application and provides a sandboxed environment with necessary dependencies.

• A Next.js API route at website/src/app/api/scraper/route.ts acts as a controller.
• It receives Python code from the client, which is then executed within a Modal sandbox.
• The sandbox has a persistent volume mounted at /data to store the scraped JSON results, identified by a session ID.
• This architecture allows for secure, on-demand execution of arbitrary scraping scripts with pre-installed libraries like beautifulsoup4, playwright, and pdfplumber.

This diagram shows the architecture of the web scraping feature.

Environment and Deployment

The project uses separate environment files for the frontend and backend to maintain a clear separation of concerns. The entire application stack is designed to be run using Docker.

Environment Configuration

There are two distinct environment files:

File	Location	Purpose	Example Variables
.env	Project Root	Configures the backend Python server for pipeline execution.	OPENAI_API_KEY, BACKEND_HOST, BACKEND_PORT
.env.local	website/	Configures the frontend Next.js application, specifically for UI-based AI features like the chatbot and prompt improver.	OPENAI_API_KEY, NEXT_PUBLIC_BACKEND_HOST, NEXT_PUBLIC_BACKEND_PORT

This separation ensures that keys for production pipeline runs are managed independently from keys used for UI assistance features.

Sources: README.md:49-53, 80-84, website/README.md:4-15

Docker Setup

The README.md describes a Docker-based setup for running the full application locally. The make docker command builds and runs the services. Although the docker-compose.yml file itself was not provided in the context, its structure can be inferred from the environment variables and documentation.

• frontend service: Runs the Next.js DocWrangler application.
• backend service: Runs the Python server for the DocETL engine.
• Volume: A Docker volume is used for persistent data.
• AWS Integration: The Docker setup supports AWS Bedrock integration by passing AWS_PROFILE and AWS_REGION environment variables.

Sources: README.md:87-129

Conclusion

The DocETL system architecture is a well-defined, modular framework that effectively separates the concerns of interactive development from production execution. The DocWrangler frontend provides a rich, AI-assisted user experience powered by a Next.js backend, while the core DocETL Python engine offers a powerful, declarative way to process documents using LLMs. The use of containerization with Docker simplifies deployment, and the integration of external services like Modal for specialized tasks like web scraping demonstrates a scalable and secure design. This architecture enables developers to build, test, and run sophisticated document processing pipelines efficiently.