Command-Line Interface (CLI)

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• website/README.md
• mkdocs.yml
• website/src/components/TutorialsDialog.tsx
• website/src/components/NaturalLanguagePipelineDialog.tsx
• docetl/moar/search_utils.py

The docetl project provides a command-line interface (CLI) for running production data processing pipelines. While the interactive DocWrangler UI is recommended for iterative development and prompt engineering, the CLI is the primary method for executing finalized pipelines from the command line or as part of a larger Python application. The CLI consumes a declarative YAML file that defines the entire pipeline, from data sources to processing operations and outputs.

The detailed API reference for the CLI is documented and can be found under the Developer Reference section of the project's documentation.

Sources: README.md:21-23, mkdocs.yml:61

Getting Started

To use docetl as a Python package and interact with its CLI, you must first install it and configure the necessary environment variables.

Sources: README.md:33-34

Installation

The package requires Python 3.10 or later. It can be installed using pip.

pip install docetl

Sources: README.md:35-38

Configuration

docetl relies on environment variables for configuration, particularly for API keys and server settings. These are managed through a .env file in the root of your project directory.

It is important to distinguish between the two environment files used in the project:

• .env (root directory): Used by the backend Python server that executes the docetl pipelines via the CLI.
• website/.env.local (website directory): Used by the frontend TypeScript code for UI-specific features in DocWrangler, such as the AI chat assistant and the "Improve Prompt" feature.

Sources: README.md:41-51, website/README.md:4-10

An example of the root .env file for the backend/CLI:

OPENAI_API_KEY=your_api_key_here  # Required for LLM operations
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True
TEXT_FILE_ENCODINGS=utf-8,latin1,cp1252,iso-8859-1

Sources: README.md:42-44, README.md:62-79

Key Environment Variables

The following table summarizes important environment variables for the backend and CLI execution.

Variable	Description	Source File
OPENAI_API_KEY	API key for OpenAI, required for any LLM-based operations. Can be substituted with a key for another supported LLM.	README.md:43
BACKEND_ALLOW_ORIGINS	Comma-separated list of allowed origins for CORS when running the backend server.	README.md:64
BACKEND_HOST	Host for the backend server.	README.md:65
BACKEND_PORT	Port for the backend server.	README.md:66
AWS_PROFILE	Your AWS CLI profile name, used for AWS Bedrock integration. Defaults to 'default'.	README.md:106
AWS_REGION	The AWS region for Bedrock services. Defaults to 'us-west-2'.	README.md:107
TEXT_FILE_ENCODINGS	Supported text file encodings.	README.md:78

Usage

The CLI executes pipelines defined in a YAML configuration file. The core function of the CLI is to parse this file and run the specified sequence of operations on the input datasets.

The following diagram illustrates the typical execution flow when using the CLI.

Sources: docetl/moar/search_utils.py:202, website/src/components/TutorialsDialog.tsx:21-171

Pipeline Configuration (YAML)

A docetl pipeline is defined in a YAML file with several key sections: datasets, default_model, operations, pipeline, and system_prompt.

Below is an example of a complete pipeline configuration for analyzing Supreme Court transcripts.

datasets:
  input:
    type: file
    path: DATASET_PATH_PLACEHOLDER
    source: local
default_model: gpt-4o-mini
operations:
  - type: map
    name: find_patterns_in_reasoning
    prompt: >-
      You are analyzing a Supreme Court oral argument transcript...
      Here is the transcript:
      {{ input.content }}
      ...
    output:
      schema:
        summary: string
        legal_domain: string
  - type: reduce
    name: analyze_common_patterns
    prompt: >-
      You're writing an engaging magazine article...
      {% for input in inputs %}
      Legal Domain: {{ input.legal_domain }}
      Case Analysis:
      {{ input.summary }}
      {% endfor %}
      ...
    output:
      schema:
        analysis: string
    reduce_key:
      - _all
pipeline:
  steps:
    - name: data_processing
      input: input
      operations:
        - find_patterns_in_reasoning
        - analyze_common_patterns
  output:
    type: file
    path: DATASET_PATH_PLACEHOLDER_OUTPUT
    intermediate_dir: DATASET_PATH_PLACEHOLDER_INTERMEDIATES
system_prompt:
  dataset_description: "a collection of Supreme Court oral argument transcripts"
  persona: "a legal analyst skilled at breaking down complex legal concepts..."

Placeholders like DATASET_PATH_PLACEHOLDER are used in templates and are replaced with actual file paths during execution.

Sources: website/src/components/TutorialsDialog.tsx:21-91, website/src/components/NaturalLanguagePipelineDialog.tsx:432-441

Local Development and Execution

To contribute to docetl or run it locally for development, you can use either Docker or a manual setup. Both methods rely on the same backend server that underpins the CLI's execution capabilities.

Sources: README.md:110-112

Docker Setup (Recommended)

The easiest way to get a local environment running is with Docker.

1. Create Environment Files: Create the root .env and website/.env.local files as described in the Configuration section.
2. Run Docker: Execute the following command from the project root.

   make docker
   

   This command builds the docetl image and runs the container, making the UI accessible.

Sources: README.md:57-88

AWS Bedrock Integration

To use models from AWS Bedrock, configure your AWS credentials and run the Docker command with your profile and region.

AWS_PROFILE=your-profile AWS_REGION=your-region make docker

Bedrock model names are prefixed with bedrock.

Sources: README.md:94-108

Manual Setup

For development without Docker:

1. Clone the repository.
2. Set up environment variables in the root .env and website/.env.local files.
3. Install dependencies:

   make install      # Install Python dependencies
   make install-ui   # Install UI dependencies
   

4. Run tests to verify the setup:

   make tests-basic
   

Sources: README.md:113-144