Pipelines: The Core Concept

Relevant source files

The following files were used as context for generating this wiki page: - [docetl/api.py](https://github.com/odra28/docetl/blob/HEAD/docetl/api.py) - [website/public/llms.txt](https://github.com/odra28/docetl/blob/HEAD/website/public/llms.txt) - [website/src/components/NaturalLanguagePipelineDialog.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/NaturalLanguagePipelineDialog.tsx) - [docetl/reasoning_optimizer/directives/README.md](https://github.com/odra28/docetl/blob/HEAD/docetl/reasoning_optimizer/directives/README.md) - [docetl/reasoning_optimizer/directives/arbitrary_rewrite.py](https://github.com/odra28/docetl/blob/HEAD/docetl/reasoning_optimizer/directives/arbitrary_rewrite.py) - [website/src/app/showcase/ai-rfi-response-analysis/page.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/app/showcase/ai-rfi-response-analysis/page.tsx)

In DocETL, a Pipeline is the central structure that defines a complete data processing workflow. It is designed for complex document processing tasks, especially those involving Large Language Models (LLMs). A pipeline is declared in a YAML file and specifies the input data, the sequence of processing operations, and the final output configuration. This declarative approach allows for both manual authoring and automated generation or optimization of workflows.

Pipelines are executed by the DSLRunner, which interprets the YAML configuration, runs the specified operations in sequence, and saves the results. The system is flexible, supporting both LLM-based and traditional code-based operations, and can be optimized for cost or accuracy through a system of rewrite "directives".

Sources: README.md, docetl/api.py:1-25

Pipeline Architecture

A DocETL pipeline is composed of several key components, defined in the Pipeline class. These components are typically specified in a YAML configuration file.

Sources: docetl/api.py:100-125

Core Components

The main Pipeline class orchestrates the entire workflow. It is initialized with the following components:

Component	Type	Description
name	str	The name of the pipeline.
datasets	dict[str, Dataset]	A dictionary of input data sources.
operations	list[OpType]	A list of all available operations in the pipeline.
steps	list[PipelineStep]	The sequence of steps that define the data flow.
output	PipelineOutput	The configuration for the final output.
default_model	str	The default LLM to use for operations.
parsing_tools	list[ParsingTool]	Custom functions for pre-processing data.
optimizer_config	dict	Configuration for the pipeline optimizer.

Sources: docetl/api.py:127-151

The following diagram illustrates the relationship between these components within the Pipeline class.

This diagram shows the composition of a Pipeline object, which aggregates datasets, operations, steps, and output configurations.

Sources: docetl/api.py

Data Flow

The data flow in a pipeline is defined by the pipeline.steps array. Each step specifies an input (which can be an initial dataset or the output of a previous step) and a list of operations to apply.

The diagram below shows a typical data flow:

This flow illustrates how data from an initial dataset is processed through a series of steps, each applying one or more operations, to produce a final output file.

Sources: docetl/api.py:159-173

Operations

Operations are the fundamental building blocks of a pipeline, defining the actual data transformations. DocETL supports a variety of operators, which can be broadly categorized into Core and Auxiliary operators.

Sources: website/public/llms.txt:16-29

Operator Types

Category	Operator	Description
Core	map	Applies an LLM prompt to each document individually.
	reduce	Aggregates a group of documents into a single output using an LLM prompt.
	resolve	Performs entity resolution to deduplicate or merge similar items.
	parallel-map	A variant of map for parallel processing.
	filter	Uses an LLM prompt to decide whether to keep or discard a document.
	equijoin	Joins two datasets based on a common key.
Auxiliary	split	Splits documents into smaller chunks.
	gather	Collects results from split operations.
	unnest	Flattens a list within documents into separate documents.
	sample	Selects a random subset of documents.
	code	Applies a user-defined Python function (code_map, code_reduce, code_filter).

Sources: website/public/llms.txt:16-29, docetl/api.py:280-297

LLM vs. Code Operations

A key feature of DocETL is the ability to mix LLM-based operations with deterministic code-based operations.

• LLM Operations (map, reduce, filter, resolve) are defined with a prompt and an output schema. They are suitable for fuzzy, qualitative, or complex reasoning tasks.
• Code Operations (code_map, code_reduce, code_filter) are defined with a code block containing a Python function. They are used for deterministic tasks like simple transformations, calculations, or regex matching.

Sources: docetl/api.py:73-90, website/public/llms.txt:28-29

Pipeline Creation

Pipelines can be authored manually in YAML or generated dynamically.

Manual Authoring (YAML)

A pipeline is defined in a YAML file with a specific structure. The example below is from the AI RFI Response Analysis showcase.

datasets:
  input:
    type: file
    path: "https://docetlcloudbank.blob.core.windows.net/demos/processed_responses.json"
    source: "remote"
default_model: gpt-4o-mini
operations:
  - type: map
    name: summarize_and_classify
    prompt: |
      ...
    output:
      schema:
        ...
pipeline:
  steps:
    - name: data_processing
      input: input
      operations:
        - summarize_and_classify
  output:
    type: file
    path: "summarized_responses.json"
    intermediate_dir: "rfi_intermediates"

This YAML defines the input dataset, default model, a map operation, and a single pipeline step to process the data and produce an output file.

Sources: website/src/app/showcase/ai-rfi-response-analysis/page.tsx:112-121

Natural Language Generation

The DocWrangler UI provides a feature to generate a pipeline from a natural language description. This process uses an LLM to construct the YAML configuration.

The flow for generating a pipeline via natural language is as follows:

1. The user uploads a dataset (JSON or CSV) and provides a text prompt describing their goal.
2. The frontend extracts a sample of the data and combines it with the user's prompt and system instructions.
3. A request is sent to the /api/generate endpoint.
4. The API uses an LLM (e.g., gpt-5) to generate a pipeline name and a complete YAML configuration.
5. The generated YAML uses placeholders like DATASET_PATH_PLACEHOLDER which are replaced by the frontend with the actual file paths before the pipeline is applied to the workspace.

Sources: website/src/components/NaturalLanguagePipelineDialog.tsx:321-429

This sequence diagram shows the interaction for generating a pipeline from natural language.

Pipeline Optimization

DocETL includes a "reasoning optimizer" that can automatically rewrite pipelines to improve cost or accuracy. This is achieved through a system of Directives.

A directive is a transformation rule that modifies the pipeline's operations. The optimizer uses a search algorithm to apply these directives and find a more efficient pipeline structure.

Sources: docetl/reasoning_optimizer/AGENT.md:4-5, docetl/reasoning_optimizer/directives/README.md:6-11

Directive Patterns

The system supports several patterns for pipeline transformation:

Pattern	Description	Example Directive
Single Operation Modification	Adds or modifies the configuration of an existing operation.	gleaning (adds validation loops), change_model
Operation Replacement	Replaces a single complex operation with a sequence of simpler ones.	chaining
Pipeline Preprocessing	Adds a new operation at the beginning of the pipeline.	doc_summarization
Arbitrary Rewrite	A flexible directive that uses search-and-replace on the pipeline's JSON representation to perform complex changes like reordering or consolidating operations.	arbitrary_rewrite

Sources: docetl/reasoning_optimizer/directives/README.md:104-115, docetl/reasoning_optimizer/directives/arbitrary_rewrite.py:15-23

The diagram below shows an example of the doc_summarization directive transforming a pipeline.

This illustrates how the doc_summarization directive inserts a new Map operation at the beginning of the pipeline to compress the document before it's processed by downstream operations.

Sources: docetl/reasoning_optimizer/directives/doc_summarization.py:19-30, 158-172

Conclusion

Pipelines are the cornerstone of the DocETL framework, providing a declarative and powerful way to define complex data processing workflows. They are composed of datasets, operations, and steps, all specified in a structured YAML format. The system's support for both LLM and code-based operations, combined with its dynamic generation and automated optimization capabilities, makes it a versatile tool for a wide range of document analysis tasks. The architecture allows developers to start with a simple pipeline and iteratively refine or automatically optimize it for production use.

Sources: README.md:10-13, docetl/api.py:26-31