Operations: An Overview

Relevant source files

The following files were used as context for generating this wiki page:

• docetl/operations/base.py
• docetl/operations/init.py
• pyproject.toml
• website/public/llms.txt
• docetl/reasoning_optimizer/directives/doc_chunking_topk.py
• tests/test_output_modes.py
• tests/reasoning_optimizer/test_directive_apply.py
• docetl/reasoning_optimizer/directives/README.md

Operations are the fundamental building blocks of data processing in the DocETL framework. They represent distinct processing steps that transform data within a pipeline. Defined in a declarative YAML configuration, operations can range from simple, code-based transformations to complex, LLM-powered analyses. This modular approach allows for the creation of sophisticated pipelines tailored to complex document processing tasks.

Each operation in a DocETL pipeline has a specific, clear purpose. The system supports a variety of operations, which can be broadly categorized into LLM-powered operators, code-based operators, and auxiliary operators for structuring data flow. These operations are dynamically loaded and registered via entry points, making the system extensible.

Sources: website/public/llms.txt:3-4, pyproject.toml:34-51

Core Concepts

At the heart of the system is the base Operation class, which all specific operations inherit from. While the base.py file itself is not provided in the context, its principles are evident in how operations are defined and used throughout the project. Key configuration parameters for an operation in the YAML file include its name, type, and other type-specific arguments like prompt, model, and output schema for LLM-based operations.

Sources: docetl/reasoning_optimizer/directives/README.md:20-31, website/public/llms.txt:104-107

Types of Operations

DocETL provides a rich set of operations that can be categorized as Core, Auxiliary, and Code-based. These are registered in the system via pyproject.toml entry points, allowing for easy extension.

Sources: pyproject.toml:34-51, website/public/llms.txt:16-29

This diagram shows the categorization of available operations.

The following table summarizes the available operations registered in the system.

Operation Type	Category	Description
map	Core	Applies an LLM prompt to each document individually.
parallel_map	Core	A variant of map that can execute multiple sub-tasks in parallel.
reduce	Core	Aggregates results from multiple documents into a single output.
resolve	Core	A specialized operator for complex reasoning and data resolution.
filter	Core	Uses an LLM prompt to decide whether to keep or discard a document.
equijoin	Core	Joins datasets based on equality conditions.
split	Auxiliary	Breaks a document into smaller chunks.
gather	Auxiliary	Collects chunks and provides context from surrounding chunks.
unnest	Auxiliary	Flattens a list within a document into multiple documents.
sample	Auxiliary	Samples a subset of documents from a dataset.
topk	Auxiliary	Selects the top-K most relevant chunks based on a query.
cluster	Auxiliary	Groups similar documents together.
extract	Auxiliary	A specialized LLM-powered extraction operator.
link_resolve	Auxiliary	A specialized version of resolve for linking entities.
code_map	Code	Applies a Python function to each document.
code_reduce	Code	Aggregates documents using a Python function.
code_filter	Code	Filters documents using a Python function that returns a boolean.
Sources: pyproject.toml:35-51, website/public/llms.txt:16-29, tests/reasoning_optimizer/test_directive_apply.py:539-543

Operation Registration

Operations are made available to the DocETL runner through Python's entry points mechanism. The pyproject.toml file defines the docetl.operation entry point group, which maps an operation's type name (e.g., map) to its corresponding class implementation.

# pyproject.toml:34-51
[project.entry-points."docetl.operation"]
map = "docetl.operations.map:MapOperation"
parallel_map = "docetl.operations.map:ParallelMapOperation"
filter = "docetl.operations.filter:FilterOperation"
unnest = "docetl.operations.unnest:UnnestOperation"
equijoin = "docetl.operations.equijoin:EquijoinOperation"
split = "docetl.operations.split:SplitOperation"
reduce = "docetl.operations.reduce:ReduceOperation"
resolve = "docetl.operations.resolve:ResolveOperation"
gather = "docetl.operations.gather:GatherOperation"
cluster = "docetl.operations.cluster:ClusterOperation"
sample = "docetl.operations.sample:SampleOperation"
topk = "docetl.operations.topk:TopKOperation"
link_resolve = "docetl.operations.link_resolve:LinkResolveOperation"
code_map = "docetl.operations.code_operations:CodeMapOperation"
code_reduce = "docetl.operations.code_operations:CodeReduceOperation"
code_filter = "docetl.operations.code_operations:CodeFilterOperation"

Sources: pyproject.toml:34-51

LLM-Powered Operations

A key feature of DocETL is its set of LLM-powered operations, such as map, reduce, and filter. These operations use a large language model to perform complex data transformations based on a natural language prompt.

Output Modes

LLM operations support two modes for generating structured output, configured via the output.mode property:

1. tools: The LLM is prompted to use a specific tool to generate the output.
2. structured_output: The LLM is directly asked to generate a JSON object matching the schema.

The choice of mode can affect the accuracy and reliability of the output, and tests are in place to compare their performance on complex schemas.

Sources: tests/test_output_modes.py:1-4, 114-117, 180-183

This sequence diagram illustrates the general flow for an LLM-powered operation.

Code-Based Operations

For deterministic transformations, DocETL provides code-based operations: code_map, code_reduce, and code_filter. These operations execute a user-defined Python function named transform.

• code_map: The transform function receives a single document and returns a modified document.
• code_reduce: The transform function receives a list of documents and returns a single aggregated result.
• code_filter: The transform function receives a single document and must return a boolean value (True to keep, False to discard).

Below is an example of a code_filter operation used to implement a boolean check after a reduce operation.

# tests/reasoning_optimizer/test_directive_apply.py:539-543
code_filter_op = {
    "name": f"code_filter_{target_op}",
    "type": "code_filter",
    "code": f"def transform(input_doc):\n    return input_doc.get('{bool_field}', False)",
}

Sources: tests/reasoning_optimizer/test_directive_apply.py:539-543, pyproject.toml:49-51

Operation Optimization with Directives

DocETL pipelines can be automatically optimized by a system of Directives. A directive is a transformation rule that modifies pipeline operations to improve cost, accuracy, or efficiency. These directives can replace a single operation with a more efficient sequence of operations.

For example, the doc_chunking_topk directive can transform a single map operation on a long document into a Split -> TopK -> Reduce pipeline. This avoids processing the entire document by splitting it into chunks, selecting only the most relevant (TopK) chunks, and then processing them.

Sources: docetl/reasoning_optimizer/directives/README.md:5-9, docetl/reasoning_optimizer/directives/doc_chunking_topk.py:17-23

The following diagram illustrates the transformation applied by the doc_chunking_topk directive.

Other examples of directives include:

• Chaining: Breaks a complex operation into a sequence of simpler ones.
• Gleaning: Adds validation loops to an operation to improve output quality.
• CascadeFiltering: Injects a series of cheaper filters before an expensive filter operation to reduce costs.

Sources: docetl/reasoning_optimizer/directives/README.md:6-9, docetl/reasoning_optimizer/directives/cascade_filtering.py:17-23

Summary

Operations are the versatile and extensible core of DocETL. They provide the expressive power to build pipelines that can handle both deterministic, code-based logic and nuanced, LLM-driven analysis. The system's architecture, with its clear separation of operation types and the powerful directive-based optimization engine, enables developers to construct, execute, and refine complex document processing workflows efficiently.