Core Operations: Map, Reduce, Filter

Relevant source files

The following files were used as context for generating this wiki page:

• docetl/operations/map.py
• docetl/operations/reduce.py
• docetl/operations/filter.py
• pyproject.toml
• tests/test_output_modes.py
• tests/reasoning_optimizer/test_directive_apply.py
• docetl/reasoning_optimizer/directives/doc_chunking.py
• docetl/reasoning_optimizer/directives/cascade_filtering.py

The core operations—map, reduce, and filter—are the fundamental building blocks for creating data processing pipelines in DocETL. They provide the primary mechanisms for transforming, aggregating, and selecting data. These operations are registered as entry points in the system, allowing them to be dynamically loaded and executed by the pipeline runner.

This document provides a technical overview of each core operation, their configuration, and how they interact within a DocETL pipeline. Other key operations include Split, Gather, and Resolve, which are often used in conjunction with these core operators.

Sources: pyproject.toml:50-58, website/public/llms.txt:20-25

Map Operation

The map operation applies a transformation to each document in a dataset individually. It is a one-to-one transformation, where each input document produces one output document. A parallel_map variant also exists for executing independent sub-tasks concurrently on the same input document.

Sources: pyproject.toml:51-52, docetl/reasoning_optimizer/op_descriptions.py

Data Flow

The diagram below illustrates the data flow for a map operation, where an LLM prompt is applied to each document to produce a new, transformed document.

Configuration

A map operation is configured with the following parameters in the pipeline YAML.

Parameter	Type	Description
name	string	A unique identifier for the operation.
type	string	Must be map or parallel_map.
prompt	string	A Jinja2 template defining the instructions for the LLM. It references input fields using {{ input.field_name }}.
model	string	The LLM to use for the operation. Can be inherited from default_model.
output	object	Defines the output schema and mode.
output.schema	dict	A dictionary defining the structure of the output. Supports types like string, list[...], and nested objects.
output.mode	string	The output mode for the LLM. Can be tools or structured_output.

Sources: tests/test_output_modes.py:120-136, tests/reasoning_optimizer/test_directive_apply.py:20-27

Example

This example demonstrates a map operation that extracts fruits and vegetables from a document's content.

operations:
  - name: extract_fruits_and_veggies
    type: map
    prompt: "Analyze the text and extract all fruits mentioned, along with the most similar vegetable for each fruit based on characteristics like color, texture, or nutritional content."
    output:
      schema:
        fruits_and_veggies: "list[{fruit: str, most_similar_veggie: str, reasoning: str}]"
      mode: tools
    model: "gpt-4o-mini"

Sources: tests/test_output_modes.py:120-136

Reduce Operation

The reduce operation aggregates multiple documents into a single output. It is a many-to-one transformation, ideal for summarizing, synthesizing, or performing calculations across a group of documents.

Sources: pyproject.toml:56, website/public/llms.txt:21

Data Flow and Folding

The reduce operation can process documents in batches using a "folding" mechanism to handle large datasets that don't fit into a single LLM context. The fold_prompt is applied iteratively to batches of documents, and the final result is produced after all batches are processed.

Sources: tests/test_output_modes.py:200-201

Configuration

Parameter	Type	Description
name	string	A unique identifier for the operation.
type	string	Must be reduce.
reduce_key	string or list[string]	The field(s) to group documents by before reducing. _all groups all documents together.
prompt	string	The main Jinja2 template for the reduction. It iterates over documents using {% for input in inputs %}.
fold_batch_size	integer	The number of documents to process in each batch for folding.
fold_prompt	string	A Jinja2 template for iterative reduction. It references the previous batch's output via {{ output }} and the current batch via {{ inputs }}.
output	object	Defines the output schema and mode.

Sources: tests/test_output_modes.py:195-212, docetl/reasoning_optimizer/directives/doc_chunking.py:255-265

Example

This example uses a reduce operation with folding to find duplicate fruits across all documents.

operations:
  - name: find_duplicate_fruits
    reduce_key: _all
    type: reduce
    prompt: "Find all fruits that are mentioned more than once across the documents in {{ inputs }}. Count the occurrences accurately."
    fold_batch_size: 3
    fold_prompt: "Given the previous output {{ output }} and the new batch of documents {{ inputs }}, summarize the fruits mentioned and their counts. Combine duplicate entries and maintain accurate counts."
    output:
      schema:
        duplicate_fruits: "list[{fruit: str, count: int, document_ids: list[int]}]"
      mode: structured_output
    model: "gpt-4o-mini"

Sources: tests/test_output_modes.py:195-212

Filter Operation

The filter operation selectively keeps or discards documents based on a condition evaluated by an LLM. The LLM prompt should be designed to return a boolean value.

Sources: pyproject.toml:53, website/public/llms.txt:24

Data Flow

A filter operation evaluates each document and passes only those that meet the specified criteria to the next stage of the pipeline.

Configuration

Parameter	Type	Description
name	string	A unique identifier for the operation.
type	string	Must be filter.
prompt	string	A Jinja2 template that instructs the LLM to return a boolean value.
model	string	The LLM to use for the operation.
output	object	The output schema, which must contain a boolean field (e.g., _bool: "bool" or is_quality: "boolean").

Sources: tests/reasoning_optimizer/test_directive_apply.py:160-167, 915-923

Example

This example defines a filter operation to keep only documents that are considered "high-quality research papers."

- name: filter_quality
  type: filter
  model: gpt-4o
  prompt: "Is this a high-quality research paper? Paper: {{ input.paper_text }}"
  output:
    schema:
      is_quality: "boolean"

Sources: tests/reasoning_optimizer/test_directive_apply.py:915-923

Operation Interaction and Optimization

Core operations are often composed to create complex workflows. The DocETL reasoning optimizer can further transform these operations to improve cost and efficiency.

Map to Split-Gather-Map-Reduce

A common optimization for a map operation on a very long document is to transform it into a Split -> Gather -> Map -> Reduce pipeline. This doc_chunking directive breaks the document into smaller chunks, processes each chunk, and then aggregates the results.

Sources: docetl/reasoning_optimizer/directives/doc_chunking.py:16-20, 273-315

Cascade Filtering

An expensive filter operation can be optimized into a "cascade" of cheaper filters. The cascade_filtering directive injects fast code_filter operations and cheaper LLM filters (e.g., using gpt-5-nano) before the original, more expensive filter to progressively winnow down the dataset.

Sources: docetl/reasoning_optimizer/directives/cascade_filtering.py:19-23, 401-423

Summary

Map, Reduce, and Filter are the foundational LLM-powered operators in DocETL. Map applies 1-to-1 transformations, Reduce performs many-to-one aggregations, and Filter selectively passes documents. These operations can be defined declaratively in a YAML pipeline and are designed to be composed and automatically optimized, enabling the construction of sophisticated and efficient document processing workflows.