Reasoning Directives

Relevant source files

The following files were used as context for generating this wiki page:

• docetl/reasoning_optimizer/AGENT.md
• docetl/reasoning_optimizer/directives/README.md
• docetl/reasoning_optimizer/directives/base.py
• docetl/reasoning_optimizer/directives/cascade_filtering.py
• docetl/reasoning_optimizer/directives/doc_chunking.py
• docetl/reasoning_optimizer/directives/doc_summarization.py
• docetl/reasoning_optimizer/directives/isolating_subtasks.py
• docetl/reasoning_optimizer/directives/reduce_gleaning.py
• tests/reasoning_optimizer/test_directive_apply.py
• docetl/reasoning_optimizer/directives/deterministic_doc_compression.py
• docetl/reasoning_optimizer/directives/take_head_tail.py

Reasoning Directives are transformation rules used by the DocETL Reasoning Optimizer to rewrite and enhance data processing pipelines. The optimizer, an LLM-based agent, employs a graph-search algorithm to apply these directives, modifying pipeline operations to improve effectiveness, reduce cost, or increase accuracy. Directives encapsulate specific patterns of pipeline transformation, such as breaking a complex operation into simpler steps, adding validation loops, or inserting preprocessing steps like document summarization.

Each directive defines a specific transformation and the conditions under which it should be applied. The core of the system is the interaction between the LLM agent and the directive's structured configuration, known as an "Instantiate Schema." This schema acts as a contract, ensuring the agent provides all necessary parameters in a valid format for the directive to execute its transformation logic. This modular approach allows for the easy addition of new optimization strategies to the system.

Sources: docetl/reasoning_optimizer/AGENT.md:5-6, docetl/reasoning_optimizer/directives/README.md:5-10

Core Concepts

Directive

A directive is a self-contained transformation rule that modifies one or more operations in a DocETL pipeline. It provides metadata about its purpose, use cases, and a method to apply the transformation.

Sources: docetl/reasoning_optimizer/directives/README.md:7-10

Instantiate Schema

The Instantiate Schema is a Pydantic model that defines the structured output an LLM agent must generate to apply a directive. It serves as a configuration blueprint that instructs the system on how to transform an operation. This schema ensures the agent provides all required parameters in the correct format, and can include validators to enforce constraints before the transformation is applied.

For example, the ReduceGleaningDirective uses a GleaningInstantiateSchema to capture the configuration for a validation loop. The LLM agent outputs a JSON object matching this schema, which the directive's apply method then uses to modify the target Reduce operation.

Sources: docetl/reasoning_optimizer/directives/README.md:14-31, docetl/reasoning_optimizer/directives/reduce_gleaning.py:22

Directive Application Workflow

The process of applying a directive involves a coordinated effort between the LLM agent and the directive's implementation.

The diagram below illustrates the general workflow for applying a directive.

Base Directive Class

All directives inherit from the Directive base class, which defines the common interface and attributes required for a directive to integrate with the reasoning optimizer.

Sources: docetl/reasoning_optimizer/directives/base.py

Attributes

Attribute	Type	Description
name	str	A unique technical identifier for the directive.
formal_description	str	A brief, formal description of the transformation pattern (e.g., "Op => Map -> Op").
nl_description	str	A natural language explanation of what the directive does.
when_to_use	str	Guidance on the specific use cases and scenarios for applying the directive.
instantiate_schema_type	Type[BaseModel]	The Pydantic model class that the LLM agent must output to configure the directive.
example	str	A string demonstrating an example of the original operation and the expected InstantiateSchema output from the agent.
test_cases	List[DirectiveTestCase]	A list of test cases to validate the directive's functionality. Each test case includes an input configuration, target operations, and expected behavior.

Sources: docetl/reasoning_optimizer/directives/base.py:17-43, docetl/reasoning_optimizer/directives/doc_summarization.py:25-151

Core Methods

• to_string_for_instantiate(...) -> str: Generates a detailed prompt for the LLM agent, explaining how to generate the required InstantiateSchema for a given operation.
• llm_instantiate(...) -> tuple: Calls the LLM with the generated prompt, requesting a structured JSON output that matches the instantiate_schema_type. It includes retry logic to handle validation errors.
• apply(...) -> List[Dict]: The core transformation logic. This method takes the current list of pipeline operations and the agent-generated InstantiateSchema (rewrite) and returns a new, modified list of operations.
• instantiate(...) -> tuple: The main entry point for the MCTS agent. It orchestrates the entire process by calling llm_instantiate to get the configuration from the agent and then apply to perform the transformation.

Sources: docetl/reasoning_optimizer/directives/README.md:110-188

Directive Patterns

Directives can be grouped into several common transformation patterns.

Sources: docetl/reasoning_optimizer/AGENT.md:52-64, docetl/reasoning_optimizer/directives/README.md:292-303

Single Operation Modification

These directives add or modify the configuration of a single existing operation without changing the pipeline structure.

• ReduceGleaningDirective: Adds a validation loop (gleaning configuration) to a Reduce operation to improve output quality.
• TakeHeadTailDirective: Inserts a code_map operation before an LLM operation to truncate a document field, keeping only the head and/or tail, thus reducing token count.

Sources: docetl/reasoning_optimizer/directives/reduce_gleaning.py:11-17, docetl/reasoning_optimizer/directives/take_head_tail.py:16-22

Operation Replacement

These directives replace a single operation with a new sequence of one or more operations.

• IsolatingSubtasksDirective: Breaks down a complex Map operation into a Parallel Map (for independent subtasks) followed by an aggregation Map.
• DocumentChunkingDirective: Transforms a Map operation on a long document into a Split -> Gather -> [Sample] -> Map -> Reduce pipeline.

Sources: docetl/reasoning_optimizer/directives/isolating_subtasks.py:17-23, docetl/reasoning_optimizer/directives/doc_chunking.py:17-23

Pipeline Preprocessing

These directives add new operations at or near the beginning of the pipeline to preprocess data for downstream tasks.

• DocSummarizationDirective: Adds a Map summarization operator at the start of the pipeline to shorten documents.
• DeterministicDocCompressionDirective: Inserts a code_map operation that uses deterministic logic (e.g., regex) to compress documents before they reach an expensive LLM operation.
• CascadeFilteringDirective: Injects a series of cheaper filters (code-based, then small LLM-based) before an expensive Filter operation to reduce costs.

Sources: docetl/reasoning_optimizer/directives/doc_summarization.py:25-33, docetl/reasoning_optimizer/directives/deterministic_doc_compression.py:25-28, docetl/reasoning_optimizer/directives/cascade_filtering.py:17-23

Detailed Directive Examples

DocSummarizationDirective

This directive adds a Map operation at the beginning of a pipeline to summarize a document. This is useful when subsequent operations do not need the full document text, thereby saving costs and improving efficiency.

The transformation flow is as follows:

The agent generates a DocSummarizationInstantiateSchema to configure the new summarization operation.

Sources: docetl/reasoning_optimizer/directives/doc_summarization.py:25-33,215-234

DocumentChunkingDirective

This directive transforms a single Map operation designed for long documents into a multi-step chunking pipeline. It splits the document, gathers context for each chunk, processes chunks in parallel, and then reduces the results.

The transformation replaces a single Map operation with the following sequence:

This diagram shows the DocumentChunkingDirective replacing a single Map operation with a sequence of Split, Gather, optional Sample, Map, and Reduce operations.

Sources: docetl/reasoning_optimizer/directives/doc_chunking.py:17-23, 314-390

CascadeFilteringDirective

This directive optimizes an expensive Filter operation by inserting a cascade of cheaper pre-filters. It starts with deterministic code_filter operations, followed by filter operations using a cheap model (like gpt-5-nano), before finally running the original expensive filter. This progressively reduces the number of documents that reach the most expensive step.

The transformation flow is as follows:

This diagram illustrates how documents flow through a series of progressively more complex and expensive filters.

Sources: docetl/reasoning_optimizer/directives/cascade_filtering.py:17-23, 313-346

Testing Directives

The framework includes robust testing mechanisms for directives, covering both the instantiation logic and the application logic.

• DirectiveTestCase: Each directive class contains a list of test_cases. A DirectiveTestCase defines an input_config, target_ops, and expected_behavior to validate that the LLM agent can correctly generate an InstantiateSchema for a given scenario.
• Apply Tests: The tests/reasoning_optimizer/test_directive_apply.py file contains a suite of tests that verify the apply() method of each directive. These tests ensure that given a valid InstantiateSchema, the apply method correctly transforms the pipeline operations without crashing.
• Test Runners:
  • experiments/reasoning/run_tests.py: A script to run instantiation tests for one or all directives.
  • tests/reasoning_optimizer/test_directive_apply.py: A script that runs the apply method tests for all directives.

Sources: docetl/reasoning_optimizer/directives/base.py:9-14, tests/reasoning_optimizer/test_directive_apply.py, docetl/reasoning_optimizer/AGENT.md:73-80

Summary

Reasoning Directives form a flexible and powerful system for the agentic optimization of DocETL pipelines. By encapsulating transformation logic into discrete, testable units, the system allows an LLM agent to strategically rewrite pipelines for improved performance, cost, and accuracy. The use of Pydantic-based Instantiate Schemas provides a robust, type-safe bridge between the agent's high-level reasoning and the concrete implementation of pipeline modifications. This architecture is extensible, enabling developers to easily introduce new optimization strategies by creating new directive classes.

Sources: docetl/reasoning_optimizer/AGENT.md:5-6, docetl/reasoning_optimizer/directives/README.md:5-10, 314-318