Creating Custom Operations
Relevant source files
The following files were used as context for generating this wiki page:The docetl framework is designed for extensibility, allowing developers to create and integrate custom operations into data processing pipelines. This is achieved through a plugin-based architecture where new operations are registered as entry points, making them discoverable and usable by the pipeline runner alongside built-in operations like map, filter, and reduce.
Creating a custom operation involves defining its logic within a Python class and registering it in the project's configuration. This guide outlines the architecture of docetl operations and provides a step-by-step process for creating your own.
Architecture of an Operation
Every operation in docetl consists of two main parts: the implementation logic and the descriptive metadata. The system uses a Pydantic model for metadata and Python's entry point mechanism for discovering and loading the operation's logic.
Operation Metadata
The metadata for an operation is defined by the Operator Pydantic model. This model captures all the necessary information about an operation, such as its name, description, parameters, and example usage. This structured metadata is used by various parts of the system, including the reasoning optimizer.
Sources: docetl/reasoning_optimizer/op_descriptions.py:4-28
The diagram below shows the structure of the Operator Pydantic model.
The fields of the Operator class are detailed in the table below.
| Field | Type | Description |
|---|---|---|
name | str | The unique name of the operation. |
type_llm_or_not | str | Indicates if the operation is "LLM-powered" or "Not LLM-powered". |
description | str | A brief description of what the operation does. |
when_to_use | str | Guidance on when the operation should be used in a pipeline. |
required_parameters | str | A description of the mandatory parameters for the operation's configuration. |
optional_parameters | Optional[str] | A description of optional parameters. |
returns | str | A description of what the operation returns or how it modifies the data. |
minimal_example_configuration | str | A minimal YAML configuration example for the operation. |
Sources: docetl/reasoning_optimizer/op_descriptions.py:10-25
Plugin Registration
docetl discovers and loads operations using a plugin system based on Python entry points. Custom operations must be registered under the docetl.operation group in the pyproject.toml file. This allows the docetl runner to dynamically load both built-in and custom operations at runtime.
The following snippet from pyproject.toml shows how built-in operations are registered.
[project.entry-points."docetl.operation"]
map = "docetl.operations.map:MapOperation"
parallel_map = "docetl.operations.map:ParallelMapOperation"
filter = "docetl.operations.filter:FilterOperation"
unnest = "docetl.operations.unnest:UnnestOperation"
equijoin = "docetl.operations.equijoin:EquijoinOperation"
split = "docetl.operations.split:SplitOperation"
reduce = "docetl.operations.reduce:ReduceOperation"
resolve = "docetl.operations.resolve:ResolveOperation"
gather = "docetl.operations.gather:GatherOperation"
cluster = "docetl.operations.cluster:ClusterOperation"
sample = "docetl.operations.sample:SampleOperation"
topk = "docetl.operations.topk:TopKOperation"
link_resolve = "docetl.operations.link_resolve:LinkResolveOperation"
code_map = "docetl.operations.code_operations:CodeMapOperation"
code_reduce = "docetl.operations.code_operations:CodeReduceOperation"
code_filter = "docetl.operations.code_operations:CodeFilterOperation"
Sources: pyproject.toml:43-58
Step-by-Step Guide to Creating a Custom Operation
This section provides a walkthrough for creating a new custom operation. The process involves defining the operation's logic, describing its metadata, and registering it as a plugin.
The overall workflow for creating a custom operation is shown below.
Step 1: Define the Operation's Logic
First, create a Python class that will contain the logic for your operation. This class should encapsulate the transformation you want to perform on the data. For deterministic, non-LLM operations, a common pattern is to have a transform method that takes a document as input and returns the modified document or a part of it.
For example, a simple CodeMapOperation is defined by a Python function named transform.
# From docetl/reasoning_optimizer/op_descriptions.py
# Example for a code_map operation
code: |
def transform(input_doc):
import re
text = input_doc.get('content', '')
# Extract words that are all caps (potential keywords)
keywords = re.findall(r'\\b[A-Z]{2,}\\b', text)
# Extract email addresses
emails = re.findall(r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b', text)
return {
'keywords': list(set(keywords)),
'email_count': len(emails),
'word_count': len(text.split())
}
Sources: docetl/reasoning_optimizer/op_descriptions.py:461-476
Step 2: Describe the Operation's Metadata
Once the logic is defined, you must create an instance of the Operator class to describe your custom operation. This metadata makes the operation understandable to the system and other developers.
# This is a hypothetical example for a new 'ToUpper' operation
from docetl.reasoning_optimizer.op_descriptions import Operator
op_to_upper = Operator(
name="ToUpper",
type_llm_or_not="Not LLM-powered",
description="Converts a specified text field in each document to uppercase.",
when_to_use="Use when you need to standardize the case of a text field for deterministic processing or comparison.",
required_parameters='''
name: Unique name for the operation
type: Must be "to_upper"
field: The document field to convert to uppercase.''',
optional_parameters=None,
returns="Each original document, with the specified field's value converted to uppercase.",
minimal_example_configuration='''
name: standardize_title
type: to_upper
field: title
'''
)
Sources: docetl/reasoning_optimizer/op_descriptions.py:7-28
Step 3: Register the Operation as a Plugin
The final step is to make your operation discoverable by the docetl runner. Add a new entry under the [project.entry-points."docetl.operation"] section in your pyproject.toml file. The key is the operation type you will use in the YAML configuration, and the value is the path to your Python class.
[project.entry-points."docetl.operation"]
# ... existing operations
to_upper = "my_custom_operations.case_operations:ToUpperOperation"
Sources: pyproject.toml:43-58
Once registered, the docetl runner can load and execute your custom operation when it encounters the corresponding type in a pipeline's YAML configuration.
Built-in Operations Reference
The following table lists the built-in operations available in docetl, which can serve as examples when creating your own.
| Operation Type | Class Path |
|---|---|
map | docetl.operations.map:MapOperation |
parallel_map | docetl.operations.map:ParallelMapOperation |
filter | docetl.operations.filter:FilterOperation |
unnest | docetl.operations.unnest:UnnestOperation |
equijoin | docetl.operations.equijoin:EquijoinOperation |
split | docetl.operations.split:SplitOperation |
reduce | docetl.operations.reduce:ReduceOperation |
resolve | docetl.operations.resolve:ResolveOperation |
gather | docetl.operations.gather:GatherOperation |
cluster | docetl.operations.cluster:ClusterOperation |
sample | docetl.operations.sample:SampleOperation |
topk | docetl.operations.topk:TopKOperation |
link_resolve | docetl.operations.link_resolve:LinkResolveOperation |
code_map | docetl.operations.code_operations:CodeMapOperation |
code_reduce | docetl.operations.code_operations:CodeReduceOperation |
code_filter | docetl.operations.code_operations:CodeFilterOperation |
Sources: pyproject.toml:44-58
Conclusion
The docetl framework provides a powerful and straightforward mechanism for extending its core functionality through custom operations. By defining the operation's logic in a Python class, describing it with the Operator metadata model, and registering it as a plugin via pyproject.toml, you can seamlessly integrate new, specialized processing steps into your docetl pipelines. This modular design ensures that the system remains flexible and adaptable to a wide range of document processing tasks.