Programmatic API (Python)
Relevant source files
The following files were used as context for generating this wiki page:
The docetl project provides a Python package for running production data processing pipelines. While the interactive DocWrangler UI is recommended for development and iterative prompt engineering, the Python package is designed for executing finalized pipelines from the command line or directly within Python applications. This allows for integration into larger automated workflows and production environments.
The primary way to use the Python API is by defining a pipeline in a declarative YAML file and then executing it using the docetl command-line tool. This approach separates the pipeline logic (the "what") from the execution engine (the "how").
Sources: README.md:12-14, README.md:38-40, mkdocs.yml:29-32
Getting Started
Installation
To use docetl as a Python package, you need Python 3.10 or later. You can install it using pip.
pip install docetl
Sources: README.md:41-43, pyproject.toml:4
Configuration
The Python package requires an environment variable for the LLM provider's API key. Create a .env file in your project's root directory.
# .env
OPENAI_API_KEY=your_api_key_here
This .env file is used by the backend Python server that executes the docetl pipelines. It is distinct from the website/.env.local file used by the DocWrangler UI.
Sources: README.md:45-52
Usage
The primary programmatic interface is the Command-Line Interface (CLI), which executes a pipeline defined in a YAML file.
Command-Line Interface (CLI)
The docetl package registers a docetl command-line script upon installation. This script is the main entry point for running pipelines.
# Example usage (conceptual)
docetl run --config my_pipeline.yaml
The CLI entry point is defined in the pyproject.toml file, which maps the docetl command to the docetl.cli:app function.
Sources: pyproject.toml:70-71
Execution Flow
The following diagram illustrates the typical execution flow when running a pipeline via the Python API.
This diagram shows how the user initiates a pipeline run via the CLI, which then parses the configuration, loads data, executes the defined sequence of operations, and writes the final result to a file.
Sources: website/public/demos/epstein_email_pipeline.yaml:238-251
Pipeline Configuration (YAML)
Pipelines are defined in YAML files. These files specify the data sources, operations, and execution flow.
Top-Level Structure
A docetl YAML file has several key sections:
| Key | Description |
|---|---|
default_model | The default LLM to use for all operations unless overridden. |
system_prompt | Contains a dataset_description and a persona for the LLM to adopt. |
datasets | Defines the input data sources, typically file paths. |
operations | A list of all processing steps (e.g., map, filter, reduce). |
pipeline | Defines the sequence of operations and the final output configuration. |
Sources: website/public/demos/epstein_email_pipeline.yaml:5-26, website/public/llms.txt:51-68
Example YAML Snippet
This snippet from an example pipeline shows how these sections are defined.
# website/public/demos/epstein_email_pipeline.yaml
default_model: azure/gpt-4.1-mini
bypass_cache: false
system_prompt:
dataset_description: Email correspondence from Jeffrey Epstein's estate...
persona: You are a data analyst...
# --------
# Datasets
# --------
datasets:
epstein_emails:
type: file
path: "epstein-files/emails_dataset.json"
# -----------
# Operations
# -----------
operations:
- name: classify_document_type
type: map
prompt: |
Determine if this document is an actual EMAIL or not.
...
output:
schema:
is_email: bool
# ... more operations
# ----------
# Pipeline
# ----------
pipeline:
steps:
- name: extract_metadata
input: epstein_emails
operations:
- classify_document_type
- filter_emails_only
# ... more operations
output:
type: file
path: "epstein-files/emails_with_metadata.json"
intermediate_dir: "epstein-files/_intermediate"
Sources: website/public/demos/epstein_email_pipeline.yaml
Core Components
Operators
Operators are the fundamental building blocks of a pipeline. They are registered as plugins and can be of various types, including LLM-powered and code-based operations.
The available operators are defined as entry points in pyproject.toml:
| Operator Type | Python Module |
|---|---|
map | docetl.operations.map:MapOperation |
parallel_map | docetl.operations.map:ParallelMapOperation |
filter | docetl.operations.filter:FilterOperation |
reduce | docetl.operations.reduce:ReduceOperation |
resolve | docetl.operations.resolve:ResolveOperation |
split | docetl.operations.split:SplitOperation |
gather | docetl.operations.gather:GatherOperation |
unnest | docetl.operations.unnest:UnnestOperation |
equijoin | docetl.operations.equijoin:EquijoinOperation |
code_map | docetl.operations.code_operations:CodeMapOperation |
code_filter | docetl.operations.code_operations:CodeFilterOperation |
code_reduce | docetl.operations.code_operations:CodeReduceOperation |
sample | docetl.operations.sample:SampleOperation |
topk | docetl.operations.topk:TopKOperation |
Sources: pyproject.toml:73-89
Directives for Optimization
Directives are transformations that can be programmatically applied to a pipeline to optimize it for cost or accuracy. They represent a more advanced programmatic feature of docetl.
An example is the cascade_filtering directive, which optimizes expensive filter operations by injecting a series of cheaper pre-filters.
- Formal Description:
Filter => (Code Filter* -> Filter(gpt-5-nano)* ->) Filter - Natural Language Description: "Optimizes filtering costs by injecting a cascade of cheaper filters before the main filter. Starts with deterministic code filters (cheapest), then gpt-5-nano filters (ordered by prompt length), before the original expensive filter."
The apply method of a directive takes a list of operations and returns a new, transformed list of operations.
# docetl/reasoning_optimizer/directives/cascade_filtering.py:382-409
def apply(
self,
global_default_model: str,
ops_list: List[Dict],
target_ops: List[str],
rewrite: CascadeFilteringInstantiateSchema,
) -> List[Dict]:
"""Apply the directive by injecting pre-filters before the target filter."""
new_ops_list = []
for op in ops_list:
if op["name"] in target_ops:
# First, add all code filters
for i, code_filter in enumerate(rewrite.code_pre_filters):
# ...
new_ops_list.append(code_filter_op)
# Then, add all LLM filters (using gpt-5-nano)
for i, llm_filter in enumerate(sorted_llm_filters):
# ...
new_ops_list.append(llm_filter_op)
# Finally, add the original filter
new_ops_list.append(deepcopy(op))
else:
# Keep other operations as-is
new_ops_list.append(deepcopy(op))
return new_ops_list
This shows how the Python API can be extended with complex, rule-based transformations for pipeline optimization.
Sources: docetl/reasoning_optimizer/directives/cascade_filtering.py:16-25, docetl/reasoning_optimizer/directives/cascade_filtering.py:382-409, tests/reasoning_optimizer/test_directive_apply.py:592-663
Summary
The docetl Python API provides a robust and flexible way to execute document processing pipelines in production environments. By leveraging a declarative YAML format, users can define complex workflows that are executed by a powerful engine. The system is extensible through custom operators and can be programmatically optimized using directives, making it suitable for a wide range of complex document processing tasks.