Data Shaping Operations: Split, Unnest, Gather
Relevant source files
The following files were used as context for generating this wiki page:
Data shaping operations are auxiliary operators within DocETL designed to restructure data before or after it is processed by core LLM operations like map or reduce. These operators are essential for handling complex data formats, such as very long documents that exceed model context windows or datasets with nested structures. The primary shaping operators are split, gather, and unnest.
These operators are registered as plugins in the system, allowing the DSLRunner to dynamically load and execute them based on the pipeline configuration. They form the building blocks for advanced patterns like document chunking, which can be automatically synthesized by the reasoning optimizer.
Sources: website/public/llms.txt:19, pyproject.toml:53-68, docetl/moar/search_utils.py:220-227
Split Operation
The Split operation is used to break down a large text field within a document into multiple, smaller documents or "chunks". This is a critical preprocessing step for handling documents that are too long for an LLM's context window.
Sources: docetl/operations/split.py, docetl/reasoning_optimizer/directives/doc_chunking.py:17-21
Configuration
The Split operation is configured with the following parameters in the pipeline YAML:
| Parameter | Type | Description |
|---|---|---|
name | string | A unique name for the operation instance. |
type | string | Must be split. |
split_key | string | The field in the input document that contains the text to be split. |
method | string | The method to use for splitting. A common value is token_count. |
method_kwargs | dict | Arguments for the splitting method. For token_count, this includes num_tokens. |
Sources: docetl/operations/split.py:16-24, tests/test_synth_gather.py:163-170
Data Flow and Output
The Split operation takes a list of documents as input and produces a larger list of chunked documents. For each chunk created, it generates a new document containing the original document's fields plus additional metadata about the split.
The diagram below illustrates the process:
This diagram shows a single document being split into multiple chunks, each with added metadata.
Output Fields:
split_<op_name>_id: A unique identifier for the original document.split_<op_name>_chunk_num: The sequential order of the chunk within the original document.<split_key>_chunk: The actual text content of the chunk.
Sources: docetl/operations/split.py:64-70
Gather Operation
The Gather operation is the counterpart to Split. It is used to re-contextualize a chunk by combining it with content from neighboring chunks. This is particularly useful for providing context to an LLM when it processes a chunk in isolation, ensuring that information from the start, end, or surrounding parts of the original document is not lost.
Sources: docetl/operations/gather.py, tests/test_synth_gather.py:187-199
Configuration
| Parameter | Type | Description |
|---|---|---|
name | string | A unique name for the operation instance. |
type | string | Must be gather. |
content_key | string | The field containing the chunk's text (e.g., content_chunk). |
doc_id_key | string | The field identifying the original document (e.g., split_doc_id). |
order_key | string | The field indicating the chunk's sequence number (e.g., split_doc_chunk_num). |
peripheral_chunks | dict | A configuration object specifying which neighboring chunks to include as context. It can contain previous and next sections, each with head and tail subsections. |
doc_header_key | string | (Optional) A field containing document-level headers or summaries to prepend to the rendered content. |
Sources: docetl/operations/gather.py:16-29, tests/test_synth_gather.py:193-199
Data Flow and Output
The Gather operation processes a list of chunked documents and outputs a list of the same size. Each output document corresponds to an input chunk but includes a new field, content_chunk_rendered, which contains the main chunk's content along with the specified peripheral context.
The diagram below shows how Gather adds context to a chunk:
This diagram illustrates the Gather operation combining a current chunk with its neighbors to create a context-rich rendered chunk.
The content_chunk_rendered field is formatted with clear delimiters:
--- Previous Context ---
...
--- Begin Main Chunk ---
...
--- End Main Chunk ---
...
--- Next Context ---
...
Sources: docetl/operations/gather.py:133-156, tests/test_synth_gather.py:223-234
Unnest Operation
The Unnest operation is designed to flatten nested data structures. It takes a field containing a list of objects and creates a new, separate document for each object in the list, copying the parent document's other fields into each new document.
Sources: docetl/operations/unnest.py
Configuration
| Parameter | Type | Description |
|---|---|---|
name | string | A unique name for the operation instance. |
type | string | Must be unnest. |
unnest_key | string | The field containing the list of objects to be unnested. |
recursive | boolean | When unnesting a list[dict], this must be set to true. |
Sources: docetl/operations/unnest.py:16-21, website/public/llms.txt:102
Data Flow and Output
The Unnest operation transforms a single document with a nested list into multiple documents.
The diagram below shows this transformation:
This diagram shows a single document with a nested list in the data field being unnested into two separate documents.
Sources: docetl/operations/unnest.py:44-59
Common Usage Pattern: Document Chunking
The Split and Gather operations are fundamental to the document chunking pattern, which is a common strategy for processing long documents. This pattern is formalized in the DocumentChunkingDirective, which can automatically transform a single map operation into a Split -> Gather -> Map -> Reduce pipeline.
This automated transformation is often triggered by setting optimize: true on a map operation that processes long documents.
Sources: docetl/reasoning_optimizer/directives/doc_chunking.py:17-24, website/public/llms.txt:104-106, tests/test_synth_gather.py:79-82
The full flow is as follows:
This diagram shows the complete document chunking pipeline, from splitting a long document to reducing the processed chunks into a final result.
This pattern is tested in tests/test_synth_gather.py, which verifies that the optimizer can synthesize a gather operation and that a manual split -> map -> gather sequence executes correctly.
Sources: tests/test_synth_gather.py:101-118, 202-209
Registration as Entry Points
These data shaping operations are registered as plugins within the docetl ecosystem via the pyproject.toml file. This allows the DSLRunner to dynamically load them based on the type field in a pipeline's YAML configuration.
[project.entry-points."docetl.operation"]
unnest = "docetl.operations.unnest:UnnestOperation"
split = "docetl.operations.split:SplitOperation"
gather = "docetl.operations.gather:GatherOperation"
Sources: pyproject.toml:58-62
Summary
The Split, Gather, and Unnest operations are essential tools for data preparation and restructuring in DocETL. Split and Gather work in tandem to enable the processing of long documents by breaking them into manageable chunks and re-introducing necessary context. Unnest provides a straightforward way to flatten nested data, making it accessible to subsequent operations. Together, they ensure that data is in the optimal shape for effective analysis by core LLM-powered operators.