Pandas Integration
Relevant source files
The following files were used as context for generating this wiki page:The docetl library provides a powerful integration with the Pandas library, allowing users to apply LLM-based and other docetl operations directly on DataFrames. This is achieved through a custom .semantic accessor, which extends Pandas DataFrames with a suite of methods that mirror docetl's core operators. This integration enables a fluid, chainable, and familiar workflow for data scientists and analysts who are accustomed to the Pandas API.
The integration is a core dependency, as indicated by the inclusion of pandas in the project's dependencies. The official documentation also dedicates a specific section to this feature, highlighting its importance.
Sources: pyproject.toml:15, mkdocs.yml:38-42
Core Component: SemanticAccessor
The entry point for all docetl operations on a Pandas DataFrame is the .semantic accessor. This accessor is registered on the pandas.DataFrame class and exposes methods for semantic mapping, filtering, aggregation, and more.
The SemanticAccessor maintains the state of the operations, including a history of applied transformations and the total cost incurred from LLM API calls. Each operation returns a new DataFrame with the .semantic accessor attached, allowing for method chaining.
Sources: tests/test_pandas_accessors.py:5
The following diagram illustrates the structure of the SemanticAccessor and its methods.
Row-wise Operations
These operations are applied to each row of the DataFrame independently.
map
The map method applies an LLM prompt to each row of a DataFrame to generate new columns. It is analogous to Pandas' apply method but is powered by an LLM.
The following diagram shows the data flow for the map operation.
| Parameter | Type | Description |
|---|---|---|
prompt | str | A Jinja2 template string that references columns in the row (e.g., {{input.text}}). |
output | dict | A dictionary specifying the output schema and mode. It must contain a schema key. Supported modes are tools (default) and structured_output. |
output_schema | dict | [DEPRECATED] A dictionary defining the new columns and their types. Use output instead. |
model | str | The LLM to use for the operation (e.g., gpt-4o-mini). |
bypass_cache | bool | If True, forces re-computation and ignores any cached results. |
Sources: tests/test_pandas_accessors.py:183-196, tests/test_pandas_accessors.py:426-483
filter
The filter method uses an LLM to decide whether to keep or discard each row based on a prompt that should evaluate to a boolean.
The following diagram shows the data flow for the filter operation.
| Parameter | Type | Description |
|---|---|---|
prompt | str | A Jinja2 template string that should ask a yes/no question about the row's content. |
model | str | The LLM to use for the filtering decision. |
bypass_cache | bool | If True, forces re-computation and ignores any cached results. |
Sources: tests/test_pandas_accessors.py:198-207, tests/test_pandas_accessors.py:270-278
DataFrame-level Operations
These operations consider the entire DataFrame or groups of rows.
merge
The merge method performs a fuzzy, semantic join between two DataFrames. It uses an LLM with a comparison_prompt to determine if rows from each DataFrame are related.
| Parameter | Type | Description |
|---|---|---|
other | pd.DataFrame | The right DataFrame to merge with. |
comparison_prompt | str | A Jinja2 template to compare a row from the left DataFrame ({{left.column}}) and a row from the right ({{right.column}}). |
fuzzy | bool | Must be True to enable semantic merging. |
Sources: tests/test_pandas_accessors.py:209-219
agg
The agg method performs a semantic aggregation on groups of rows. It can perform simple grouping, or fuzzy grouping using an LLM to resolve similar keys.
| Parameter | Type | Description |
|---|---|---|
reduce_prompt | str | A Jinja2 template for the LLM to summarize or aggregate a group of rows. |
reduce_keys | str or list[str] | The column(s) to group by. Use "_all" for a global aggregation. |
output | dict | A dictionary specifying the output schema for the aggregated data. |
fuzzy | bool | If True, enables semantic key resolution before aggregation. |
comparison_prompt | str | (Optional, for fuzzy) Prompt to compare two keys for similarity. |
resolution_prompt | str | (Optional, for fuzzy) Prompt to standardize a group of similar keys. |
resolution_output | dict | (Optional, for fuzzy) The output schema for the resolution_prompt. |
Sources: tests/test_pandas_accessors.py:221-267
The fuzzy aggregation workflow is shown below.
Document Chunking Operations
These operations are designed to handle long documents by splitting them into smaller pieces and then reconstructing context.
split
The split method breaks down the content of a specified column into smaller chunks, creating a new DataFrame where each row represents a chunk.
| Parameter | Type | Description |
|---|---|---|
split_key | str | The name of the column containing the text to be split. |
method | str | The splitting method. Supported values include token_count and delimiter. |
method_kwargs | dict | Arguments for the splitting method, e.g., {'num_tokens': 10} or {'delimiter': '\n\n'}. |
The operation adds new columns to the DataFrame:
<split_key>_chunk: The content of the chunk.semantic_split_<n>_id: A unique identifier for the original document.semantic_split_<n>_chunk_num: The sequential number of the chunk within the document.
Sources: tests/test_pandas_accessors.py:326-362
gather
The gather method reconstructs the context for each chunk created by split. It can include the content of the main chunk as well as "peripheral" content from preceding or succeeding chunks.
| Parameter | Type | Description |
|---|---|---|
content_key | str | The column containing the chunk content (e.g., text_chunk). |
doc_id_key | str | The column identifying the original document. |
order_key | str | The column specifying the chunk order. |
peripheral_chunks | dict | (Optional) Configuration to include context from previous and next chunks. |
This method adds a <content_key>_rendered column containing the main chunk and its surrounding context.
Sources: tests/test_pandas_accessors.py:364-406
Data Restructuring
unnest
The unnest method is used to flatten columns that contain lists or dictionaries into multiple rows or columns.
| Parameter | Type | Description |
|---|---|---|
unnest_key | str | The column to unnest. |
expand_fields | list[str] | (For dicts) A list of dictionary keys to expand into new columns. |
recursive | bool | If True, recursively unnests nested lists/dicts. |
keep_empty | bool | If True, preserves rows where the unnest_key is an empty list. |
Sources: tests/test_pandas_accessors.py:408-483
State and Configuration Management
The SemanticAccessor also provides methods and properties for managing the execution environment and tracking metadata.
Configuration
The set_config method allows for updating the underlying docetl runner configuration.
# Example from tests/test_pandas_accessors.py
df.semantic.set_config(default_model="gpt-4o-mini", optimizer_config={"rewrite_agent_model": "gpt-4o"})
Sources: tests/test_pandas_accessors.py:280-286
Operation History
The history property is a list that records each semantic operation applied to the DataFrame. This provides a complete audit trail of the transformations.
Sources: tests/test_pandas_accessors.py:296-311
Cost Tracking
The total_cost property tracks the cumulative cost of all LLM API calls made through the accessor. This is useful for monitoring and managing expenses.
Sources: tests/test_pandas_accessors.py:269-278, tests/test_pandas_accessors.py:313-324
Conclusion
The Pandas integration in docetl offers a seamless and powerful way to blend traditional data manipulation with advanced LLM-powered semantic operations. By extending the familiar DataFrame object with a .semantic accessor, it provides a natural and efficient workflow for complex data processing tasks involving unstructured text, all while maintaining state, tracking costs, and preserving a history of operations.