Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• tests/reasoning_optimizer/test_directive_apply.py
• tests/test_output_modes.py
• docetl/reasoning_optimizer/AGENT.md
• docetl/reasoning_optimizer/directives/README.md
• experiments/structured_outputs.py

The testing strategy for docetl is multi-faceted, encompassing simple crash tests, feature-specific validation, and performance experiments. The primary focus is on ensuring the correctness and stability of the Reasoning Optimizer's directives, which are responsible for transforming and optimizing data processing pipelines. Additionally, core functionalities like different LLM output modes are rigorously tested, and experimental scripts are used to compare the performance of different implementation approaches.

The testing framework uses a combination of standard Python scripts for simple checks and pytest for more structured feature testing. Documentation within the codebase also outlines a clear process for testing newly created directives, covering both instantiation logic and application.

Sources: tests/reasoning_optimizer/test_directive_apply.py, tests/test_output_modes.py, docetl/reasoning_optimizer/AGENT.md, docetl/reasoning_optimizer/directives/README.md

Directive Testing

A significant portion of the testing effort is dedicated to validating the behavior of pipeline transformation directives. This is broken down into two main areas: apply() method verification and instantiation logic testing.

Sources: tests/reasoning_optimizer/test_directive_apply.py, docetl/reasoning_optimizer/directives/README.md:213-231

apply() Method Verification

The test_directive_apply.py script contains a suite of simple tests designed to ensure that the apply() method of each directive does not crash and correctly modifies the pipeline structure. These tests do not use pytest but are run as a standard Python script. Each test function sets up a mock pipeline configuration (ops_list) and a mock rewrite schema, then calls the directive's apply() method and asserts the expected outcome, such as the number and type of operations in the resulting pipeline.

Sources: tests/reasoning_optimizer/test_directive_apply.py:2-5, tests/reasoning_optimizer/test_directive_apply.py:1001-1033

The following diagram illustrates the typical flow for an apply() method test.

The table below summarizes the directives tested and the structural changes they are expected to make.

Directive Tested	Expected Structural Change	Source File
ChainingDirective	Replaces 1 map op with 2 map ops.	test_directive_apply.py:40-41
GleaningDirective	Adds a gleaning configuration to a map op.	test_directive_apply.py:65-67
ChangeModelDirective	Changes the model field of a map op.	test_directive_apply.py:86-87
OperatorFusionDirective	Fuses 2 map ops into 1 map op.	test_directive_apply.py:113-114
DocSummarizationDirective	Adds a summarization map op before the target op.	test_directive_apply.py:293-294
IsolatingSubtasksDirective	Replaces 1 map op with a parallel_map and a map op.	test_directive_apply.py:328-329
ReduceChainingDirective	Adds a map op before a reduce op.	test_directive_apply.py:382-383
DocCompressionDirective	Adds an extract op before the target op.	test_directive_apply.py:431-432
DeterministicDocCompressionDirective	Adds a code_map op before the target op.	test_directive_apply.py:461-462
DocumentChunkingDirective	Replaces 1 map op with a split -> gather -> map -> reduce sequence.	test_directive_applypy:497-498
DocumentChunkingTopKDirective	Replaces 1 map op with a split -> topk -> reduce sequence.	test_directive_apply.py:569-570
ChunkHeaderSummaryDirective	Inserts a parallel_map op between split and gather ops.	test_directive_apply.py:716-717
TakeHeadTailDirective	Adds a code_map op before the target map or filter op.	test_directive_apply.py:748-752
ClarifyInstructionsDirective	Modifies the prompt of an existing map op.	test_directive_apply.py:790-793
SwapWithCodeDirective	Replaces a reduce op with a code_reduce op (and optionally a map op).	test_directive_apply.py:821-825
HierarchicalReduceDirective	Replaces 1 reduce op with 2 reduce ops (and optionally a map op).	test_directive_apply.py:898-901
CascadeFilteringDirective	Injects code_filter and filter ops before the target filter op.	test_directive_apply.py:969-970

Instantiation and Logic Testing

Beyond simple apply() verification, the project has a testing framework for a directive's instantiation logic. This involves ensuring the llm_instantiate method works correctly and that the agent can generate a valid InstantiateSchema. Each directive class defines a list of DirectiveTestCase objects, which specify input configurations and expected behaviors.

These tests can be run from the command line.

Command	Description
python experiments/reasoning/run_tests.py --directive=<name>	Runs instantiation tests for a single, specific directive.
python experiments/reasoning/run_tests.py	Runs instantiation tests for all directives.

Sources: docetl/reasoning_optimizer/AGENT.md:52-55, docetl/reasoning_optimizer/directives/README.md:218-226

Core Feature Testing

The project uses pytest to test core features, such as the different output modes for LLM operations.

Output Mode Comparison

The tests/test_output_modes.py file contains tests for comparing the tools and structured_output modes for map and reduce operations. These tests use a synthetic dataset of documents with embedded fruit and vegetable names.

The test setup involves:

1. Creating a temporary JSON file for the synthetic dataset using a pytest fixture (temp_dataset_file).
2. Creating a temporary output file for the pipeline results (temp_output_file).
3. Executing a DSLRunner with a pipeline configuration parameterized by the output_mode.
4. Asserting that the results are a non-empty list.
5. Counting the number of extracted items and assessing accuracy.

Sources: tests/test_output_modes.py:37-46, tests/test_output_modes.py:114-149, tests/test_output_modes.py:175-210

The test flow for comparing output modes is as follows:

Experimental Validation

The experiments/ directory contains scripts for performance validation, which are not part of the formal test suite but serve to compare different implementation strategies.

Structured Output vs. Tool Calling Performance

The experiments/structured_outputs.py script runs an experiment to compare the performance of structured_output versus tool_calling for litellm.completion. The experiment measures precision, recall, F1 score, average runtime, and average cost.

The process is as follows:

1. Data Augmentation: Presidential debate transcripts are loaded, and a random number of fruit/vegetable names are inserted into the text. The set of inserted items serves as the ground truth.
2. Parallel Execution: For each model and document, two functions are called in parallel: evaluate_structured_output and evaluate_tool_calling.
3. Evaluation: Each function calls the LLM with the respective output format (response_format for structured, tools for tool calling) and extracts the items.
4. Metrics Calculation: The extracted items are compared against the ground truth to calculate precision, recall, and F1 score. Runtime and cost are also recorded.
5. Results Aggregation: Results are aggregated and can be displayed in a rich table.

Sources: experiments/structured_outputs.py:57-81, experiments/structured_outputs.py:159-170, experiments/structured_outputs.py:172-202

This diagram shows the sequence of the experimental validation process.

Test Execution

The project provides several ways to run tests, catering to different needs from quick checks to full suite runs.

Command	Description	Source
python tests/reasoning_optimizer/test_directive_apply.py	Runs simple crash tests for all directive apply() methods.	docetl/reasoning_optimizer/AGENT.md:58-59
python experiments/reasoning/run_tests.py	Runs instantiation tests for all directives.	docetl/reasoning_optimizer/AGENT.md:55-56
python experiments/reasoning/run_tests.py --directive=<name>	Runs instantiation tests for a specific directive.	docetl/reasoning_optimizer/AGENT.md:54-55
make tests-basic	Runs the basic pytest test suite.	README.md:204

Conclusion

The testing strategy in docetl is comprehensive, covering multiple layers of the application. It combines simple, fast-running script-based tests for directive application, a formal pytest suite for core features like output modes, and detailed experimental scripts for performance and accuracy validation of key components. This layered approach ensures both the stability of individual components and the overall quality and efficiency of the pipeline execution and optimization processes. The clear documentation on how to test new directives further supports the project's maintainability and extensibility.