Tutorials and Examples

Relevant source files

The following files were used as context for generating this wiki page: - [README.md](https://github.com/odra28/docetl/blob/HEAD/README.md) - [website/src/components/TutorialsDialog.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/TutorialsDialog.tsx) - [website/src/components/NaturalLanguagePipelineDialog.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/NaturalLanguagePipelineDialog.tsx) - [website/src/components/PromptImprovementDialog.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/PromptImprovementDialog.tsx) - [website/src/components/PresidentialDebateDemo.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/components/PresidentialDebateDemo.tsx) - [website/src/app/showcase/ai-rfi-response-analysis/page.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/app/showcase/ai-rfi-response-analysis/page.tsx) - [website/src/app/showcase/ai-system-prompts-analysis/page.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/app/showcase/ai-system-prompts-analysis/page.tsx) - [website/src/app/showcase/lease-contract-red-flags/page.tsx](https://github.com/odra28/docetl/blob/HEAD/website/src/app/showcase/lease-contract-red-flags/page.tsx) - [mkdocs.yml](https://github.com/odra28/docetl/blob/HEAD/mkdocs.yml) - [website/public/llms.txt](https://github.com/odra28/docetl/blob/HEAD/website/public/llms.txt)

DocETL provides a rich set of tutorials, examples, and AI-assisted features to help users learn the system and accelerate pipeline development. These resources are primarily available through the DocWrangler interactive UI, and range from pre-built example pipelines to AI-powered assistants that help with pipeline generation and prompt engineering.

The project also includes several showcase pages that demonstrate the power of DocETL on real-world complex document processing tasks. These showcases serve as inspirational examples and provide ready-to-use pipeline configurations that users can download and adapt.

DocWrangler Interactive Tutorials

The DocWrangler UI includes a "Load Example Pipeline" feature that allows users to load pre-configured tutorials directly into their workspace. This is managed by the TutorialsDialog component.

Available Tutorials

Two main tutorials are available, each demonstrating a map-reduce pattern for analysis.

Tutorial	Description	Dataset	Operations
Supreme Court Transcript Analysis	Analyze Supreme Court oral arguments to understand justice reasoning and lawyer arguments.	2024 oral argument transcripts.	1. Map: Analyze patterns in individual transcripts. 2. Reduce: Synthesize patterns into a magazine-style article.
Airline Customer Support Analysis	Analyze customer support chats for common complaints, frustration levels, and satisfaction patterns.	Sample airline customer service chat data.	1. Map: Identify complaints, categorize issues, and assess frustration. 2. Reduce: Summarize common complaints across all chats.

Sources: website/src/components/TutorialsDialog.tsx:25-198

Tutorial Loading Process

When a user selects a tutorial, a process is initiated to download the dataset, upload it to the user's workspace, and then load the corresponding pipeline configuration.

The following diagram illustrates the tutorial loading sequence:

This flow is orchestrated within the loadTutorial function and the useEffect hook that watches for the dataset to be uploaded before restoring the pipeline.

Sources: website/src/components/TutorialsDialog.tsx:320-410

Showcase Demos

The project website features several showcase pages, each demonstrating a complete DocETL pipeline applied to a specific use case. These pages provide context, processing details, and downloadable assets (input data, output data, and the pipeline YAML).

Showcase	Description	Pipeline Goal
AI RFI Response Analysis	Analyzes 10,000+ public responses to a U.S. government Request for Information on AI.	Extracts proposals, demographics, and topics of concern.
Lease Contract Red-Flag Analysis	Ranks 179 lease agreements by the severity of "red-flag" clauses.	Extracts and ranks risky provisions to help sort legal documents.
AI System Prompts Analysis	Analyzes leaked system prompts from popular AI assistants.	Identifies common prompt engineering strategies and patterns.
US Presidential Debate Analysis	Analyzes themes in US presidential debates since 1960.	Summarizes the evolution of party viewpoints on 152 distinct themes.

Sources: website/src/app/showcase/ai-rfi-response-analysis/page.tsx:75-92, website/src/app/showcase/lease-contract-red-flags/page.tsx:78-93, website/src/app/showcase/ai-system-prompts-analysis/page.tsx:71-87, website/src/components/PresidentialDebateDemo.tsx:323-332

Example Showcase Pipeline: Presidential Debate Analysis

The "US Presidential Debate Analysis" demo uses a multi-step pipeline to transform raw debate transcripts into thematic reports.

This diagram shows the data flow for the debate analysis pipeline:

AI-Assisted Pipeline Generation

DocWrangler includes a feature to generate a full DocETL pipeline from a natural language description. This is handled by the NaturalLanguagePipelineDialog component.

The process involves the user uploading a dataset (JSON or CSV) and describing their goal. An AI assistant then generates a complete YAML pipeline configuration.

The following diagram shows the AI-assisted generation flow:

The prompt sent to the /api/generate endpoint is constructed from a base set of instructions, a sample of the user's data, and the user's specific request.

Sources: website/src/components/NaturalLanguagePipelineDialog.tsx:288-403, website/public/llms.txt

A snippet of the prompt construction logic:

// website/src/components/NaturalLanguagePipelineDialog.tsx:348-378
const sampleData = extractSampleData(datasetRecords);

// Create the combined user message
const promptText = `You are a DocETL pipeline generator. Your task is to create a YAML pipeline configuration based on the user's requirements and data.

Follow these DocETL guidelines:
${instructions}

I need to create a DocETL pipeline for the following data:

${sampleData}

The task I need a pipeline for is: ${userPrompt}

// ... more instructions ...
`;

AI-Powered Prompt Improvement

To help users write effective prompts for their operations, DocWrangler provides a "Prompt Improvement" assistant. This feature, implemented in PromptImprovementDialog.tsx, uses an AI to analyze and suggest revisions for existing prompts.

Workflow

1. Select Operation: The user selects an LLM-based operation (map, reduce, resolve, etc.) to improve.
2. Analyze: The assistant analyzes the current prompt, schema, and surrounding pipeline context. It also considers any bookmarked notes the user has made for that operation.
3. Suggest: The AI provides a revised prompt, explaining its reasoning. Changes are highlighted in a diff view.
4. Iterate: The user can provide natural language feedback (e.g., "make it more concise") to further refine the prompt, or edit it directly.
5. Save: Once satisfied, the user can save the new prompt, which overwrites the old one in the pipeline configuration.

This diagram illustrates the prompt improvement interaction:

The system prompt for the assistant is dynamically generated to include the full pipeline context, the specific operation's prompt, and instructions on how to provide suggestions.

Sources: website/src/components/PromptImprovementDialog.tsx:143-231, website/src/components/PromptImprovementDialog.tsx:613-689

Other Educational Resources

The project provides additional resources to guide users:

• Documentation: A full documentation site is available, with guides on installation, core concepts, operators, and optimization.
• Community Projects: The README.md links to community-contributed example projects.
• LLM Instructions: A detailed text file (llms.txt) is provided for users who want to use an external LLM like ChatGPT to help write a pipeline from scratch. It describes the DocETL system, operators, and best practices.

Sources: mkdocs.yml:9-70, README.md:21-23, website/public/llms.txt:3-10