Utility Shell Scripts

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• scripts/normalize_task_operations.py
• scripts/build_benchmark_inventory.py
• scripts/build_operation_hierarchy.py
• scripts/round2_knowledge_screen.py
• scripts/build_operation_examples_table.py

The utility scripts in the work-validity-eval repository comprise a reproduction pipeline designed to transform raw occupational data from O*NET and ESCO into a structured taxonomy of 18 knowledge-work operations. These scripts automate stages such as data ingestion, task-level screening, embedding generation, clustering, and consolidation.

Sources: README.md:144-162

Data Processing Pipeline

The project utilizes a multi-stage pipeline where each script's output serves as the input for the subsequent stage. This pipeline is responsible for filtering thousands of O*NET task statements to identify cognitively-intensive "knowledge work" and then clustering those tasks into recognizable professional operations.

Knowledge Work Screening

The pipeline begins with a two-round screening process. The first round occurs during initial ingestion, while the second round (scripts/round2_knowledge_screen.py) performs a granular, task-level evaluation using LLMs.

• Round 1: Occupation-level gating based on Job Zones (3-5).
• Round 2: Task-level evaluation to exclude direct physical, clinical, or routine manual tasks (e.g., "operate on a patient" or "clean equipment") while including judgment-bearing cognitive tasks (e.g., "analyze market conditions").

Sources: scripts/round2_knowledge_screen.py:7-40, README.md:146-147

Normalization and Embedding

Once tasks are screened, they are normalized by stripping domain-specific nouns to focus on the underlying operation. The scripts/normalize_task_operations.py script rewrites O*NET tasks into "operation phrases" (verb + generic object type) to ensure that tasks like "designing a solar system" and "designing a vacuum tube" cluster together as "designing a hardware system."

Sources: scripts/normalize_task_operations.py:7-25

Core Script Functionality

Script	Purpose	Key Inputs/Outputs
normalize_task_operations.py	Strips domain nouns from tasks to focus on the core action.	In: onet_tasks_prepared.parquet Out: task_operations_normalized.parquet
round2_knowledge_screen.py	Filters for cognitive, judgment-bearing tasks using LLM triage.	In: onet_tasks_prepared.parquet Out: onet_tasks_prepared_round2.parquet
build_operation_hierarchy.py	Uses agglomerative clustering to merge sub-ops into top-level operations.	In: work_pattern_labels_by_cluster.json Out: operation_hierarchy.json
build_benchmark_inventory.py	Populates the inventory of evaluated AI benchmarks (e.g., GAIA, WebArena).	In: Hand-curated dicts Out: benchmarks.csv, labels.csv

Sources: scripts/normalize_task_operations.py:46-52, scripts/round2_knowledge_screen.py:34-40, scripts/build_operation_hierarchy.py:16-25, scripts/build_benchmark_inventory.py:9-15

Architectural Logic

Clustering and Agglomeration

The system uses HDBSCAN for initial clustering, followed by an agglomerative merge in scripts/build_operation_hierarchy.py. A key feature of this script is the Cross-Job Absorption Pass, which prevents the creation of job-specific operations.

Normalization Logic

Normalization uses a specific schema to categorize the "Object Type" of a task. The LLM is constrained to an enum including: code, hardware, material, document, plan, data, image_or_media, person, organization, environment, money, schedule, procedure, and other.

Sources: scripts/normalize_task_operations.py:65-80

Benchmark Integration

The script scripts/build_benchmark_inventory.py maps scored units from existing AI benchmarks to the project's operation taxonomy. For example, FinanceBench is mapped to analyze_data_for_decisions and extract_or_verify_information_from_documents.

Sources: scripts/build_benchmark_inventory.py:65-75

Configuration and Environment

Scripts rely on environment variables for model selection and API access.

• OPENAI_API_KEY: Required for embedding and LLM calls.
• NORMALIZE_TASK_MODEL: Defaults to gpt-5.4-mini.
• ROUND2_KSCREEN_MODEL: Defaults to gpt-5.4-mini.
• OPERATION_HIERARCHY_EMBED_MODEL: Defaults to text-embedding-3-small.

Sources: scripts/normalize_task_operations.py:63, scripts/round2_knowledge_screen.py:53, scripts/build_operation_hierarchy.py:46-48

Conclusion

The utility scripts form the backbone of the "From Scores to Knowledge Work" research, providing a reproducible pathway from raw task data to a validated 18-operation taxonomy. By combining LLM-based triage with traditional clustering and domain-stripping normalization, the scripts ensure the resulting taxonomy is both technically accurate and professionally relevant across multiple occupations.

Sources: README.md:144-162, scripts/build_operation_examples_table.py:116-125