Pipeline Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The pipeline within the work-validity-eval repository is designed to transform raw occupational data from O*NET and ESCO into a structured taxonomy of 18-19 "operations." These operations represent the smallest recurring activity patterns across knowledge-work occupations. The architecture facilitates a rigorous evaluation of AI agents by shifting the focus from primitive benchmark scores to the validity of knowledge-work outputs.
The pipeline operates in three major phases: data ingestion and screening, operational normalization and clustering, and final taxonomy consolidation. This process allows researchers to audit AI benchmark suites against a "strict-cover rule" and four validity criteria (grounding, boundedness, handoff, and organizational accountability).
Sources: README.md:1-50, README.md:188-212
Data Ingestion and Screening
The initial phase focuses on retrieving and filtering raw task statements to isolate "knowledge work." The pipeline processes two primary corpora: O*NET 30.2 and ESCO v1.1.1.
O*NET Processing
O*NET tasks undergo a two-layer screening process. The first layer is an occupation-level gate (Job Zones 3-5). The second layer, handled by scripts/round2_knowledge_screen.py, is a task-level screen that filters out direct physical, manual, clinical, or routine clerical tasks that may exist within otherwise high-level occupations.
Sources: scripts/round2_knowledge_screen.py:1-40, README.md:215-220
ESCO Processing
The ESCO pipeline, primarily driven by scripts/esco/prepare_esco_items.py, scopes skills and competences to knowledge work. It filters for items linked as "essential" to occupations in ISCO-08 groups 1, 2, and 3 (Managers, Professionals, and Technicians).
The diagram shows the parallel ingestion and screening paths for O
NET and ESCO data sources.* Sources: README.md:188-212, scripts/round2_knowledge_screen.py
Operation Normalization and Clustering
Once tasks are screened, the pipeline normalizes them to focus on the "operation" (the verb and generic object) while stripping away domain-specific nouns (e.g., "solar," "robotics").
Text Normalization
The scripts/normalize_task_operations.py script uses LLMs to rewrite task statements into domain-stripped operation phrases. This prevents the embedding model from clustering tasks based on industry rather than activity type.
| Field | Description |
|---|---|
operation_phrase | 3-10 word phrase: <verb> <generic object> |
operation_verbs | 1-3 canonical verbs in infinitive form |
operation_object_type | Enum (e.g., code, hardware, document, person) |
Sources: scripts/normalize_task_operations.py:1-60
Hierarchy Building
scripts/build_operation_hierarchy.py constructs a two-level hierarchy. It uses HDBSCAN to create dense clusters, followed by Agglomerative Clustering to merge clusters into top-level operations based on a cosine-distance threshold.
The clustering logic ensures operations are "cross-job" by requiring they apply to at least five distinct O
NET occupations.* Sources: scripts/build_operation_hierarchy.py:1-100
Benchmark Inventory and Audit
The final stage of the pipeline involves mapping existing AI benchmarks to the derived taxonomy.
Inventory Population
scripts/build_benchmark_inventory.py populates a registry of "scored units." It maps benchmarks like LegalBench, FinanceBench, and WebArena to specific operation labels and documents their task forms and scoring rules.
Sources: scripts/build_benchmark_inventory.py:1-50
Operation Taxonomy Structure
The final taxonomy is stored in data/derived/manual_labeling/operation_taxonomy.json. Each operation includes its definition, input/output specifications, and differentiation from adjacent operations.
{
"operation_id": "analysis",
"one_line": "Take raw evidence and produce findings or written interpretation.",
"input": "Raw evidence / data.",
"output": "A report, paper, memo, brief, or interpretive finding.",
"distinct_from": {
"appraisal": "appraisal uses a pre-specified rubric to produce a score."
}
}
Sources: data/derived/manual_labeling/operation_taxonomy.json:18-35
Conclusion
The pipeline architecture provides a standardized method for classifying knowledge work into 18 discrete operations. By combining automated LLM-based normalization with rigorous clustering and manual auditing, the system creates a foundation for evaluating whether AI agent benchmarks truly exercise the "minimum task contract" required for professional knowledge work.
Sources: README.md:25-45, scripts/build_operation_hierarchy.py:30-50