Home
Relevant source files
The following files were used as context for generating this wiki page:
The work-validity-eval project provides a framework for rethinking AI agent evaluation by moving beyond primitive scores toward a "knowledge work" validity account. It argues that high scores on public benchmarks do not necessarily demonstrate readiness for real-world professional workflows. The project establishes this through a two-layer validity account (Task Validity and Organizational Validity) and an empirically derived taxonomy of 18 knowledge-work operations built from O*NET 30.2 and validated against ESCO v1.1.1.
Sources: README.md:1-26
Core Methodology
The project's position is that knowledge work evaluation must be based on operations—the smallest recurring activity patterns across occupations—and that output must be validated across specific criteria. Primitive scores (like exact-match patches or retrieval) are insufficient unless evaluation establishes both operation coverage and output validity.
The Validity Framework
The project defines a two-layer validity account for knowledge-work outputs:
- Task Validity: Includes grounding, boundedness, and handoff.
- Organizational Validity: Includes authorized use, locatable accountability, and integrability.
Sources: README.md:15-26
Technical Pipeline Architecture
The repository implements a complex reproduction pipeline to process occupational data into a stabilized taxonomy. This involves ingestion, screening, embedding, clustering, and consolidation.
Processing Workflow
The diagram below illustrates the flow from raw data sources to the final operation taxonomy and audit.
The pipeline uses LLM-based screening and normalization to strip domain-specific nouns, leaving only the underlying cognitive operation.
Sources: scripts/round2_knowledge_screen.py:1-41, scripts/normalize_task_operations.py:1-38, scripts/build_operation_hierarchy.py:1-27
Knowledge Screening and Normalization
The project employs a two-stage screening process. An initial occupation-level gate is followed by a task-level screen to exclude direct physical, manual, or routine clerical tasks, even if they appear in knowledge-work occupations.
| Component | Description | Key Script |
|---|---|---|
| Knowledge Screen | Filters tasks for primarily cognitive, judgment-bearing work. | round2_knowledge_screen.py |
| Normalization | Strips domain/industry nouns to reveal the generic operation. | normalize_task_operations.py |
| Hierarchical Build | Agglomerates clusters into cross-job top-level operations. | build_operation_hierarchy.py |
Sources: scripts/round2_knowledge_screen.py:1-41, scripts/normalize_task_operations.py:1-38, scripts/build_operation_hierarchy.py:65-114
The 18 Knowledge-Work Operations
The pipeline results in an 18-operation taxonomy. These operations represent the "primitives" of knowledge work that an AI agent should be evaluated against.
| Operation | Definition | Task Count |
|---|---|---|
analysis | Evidence → findings / interpretation. | 1,732 |
administration | Run HR / finance / procurement / operations. | 1,517 |
design | Specify a new system / plan / service. | 992 |
inspection | Present-state compliance check against rules or specs. | 963 |
troubleshooting | Diagnose a malfunction and apply a remediation. | 902 |
record_keeping | Produce, store, retrieve records (retrieval/RAG). | 882 |
Sources: README.md:46-64, data/derived/work_pattern_labels_frozen.json:1-450
Operational Specification
Each operation in the taxonomy is defined by specific metadata, including an agent_doability rating and an evaluation_artifact. For example, Inspect, Evaluate, and Verify Compliance produces an "inspection/audit/review report" and is considered "high" doability for AI agents.
Sources: data/derived/work_pattern_labels_frozen.json:3-20
Benchmark Audit and Coverage
The project includes a methodology for auditing frontier suites (e.g., GDPval, $\tau^2$-Bench, APEX-Agents) against the taxonomy and validity criteria. This involves mapping benchmark "scored units" to work-pattern labels.
Benchmark Inventory Logic
Benchmark units are categorized by their task form, input form, and scoring rule.
Coverage tiers are assigned based on the number of scored units:
- Broad: ≥ 4 units
- Moderate: 2–3 units
- Weak: 1 unit
- Missing: 0 units
Sources: scripts/build_comparison.py:46-60, scripts/build_benchmark_inventory.py:21-170
Audit Summary of Frontier Suites
Current benchmarks often exhibit a "validity gap" where the handoff to organizational systems (sign-off, liability, accountability) is not exercised.
| Suite | Coverage Verdict | Validity Gap |
|---|---|---|
| GDPval | Partial cover | Handoff does not test admissibility or liability. |
| $\tau^2$-Bench | Partial cover | Handoff terminates inside the simulator. |
| APEX-Agents | Partial cover | Institutional layer (sign-off, filing) not exercised. |
Sources: README.md:105-115
Repository Structure
The project is organized into reproduction scripts, data artifacts, and documentation.
scripts/: Contains the reproduction pipeline (O*NET + ESCO).data/: Contains raw inputs and derived analysis artifacts.docs/: Includes codebooks, audit notes, and figures like the Operation Atlas.main.tex: LaTeX source for the NeurIPS 2026 position paper.
Sources: README.md:126-150
The work-validity-eval project provides the technical foundation for a more rigorous evaluation of AI agents by shifting focus from benchmark scores to the operational validity of knowledge work.