Taxonomy Finalization
Relevant source files
The following files were used as context for generating this wiki page:
Taxonomy Finalization is the terminal stage of the construction pipeline in the work-validity-eval project. Its purpose is to transform raw O*NET occupational task statements into a stable, 18-to-24-operation taxonomy of knowledge work. This process involves stripping domain-specific terminology, clustering similar tasks using high-dimensional embeddings (HDBSCAN), and consolidating these clusters into "frozen" labels that represent cross-job operations suitable for AI agent evaluation.
The finalization logic ensures that the resulting taxonomy is operation-first rather than tool- or occupation-first. It establishes a "minimum task contract" for each operation, defining the properties a benchmark must preserve to validly measure that specific pattern. This stage is critical for distinguishing between "knowledge work" (judgement-bearing cognitive tasks) and "professional work" (tasks requiring formal credentials or liability).
Sources: README.md:14-25, scripts/finalize_work_patterns.py:7-22
Architecture and Data Flow
The finalization process follows a multi-stage pipeline where data is progressively refined from granular task statements to high-level operational categories.
Construction Pipeline
The flow starts with the normalization of O*NET tasks, where domain nouns are stripped to highlight the underlying worker operation. These normalized phrases are embedded and clustered. The finalization scripts then ingest these clusters and use LLMs to synthesize them into the final taxonomy.
This diagram shows the transition from raw data to the finalized taxonomy through normalization, clustering, and consolidation.
Sources: README.md:104-121, scripts/normalize_task_operations.py:11-30
Logic for Emergent Hierarchy
The project employs an "emergent" strategy where the number of top-level operations is not prespecified. Instead, an agglomerative merge is applied to cluster-theme embeddings based on a cosine-distance threshold (typically 0.55). This allows the structure of the data to define the taxonomy breadth.
Sources: scripts/build_operation_hierarchy.py:11-18, scripts/finalize_work_patterns.py:11-16
Key Components and Logic
1. Domain-Stripping and Normalization
Before finalization, tasks are rewritten into domain-stripped operation phrases. This prevents the model from clustering by industry (e.g., "solar" or "medical") and instead groups by the cognitive operation (e.g., "designing hardware" or "analyzing data").
Sources: scripts/normalize_task_operations.py:15-25
2. Agglomerative Merging and Absorption
To ensure operations are "cross-job" (performing across at least five distinct O*NET occupations), the system performs an iterative absorption pass. Any bucket that is too narrow—meaning it has too few unique occupations or sub-operations—is merged into its nearest neighbor based on centroid cosine distance.
Sources: scripts/build_operation_hierarchy.py:173-195, scripts/build_operation_hierarchy.py:220-240
3. Manual Refinement and Splitting
While the process is largely automated, the finalize_work_patterns.py script includes hard-coded logic for manual overrides based on research requirements. For example, a single emergent bucket like "Produce and Communicate Research Reports" is manually split into "Prepare Reports and Presentations" and "Conduct Scientific Research" to separate research execution from communication for audit purposes.
Sources: scripts/finalize_work_patterns.py:17-22, scripts/finalize_work_patterns.py:118-135
Data Structures
The finalized taxonomy is stored in a JSON format that includes detailed metadata for each operation.
Operation Label Schema
| Field | Type | Description |
|---|---|---|
label_id | string | Snake-case stable identifier for the operation. |
label_name | string | Title-case name for display. |
definition | string | 1-2 sentence description of what the worker does. |
minimum_task_contract | string | The specific properties a benchmark must preserve to measure this. |
agent_doability | string | Assessment of AI capability (high, medium, low). |
inclusion_examples | array | Paraphrased examples of tasks that fit the label. |
sub_operation_cluster_ids | array | IDs of the HDBSCAN clusters merged into this label. |
Sources: data/derived/work_pattern_labels_frozen.json:3-23, scripts/freeze_work_patterns.py:40-48
Example: Inspection and Compliance
{
"label_id": "inspect_evaluate_and_verify_compliance_and_performance",
"label_name": "Inspect, Evaluate, and Verify Compliance and Performance",
"definition": "Examine systems, sites, documents, activities, or test results against explicit standards...",
"minimum_task_contract": "The item must provide a concrete subject to examine plus an explicit standard...",
"agent_doability": "high",
"n_tasks": 952
}
Sources: data/derived/work_pattern_labels_frozen.json:3-13
Hierarchy Generation Process
The generation of the hierarchy involves a sequence between local data processing and LLM-assisted description.
This diagram illustrates the interaction between script logic and LLM calls to arrive at the final taxonomy.
Sources: scripts/build_operation_hierarchy.py:441-470, scripts/finalize_work_patterns.py:112-140
Conclusion
Taxonomy Finalization bridges the gap between raw occupational data and an evaluable framework for AI agents. By utilizing domain-stripping, cosine-distance clustering, and cross-job validation, the project produces a stable set of 18–24 operations. This finalized output provides the ground truth necessary for auditing frontier AI evaluation suites against real-world knowledge-work requirements.
Sources: README.md:14-35, scripts/finalize_work_patterns.py:10-16