Task Clustering & Hierarchy
Relevant source files
The following files were used as context for generating this wiki page:
The Task Clustering & Hierarchy system is a multi-stage pipeline designed to categorize U.S. O*NET occupational task statements into a cross-job taxonomy of knowledge work operations. By leveraging embedding-distance agglomeration and LLM-driven synthesis, the system transforms raw task data into a structured hierarchy consisting of high-level "operations" and constituent "sub-operations" (clusters).
The system ensures that the resulting taxonomy is "agent-doable" and "evaluable," focusing on activity patterns that recur across occupations rather than profession-specific jargon. This process is critical for the project's goal of evaluating AI agents against realistic knowledge-work constructs.
Sources: scripts/build_operation_hierarchy.py:1-40, README.md:16-25
Pipeline Architecture
The construction pipeline follows a bottom-up approach, starting with raw task ingestion and moving through embedding, clustering, and hierarchical consolidation.
The diagram above illustrates the data flow from raw data ingestion to the final 24-label frozen taxonomy.
Sources: scripts/finalize_work_patterns.py:27-35, scripts/cluster_onet_tasks.py:10-30
Data Preparation and Normalization
Task Screening
Raw tasks are first filtered by Job Zone (typically 3-5) and subjected to a two-round knowledge-work screen. The first round uses keyword-based heuristics to include professional terms and exclude manual labor. The second round uses an LLM to refine this at the task level, excluding physical procedures or routine clerical work.
Sources: scripts/prepare_onet_tasks.py:23-100, scripts/round2_knowledge_screen.py:34-100
Domain-Stripped Normalization
To prevent clustering by industry (e.g., "solar" or "robotics"), tasks are normalized into domain-stripped operation phrases. This process preserves the core verb and generic object while removing proper nouns and industry-specific terminology.
| Object Type | Description |
|---|---|
code | Programming and software-related tasks |
hardware | Physical systems and machinery |
plan | Strategic or operational frameworks |
data | Numerical or qualitative information for analysis |
person | Direct interpersonal operations |
Sources: scripts/normalize_task_operations.py:27-50
Clustering Logic
The system utilizes HDBSCAN on UMAP-reduced dimensions to identify dense groups of similar task operations.
Dimensionality Reduction and Density Clustering
- UMAP Reduction: Embeddings (generated via
text-embedding-3-small) are L2-normalized and reduced to 15 dimensions using a cosine metric to facilitate density clustering. - HDBSCAN: Clusters are identified with
min_cluster_size=30andmin_samples=10. - Soft-Assignment: Noise points (unassigned by HDBSCAN) are routed to their nearest cluster using membership vectors to ensure 100% corpus coverage.
Sources: scripts/cluster_onet_tasks.py:10-30, scripts/finalize_work_patterns.py:27-35
Hierarchical Consolidation
The 68 emergent clusters (sub-operations) are merged into top-level operations using an agglomerative merge threshold.
Agglomerative Merge Process
The system applies a cosine-distance threshold (default 0.55) to the embeddings of cluster-theme descriptions. This allows the number of top-level operations to emerge from the data rather than being pre-specified.
This diagram shows the iterative process of consolidating sub-op clusters into a high-level taxonomy.
Sources: scripts/build_operation_hierarchy.py:1-35, scripts/build_operation_hierarchy.py:284-300
Hierarchy Validity Constraints
The system enforces strict requirements for top-level operations:
- Cross-Job Requirement: Operations must plausibly apply to at least five distinct O*NET occupations.
- Agent-Doability: An LLM-agent with standard tools must be able to execute the operation.
- Evaluable: Success must be checkable against a concrete artifact or rubric.
Sources: scripts/build_operation_hierarchy.py:44-90
The Frozen Taxonomy
The final output is a 24-label frozen taxonomy. A significant manual split was performed on the emergent "Produce and Communicate Research Reports" bucket to separate "Conduct Scientific Research" from "Prepare Reports and Presentations."
Taxonomy Metadata Structure
Each operation in the hierarchy includes specific metadata used for validity analysis:
| Field | Purpose |
|---|---|
operation_id | Stable snake_case identifier |
agent_doability | Assessment (High/Medium/Low) of LLM capability |
minimum_task_contract | Load-bearing validity constraint for benchmarks |
evaluation_artifact | The concrete output to be graded (e.g., "written report") |
n_tasks | Total number of O*NET tasks covered |
Sources: scripts/finalize_work_patterns.py:53-150, scripts/build_operation_hierarchy.py:92-120
Conclusion
The Task Clustering & Hierarchy system effectively bridges the gap between raw occupational data and a functional evaluation framework for AI agents. By combining density-based clustering with LLM-guided synthesis and strict cross-job constraints, the project creates a taxonomy of 24 knowledge-work operations that are grounded in empirical labor data and suitable for rigorous agent capability testing.
Sources: README.md:16-25, scripts/finalize_work_patterns.py:34-35