Proposing Work Patterns
Relevant source files
The following files were used as context for generating this wiki page:
The "Proposing Work Patterns" phase is a critical stage in the construction of an empirically derived taxonomy of knowledge-work operations. This process transitions from raw O*NET task statements and HDBSCAN clusters to a structured set of 18–24 "work patterns" or "operations." These patterns represent the smallest recurring activity units that carry professional constructs across different occupations. Sources: README.md:16-24, scripts/propose_work_patterns.py:11-15
The primary objective is to draft neutral, operation-first labels grounded in job-task language rather than AI or benchmark terminology. This ensures that the evaluation of AI agents is based on real-world professional requirements such as drafting, analysis, and troubleshooting. Sources: scripts/propose_work_patterns.py:18-21, README.md:12-15
Pipeline Overview
The proposal pipeline consists of sampling clustered tasks, utilizing Large Language Models (LLMs) to draft candidate labels, and consolidating those candidates into a stabilized, frozen taxonomy.
Logic Flow of Pattern Proposal
The flow below illustrates how task data is transformed into a finalized taxonomy.
The pipeline supports two main paths: a stratified sampling pass for broad drafting and a per-cluster refresh pass for granular thematic extraction. Sources: scripts/propose_work_patterns.py:10-23, scripts/refresh_work_patterns.py:12-25
Draft Generation and Refresh
Drafting work patterns involves interpreting the "dominant theme" of a task cluster. The system uses specific prompts to ensure labels are "operation-first" (e.g., "Compare records against standards" instead of "Uses Excel"). Sources: scripts/refresh_work_patterns.py:38-42, scripts/propose_work_patterns.py:38-45
Proposal Strategies
| Strategy | Description | Key Script |
|---|---|---|
| Stratified Sampling | Samples ~12 tasks per cluster, weighting by importance and relevance to surface high-salience tasks. | propose_work_patterns.py |
| Cluster Refresh | Sends every task in a cluster to the LLM to identify a dominant theme and pick a "best theme task" exemplar. | refresh_work_patterns.py |
Sources: scripts/propose_work_patterns.py:10-13, 85-95, scripts/refresh_work_patterns.py:10-20, 115-120
LLM Output Schema
The LLM is required to return a JSON object with the following structure for each proposed pattern:
- theme_label: A 2-6 word sentence-case phrase.
- definition: A one-sentence, job-task-side description.
- minimum_task_contract: The properties a benchmark must preserve to measure this pattern.
- examples: Inclusion and exclusion task IDs.
Sources: scripts/refresh_work_patterns.py:84-95, scripts/propose_work_patterns.py:110-115
Consolidation and Freezing
Once candidate labels are generated, they undergo a "tightening" or "consolidation" pass. This merges near-duplicates (e.g., merging "compare to standard" and "inspect for defects") and enforces a target count of labels, typically between 22 and 32. Sources: scripts/propose_work_patterns.py:65-72, scripts/freeze_work_patterns.py:33-40
Sequence of Consolidation
Final Taxonomy Promotion
The final step, finalize_work_patterns.py, produces the authoritative set of labels used for evaluation. This script includes manual adjustments, such as splitting the "Produce and Communicate Research Reports" bucket into "Prepare Reports and Presentations" and "Conduct Scientific Research" to better align with benchmark coverage needs. Sources: scripts/finalize_work_patterns.py:14-22, 133-145
Data Structures
The outputs of the proposal process are stored in data/derived/ to facilitate manual review and downstream analysis.
Candidate Label Model
| Field | Type | Description |
|---|---|---|
label_id | string | stable snake_case identifier |
label_name | string | Concise title-case name |
definition | string | 1–2 sentence professional description |
agent_doability | enum | High, Medium, or Low assessment for LLMs |
minimum_task_contract | string | Validity constraint for benchmarks |
Sources: data/derived/work_pattern_labels_frozen.json:3-12, scripts/freeze_work_patterns.py:43-50
{
"label_id": "analyze_and_interpret_data_for_decisions",
"label_name": "Analyze and Interpret Data for Decisions",
"definition": "Examine data, records, or test outputs to identify patterns supporting a decision or plan.",
"minimum_task_contract": "Must provide concrete data and require an explicit interpretive conclusion.",
"agent_doability": "high"
}
Sources: data/derived/work_pattern_labels_frozen.json:164-173
Conclusion
Proposing work patterns is an iterative process that converts high-dimensional task clusters into a human-interpretable and machine-evaluable taxonomy. By grounding these patterns in O*NET task language and requiring a "minimum task contract," the system establishes a valid framework for assessing AI agent capabilities in professional knowledge-work domains. Sources: README.md:12-20, scripts/finalize_work_patterns.py:27-38