Consolidating Patterns
Relevant source files
The following files were used as context for generating this wiki page:
The consolidation process in the work-validity-eval project is a multi-stage pipeline designed to transform raw, high-dimensional occupational task data into a stable, operation-first taxonomy of knowledge work. This system bridges the gap between raw O*NET task statements and a structured set of "operations"—the smallest recurring activity patterns across occupations that carry professional construct.
Sources: README.md:21-27, scripts/freeze_work_patterns.py:1-8
Taxonomy Construction Pipeline
The consolidation logic follows a bottom-up approach, starting with raw task data and narrowing down to a frozen taxonomy of approximately 18–28 operations. The pipeline utilizes a combination of vector embeddings, clustering algorithms (HDBSCAN), and Large Language Model (LLM) refinement to ensure the resulting patterns are cross-job, agent-doable, and evaluable.
Sources: scripts/build_operation_hierarchy.py:1-15, scripts/freeze_work_patterns.py:30-55
Data Flow and Transformation
The process begins with "proposing" candidate labels and proceeds through a "tightening" or "freezing" phase.
The diagram above shows the progression from raw tasks to the final frozen taxonomy.
Sources: scripts/propose_work_patterns.py:12-25, scripts/finalize_work_patterns.py:10-25
Core Components and Logic
1. Proposal and Thematic Extraction
In the initial proposal phase, the system stratifies tasks by cluster and job zone. The propose_work_patterns.py script and its successor refresh_work_patterns.py use LLMs to extract "neutral work-pattern labels" grounded in job-task language. These labels must be "operation-first," meaning they describe what the worker does (e.g., "Compare records") rather than the tool used or the specific occupation.
Sources: scripts/propose_work_patterns.py:12-25, scripts/refresh_work_patterns.py:38-75
2. Hierarchical Agglomeration
Instead of prespecifying a target count, the system uses build_operation_hierarchy.py to apply a cosine-distance threshold to the agglomerative merge of cluster-theme embeddings. This allows the number of top-level operations to emerge naturally from the data.
Key logic includes:
- Centroid Calculation: Determining the average vector for a bucket of sub-operations.
- Absorption Pass: Merging "job-specific" buckets (those with too few pooled occupations) into their nearest neighbor to ensure cross-job applicability.
Sources: scripts/build_operation_hierarchy.py:10-25, scripts/build_operation_hierarchy.py:146-160, scripts/build_operation_hierarchy.py:170-200
3. Tightening and Freezing
The final stage involves "tightening" the taxonomy. The freeze_work_patterns.py script takes candidate labels and produces a "CLEAN, FROZEN taxonomy" of exactly 24–28 labels. Each label is required to have a specific schema.
| Field | Description |
|---|---|
label_id | stable snake_case identifier |
label_name | Title Case name |
definition | 1–2 sentence scope description |
inclusion_examples | 3 strings drawn/paraphrased from O*NET |
minimum_task_contract | The property a benchmark must preserve to measure the pattern |
Sources: scripts/freeze_work_patterns.py:32-50, scripts/finalize_work_patterns.py:46-110
Implementation Details
LLM Prompting Strategy
The consolidation relies on specific system prompts that enforce "Cross-Job" constraints. For example, the DESCRIBE_SYSTEM_PROMPT in build_operation_hierarchy.py explicitly forbids profession-specific jargon (e.g., "prescribe medications") in favor of abstract operations (e.g., "recommend and document intervention plans").
Sources: scripts/build_operation_hierarchy.py:53-85
Manual Splits and Finalization
While the process is largely automated, the finalize_work_patterns.py script allows for manual overrides based on research requirements. For instance, the "Produce and Communicate Research Reports" bucket was split into "Prepare Reports and Presentations" and "Conduct Scientific Research" to better separate research execution from communication in benchmark audits.
Sequence of data loading and finalization within the finalize script.
Sources: scripts/finalize_work_patterns.py:130-170
Conclusion
Consolidating patterns is the central mechanism for reducing 12,464 O*NET task statements into a manageable taxonomy of 18–24 operations. By combining unsupervised clustering with LLM-guided thematic consolidation, the system ensures that the final "Operation Atlas" represents distinct, evaluable, and cross-occupational knowledge work patterns.
Sources: README.md:46-55, scripts/finalize_work_patterns.py:30-40