Cluster Review Exports
Relevant source files
The following files were used as context for generating this wiki page:
The Cluster Review Export system is a critical component of the work-validity-eval pipeline, responsible for generating human-readable artifacts that allow for the auditing and refinement of machine-generated work patterns. It bridge the gap between high-dimensional task embeddings and the finalized 18-operation taxonomy by exporting cluster data into formats suitable for manual review, documentation, and the scientific paper's supplementary materials.
The primary purpose of these exports is to validate the coherence of HDBSCAN-generated clusters. Each export typically includes representative task statements, occupation titles, and automated "theme labels" generated by LLMs. This ensures that the derived operations—the smallest activity patterns recurring across occupations—are grounded in empirical O*NET data.
Sources: README.md:21-25, scripts/refresh_work_patterns.py:27-32, scripts/build_operation_examples_table.py:15-20
Architecture and Data Flow
The export process follows a structured pipeline where raw O*NET tasks are embedded, clustered, and then enriched with metadata before being formatted into Markdown or JSON files.
The diagram above shows the flow from raw task ingestion to the various export formats used for taxonomy validation.
Sources: scripts/cluster_onet_tasks.py:23-40, scripts/refresh_work_patterns.py:38-42
Key Export Components
1. Operations Examples Table
This export generates a compact Markdown table used for paper captions and documentation. It performs a deterministic selection of "prototypical" tasks for each operation by calculating the median of UMAP coordinates (centroid) and picking tasks closest to that center.
| Component | Description |
|---|---|
| Logic | Ranks tasks by closeness to cluster centroid in 2D UMAP plane. |
| Output | docs/figures/operation_examples_table.md |
| Metadata | Includes op name, cluster counts, task counts, and example statements. |
Sources: scripts/build_operation_examples_table.py:48-60, scripts/build_operation_examples_table.py:108-115
2. Cluster Theme Refresh
The refresh system exports detailed JSON and CSV logs for every HDBSCAN cluster. It utilizes gpt-5.4-mini to analyze the full task list within a cluster to identify a dominant theme, a "minimum task contract," and representative examples.
- Best Theme Task: A single verbatim
task_idthat exemplifies the cluster. - Inclusion/Exclusion Examples: 2-3 tasks fitting the theme and up to 3 outliers.
- Minimum Task Contract: A one-sentence validity constraint for benchmark items.
Sources: scripts/refresh_work_patterns.py:112-132, scripts/refresh_work_patterns.py:150-160
3. Operation Hierarchy Exports
This module exports the 2-level hierarchy (Top-level Operations → Sub-operations/Clusters). It features a "Cross-job Absorption" pass that prevents the export of job-specific operations by merging narrow clusters into their nearest neighbors.
The sequence shows the logic for ensuring exported operations are sufficiently broad for a cross-job taxonomy.
Sources: scripts/build_operation_hierarchy.py:152-165, scripts/build_operation_hierarchy.py:210-225
Data Structures and Field Definitions
The exported JSON files follow strict schemas to ensure downstream compatibility with the taxonomy freezing process.
| Field Name | Type | Source / Purpose |
|---|---|---|
operation_id | String | Snake-case stable identifier (e.g., analysis). |
agent_doability | Enum | Assessment of LLM capability (high/medium/low). |
n_pooled_occupations | Integer | Count of unique O*NET codes in the export bucket. |
evaluation_artifact | String | The concrete output to be graded (e.g., "written report"). |
n_tasks_covered_estimate | Float | Share of cluster tasks the theme covers (0.0 - 1.0). |
Sources: scripts/build_operation_hierarchy.py:100-120, scripts/refresh_work_patterns.py:130-140
Export Implementation Detail
The system uses specific filtering and sampling logic to ensure high-quality exports. For instance, the _pick_examples function in the example table builder ensures that representative tasks are selected from distinct occupations to demonstrate the cross-job nature of the operation.
# scripts/build_operation_examples_table.py:53-62
def _pick_examples(sub: pd.DataFrame, n: int) -> list[tuple[str, str]]:
rows: list[tuple[str, str]] = []
seen_occ: set[str] = set()
for cid, grp in sub.groupby("cluster_id", sort=False):
cx = grp["umap_x"].median(); cy = grp["umap_y"].median()
d = np.hypot(grp["umap_x"] - cx, grp["umap_y"] - cy)
for i in d.argsort().to_numpy():
r = grp.iloc[int(i)]
# ... ensures distinct occupations ...
Sources: scripts/build_operation_examples_table.py:53-62
Summary
Cluster Review Exports provide the load-bearing empirical evidence for the project's 18-operation taxonomy. By generating a variety of artifacts—from raw theme logs to filtered example tables—the system enables a high-fidelity audit of how high-level knowledge-work operations map to specific occupational tasks in the O*NET database.