Derived Data Artifacts
Relevant source files
The following files were used as context for generating this wiki page:
Derived data artifacts represent the secondary data structures, taxonomies, and analysis results generated from raw occupational data (O*NET 30.2 and ESCO v1.1.1). These artifacts form the basis for auditing AI agent evaluation suites by establishing a structured taxonomy of knowledge-work operations and mapping them to benchmark coverage.
The project transforms thousands of raw task statements into a stabilized 18-operation taxonomy. This process involves embedding, clustering, and LLM-assisted consolidation to ensure the resulting categories are cross-job, agent-doable, and evaluable.
Sources: README.md:16-36, data/derived/esco/README.md:1-12
Data Transformation Pipeline
The generation of derived artifacts follows a multi-stage pipeline that converts raw task descriptions into structured operational definitions.
The diagram shows the logical flow from raw occupational data to the final benchmark audit results.
Sources: README.md:126-143, scripts/cluster_onet_tasks.py:44-70
Preparation and Screening
Raw O*NET tasks are processed and filtered through a "knowledge-work screen." This retains tasks from Job Zones 3-5, resulting in approximately 8,372 retained tasks for the operation atlas. Similarly, ESCO items are scoped to ISCO major groups 1-3.
Sources: README.md:58-64, scripts/esco/prepare_esco_items.py:16-24
Operation Taxonomy Artifacts
The core derived artifact is the 18-operation taxonomy. Each operation is defined with specific attributes to facilitate AI evaluation.
Taxonomy Data Model
The taxonomy is primarily stored in operation_taxonomy.json (and its frozen variant). Each entry contains:
| Field | Type | Description |
|---|---|---|
label_id | string | Unique snake_case identifier (e.g., analysis). |
label_name | string | Human-readable title of the operation. |
definition | string | Technical description of the work pattern. |
minimum_task_contract | string | The load-bearing validity constraint for benchmark items. |
agent_doability | enum | Rating (high, medium, low) of LLM capability. |
evaluation_method | enum | Primary method (e.g., rubric_on_artifact, expert_review). |
sub_operation_cluster_ids | list | IDs of HDBSCAN clusters merged into this operation. |
Sources: data/derived/work_pattern_labels_frozen.json:3-50, scripts/build_operation_hierarchy.py:65-100
Hierarchy Construction Logic
Operations are derived by applying a cosine-distance threshold to agglomerative merges of cluster-theme embeddings. A critical "absorption" pass merges "job-specific" buckets (those with too few pooled occupations) into nearest neighbors to ensure the taxonomy is truly "cross-job."
Sources: scripts/build_operation_hierarchy.py:12-25, scripts/build_operation_hierarchy.py:163-180
Clustering and Embedding Artifacts
Clustering artifacts provide the spatial and statistical grouping of tasks.
Cluster Metadata
The project generates onet_task_clusters.parquet, which includes:
- UMAP Coordinates: 2D and 3D projections (
umap_x,umap_y) for visualization. - HDBSCAN Labels: Cluster assignments, including soft-assigned noise points to ensure maximum corpus coverage.
- Stability Metrics: Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) against KMeans cross-checks.
Sources: scripts/cluster_onet_tasks.py:11-30, scripts/cluster_onet_tasks.py:270-285
Visualization Projections
Visual artifacts include PNG projections used to Separation operations that overlap in lower dimensions.
This diagram illustrates how high-dimensional embeddings are reduced and clustered to produce visual atlas artifacts.
Sources: scripts/cluster_onet_tasks.py:114-135, README.md:73-82
Benchmark Coverage Artifacts
The final set of derived artifacts maps the derived taxonomy to existing AI benchmarks.
Coverage Summary
The coverage_by_label.csv file summarizes how well current benchmarks address the 18 operations. It defines coverage tiers based on the number of scored units:
- Broad: $\ge$ 4 scored units.
- Moderate: 2-3 scored units.
- Weak: 1 scored unit.
- Missing: 0 scored units.
Sources: scripts/build_comparison.py:27-33, scripts/build_comparison.py:75-84
Salience Weighting
Coverage is compared against "Salience Weight," calculated as:
importance_mean * (relevance_pct / 100.0) * core_weight
This ensures the analysis prioritizes operations that are more central to professional knowledge work.
Sources: scripts/build_comparison.py:48-53, scripts/build_comparison.py:108-115
Conclusion
Derived Data Artifacts serve as the bridge between raw occupational data and actionable AI evaluation frameworks. By systematizing task statements into a cross-job hierarchy, the project provides a rigorous structure for identifying "validity gaps" in contemporary AI benchmarks. These artifacts—ranging from the operation taxonomy to the ESCO robustness study—ensure that AI capability claims are grounded in recognizable professional constructs.