ESCO Coverage Analysis
Relevant source files
The following files were used as context for generating this wiki page:
The ESCO Coverage Analysis is a validation study designed to test the robustness of the project's 18-operation knowledge-work taxonomy against an external, independently built occupational ontology. While O*NET serves as the primary derivation dataset for the taxonomy, the European Skills, Competences, Qualifications and Occupations (ESCO) v1.1.1 catalog is used as a test bed to determine if the identified work patterns are recoverable in a different framework.
Sources: data/derived/esco/README.md:1-10, README.md:118-125
Data Preparation and Scoping
The analysis begins by filtering the ESCO dataset to focus on knowledge-work relevant items, mirroring the Job Zone 3-5 filter used for O*NET. The process ingests raw CSV files (skills, occupations, and relations) and applies specific inclusion criteria.
Inclusion Criteria
- Skill Type: Only items with
skillType == "skill/competence"are included. Items typed asknowledgedescribe subject-matter domains and are excluded from the primary operational analysis. - ISCO Linkage: The skill must be linked as "essential" to at least one occupation within ISCO-08 major groups 1 (Managers), 2 (Professionals), or 3 (Technicians and Associate Professionals).
Sources: scripts/esco/prepare_esco_items.py:7-22, data/derived/esco/README.md:11-18
Diagram shows the scoping logic for ESCO skills to ensure alignment with professional knowledge-work definitions.
Sources: scripts/esco/prepare_esco_items.py:38-65
Taxonomy Mapping Methodology
The core of the coverage analysis involves mapping the scoped ESCO items to the frozen 18-operation taxonomy using two independent methods: Geometric Centroid Mapping and LLM-based Semantic Classification.
Mapping Approaches
| Method | Description | Implementation Detail |
|---|---|---|
| Centroid Mapping | Calculates cosine similarity between ESCO item embeddings and O*NET operation centroids. | Uses text-embedding-3-small (1,536-d) L2-normalized vectors. |
| LLM Assignment | Semantic classification using a zero-shot LLM prompt with op definitions. | Utilizes gpt-5.4-mini with a JSON schema output for consistency. |
| Consensus | Final assignment rule based on agreement between methods. | Agreement creates a consensus label; disagreement triggers a review flag. |
Sources: scripts/esco/map_esco_to_ops.py:9-35, data/derived/esco/README.md:19-25
Mapping Logic and Agreement
The system calculates a "consensus" by comparing the results of the centroid-based geometric check and the LLM's semantic classification. If the LLM returns none_of_the_above or if the geometric mapping falls below the 5th percentile of within-op O*NET cosines (unmappable_geometric), the item is considered unmapped.
Sources: scripts/esco/map_esco_to_ops.py:246-260, scripts/esco/build_esco_coverage.py:53-64
Diagram illustrates the dual-path mapping architecture to ensure cross-method validation of operation assignments.
Sources: scripts/esco/map_esco_to_ops.py:220-275
Clustering and Structural Recoverability
To further validate the taxonomy, the analysis performs independent clustering of ESCO items without prior knowledge of the 18-operation labels. This is used to calculate structural recoverability metrics.
Clustering Pipeline
- Normalization: L2-normalization of ESCO embeddings.
- Dimension Reduction: UMAP reduction to 15-d cosine space.
- Density Clustering: HDBSCAN with
min_cluster_size=40andmin_samples=10. - Matched-Granularity Check: KMeans ($k=18$) run to provide a fair comparison against the 18-op taxonomy.
Sources: scripts/esco/cluster_esco_items.py:7-27, data/derived/esco/README.md:26-32
Evaluation Metrics
The analysis produces several metrics to quantify how well the 18 operations reflect the natural structure of ESCO:
- ARI/NMI: Adjusted Rand Index and Normalized Mutual Information between HDBSCAN/KMeans partitions and LLM labels.
- kNN Consistency: Fraction of the $k$ nearest neighbors that share the same operation label.
- Purity/Homogeneity: Measures how dominant a single operation is within an unsupervised cluster.
Sources: scripts/esco/cluster_esco_items.py:118-175, scripts/esco/build_esco_coverage.py:100-112
Coverage Metrics and Outputs
The final stage of the analysis aggregates the mapping results into coverage reports that back the external-robustness sections of the project's research.
Summary Statistics
The esco_coverage_summary.json file contains aggregate metrics, including:
- Total number of ESCO items processed.
- Percentage of items covered by any operation (LLM vs. Consensus).
- Rate of strict agreement between centroid and LLM mapping.
- LLM confidence distribution (high, medium, low).
Sources: scripts/esco/build_esco_coverage.py:80-98, data/derived/esco/esco_coverage_summary.json
Per-Operation Analysis
Detailed statistics are generated for each of the 18 operations, including O*NET task counts versus ESCO item counts to compare "salience share" across the two corpora.
Sources: scripts/esco/build_esco_coverage.py:53-78, README.md:123-125
# Per-op coverage calculation logic
for op in ONET_OP_TASK_COUNTS.keys():
llm_mask = df["llm_op_id"] == op
sub = df[llm_mask]
n_items = int(llm_mask.sum())
agree = int(((df["centroid_op_id"] == op) & llm_mask).sum())
top2 = int(
(((df["centroid_op_id"] == op) | (df["centroid_top2_op_id"] == op)) & llm_mask).sum()
)
# ... aggregation and rounding logic ...
Source: scripts/esco/build_esco_coverage.py:53-65
Conclusion
The ESCO Coverage Analysis provides the empirical foundation for claiming that the 18-operation taxonomy is a universal professional construct. By demonstrating high recoverability and mapping consistency in an independent ontology, the project establishes that these work patterns are not artifacts of the O*NET data structure but represent recurring activities in high-skill knowledge work.