ESCO Data Preparation
Relevant source files
The following files were used as context for generating this wiki page:
The ESCO Data Preparation module is a critical component of the work-validity-eval project, serving as the primary mechanism for external-taxonomy robustness testing. Its purpose is to determine if the 18-operation taxonomy, originally derived from O*NET task statements, is recoverable and valid when applied to the European Skills, Competences, Qualifications and Occupations (ESCO) v1.1.1 framework. Sources: data/derived/esco/README.md:3-9, README.md:144-149
The preparation pipeline ingests raw ESCO CSV data, filters it to define a "knowledge work" scope (matching O*NET Job Zones 3-5), and transforms these items into a format suitable for embedding, clustering, and LLM-based operation assignment. This process ensures that ESCO serves as an independent test bed for the project's operation atlas. Sources: scripts/esco/prepare_esco_items.py:7-14, data/derived/esco/README.md:12-21
Pipeline Architecture
The ESCO preparation and analysis workflow follows a linear sequence of data acquisition, scoping, transformation, and evaluation.
This diagram illustrates the flow of data from raw external source acquisition to the generation of coverage and recoverability reports. Sources: data/derived/esco/README.md:52-61, README.md:151-163
Data Acquisition and Ingestion
The initial stage involves retrieving the ESCO v1.1.1 English classification and relations dataset. This is handled by a shell script that fetches zipped CSV files from a public mirror. Sources: scripts/esco/download_esco.sh:1-20
| File Name | Source Description |
|---|---|
skills_en.csv | Core skill/competence labels and descriptions. |
occupations_en.csv | Occupation labels and ISCO-08 group mappings. |
occupationSkillRelations.csv | Links between occupations and skills (essential/optional). |
Sources: scripts/esco/prepare_esco_items.py:84-86, scripts/esco/download_esco.sh:22-23
Scoping and Filtering Logic
To ensure comparability with the ONET derivation set, the ESCO data is filtered to a specific "knowledge work" scope. This is the ESCO analogue of the ONET Job Zone 3-5 filter. Sources: scripts/esco/prepare_esco_items.py:16-19
Filtering Criteria
- Skill Type: Only rows where
skillType == "skill/competence"are included in the primary pass. Items typed asknowledge(subject-matter domains like "anatomy") are excluded from primary analysis. Sources: scripts/esco/prepare_esco_items.py:11-14 - ISCO-08 Mapping: The skill must be linked as
essentialto at least one occupation in ISCO major groups 1, 2, or 3 (Managers, Professionals, and Technicians/Associate Professionals). Sources: scripts/esco/prepare_esco_items.py:101-105
Scoped Data Model
The output of the preparation script is a Parquet file (esco_items_scoped.parquet) containing the following fields:
| Field | Type | Description |
|---|---|---|
esco_uri | String | Unique concept URI for the ESCO item. |
preferred_label | String | The primary English name of the skill. |
description | String | Detailed text description of the skill. |
n_occupations_essential | Integer | Count of knowledge-work occupations requiring this skill. |
isco_major_groups | String | Comma-separated list of ISCO groups (1, 2, 3) linked. |
anchor_occupations | String | Examples of occupations where the skill is essential. |
Sources: scripts/esco/prepare_esco_items.py:108-125, data/derived/esco/README.md:29-32
Mapping and Assignment
Once prepared, ESCO items are mapped to the 18-operation taxonomy using two independent methods to ensure robustness. Sources: scripts/esco/map_esco_to_ops.py:7-13
This diagram shows the dual-path assignment strategy used to validate ESCO items against the O*NET taxonomy. Sources: scripts/esco/map_esco_to_ops.py:15-30
Methods of Assignment
- Centroid Assignment: Calculates cosine similarity between the ESCO item embedding and the centroids of the 18 ONET operations. An item is flagged as
unmappable_geometricif its top-1 cosine is below the 5th percentile of within-op ONET member cosines. Sources: scripts/esco/map_esco_to_ops.py:17-23, scripts/esco/map_esco_to_ops.py:270-276 - LLM Assignment: A single-shot call to
gpt-5.4-miniusing frozen operation definitions and anone_of_the_aboveoption. Sources: scripts/esco/map_esco_to_ops.py:25-30, scripts/esco/map_esco_to_ops.py:280-313
Structural Recoverability Analysis
The module includes a clustering stage to test if the 18-operation taxonomy is structurally inherent in the ESCO data. This involves running HDBSCAN on ESCO embeddings without knowledge of the taxonomy labels and then comparing the resulting clusters to the assigned operation labels. Sources: scripts/esco/cluster_esco_items.py:7-15
Key metrics produced include:
- ARI/NMI: Adjusted Rand Index and Normalized Mutual Information against LLM labels. Sources: scripts/esco/cluster_esco_items.py:44-46, scripts/esco/cluster_esco_items.py:104-106
- kNN Consistency: The fraction of an item's nearest neighbors that share the same operation label. Sources: scripts/esco/cluster_esco_items.py:141-145
- Silhouette Score: Measures the separation of operation-labeled clusters in the embedding space. Sources: scripts/esco/cluster_esco_items.py:161-164
Coverage Assembly
The final stage of preparation assembles coverage tables that mirror the O*NET side analysis. This calculates the percentage of the ESCO corpus covered by each operation and the level of agreement between methods. Sources: scripts/esco/build_esco_coverage.py:7-13
# Logic for calculating per-op agreement stats
for op in ONET_OP_TASK_COUNTS.keys():
llm_mask = df["llm_op_id"] == op
sub = df[llm_mask]
n_items = int(llm_mask.sum())
agree = int(((df["centroid_op_id"] == op) & llm_mask).sum())
# ... aggregation of confidence and cosine stats
Sources: scripts/esco/build_esco_coverage.py:53-60
Conclusion
The ESCO Data Preparation module transforms a large-scale international skill taxonomy into a validated test bed. By scoping skills to the professional/technical ISCO groups and applying dual-method assignment (Geometric and LLM), the project establishes a robust baseline for the 18-operation taxonomy's external validity and cross-taxonomy recoverability. Sources: data/derived/esco/README.md:3-9, scripts/esco/build_esco_coverage.py:90-100