ESCO Embed & Cluster
Relevant source files
The following files were used as context for generating this wiki page:
The ESCO Embed & Cluster system is a validation pipeline designed to test the external-taxonomy robustness of the project's 18-operation knowledge-work taxonomy. By using the European Skills, Competences, Qualifications and Occupations (ESCO) v1.1.1 catalog as an independent test bed, the system evaluates whether the taxonomy derived from O*NET task statements is recoverable and consistent across different occupational ontologies.
Sources: data/derived/esco/README.md:3-10, README.md:143-157
The pipeline encompasses data ingestion, scoping based on International Standard Classification of Occupations (ISCO) groups, high-dimensional embedding via OpenAI models, density-based clustering, and mapping to the frozen taxonomy using both geometric and LLM-based methods.
Sources: scripts/esco/prepare_esco_items.py:8-18, scripts/esco/map_esco_to_ops.py:12-30
Data Preparation and Scoping
The process begins by filtering the ESCO skills catalog to match the knowledge-work scope used on the O*NET side (Job Zones 3-5). Items are retained if they are classified as a skill/competence and are linked as essential to at least one occupation within ISCO-08 major groups 1 (Managers), 2 (Professionals), or 3 (Technicians and Associate Professionals).
Sources: scripts/esco/prepare_esco_items.py:22-26, data/derived/esco/README.md:15-18
Scoping Parameters
| Parameter | Value | Description |
|---|---|---|
| ESCO Version | v1.1.1 | The specific version of the ESCO English CSV package. |
| ISCO Groups | 1, 2, 3 | Managers, Professionals, and Technicians/Associate Professionals. |
| Relation Type | essential | Only skills vital to the occupation are included. |
| Item Count | 5,826 | The resulting count of scoped skill/competence items. |
Sources: data/derived/esco/README.md:13-21, scripts/esco/prepare_esco_items.py:108-112
Embedding and Dimensionality Reduction
The system generates $L_2$-normalized embeddings for each scoped item using the text-embedding-3-small model. Two variants are typically produced: a primary variant using only the preferredLabel and a sensitivity variant combining the label and the full description.
Sources: scripts/esco/embed_esco_items.py, data/derived/esco/README.md:29-33
The following diagram illustrates the data flow from raw ESCO files to processed embeddings:
The embedding process ensures that items are represented in a 1,536-dimensional space suitable for downstream clustering and similarity analysis.
Sources: scripts/esco/cluster_esco_items.py:34-36, data/derived/esco/README.md:20
Density-Based Clustering
The pipeline performs an independent partition of ESCO items using HDBSCAN to test structural recoverability without prior knowledge of the target taxonomy. This process involves reducing the 1,536-d embeddings to a 15-d manifold using UMAP (Uniform Manifold Approximation and Projection).
Sources: scripts/esco/cluster_esco_items.py:10-25
Clustering Configuration
| Component | Setting | Description |
|---|---|---|
| UMAP Reduction | 15-d | Reduced space for density clustering using cosine metric. |
| HDBSCAN min_cluster_size | 40 | Minimum points to form a cluster. |
| HDBSCAN min_samples | 10 | Parameter for cluster noise resilience. |
| UMAP neighbors | 30 | Local neighborhood size for manifold learning. |
Sources: data/derived/esco/README.md:22, scripts/esco/cluster_esco_items.py:44-50
Noise points identified by HDBSCAN are soft-assigned to their nearest non-noise cluster centroid in the 15-d space to ensure full corpus coverage for mapping.
Sources: scripts/esco/cluster_esco_items.py:77-88
Taxonomy Mapping and Consensus
Items are mapped to the frozen 18-operation taxonomy using two independent methods to ensure robustness:
- Centroid Assignment: Calculating the cosine similarity between the ESCO item embedding and the centroids of O*NET tasks within each operation.
- LLM Assignment: A single-shot call to
gpt-5.4-miniproviding operation definitions, inputs, and outputs to classify the item.
Recoverability Metrics
The system calculates several metrics to evaluate taxonomy recovery, including Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) against LLM labels, as well as k-Nearest Neighbor (kNN) consistency.
Sources: scripts/esco/cluster_esco_items.py:120-135, data/derived/esco/README.md:39-41
Summary of Outputs
The pipeline produces several artifacts in data/derived/esco/ for analysis and reporting in the final paper.
| File | Purpose |
|---|---|
esco_op_assignments.parquet | Contains centroid, LLM, and consensus assignments. |
esco_hdbscan_clusters.parquet | HDBSCAN labels and UMAP 2-d coordinates for visualization. |
esco_structural_recoverability.json | ARI, NMI, and kNN consistency statistics. |
esco_coverage_by_op.csv | Per-operation counts and agreement percentages. |
Sources: data/derived/esco/README.md:26-44
The "ESCO Embed & Cluster" system provides the empirical foundation for claiming that the project's operation taxonomy is a robust, cross-taxonomy construct capable of organizing work patterns from multiple independent sources.
Sources: data/derived/esco/README.md:3-8