Text Embeddings Generation
Relevant source files
The following files were used as context for generating this wiki page:
The text embeddings generation system is a critical component of the work-validity-eval pipeline, responsible for transforming raw occupational task statements and skill descriptions into high-dimensional vector representations. These embeddings enable downstream analysis tasks such as clustering, operation-first taxonomy construction, and cross-taxonomy robustness studies (e.g., O*NET vs. ESCO).
The system utilizes OpenAI's text-embedding-3-small model to produce 1,536-dimensional, $L_2$-normalized vectors. By embedding different variants of task text—including raw task statements and domain-stripped operation phrases—the project can perform sensitivity analysis and evaluate work patterns independently of specific industry jargon.
Sources: scripts/embed_onet_tasks.py:1-25, README.md:143-160
Pipeline Architecture
The embedding generation process follows a structured workflow that involves data ingestion, optional domain normalization, batch embedding via API, and compressed storage.
The diagram shows the flow from raw data sources through optional normalization to final vector storage.
Sources: scripts/embed_onet_tasks.py:46-130, scripts/normalize_task_operations.py:15-40
Embedding Variants
The system supports multiple embedding "variants" to facilitate different analytical goals. These variants determine what text is ultimately sent to the embedding model.
| Variant | Input Logic | Purpose |
|---|---|---|
task_only | Uses task_text_clean directly. | Primary baseline for O*NET task clustering. |
title_plus_task | Concatenates "{occupation_title}: {task_text_clean}". | Legacy comparison for occupation-aware clustering. |
operation_only | Uses domain-stripped operation_phrase. | Enables clustering by shared operation across different domains. |
label_only | Uses ESCO preferredLabel. | Primary embedding for ESCO cross-taxonomy validation. |
Sources: scripts/embed_onet_tasks.py:17-25, scripts/embed_onet_tasks.py:84-105, data/derived/esco/README.md:15-30
Domain Normalization
To ensure that tasks are clustered by the work performed (the "operation") rather than the industry domain (e.g., "solar" or "robotics"), the system includes a normalization step. This step uses an LLM to strip domain-specific terminology while preserving the core cognitive or physical operation.
Normalization Logic
The normalization script transforms a specific task like "Design vacuum tube collector systems for solar applications" into a generic phrase like "design hardware system". This process is governed by a SYSTEM_PROMPT that enforces canonical object types and active verbs.
Canonical Object Types:
- code, hardware, material, document, plan, data
- image_or_media, person, organization, environment
- money, schedule, procedure, other
Sources: scripts/normalize_task_operations.py:15-40, scripts/normalize_task_operations.py:53-85
Implementation Details
Batch Processing and Resilience
Embeddings are generated in batches (default size 256) to optimize API throughput. The implementation includes exponential backoff retries to handle transient API failures.
# scripts/embed_onet_tasks.py:118-128
for start in tqdm(range(0, len(inputs), BATCH), desc="embed"):
chunk = inputs[start:start + BATCH]
for attempt in range(5):
try:
resp = client.embeddings.create(model=MODEL_ID, input=chunk)
break
except Exception as exc:
wait = 2 ** attempt
time.sleep(wait)
Sources: scripts/embed_onet_tasks.py:118-132
Storage and Metadata
Generated vectors are stored in compressed NumPy (.npz) format. Each .npz file is accompanied by a metadata Parquet file that maps row_index to the original task identifiers (onetsoc_code, task_id).
- Vector file:
onet_task_embeddings__<variant>.npz - Metadata file:
onet_task_embeddings__<variant>_meta.parquet
Sources: scripts/embed_onet_tasks.py:144-155
Summary of Configuration
The system relies on specific environmental and script-level parameters to ensure reproducible results.
| Parameter | Default Value | Source |
|---|---|---|
| Embedding Model | text-embedding-3-small | scripts/embed_onet_tasks.py |
| Dimensions | 1,536 | README.md |
| Normalization Model | gpt-5.4-mini | scripts/normalize_task_operations.py |
| UMAP Reduction (Post-embed) | 15-d cosine | data/derived/esco/README.md |
| ISCO Scope (ESCO) | Major Groups 1, 2, 3 | scripts/esco/prepare_esco_items.py |
Sources: scripts/embed_onet_tasks.py:34, README.md:144, data/derived/esco/README.md:21, scripts/esco/prepare_esco_items.py:15-20
The text embedding generation system provides the quantitative foundation for the project's validity accounts. By supporting domain-normalized variants and batch-processing across O*NET and ESCO datasets, it enables the empirical derivation of a knowledge-work operation taxonomy.