O*NET Data Ingestion
Relevant source files
The following files were used as context for generating this wiki page:
ONET Data Ingestion is the foundational stage of the project's data pipeline, responsible for retrieving and processing the ONET 30.2 database. This system extracts occupational task statements, ratings, and job zone information to create a curated corpus of "knowledge work" tasks. These tasks serve as the raw material for the empirically derived 18-operation taxonomy used to evaluate AI agent capabilities.
The ingestion process involves automated downloading, schema normalization, and a multi-stage screening process to filter out manual or routine labor, ensuring the dataset remains focused on cognitive, professional, and technical work patterns.
Sources: README.md:16-24, scripts/prepare_onet_tasks.py:1-12
Data Retrieval and Preparation
The ingestion begins with the execution of a shell script that fetches the core O*NET 30.2 tab-delimited text files. These files contain the structural data required for task analysis, including statements, ratings (importance and relevance), and job zone classifications.
Core Data Sources
The system specifically targets the following O*NET files:
| File Name | Description | Source Mapping |
|---|---|---|
Task Statements.txt | Raw task descriptions and unique IDs | Task ID, Task |
Task Ratings.txt | Importance (IM) and Relevance (RT) metrics | Scale ID, Data Value |
Occupation Data.txt | Titles and descriptions for SOC codes | Title, Description |
Job Zones.txt | Experience and education level requirements | Job Zone |
Sources: scripts/download_onet_core.sh:16-23, scripts/prepare_onet_tasks.py:173-205
Processing Flow
The prepare_onet_tasks.py script joins these disparate files into a unified dataset. It cleans task text by removing extra whitespace and normalizes ratings. Frequency distributions across categories (1–7) are pivoted into distinct columns for detailed analysis.
Multi-Stage Knowledge Screening
A critical component of ingestion is the "knowledge work" screen. This ensures that the downstream analysis focuses on tasks that are primarily cognitive and judgment-bearing.
Round 1: Heuristic Screening
The initial screen uses keyword-based heuristics to classify both occupations and tasks. It prioritizes O*NET Job Zones 4 and 5 (higher education/expertise) and applies inclusive/exclusive term lists to identify professional vs. manual labor.
- Occupation-Level Hits: Terms like
analys,architect,engineer, andspecialistincrease inclusion scores. - Task-Level Exclusions: Terms like
assemble,clean,drill, andweldtrigger exclusions.
Sources: scripts/prepare_onet_tasks.py:24-110
Round 2: LLM-Based Triage
Because heuristic screens can miss task-level manual labor within knowledge-work occupations, a second pass is performed using gpt-5.4-mini. This pass triages statements into include, exclude, or uncertain based on whether the task produces a "judgment-bearing artifact."
Task Operation Normalization
Once screened, tasks are normalized to strip domain-specific terminology. This process isolates the Operation (verb + generic object) from the specific industry or profession. For example, "Debug robotics programs" is normalized to "debug software" to allow clustering with other software debugging tasks from different industries.
Canonical Object Types
The normalization process maps task objects to a fixed enum:
code,hardware,material,document,plan,dataimage_or_media,person,organization,environment,moneyschedule,procedure,other
Sources: scripts/normalize_task_operations.py:44-59
Normalization Logic
The system calls an LLM to rewrite task statements into 3-10 word "operation phrases." This ensures that the embedding and clustering stages group tasks by shared professional construct rather than shared domain nouns.
# Example of normalization logic from scripts/normalize_task_operations.py:112-117
source: "Design or develop vacuum tube collector systems for solar applications."
-> {"operation_phrase": "design hardware system",
"operation_verbs": ["design", "develop"],
"operation_object_type": "hardware",
"domain_stripped": ["vacuum tube", "collector", "solar"]}
Sources: scripts/normalize_task_operations.py:11-38, scripts/normalize_task_operations.py:112-117
Conclusion
The O*NET Data Ingestion system transforms raw, messy occupational data into a high-signal corpus of knowledge-work operations. By combining traditional data engineering (joining and pivoting TSVs) with modern LLM-based screening and normalization, the pipeline creates a clean, domain-agnostic foundation for the project's 18-operation taxonomy. This allows for rigorous evaluation of AI agents across diverse professional constructs without being biased by specific industry terminology.