LLM Integration and Labeling
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The "LLM Integration and Labeling" system serves as a critical automated and semi-automated pipeline for classifying, normalizing, and auditing knowledge-work operations within the project. It utilizes Large Language Models (LLMs)—specifically OpenAI's gpt-5.4-mini—to transform raw occupational data from O*NET and ESCO into a structured taxonomy of 18 knowledge-work operations. This process ensures that complex task statements are stripped of domain-specific noise and categorized based on cognitive primitives rather than industry-specific terminology.
The system is designed to bridge the gap between high-level benchmark scores and actual "knowledge work" capability. By integrating LLMs into the data processing pipeline, the project can handle thousands of task statements to determine Task Validity and organizational fit. This integration covers multiple stages, including task-level screening, domain-stripping normalization, and mapping external skill catalogs (ESCO) to the project's internal operational taxonomy.
Sources: README.md:16-35, scripts/normalize_task_operations.py:11-35
Core LLM Operations and Pipeline
Task Normalization and Domain Stripping
The normalization process uses LLMs to rewrite O*NET task statements into domain-stripped operation phrases. The goal is to preserve the cognitive or physical operation (the "doing") while removing industry-specific nouns like "solar," "robotic," or "patient." This ensures that tasks like "Debug robotics programs" and "Fix bugs in web code" are both recognized as the same underlying operation: "debug software."
The diagram above shows the flow of a task through the normalization script.
Key Data Structure:
The LLM returns a JSON object with the following schema for every task:
| Field | Type | Description |
|---|---|---|
operation_phrase | string | 3-10 word sentence-case phrase (e.g., "design hardware system"). |
operation_verbs | array | 1-3 canonical infinitive verbs (e.g., ["design", "develop"]). |
operation_object_type | enum | Dominant object kind (code, hardware, document, person, etc.). |
domain_stripped | array | Specific terms removed (e.g., ["nanoscale", "solar"]). |
Sources: scripts/normalize_task_operations.py:11-65
Knowledge Work Screening (Round-2)
A secondary task-level screen is performed to filter out non-cognitive tasks that might exist within professional occupations (e.g., "washing surgical equipment" within a medical job). The LLM triages tasks into "include," "exclude," or "uncertain."
The logic used to refine the knowledge-work corpus after initial occupation-level gating.
Taxonomy Generation and Mapping
Work Pattern Proposal
The LLM is utilized to draft the initial work-pattern labels by analyzing stratified samples of clustered O*NET tasks. It is instructed to use "neutral, job-task-side language" rather than benchmark terminology.
Proposal Strategy:
- Stratified Sampling: Tasks are sampled by HDBSCAN cluster and job zone.
- LLM Draft: LLM generates ~25-40 candidate labels with definitions, inclusion examples, and exclusion examples.
- Consolidation: A second "Merge Pass" consolidates duplicates and enforces operation-first naming (e.g.,
compare_against_standard).
Sources: scripts/propose_work_patterns.py:11-40, scripts/propose_work_patterns.py:65-80
ESCO to Operation Mapping
The system maps ESCO skill/competence items to the frozen 18-operation taxonomy using a dual-method approach: Centroid Assignment (geometric similarity) and LLM Assignment (semantic classification).
| Component | Method | Logic |
|---|---|---|
| Centroid Assignment | Vector Math | Cosine similarity to the mean of L2-normalized O*NET task embeddings. |
| LLM Assignment | Semantic | Single-shot GPT call using frozen op definitions and none_of_the_above option. |
| Consensus | Logic | If methods agree, label is final. If they disagree, the item is flagged for review. |
The consensus mapping process for ESCO items.
Sources: scripts/esco/map_esco_to_ops.py:11-45, scripts/esco/map_esco_to_ops.py:315-350
Implementation Details
Configuration and Environment
LLM calls are standardized across scripts using the OpenAI API.
- Model:
gpt-5.4-mini(Default) - Temperature: Typically
0.0for classification/normalization and0.2for taxonomy proposal. - Caching: Append-only JSONL files (e.g.,
esco_llm_cache.jsonl,round2_knowledge_screen_cache.jsonl) are used to make the pipeline resumable and avoid redundant API costs.
Sources: scripts/normalize_task_operations.py:145-160, scripts/round2_knowledge_screen.py:130-150, README.md:144-155
Benchmark Inventory Labeling
The script build_benchmark_inventory.py maps scored units from AI benchmarks (like LegalBench or FinanceBench) to work-pattern labels. This creates a link between existing AI evaluation suites and the empirically derived 18 operations.
Examples of Mapped Benchmarks:
- LegalBench: Mapped to
compare_records_against_standards. - FinanceBench: Mapped to
extract_or_verify_information_from_documentsandanalyze_data_for_decisions. - GAIA: Mapped to
gather_field_evidenceandanalyze_data_for_decisions.
Sources: scripts/build_benchmark_inventory.py:25-95
Conclusion
LLM Integration and Labeling is the structural backbone for the project's validity analysis. By moving from raw task text to normalized, cognitive operations, the system enables a rigorous audit of whether AI agents are truly performing "knowledge work" or merely solving "primitive" classification and retrieval tasks. The combination of semantic LLM classification and geometric embedding similarity provides a robust, cross-validated framework for occupational taxonomy mapping.