Knowledge-Work Screen
Relevant source files
The following files were used as context for generating this wiki page:
The Knowledge-Work Screen is a multi-stage filtering process designed to isolate "knowledge work" tasks from the broader O*NET and ESCO occupational databases. The system defines knowledge work as tasks that are primarily cognitive, produce judgment-bearing artifacts (reports, plans, designs), and involve non-routine judgment. This screening ensures that downstream AI agent evaluations focus on operations that an AI could plausibly execute or be evaluated on, such as analysis, advisory, or troubleshooting, while excluding manual, physical, or routine clerical tasks.
Sources: README.md:21-26, scripts/round2_knowledge_screen.py:73-80
Screening Architecture
The screening process is divided into two primary rounds for ONET data and a targeted scope filter for ESCO data. The ONET pipeline moves from a coarse occupation-level filter to a fine-grained, LLM-driven task-level analysis.
Multi-Stage Screening Flow
The following diagram illustrates the progression of data through the O*NET screening pipeline.
Round 1: Heuristic Occupation & Task Screen
The initial screen applies keyword-based heuristics and O*NET Job Zone filters. It targets occupations in Job Zones 4 and 5 (higher levels of preparation) and uses inclusion/exclusion term lists to categorize task statements and occupation titles.
Inclusion and Exclusion Criteria
The heuristic screen identifies "knowledge work" by looking for specific professional and cognitive terms while filtering out manual labor.
| Type | Inclusion Examples | Exclusion Examples |
|---|---|---|
| Occupations | analyst, engineer, manager, scientist, strategist | assembler, cook, laborer, operator, welder |
| Tasks | analyze, assess, budget, coordinate, design, plan | clean, drill, harvest, lift, repair, wash |
Sources: scripts/prepare_onet_tasks.py:18-120
Logic Components
- Job Zone Filter: Retains tasks from occupations requiring significant preparation (typically zones 3-5). Sources: scripts/prepare_onet_tasks.py:157-160
- Score Calculation: Occupation scores are calculated based on inclusion hits + Job Zone bonuses - exclusion hits. A score of $\ge 2$ leads to inclusion. Sources: scripts/prepare_onet_tasks.py:182-192
Round 2: LLM Task-Level Screen
Round 2 is a "stabilized" screen that performs a deep analysis of individual task statements using a Large Language Model (e.g., gpt-5.4-mini). This stage addresses the "meaningful tail" of non-knowledge work items—such as lab procedures or physical repairs—that exist within otherwise cognitive occupations.
LLM Decision Logic
The LLM evaluates if a task is primarily cognitive and produces a judgment-bearing artifact.
Exclusion Categories
The Round 2 screen categorizes excluded tasks into specific groups:
- physical_procedure: Direct manual, clinical, surgical, or lab procedures.
- installation_or_repair: Adjusting hardware or repairing equipment.
- routine_clerical: Ordering supplies or routine correspondence.
- performance_or_ritual: Narrations, announcements, or ritualistic actions.
Sources: scripts/round2_knowledge_screen.py:133-145
ESCO Scope Filtering
The ESCO screening process mirrors the O*NET Job Zone filter by utilizing the International Standard Classification of Occupations (ISCO) major groups.
- ISCO Scope: Only items associated with ISCO groups 1 (Managers), 2 (Professionals), and 3 (Technicians and Associate Professionals) are included.
- Skill Type Filter: Specifically includes items categorized as
skill/competence, while excludingknowledgerows (which describe subject-matter domains rather than operations).
Sources: scripts/esco/prepare_esco_items.py:11-20, README.md:144-150
Data Models and Outputs
The screening pipeline produces enriched datasets that track the reasoning behind every inclusion or exclusion decision.
Result Schema (O*NET Round 2)
| Field | Type | Description |
|---|---|---|
round2_knowledge_decision | Enum | "include", "exclude", or "uncertain". |
round2_knowledge_category | String | The specific category (e.g., clinical_procedure, cognitive_knowledge_work). |
round2_knowledge_reason | String | A short sentence citing the decision rule applied by the LLM. |
Sources: scripts/round2_knowledge_screen.py:34-40, scripts/round2_knowledge_screen.py:146-150
Final Knowledge Work Statistics
As of the latest freeze, the screening pipeline successfully isolated a refined corpus of 8,372 O*NET tasks from an original set of 12,464 tasks. This subset forms the basis for the "Operation Atlas" and the 18-operation taxonomy.
Sources: README.md:65-68, scripts/build_operation_atlas.py:133-140
Conclusion
The Knowledge-Work Screen is critical for ensuring the technical validity of AI agent evaluations. By moving from heuristic filters to LLM-driven task analysis, the system removes physical and routine clerical "noise," allowing the project to focus on high-level cognitive operations like design, analysis, and advisory work. This filtered dataset directly powers the creation of the operation atlas and the subsequent auditing of benchmark suites.
Sources: README.md:28-35, scripts/round2_knowledge_screen.py:73-80