The 18-Operation Taxonomy
Relevant source files
The following files were used as context for generating this wiki page:
The 18-Operation Taxonomy is an empirically derived classification system for knowledge-work operations, built from O*NET 30.2 task statements and validated against the ESCO v1.1.1 skill catalog. It serves as a foundational layer for rethinking AI agent evaluation by moving away from "primitive" scores (like retrieval or classification) toward "operations"—the smallest activity patterns that recur across occupations and carry recognizable professional constructs.
This taxonomy distinguishes between knowledge work (defined by non-routine judgment and knowledge as raw material) and professional work (gated by formal credentialing and liability). The system identifies 18 core operations that determine whether an AI agent's output can be acted upon within a real-world professional workflow.
Sources: README.md:21-42, README.md:65-71
Construction Pipeline
The taxonomy is generated through a multi-stage pipeline that transforms raw occupational data into a structured hierarchy. This process involves rigorous screening for "knowledge-work" relevance and domain-stripping to ensure operations are cross-job rather than industry-specific.
The diagram shows the data flow from raw O
NET tasks to the final consolidated taxonomy.*
Sources: README.md:155-168, scripts/finalize_work_patterns.py:30-40, scripts/round2_knowledge_screen.py:12-32
1. Knowledge-Work Screening
A "Round-2" task-level screen filters previously included O*NET occupations to remove direct manual, clinical, or routine clerical tasks. The screen gates tasks based on whether they are primarily cognitive and result in a judgment-bearing artifact.
- Include: Cognitive, judgment-bearing tasks (e.g., "Record health histories").
- Exclude: Physical procedures, manual installation, or performative rituals (e.g., "Take fingerprints", "Stain slides").
Sources: scripts/round2_knowledge_screen.py:12-25, scripts/round2_knowledge_screen.py:65-95
2. Domain-Stripping and Normalization
To group shared operations across different domains, task statements are rewritten into domain-stripped phrases. This process preserves the worker's action (verb + generic object) while removing industry-specific terminology.
- Original: "Design vacuum tube collector systems for solar applications."
- Stripped: "Design hardware system."
Sources: scripts/normalize_task_operations.py:15-28, scripts/normalize_task_operations.py:120-135
Taxonomy Structure and Operation Schema
The taxonomy consists of 18 top-level operations (reduced from 19 after merging assistance into protocol_execution). Each operation is a distinct technical construct with a specific schema.
| Operation | One-line Definition | Primary Input | Primary Output |
|---|---|---|---|
analysis | Evidence → findings / interpretation | Raw evidence / data | Report, memo, or brief |
administration | Run HR / finance / procurement | Organizational functions | Functional status quo |
design | Specify a new system / plan | Requirements / goals | Specifications / Blueprints |
inspection | Compliance check against rules | Subject + Standards | Pass/Fail + Violations |
troubleshooting | Diagnose and remediate malfunctions | Malfunctioning subject | Remediated subject |
record_keeping | Store and retrieve records | Source documents | Maintained record set |
teaching | Transfer skill to learners | Learners + Curriculum | Learner capability |
protocol_exec | Follow procedure exactly | Predefined procedure | Completed procedure |
representation | Engage externals on behalf of org | Audience + Message | Transaction / Commitment |
coordination | Align internal parties | Shared deliverable | Aligned next actions |
creative_prod | Original creative media output | Creative brief | Media work |
fabrication | Make or install tangible assets | Materials + Design | Working physical system |
appraisal | Score/value via a rubric | Submitted item + Rubric | Score / Grade / Value |
advising | 1:1 personalized recommendation | Client situation | Personalized advice |
investigation | Reconstruct a past event | Incident / Allegation | Factual narrative |
emergency_resp | Stabilize an acute crisis | Acute incident | Stabilized safety |
self_study | Maintain professional competency | Literature / CE courses | Current practitioner |
rule_enforce | Uphold behavioral rules | Setting + Rules | Compliant behavior |
Sources: README.md:44-63, data/derived/manual_labeling/operation_taxonomy.json:10-250, scripts/build_operation_atlas.py:44-50
Operation Metadata Schema
Every operation in the taxonomy is defined by several critical metadata fields that establish its validity for evaluation.
The class diagram represents the data structure used for each operation in the work_pattern_labels_frozen.json and operation_taxonomy.json files.
Sources: data/derived/manual_labeling/operation_taxonomy.json:11-30, data/derived/work_pattern_labels_frozen.json:5-25
Technical Implementation and Logic
The taxonomy is implemented using a set of scripts that manage embeddings, clustering, and hierarchical merging.
Agglomerative Merging
The taxonomy is not pre-specified by count but emerges from the data. The system applies a cosine-distance threshold to an agglomerative merge of cluster-theme embeddings.
# Agglomerative clustering logic from scripts/build_operation_hierarchy.py
labels_arr = AgglomerativeClustering(
metric="cosine",
linkage="average",
distance_threshold=args.threshold,
n_clusters=None,
).fit_predict(vecs)
Sources: scripts/build_operation_hierarchy.py:15-25, scripts/build_operation_hierarchy.py:345-353
The Minimum Task Contract
A "Minimum Task Contract" is a critical validity constraint for each operation. It defines exactly what a benchmark item must expose to legitimately test that specific operation. For example, for inspection, the item must provide both a subject to examine and an explicit standard or protocol to determine compliance.
Sources: data/derived/work_pattern_labels_frozen.json:7-10, scripts/build_operation_hierarchy.py:112-115
Agent Doability Assessment
Each operation is tagged with a "doability" rating (high, medium, low) indicating whether an AI agent with standard tools (file I/O, browser, APIs) could execute it end-to-end.
- High: Text-based management (e.g.,
coordination). - Low: Requires professional licensure or physical risk (e.g.,
prescribe_treatments).
Sources: scripts/build_operation_hierarchy.py:102-109, data/derived/work_pattern_labels_frozen.json:133-140
Summary
The 18-Operation Taxonomy provides a structured bridge between raw occupational tasks and AI capability evaluation. By decomposing professional work into 18 cross-job operations, the system enables researchers to audit benchmark coverage and identify validity gaps where current AI evaluations fail to exercise the institutional, credentialing, or liability-based layers of real-world knowledge work.
Sources: README.md:25-38, README.md:143-153