Operation Atlas
Relevant source files
The following files were used as context for generating this wiki page:
Operation Atlas is a high-dimensional visualization and structural framework used to map knowledge-work activities. It primarily consists of a UMAP (Uniform Manifold Approximation and Projection) projection of 8,372 O*NET task statements, categorized into an 18-operation taxonomy. The system serves as a diagnostic tool to distinguish between general "knowledge work" and credentialed "professional work," enabling an evaluation of AI agent capabilities against recurring activity patterns rather than isolated benchmark scores.
Sources: README.md:15-32, scripts/build_operation_atlas.py:1-15
System Architecture and Pipeline
The Operation Atlas is constructed through a multi-stage pipeline that transforms raw occupational data into a semantically meaningful 2D and 3D map. This process involves normalization of task statements, embedding into high-dimensional space, clustering, and final visualization.
Data Flow Overview
The following diagram illustrates the transformation of raw O*NET tasks into the final Atlas visualization:
The pipeline ensures that tasks are grouped by their underlying operation (e.g., "design," "troubleshooting") rather than their specific industry domain (e.g., "robotics," "solar").
Sources: scripts/normalize_task_operations.py:10-30, scripts/build_operation_atlas.py:155-180
Task Normalization
To prevent clustering by job domain, the system uses an LLM-based normalization step. Task statements are rewritten into "domain-stripped" phrases consisting of a verb and a generic object type. For example, "Debug robotics programs" is normalized to "debug software."
Sources: scripts/normalize_task_operations.py:18-25, scripts/normalize_task_operations.py:133-145
| Component | Description |
|---|---|
operation_phrase | A 3-10 word sentence-case phrase: <verb> <generic object>. |
operation_verbs | Array of 1-3 canonical verbs (e.g., ["design"]). |
operation_object_type | Enum mapping (e.g., code, hardware, document, data). |
domain_stripped | Array of specific terms removed (e.g., ["robotics"]). |
Sources: scripts/normalize_task_operations.py:53-75
Operation Taxonomy
The Atlas organizes tasks into 18 primary operations. These operations are designed to be cross-job, meaning they apply to at least five distinct O*NET occupations at Job Zone 3-5.
Taxonomy Structure
The taxonomy is defined in operation_taxonomy.json and categorized by unique IDs, definitions, and cross-job examples.
Primary Operations and Definitions
| Operation | One-Line Definition | Total Tasks |
|---|---|---|
analysis | Evidence → findings / interpretation. | 1,732 |
administration | Run internal HR / finance / procurement operations. | 1,517 |
design | Specify a new system / plan / service. | 992 |
inspection | Present-state compliance check against rules or specs. | 963 |
troubleshooting | Diagnose a malfunction and apply a remediation. | 902 |
record_keeping | Produce, store, and retrieve records. | 882 |
Sources: README.md:46-65, data/derived/manual_labeling/operation_taxonomy.json:15-130
Visualization and Mapping Logic
The visual "Atlas" is generated as a UMAP projection. The Largest operations often occupy multiple disjoint regions because their constituent tasks share an abstraction despite coming from different occupations.
Dimensionality Reduction Parameters
The 3D Atlas is generated using specific hyperparameters to ensure stability and reproducibility:
- Embedding Model:
text-embedding-3-small(1,536-d). - UMAP Configuration: 15-d cosine reduction,
n_neighbors = 30, seed 42. - Cluster Filtering: Uses Median Absolute Deviation (MAD) to remove outliers beyond 2 robust sigmas of the cluster centroid.
Sources: README.md:183-195, scripts/build_operation_atlas.py:228-245
Interactive Features
The build_operation_atlas.py script generates an interactive Plotly HTML file with the following capabilities:
- View Toggle: Three-way selector for "Included," "Excluded," or "All" tasks.
- Hover Cards: Displays occupation title, original task text, normalized phrase, and sub-op cluster ID.
- Legend Grouping: Toggling a legend item hides/shows all tasks associated with that operation across the 3D space.
Sources: scripts/build_operation_atlas.py:270-320, scripts/build_operation_atlas.py:380-410
Implementation Details
The core logic for assigning colors and generating example tables relies on deterministic selection from cluster centers.
Prototypical Example Selection
To generate representative tables for the paper, the system picks example tasks that are closest to the cluster centroid in the 2D UMAP plane.
def _pick_examples(sub: pd.DataFrame, n: int) -> list[tuple[str, str]]:
# Rank tasks by closeness to cluster centroid in 2D UMAP plane
# so we pick prototypical task statements.
rows: list[tuple[str, str]] = []
for cid, grp in sub.groupby("cluster_id", sort=False):
cx = grp["umap_x"].median(); cy = grp["umap_y"].median()
d = np.hypot(grp["umap_x"] - cx, grp["umap_y"] - cy)
for i in d.argsort().to_numpy():
# Selection logic for distinct occupations...
break
Sources: scripts/build_operation_examples_table.py:46-65
Summary
Operation Atlas provides a structured, empirical map of knowledge work. By combining LLM-based task normalization with high-dimensional clustering and UMAP visualization, it creates a framework where AI capabilities can be audited against 18 distinct, cross-job operations. This shifts evaluation from primitive scores to operational validity in real-world workflows.