Tables and Benchmarks Generation
Relevant source files
The following files were used as context for generating this wiki page:
This module manages the systematization of AI agent benchmarks and their alignment with the project's operation taxonomy. It involves populating a canonical inventory of benchmarks, calculating coverage across work operations, and generating human-readable artifacts such as comparison tables, coverage reports, and representative task examples for academic and technical documentation.
The system bridges raw occupational data (O*NET/ESCO) with existing AI evaluation suites to determine the "coverage tier" of current benchmarks. This process identifies gaps where real-world knowledge work operations are under-represented or completely missing in current evaluation frameworks.
Benchmark Inventory and Mapping
The benchmark inventory serves as a centralized registry of scored units and task templates used in AI agent evaluation. Each entry includes metadata regarding the source, task form, input form, and scoring rules, alongside manual labels that link benchmarks to specific work-pattern operations.
Inventory Data Structure
The inventory is populated primarily via scripts/build_benchmark_inventory.py, which processes a curated list of benchmarks such as LegalBench, FinanceBench, and WebArena.
| Field | Description |
|---|---|
benchmark_id | Unique identifier for the benchmark unit (e.g., bench_legalbench_issue_spotting). |
source | The name of the benchmark suite. |
source_ref | URL or ArXiv ID of the original paper. |
scored_unit | Specific unit or prompt type being measured. |
task_form | The nature of the task (e.g., binary classification, multi-step tool use). |
scoring_rule | Methodology for evaluation (e.g., exact-match, expert-labeled key). |
labels | Work-pattern labels mapped to the benchmark (e.g., evaluate_compliance). |
Sources: scripts/build_benchmark_inventory.py:22-146, README.md:148-180
Processing Flow
The inventory script performs several normalization and join operations to produce consistent datasets for downstream analysis.
The diagram shows the data flow from manual curation and existing labels to final CSV artifacts.
Sources: scripts/build_benchmark_inventory.py:161-218
Coverage Analysis and Tiers
The system calculates a "Coverage Tier" for each operation in the taxonomy based on the number of unique scored units in the benchmark inventory that target it.
Coverage Tiers Definition
Coverage is categorized into four distinct tiers based on the frequency of benchmark representation:
| Tier | Condition |
|---|---|
coverage_broad | ≥ 4 scored units |
coverage_moderate | 2–3 scored units |
coverage_weak | 1 scored unit |
coverage_missing | 0 scored units |
Sources: scripts/build_comparison.py:27-31, scripts/plot_coverage.py:30-35
Salience and Coverage Synthesis
The scripts/build_comparison.py script joins benchmark coverage with O*NET task salience. Salience is calculated by weighting tasks based on their importance mean, relevance percentage, and whether they are classified as "Core" tasks.
# scripts/build_comparison.py:46-51
coded["core_weight"] = coded["task_type"].map({"Core": 1.0}).fillna(0.5)
coded["salience_weight"] = (
coded["importance_mean"].fillna(3.0)
* (coded["relevance_pct"].fillna(50.0) / 100.0)
* coded["core_weight"]
)
Sources: scripts/build_comparison.py:46-51, scripts/plot_coverage.py:41-65
Operation Hierarchy and Examples
The system generates hierarchical structures and representative examples to define the 18 operations (originally 24 in some development phases) within the taxonomy.
Hierarchy Generation
The scripts/build_operation_hierarchy.py script uses a bottom-up approach to group cluster themes into top-level operations using agglomerative clustering on theme embeddings.
This sequence illustrates the transition from raw clusters to a structured, LLM-described hierarchy.
Sources: scripts/build_operation_hierarchy.py:183-294
Representative Example Selection
To make the taxonomy concrete, scripts/build_operation_examples_table.py selects "representative" task statements for each operation. It uses UMAP coordinates to find tasks closest to the cluster centroid, ensuring examples are prototypical rather than outliers.
# scripts/build_operation_examples_table.py:59-65
for cid, grp in sub.groupby("cluster_id", sort=False):
cx = grp["umap_x"].median(); cy = grp["umap_y"].median()
d = np.hypot(grp["umap_x"] - cx, grp["umap_y"] - cy)
for i in d.argsort().to_numpy():
r = grp.iloc[int(i)]
# ... pick closest task ...
Sources: scripts/build_operation_examples_table.py:53-83
Visual Artifacts
The project produces several visualization artifacts to support its findings on benchmark validity and coverage:
- Operation Examples Table: A Markdown table (
docs/figures/operation_examples_table.md) containing color-coded swatches, task counts, and deterministic representative examples. - Coverage vs. Salience Plot: A horizontal bar chart (
coverage_vs_salience.png) showing the salience share of O*NET tasks against their corresponding benchmark coverage tiers. - Operation Hierarchy Tree: A human-readable text representation (
operation_hierarchy__tree.txt) of the sub-operation to top-level operation mappings.
Sources: scripts/build_operation_examples_table.py:122-143, scripts/plot_coverage.py:38-65, scripts/build_operation_hierarchy.py:381-409
Summary
The Tables and Benchmarks Generation system provides a quantitative audit of the gap between actual knowledge work and current AI evaluation datasets. By mapping benchmarks to a validated operation taxonomy and calculating coverage tiers weighted by occupational salience, it establishes a rigorous framework for identifying where AI agent evaluation must evolve to reflect real-world professional requirements.