Frontier Suites Audit
Relevant source files
The following files were used as context for generating this wiki page:
The Frontier Suites Audit is a critical evaluative component of the project designed to assess the capability of leading AI agent evaluation frameworks (frontier suites) against a standardized taxonomy of knowledge-work operations. The audit evaluates these suites based on two primary dimensions: Operation Coverage (whether the benchmark exercises the specific cognitive patterns found in professional work) and Output Validity (whether the benchmark results are predictive of real-world readiness).
Sources: README.md:1-20, README.md:95-103
This audit addresses the "validity gap" where AI agents score high on primitive tasks (retrieval, classification) but fail to demonstrate readiness for complex workflows involving professional constructs such as accountability, sign-off authority, and liability.
Sources: README.md:22-35
Audit Methodology and Taxonomy
The audit relies on an empirically derived 18-operation taxonomy of knowledge-work, constructed from O*NET 30.2 task statements and validated against the ESCO v1.1.1 catalog. The auditing process maps benchmark "scored units" to these operations to determine the breadth and depth of coverage.
Sources: README.md:31-35, scripts/build_comparison.py:10-25
The 18 Knowledge-Work Operations
| Operation | Definition | Scored Units (Typical) |
|---|---|---|
analysis | Evidence → findings / interpretation. | 1,732 |
administration | Run HR / finance / procurement / operations. | 1,517 |
design | Specify a new system / plan / service. | 992 |
inspection | Present-state compliance check against rules or specs. | 963 |
troubleshooting | Diagnose a malfunction and apply a remediation. | 902 |
record_keeping | Produce, store, retrieve, and curate records. | 882 |
coordination | Align internal parties on a shared task. | 603 |
appraisal | Score / grade / value a submission on a rubric. | 453 |
Sources: README.md:46-70, data/derived/manual_labeling/operation_taxonomy.json:20-150
Audit Data Flow
The following diagram illustrates how raw task data and benchmark units are processed to generate the audit verdict.
The audit pipeline transforms raw occupational data into a taxonomy which is then used as a reference to evaluate the "Scored Units" of various frontier suites.
Sources: scripts/normalize_task_operations.py:10-30, scripts/build_comparison.py:40-60, README.md:118-135
Audited Frontier Suites
The audit specifically targets three major frontier evaluation suites: GDPval, $\tau^2$-Bench, and APEX-Agents. Each suite was analyzed for its ability to cover the taxonomy and its adherence to four validity criteria (Grounding, Boundedness, Hand-off, and Organizational Validity).
Sources: README.md:104-115
Audit Verdict Summary
| Suite | Coverage Verdict | Identified Validity Gap |
|---|---|---|
| GDPval | Partial Cover | Grounding only partly tested; Handoff fails to test admissibility or liability. |
| $\tau^2$-Bench | Partial Cover | Limited to single domain (customer service); Handoff terminates in simulator. |
| APEX-Agents | Partial Cover | Covers three professions (law, consulting, banking); lacks institutional layer (sign-off/filing). |
Sources: README.md:105-115
Scored Unit Analysis
The audit breaks down benchmarks into "Scored Units"—the smallest evaluable task templates. For example, the audit of LegalBench identifies units such as "Issue-spotting" and "Rule-application," mapping them to the inspection or analysis operations.
Sources: scripts/build_benchmark_inventory.py:25-50
Coverage Tiers
Benchmarks are categorized into tiers based on the number of scored units mapped to an operation:
- Coverage Broad: $\ge$ 4 scored units.
- Coverage Moderate: 2-3 scored units.
- Coverage Weak: 1 scored unit.
- Coverage Missing: 0 scored units.
Sources: scripts/build_comparison.py:38-45
The sequence shows the process of auditing a suite by mapping its units to the taxonomy and testing them against the "minimum task contract" required for validity.
Sources: scripts/build_benchmark_inventory.py:15-20, data/derived/work_pattern_labels_frozen.json:15-30
Validity Criteria
A central finding of the audit is that high benchmark scores do not equate to operational validity. The audit identifies a significant "Validity Gap" in current suites across two layers:
1. Task Validity Layer
- Grounding: Is the output based on provided evidence?
- Boundedness: Is the agent constrained within the authorized task scope?
- Hand-off: Is the output in a format/state ready for professional action (e.g., admissible in court)?
Sources: README.md:26-30, data/derived/work_pattern_labels_frozen.json:10-15
2. Organizational Validity Layer
- Authorized Use: Is the agent's action permitted by organizational policy?
- Locatable Accountability: Can a human be held responsible for the agent's output?
- Integrability: Does the output flow into downstream workflows (e.g., sign-off, filing, liability)?
Sources: README.md:28-32
Conclusion
The Frontier Suites Audit concludes that while modern evaluation frameworks provide "partial covers" of knowledge-work operations, they currently achieve zero strict covers. Most operations—such as administration, coordination, and emergency_response—remain largely unaddressed by the audited suites. The audit serves as a diagnostic tool, highlighting that AI capability claims stated in professional-work language are often supported by benchmarks that lack the necessary credentialing, sign-off, or liability frameworks required for actual professional deployment.
Sources: README.md:110-117, scripts/build_comparison.py:120-135