Testing & Benchmarks Suite

Relevant source files

The following files were used as context for generating this wiki page:

• plugins/token-optimizer/skills/token-optimizer/SKILL.md
• plugins/token-optimizer/skills/token-optimizer/references/agent-prompts.md
• plugins/token-optimizer/skills/token-optimizer/references/implementation-playbook.md
• skills/token-coach/SKILL.md
• skills/token-coach/references/coaching-scripts.md

The Testing & Benchmarks Suite in Token Optimizer is a multi-layered verification and measurement framework designed to quantify token overhead and validate optimization efficacy. It operates across several phases, utilizing automated audit agents to establish a baseline, implementing changes through an orchestrator, and performing final verification to calculate realized savings.

The suite ensures that optimizations—ranging from configuration cleanups to behavioral habit changes—actually recover context window space (target: 5-25%+) without degrading system quality. It relies heavily on measure.py for data collection, trend analysis, and quality scoring.

Measurement and Data Collection Architecture

The core of the testing suite is the measure.py script, which serves as the authoritative engine for gathering runtime data, calculating token costs, and tracking historical performance trends.

Key Metrics and Data Sources

The suite collects data from various components to build a comprehensive token profile:

• Startup Overhead: Measures tokens consumed by CLAUDE.md, MEMORY.md, skills menus, and MCP server definitions.
• Historical Trends: Extracted from trends.db, including quality scores, session durations, and cache hit rates.
• Quality Scoring: A v2.0 feature that identifies "rot" in the context window, such as stale file reads or bloated tool results.

The diagram shows how measure.py aggregates current snapshots, historical trends, and quality metrics to inform the optimization process.
Sources: skills/token-coach/SKILL.md:68-96, plugins/token-optimizer/skills/token-optimizer/SKILL.md:275-279

Verification Workflow

Verification is performed by a dedicated agent (typically using the Haiku model) that re-measures the environment after optimizations are applied.

Component	Measurement Method	Target Outcome
CLAUDE.md	Line count & Token estimation	Reduction in always-loaded tokens
Skills	Frontmatter & Count audit	Archival of unused skills
MCP Menu	ToolSearch status & Definition size	Deferral of tool definitions
Savings	Delta calculation	Percentage reduction in overhead

Sources: plugins/token-optimizer/skills/token-optimizer/references/agent-prompts.md:460-505, plugins/token-optimizer/skills/token-optimizer/SKILL.md:12-13

Audit and Analysis Suite

The suite dispatches parallel agents to audit specific modules. Each agent follows a strict template to ensure technical accuracy and prevent prompt injection during analysis.

Parallel Audit Agents

• CLAUDE.md Auditor: Analyzes global and project-specific configuration for duplication and verbose sections.
• Skills Auditor: Inventories skills, identifying frontmatter overhead and truncated descriptions (limit: 1,536 characters).
• MCP Auditor: Checks if Tool Search is active (which defers full definitions) and identifies forked/duplicate tool scopes.
• Advanced Auditor: Evaluates hooks, prompt caching structure, and model routing effectiveness.

The sequence illustrates the transition from parallel data gathering to centralized synthesis.
Sources: plugins/token-optimizer/skills/token-optimizer/references/agent-prompts.md:15-300

Behavioral Benchmarking

The suite also benchmarks behavioral habits, which are cited as potentially saving more than configuration changes. This includes:

• Plan Mode Efficiency: Measuring iteration cycle reduction (target: 50-70%).
• Model Routing ROI: Benchmarking the cost differential between Haiku (60x cheaper) and Opus for data-gathering tasks.
• Compaction Timing: Tracking the impact of manual /compact vs auto-compaction.

Sources: plugins/token-optimizer/skills/token-optimizer/references/agent-prompts.md:428-445, skills/token-coach/references/coaching-scripts.md:115-130

Implementation and Verification Safeguards

To prevent regressions during testing and implementation, the suite enforces strict operational rules and dependency checks.

Safety Procedures

1. Mandatory Backups: Every optimization phase (4A-4P) requires a backup to ~/.claude/_backups/ before mutation.
2. Dependency Checks: Before archiving a skill or disabling an MCP server, the suite searches for hardcoded references in CLAUDE.md and other skill files.
3. Token-Cost Warning: The suite monitors for broad permissions.deny rules that may cause "permission denied" feedback loops, which paradoxically consume more tokens.

The flowchart details the safety-first approach to applying optimizations and verifying them.
Sources: plugins/token-optimizer/skills/token-optimizer/references/implementation-playbook.md:10-90, plugins/token-optimizer/skills/token-optimizer/references/implementation-playbook.md:280-305

Version-Specific Benchmarks

Testing logic adapts based on the detected Claude Code version:

• v2.1.86+: Native deduplication of unchanged re-reads is benchmarked against the read_cache.py hook.
• v2.1.85+: The suite tests for the presence of PostCompact hooks for analytics and conditional if fields in hooks.
• v2.1.82: Checks for "phantom skills" loading from gitignored directories.

Sources: plugins/token-optimizer/skills/token-optimizer/references/implementation-playbook.md:235-265, plugins/token-optimizer/skills/token-optimizer/references/agent-prompts.md:120-135

Summary

The Testing & Benchmarks Suite provides the necessary data infrastructure to transform token optimization from guesswork into a precise technical process. By combining parallel auditing, historical trend analysis via measure.py, and strict verification agents, the project ensures that any context recovery is measurable and sustainable across long development sessions.