Benchmarks & Savings Tracking

Relevant source files

The following files were used as context for generating this wiki page:

• skills/token-optimizer/SKILL.md
• skills/token-optimizer/references/agent-prompts.md
• skills/token-optimizer/references/optimization-checklist.md
• skills/token-coach/SKILL.md
• skills/token-optimizer/references/implementation-playbook.md

Benchmarks & Savings Tracking is a critical component of the Token Optimizer system designed to quantify context window waste, measure the impact of applied optimizations, and track historical usage trends. The system targets a 5-15% context recovery through configuration cleanup and up to 25%+ with advanced management like autocompact control.

Sources: skills/token-optimizer/SKILL.md:12-13

The framework utilizes a core script, measure.py, to perform audits, generate dashboards, and collect data across different runtimes including Claude Code, Codex, and OpenCode. By baseline-ing token usage and monitoring quality metrics, the system provides developers with actionable data to maintain high-performance LLM interactions.

Sources: skills/token-optimizer/SKILL.md:42-80, skills/token-coach/SKILL.md:37-64

Core Measurement Architecture

The measurement system operates through a multi-phase workflow that transitions from initial environment detection to final verification of savings.

Lifecycle of a Measurement Session

The workflow ensures that every optimization is preceded by a baseline and followed by a verification step to confirm actual savings.
Sources: skills/token-optimizer/SKILL.md:105-188

Key Data Collection Components

• measure.py report: The authoritative check for environment detection and baseline reporting. Sources: skills/token-optimizer/SKILL.md:82-84
• measure.py collect: A hook-based collector (typically SessionEnd) that stores usage data into a local SQLite trends.db. Sources: skills/token-optimizer/references/implementation-playbook.md:92-98
• measure.py quality: Analyzes session JSONL to score content quality (0-100) based on stale reads, bloat, and information loss. Sources: skills/token-coach/SKILL.md:76-80, skills/token-optimizer/references/optimization-checklist.md:301-314

Savings Metrics & Benchmarks

The system categorizes savings into "Config Savings" (permanent overhead reduction) and "Behavioral Savings" (long-term cost reduction).

Optimization Target Benchmarks

Category	Component	Potential Savings	Derived From
Config	CLAUDE.md	1,000 - 3,000+ tokens	tiered architecture, content migration
Config	MEMORY.md	400 - 800 tokens	deduplication with CLAUDE.md
Config	Skills Menu	~100 tokens / skill	archiving unused skills
Config	MCP Tools	~15 tokens / tool	removing unused deferred tools
Behavioral	Model Routing	50 - 75% cost reduction	defaulting subagents to Haiku
Behavioral	/compact hygiene	up to 18x reduction	clearing conversation history at ~70%
Behavioral	Batching	2x - 3x reduction	reducing context re-sends

Sources: skills/token-optimizer/references/optimization-checklist.md:37-185, skills/token-optimizer/references/optimization-checklist.md:376-385

Quality Scoring

The system uses a quality score to identify session rot even when context window space is available:

• 85-100 (Excellent): Clean session, minimal rot.
• 70-84 (Good): Some bloat; manual /compact recommended.
• 50-69 (Degraded): Danger zone; stale reads and bloat impacting performance.
• <50 (Critical): Heavy rot; immediate /clear or checkpointed restart required.

Sources: skills/token-optimizer/references/optimization-checklist.md:301-314

Historical Tracking and Trends

The measure.py coach --json command extracts historical data from trends.db to provide trend-based insights.

Trend Data Points

• Quality Averages: Comparison of recent (7-day) vs. older quality scores.
• Cache Hit Rates: Monitoring the hit rate to detect invalidation causes (e.g., frequent model switching).
• Cost Analysis: Spend summary including total USD, cost per session, and sessions in period.
• Optimal Session Hint: Correlation between session duration and quality degradation.

Sources: skills/token-coach/SKILL.md:37-56

This diagram shows how historical data is retrieved to inform the synthesis of optimization plans.
Sources: skills/token-optimizer/references/agent-prompts.md:158-175

Verification Process

Phase 5 (Verification) utilizes a haiku model agent to re-measure all components after implementation.

Results Comparison

The verification agent compares "Before" and "After" states across:

1. CLAUDE.md estimated tokens.
2. MEMORY.md size.
3. Skills count in the menu.
4. MCP deferred tools count.
5. Calculated reduction percentage and delta tokens saved per message.

Sources: skills/token-optimizer/references/agent-prompts.md:237-268

Summary

Benchmarks & Savings Tracking provides the empirical foundation for the Token Optimizer project. By integrating runtime detection, automated quality scoring, and historical trend analysis via measure.py, the system transforms subjective "sluggishness" into quantifiable metrics. This data-driven approach allows users to validate that config changes successfully recover 5-15% of the context window while behavioral changes like model routing slash automation costs by up to 75%.