Database & Storage
Relevant source files
The following files were used as context for generating this wiki page:
The Database & Storage system in Token Optimizer is designed to provide persistent tracking of token usage, session metadata, and historical trends. Its primary purpose is to enable data-driven optimization by measuring context overhead, historical spend, and the effectiveness of implemented fixes.
The system utilizes a combination of local SQLite databases for structured historical data and a coordination directory structure for active session audits. This architecture allows the project to quantify context recovery (typically 5-25%) and project savings based on a user's specific interaction patterns.
Sources: plugins/token-optimizer/skills/token-optimizer/SKILL.md:14-16, skills/token-coach/SKILL.md:46-56
Data Persistence Architecture
The storage layer is bifurcated into long-term historical storage and short-term session coordination. Long-term data is stored in a SQLite database, while active audits utilize a temporary coordination path (COORD_PATH) to pass information between parallel analysis agents.
Historical Trends Database
The trends.db file serves as the primary repository for historical session data. It is located within the backup and data directory at ~/.claude/_backups/token-optimizer/trends.db. This database is populated via a SessionEnd hook that executes measure.py collect.
The diagram shows the flow of data from session termination to historical analysis.
Sources: skills/token-optimizer/references/agent-prompts.md:204-208, skills/token-coach/SKILL.md:46-56
Coordination Storage (COORD_PATH)
During an active optimization audit, the system creates a coordination folder. This folder acts as a shared storage space for parallel subagents to write their findings.
| Storage Component | Path / Location | Purpose |
|---|---|---|
| Audit Reports | {COORD_PATH}/audit/ | Raw analysis from parallel agents (CLAUDE.md, MCP, etc.) |
| Optimization Plan | {COORD_PATH}/analysis/ | Synthesized plan generated by the Synthesis Agent |
| Planned Changes | {COORD_PATH}/plan/ | Optimized versions of files (e.g., CLAUDE.md.optimized) |
| Verification | {COORD_PATH}/verification/ | Post-optimization measurements and delta results |
| Sources: skills/token-optimizer/references/agent-prompts.md:8-15, plugins/token-optimizer/skills/token-optimizer/references/implementation-playbook.md:12-25 |
Session & Quality Metrics
The storage system tracks specific metrics to assess "Context Quality," which moves beyond simple token counts to measure the "rot" or relevance of the stored context.
Quality Indicators
The system measures several data points to generate a quality score:
- Stale Reads: Ratio of files read that were subsequently edited without a re-read.
- Tool Result Bloat: Large tool outputs that are never referenced again in the session.
- Compaction Depth: Information loss incurred during auto-compaction events.
- Decision Density: The ratio of substantive content to configuration overhead.
Sources: skills/token-optimizer/references/optimization-checklist.md:389-405
Historical Data Fields
The database tracks the following trends used by the Token Coach:
| Field | Description |
|---|---|
quality_recent_avg | Average quality score over the last 7 days. |
cache_hit_recent_avg | Percentage of tokens served via prompt caching. |
total_cost_usd | Cumulative spend calculated from model-specific pricing. |
multi_model_session_pct | Frequency of mid-session model switches (which invalidates cache). |
| Sources: skills/token-coach/SKILL.md:58-75 |
Backup & Recovery System
Storage reliability is ensured through a mandatory backup system. Before any modifications are applied to configuration files (like CLAUDE.md or MEMORY.md), the system archives the current state to a timestamped backup directory.
This diagram illustrates the safety protocol for configuration storage.
Sources: plugins/token-optimizer/skills/token-optimizer/references/implementation-playbook.md:5-10, plugins/token-optimizer/skills/token-optimizer/SKILL.md:253-256
Smart Compaction & Checkpointing
The storage system implements a "checkpointing" mechanism to mitigate data loss during context compaction.
- Pre-Compaction Capture: A hook snapshots structured session state (decisions, active errors, agent state) to disk.
- Post-Compaction Injection: The stored state is re-injected into the fresh context, ensuring continuity.
- Storage Location: Checkpoints are stored in
~/.claude/token-optimizer/checkpoints/and are automatically rotated (keeping the last 50 or 7 days).
Sources: skills/token-optimizer/references/optimization-checklist.md:368-385
Summary
The Database & Storage system facilitates a feedback loop between measurement and optimization. By combining temporary coordination files for active audits with a SQLite-backed historical trends database, the system allows users to move from reactive context management to proactive, data-driven token efficiency. This structured approach to data persistence ensures that optimizations are based on actual usage patterns while maintaining safety through a robust backup and checkpointing architecture.
Sources: skills/token-optimizer/references/optimization-checklist.md:410-435