Compaction Memory Benchmarks
Relevant source files
The following files were used as context for generating this wiki page:
The Compaction Memory Benchmarks represent a suite of evaluative tools and automated tests designed to measure the efficiency of context management within the plugsuits ecosystem. These benchmarks specifically focus on the system's ability to maintain high information retention while operating under strict token limits by utilizing various compaction strategies. The primary goal is to ensure that agents can recall critical user facts, technical decisions, and conversation history even after multiple rounds of summarization and history pruning.
Sources: packages/minimal-agent/benchmark.ts:468-472, packages/minimal-agent/BENCHMARK-RESULTS.md:1-5
Benchmark Architecture and Execution Flow
The benchmarking system operates by simulating multi-turn conversations and periodically issuing "probes"—specific questions designed to test the model's memory of previously shared information. The system tracks token usage across four critical boundaries: current context, speculative threshold, blocking threshold, and hard context limit.
The benchmark lifecycle manages the transition from active conversation to summarized state, ensuring that tools like CheckpointHistory and CompactionOrchestrator interact correctly to preserve context.
Sources: packages/minimal-agent/benchmark.ts:489-565, packages/harness/src/checkpoint-history.ts:470-495
Core Metrics and Analysis
The system captures granular metrics for every turn to visualize how compaction events impact context usage.
| Metric | Description | Source File |
|---|---|---|
| Estimated Tokens | Predicted token count based on internal heuristics before LLM call. | analyze-compaction-metrics.ts |
| Actual Tokens | Verified token count returned by the Model provider API. | analyze-compaction-metrics.ts |
| Retention Pct | Percentage of "expected facts" successfully recalled during probes. | benchmark.ts |
| Compaction Cycles | Number of times a full summarization event was triggered. | benchmark.ts |
| Peak Tokens | The maximum number of tokens utilized during the session. | benchmark.ts |
| Blocking Duration | Latency introduced when an agent must wait for compaction. | analyze-compaction-metrics.ts |
The analyze-compaction-metrics.ts script aggregates these values into detailed Markdown tables, calculating the "Delta%" between estimated and actual token counts to calibrate internal tokenizers.
Sources: scripts/analyze-compaction-metrics.ts:133-157, packages/minimal-agent/benchmark.ts:592-615
Compaction Strategies and Triggers
Benchmarks evaluate multiple compaction techniques that the CheckpointHistory class implements to prevent context overflow.
Threshold Management
The system uses ratios relative to the contextLimit to trigger different behaviors:
- Speculative Threshold: Often 75% of the blocking threshold. Triggers a background summarization task without stopping the agent.
- Blocking Threshold: The point at which the agent must pause to compact history before sending the next message.
- Keep Recent Tokens: A reserved buffer (e.g., 2000 tokens) of the most recent messages that are never summarized to preserve immediate conversation flow.
Sources: packages/minimal-agent/benchmark.ts:474-482, packages/harness/src/checkpoint-history.ts:430-465
Compaction Prompt Optimization
Retention is heavily influenced by the prompt used during summarization. The "Chatbot Compaction Prompt" utilizes a four-step analysis scratchpad:
- Chronological Review: Identifying personal facts shared by the user.
- Previous Summary Check: Extracting facts from existing
<previous-summary>tags. - Fact Merging: Ensuring no information is lost during the merge of old and new facts.
- Verification: Checking for completeness against known categories (pets, jobs, etc.).
Sources: packages/minimal-agent/benchmark.ts:31-54, packages/harness/src/compaction-prompts.ts:19-45
Benchmark Results and Findings
Testing across various models and context limits (2000 to 32768 tokens) has yielded several key performance indicators for the compaction system.
Retention Ceiling*: Benchmarks show that model quality (e.g., Claude 3 Opus vs Sonnet) is the primary factor in memory retention, with Opus achieving up to 94% retention on 30-turn tests.
- Context Collapse: The use of
microCompactand context collapse techniques allows 80-turn conversations to stay within a 2000-token limit without hitting the hard threshold. - Compaction Impact: Turn 35 probes (post-compaction) showed a jump from 0/4 recall to 4/4 recall after prompt optimizations were implemented.
Sources: packages/minimal-agent/BENCHMARK-RESULTS.md:15-55, packages/cea/benchmark/cea-memory-bench.sh:44-70
Conclusion
Compaction Memory Benchmarks are vital for validating the autonomous context-management lifecycle. By combining threshold-based triggers, optimized summarization prompts, and real-time fact extraction, the system achieves a balance between operating in constrained memory environments and maintaining high information fidelity. These metrics provide the empirical data necessary to tune plugsuits agents for production workflows where long-running sessions are standard.
Sources: packages/minimal-agent/BENCHMARK-RESULTS.md:57-61, packages/harness/src/checkpoint-history.ts:16-25