Token Estimation
Relevant source files
The following files were used as context for generating this wiki page:
Token Estimation in M31A is a critical system responsible for calculating and managing the context window usage of Large Language Models (LLMs) across different providers. It ensures that the agent operates within the physical limits of the underlying models while providing real-time feedback to the user regarding context density and potential overflows.
The system utilizes tiktoken-based estimation combined with an Exponential Moving Average (EMA) self-calibration mechanism. This allows the agent to maintain high accuracy even when switching between providers like OpenRouter, Zen, and Nvidia, where tokenization strategies may vary slightly.
Sources: README.md:154-158, CONTRIBUTING.md:46
Core Architecture
The token estimation logic is encapsulated within the internal/tokens package. It provides a specialized Estimator struct that handles the tracking of tokens for a specific model context.
The Estimator Component
The Estimator is initialized with a specific model ID and tracks context usage throughout a session. It is integrated directly into the workflow engine to monitor tool calls, prompt generation, and model responses.
The diagram above illustrates how the Estimator interacts with the workflow engine and external providers to calibrate and report context usage.
Sources: internal/engine/tokens/estimator.go, README.md:154-158
EMA Self-Calibration
M31A employs Exponential Moving Average (EMA) to adjust its estimates based on actual usage reported by provider APIs. This prevents cumulative errors in long-running conversations where context compaction (via AutoDream) or complex tool outputs might otherwise lead to inaccurate tracking.
| Component | Description | Default/Threshold |
|---|---|---|
| EMA Alpha | Smoothing factor for calibration | 0.3 |
| Warning Threshold | Point where UI displays context warnings | 80% |
| Reject Threshold | Point where further requests are blocked | 95% |
| Compaction Trigger | Automatic context consolidation point | Overflow Detection |
Sources: README.md:154-158, README.md:278-285
Integration with Workflow Phases
Token estimation is performed during multiple phases of the seven-phase workflow, notably during Execute and Plan.
Execution Monitoring
During the Execute phase, the engine tracks tool call counts and content sizes. If proactive compaction is enabled, the system checks the current token density against a threshold (defined by tool_calls_threshold) before deciding whether to trigger a compaction cycle to free up space.
The sequence diagram shows the calibration loop where actual provider usage stats are used to refine the internal estimator.
Sources: internal/engine/workflow/execute.go:417-422, internal/engine/workflow/integration_test.go:76
Multi-Provider Support
Because M31A supports three primary providers (OpenRouter, Zen, Nvidia), the estimation system must handle runtime capability detection.
- tiktoken-go: Used for models compatible with OpenAI-style tokenization.
- Fallback Strategies: Used when provider-specific tokenizers are unavailable or when using custom base URLs.
Sources: README.md:154-161
Implementation Details
The Estimator is instantiated within the workflow engine as part of the session initialization.
// From internal/engine/workflow/integration_test.go
est := tokens.NewEstimator("test-model")
engine, _ := NewEngine(s.ID, dir, backups, planningDir,
mockP, "test-model", dispatcher, est, mgr, nil)
Sources: internal/engine/workflow/integration_test.go:76-78
Critical Thresholds
The system enforces strict limits to prevent API failures:
- Warning (80%): The TUI displays a context warning banner.
- Critical (95%): Hard rejection of new requests to ensure the final "Ship" phase has enough remaining context to generate walkthroughs and archive logs.
Sources: README.md:156-158
Summary
Token Estimation serves as the "fuel gauge" for M31A. By combining static calculation via tiktoken with dynamic EMA calibration, the system provides a reliable mechanism for managing large codebase contexts, enabling features like AutoDream context compression and intelligent model arbitrage.