Model Arbitrage
Relevant source files
The following files were used as context for generating this wiki page:
Model Arbitrage is a cost-optimization system within M31 Autonomous designed to classify the complexity of coding tasks and recommend or automatically switch to the most cost-effective Large Language Model (LLM) that meets the task's technical requirements. By analyzing task descriptions, file counts, and dependencies, the system ensures that powerful, expensive models are reserved for complex architectural changes while cheaper models handle routine updates and fixes.
The system operates primarily during the transition between workflow phases, such as the transition to the Execute phase. It leverages metadata from multiple providers (OpenRouter, Zen, Nvidia) to compare real-time pricing and context window capabilities, ensuring that selected models have sufficient capacity (e.g., 64K+ context for complex tasks) without unnecessary expenditure.
Sources: README.md:144-147, internal/integrations/arbitrage/arbitrage.go:121-135, internal/ui/tui/app.go:585-600
Architecture and Core Logic
The arbitrage system is built around a scoring engine that evaluates task complexity and a recommendation engine that compares available models based on those scores.
Task Complexity Classification
The system classifies tasks into three distinct levels: Simple, Moderate, and Complex. This classification is driven by keyword analysis of the task action and description, followed by "boost rules" based on the scale of the change.
The classification logic prioritizes complex keywords (e.g., "architect", "migrate") over moderate (e.g., "refactor", "database") and simple ones (e.g., "fix", "typo").
Sources: internal/integrations/arbitrage/arbitrage.go:73-90, internal/integrations/arbitrage/arbitrage.go:215-245
Token Estimation
Once a complexity level is established, the Scorer estimates the required input and output tokens. These estimates are used to calculate the projected cost across all cached models.
| Complexity | Base Input Tokens | Base Output Tokens |
|---|---|---|
| Simple | 2,000 | 1,000 |
| Moderate | 5,500 | 2,750 |
| Complex | 14,000 | 7,000 |
Note: Each file involved in the task adds an adjustment of 500 tokens to both input and output estimates.
Sources: internal/integrations/arbitrage/arbitrage.go:94-111, internal/integrations/arbitrage/arbitrage_test.go:87-105
Recommendation Engine
The recommendation engine filters available models by their pricing and technical constraints. For Complex tasks, the system imposes a hard constraint: the model must have a context window greater than 64,000 tokens.
Logic Flow for Model Selection
- Filter: Remove models with no pricing data.
- Sort: Order models by total estimated cost (ascending).
- Validate:
- If task is Complex: Find the cheapest model with >64K context that is within the user-defined
ArbitrageThreshold(default 20%) of the absolute cheapest model. - If task is Simple/Moderate: Select the absolute cheapest model.
- If task is Complex: Find the cheapest model with >64K context that is within the user-defined
- Alternatives: Identify up to 3 alternative models within the price threshold.
Sources: internal/integrations/arbitrage/arbitrage.go:138-175, internal/integrations/arbitrage/arbitrage_test.go:167-185
Data Structures
The system utilizes structured "Decision Receipts" and recommendations to track optimization logic.
TUI Integration and Auto-Arbitrage
The Terminal User Interface (TUI) implements "Auto-Arbitrage," which can automatically switch the active model before a phase begins if significant savings are detected.
Automatic Switching Logic
The checkAutoArbitrage function in the TUI application state evaluates whether a switch is necessary. It calculates the currentCost of the active model and compares it to the recommended model's cost. A switch only occurs if the savings exceed the configured arbitrage_threshold.
Configuration Options
Model Arbitrage behavior is controlled via the [model] section of the configuration file.
| Option | Type | Default | Description |
|---|---|---|---|
auto_arbitrage | boolean | false | Enables automatic model switching between phases. |
arbitrage_threshold | float | 0.1 | The minimum percentage of savings required to trigger a switch. |
default | string | "" | The fallback model ID used if arbitrage is disabled. |
Sources: README.md:323-328, internal/ui/tui/app.go:586-595
Conclusion
Model Arbitrage serves as a critical economic layer in M31 Autonomous, preventing the wasteful use of high-tier models on low-complexity tasks. By combining semantic task analysis with hardware constraints like context window size, it provides a balanced approach that prioritizes both cost efficiency and implementation quality.
Sources: README.md:144-147, internal/integrations/arbitrage/arbitrage.go:121-135