LLM Budget Tracker
Relevant source files
The following files were used as context for generating this wiki page:
The LLM Budget Tracker manages Large Language Model (LLM) token consumption and implements cost-control policies. It tracks token usage per model and per feature on a monthly basis to ensure the bot operates within a target budget of approximately $5 USD per month. The system identifies when usage exceeds predefined "soft limits" and triggers graceful degradation of bot features to conserve resources.
Sources: src/core/llm/budget-tracker.ts:1-12, README.md:1-10
Architecture and Data Flow
The tracker functions as a centralized utility within the src/core/llm module. Features like the CriminalCodeAnalyzerDO report token usage to the tracker after every AI provider interaction. The tracker updates in-memory counters and periodically flushes data to Cloudflare KV storage for persistence.
Sources: src/core/llm/index.ts:1-20, src/durable-objects/criminal-code-analyzer-do.ts:553-568
This diagram shows how token usage flows from an active feature through the budget tracker to influence bot behavior.
Explanation: The Feature records usage in the Tracker; the Tracker checks limits; the resulting status informs whether the feature should continue in "Full" or "Degraded" mode.
Sources: src/core/llm/budget-tracker.ts:150-185, src/core/llm/feature-modes.ts:74-95
Key Data Structures
TokenUsage
This interface represents the raw token counts for a single interaction or an accumulated total.
| Field | Type | Description |
|---|---|---|
promptTokens | number | Tokens sent to the model |
completionTokens | number | Tokens generated by the model |
totalTokens | number | Sum of prompt and completion tokens |
Sources: src/core/llm/budget-tracker.ts:21-25
MonthlyUsageData
The tracker persists this object to KV storage using a key format like llm_budget_YYYY-MM.
| Field | Type | Description |
|---|---|---|
month | string | YYYY-MM format identifier |
models | Record<string, TokenUsage> | Usage mapped by specific model name |
features | Record<FeatureType, TokenUsage> | Usage mapped by feature (summary, profanity, criminal) |
lastUpdated | number | Unix timestamp of the last update |
Sources: src/core/llm/budget-tracker.ts:44-50
Configuration and Soft Limits
The system defines default soft limits based on the model tier (Nano vs. Mini). These limits represent the total tokens allowed per month before degradation occurs.
| Model Tier | Default Limit | Environment Variable |
|---|---|---|
| Nano (e.g., gpt-4.1-nano) | 20,000,000 | NANO_TOKENS_SOFT_LIMIT |
| Mini (e.g., gpt-4.1-mini) | 3,000,000 | MINI_TOKENS_SOFT_LIMIT |
| Default (Fallback) | 5,000,000 | N/A |
Sources: src/core/llm/budget-tracker.ts:57-63, src/core/llm/budget-tracker.ts:124-148
Feature Degradation Logic
The getLLMFeatureModes function computes the current operational state for each feature based on the tracker's status.
Summary Feature Modes
- full: Normal summarization.
- economy: Triggered when Mini models exceed limits. Reduces token usage.
- disabled: Triggered when both Nano and Mini limits are exceeded.
Profanity Feature Modes
- full: AI-powered detection.
- local-only: Triggered when Nano models exceed limits. Falls back to local dictionary checks.
Criminal Feature Modes
- full: Comprehensive legal analysis.
- light: Triggered when Mini models exceed limits.
- disabled: Triggered when both limits are exceeded.
Sources: src/core/llm/feature-modes.ts:14-42, src/core/llm/feature-modes.ts:88-115
Core Implementation Details
The LLMBudgetTracker class provides the primary interface for usage management.
// src/core/llm/budget-tracker.ts:150-185
recordUsage(model: string, feature: FeatureType, usage: TokenUsage): void {
const currentMonth = getCurrentMonth();
if (this.data.month !== currentMonth) {
this.resetForNewMonth(currentMonth);
}
if (!this.data.models[model]) {
this.data.models[model] = emptyUsage();
}
this.data.models[model] = addUsage(this.data.models[model], usage);
this.data.features[feature] = addUsage(this.data.features[feature], usage);
this.data.lastUpdated = Date.now();
this.isDirty = true;
}
Sources: src/core/llm/budget-tracker.ts:150-180
The isOverSoftLimit method determines if a model has reached its threshold. It uses pattern matching to identify if a model belongs to the "nano" or "mini" tier if an exact name match is not found in the configuration.
Sources: src/core/llm/budget-tracker.ts:197-230
Usage in System Components
The CriminalCodeAnalyzerDO utilizes the budget tracker to monitor the costs of legal analysis. It captures usage from the AI provider's response and reports it immediately.
// src/durable-objects/criminal-code-analyzer-do.ts:553-568
private recordOpenAIUsage(model: string, feature: string, usage: any): void {
if (!usage) return;
const promptTokens = usage.prompt_tokens || usage.input_tokens || 0;
const completionTokens = usage.completion_tokens || usage.output_tokens || 0;
const totalTokens = usage.total_tokens || (promptTokens + completionTokens);
getBudgetTracker(this.env).recordUsage(model, 'criminal', {
promptTokens,
completionTokens,
totalTokens,
});
}
Sources: src/durable-objects/criminal-code-analyzer-do.ts:553-568
Conclusion
The LLM Budget Tracker provides a critical safety layer for the bot's operation. By combining real-time tracking with a structured degradation policy, it prevents unexpected costs while maintaining essential bot functionality (such as local profanity detection) even when AI budgets are exhausted.