Resource Governance & Rate Limiting
Relevant source files
The following files were used as context for generating this wiki page:
The Resource Governance and Rate Limiting modules in the tldw_server project provide a centralized framework for managing system resources, enforcing consumption quotas, and preventing service abuse. These systems ensure that high-cost operations, such as LLM (Large Language Model) calls and embedding generation, are controlled via policies that govern token usage, concurrency, and request frequency.
The architecture is designed to support both single-user and multi-tenant environments, providing granular control over how resources are allocated to different users or roles. This includes monitoring rate limit events and providing administrators with visibility into policy enforcement through specialized management interfaces.
Sources: Docs/Architecture.md:73-75, AGENTS.md:74-75, Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_04_ai_model_governance.md:16-20
Resource Governance Framework
The Resource Governor serves as a centralized controller for managing quotas and concurrency across the platform. It supports Redis for distributed state management, allowing for scaled deployments where resource usage must be tracked across multiple server instances.
Key Components
| Component | Description |
|---|---|
| Governor | The core logic responsible for evaluating resource requests against active policies. |
| Policy Store | Manages the CRUD operations for resource policies, including scope and rate limits. |
| Policy Simulator | A tool for administrators to preview the impact of a policy on users/requests before application. |
| Resolution Tool | An evaluation chain that determines which policy wins based on priority and scope. |
Sources: Docs/Architecture.md:73-75, Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_04_ai_model_governance.md:23-28
Policy Resolution Logic
Resource governance follows an evaluation chain where multiple policies might match a single request. The system resolves these using priority and scope matching. For example, a "User" specific policy with a higher priority will override a "Global" default policy.
The diagram shows how the evaluation chain resolves multiple matching policies to a single "Winner" based on priority.
Sources: Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_04_ai_model_governance.md:26-28
Rate Limiting Systems
Rate limiting is implemented at multiple levels: the endpoint level via standard rate limiters and the module level for specific high-intensity features like embeddings (using slowapi) and chat/evaluations.
Types of Rate Limiting
- Endpoint-level: Protects REST and WebSocket APIs from general abuse.
- Module-level: Found in specific modules like
core/Embeddings/andcore/Chat/to handle domain-specific constraints. - Programmatic Access: Virtual keys and API keys are mapped to specific scopes and limits, ensuring that programmatic usage stays within budget.
Sources: AGENTS.md:126-128, Docs/Architecture.md:73-75, Docs/superpowers/plans/2026-03-23-authnz-sequential-review.md:52-54
Rate Limit Monitoring
Administrators can monitor rate limit enforcement through the admin-ui. This interface provides visibility into rejection counts and specific policy violations.
| Feature | Monitoring Detail |
|---|---|
| Rejection Count | Number of requests denied in the last 24 hours. |
| Event Logging | User/role and policy identification for every rejection event. |
| Metrics Integration | Use of Prometheus counters to track rate-limit hits for external observability. |
Sources: Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_04_ai_model_governance.md:37-40, Docs/superpowers/specs/2026-04-07-metrics-module-review-design.md:36-40
AI Model Governance
Specific governance is applied to AI models to manage the high costs associated with external providers. This involves tracking token usage, costs, and error rates per model and provider.
This flowchart illustrates the connection between model usage tracking and the resource governor's monitoring of rate limit events.
Sources: Docs/Product/Completed/Plans/IMPLEMENTATION_PLAN_hci_04_ai_model_governance.md:16-40
Governance Configuration
Governance policies are configurable through the Config_Files/resource_governor_policies.yaml. This file defines the baseline rules for different resources, including:
- LLM Tokens: Input and output token limits.
- Concurrency: Maximum simultaneous requests allowed per user or role.
- Budgeting: Financial caps on model consumption over specific time periods.
Sources: AGENTS.md:275-277, Docs/superpowers/plans/2026-03-23-characters-backend-sequential-review.md:144-148
Conclusion
The Resource Governance and Rate Limiting infrastructure is a critical layer for maintaining the stability and financial predictability of the tldw_server platform. By combining priority-based policy resolution with granular monitoring of AI model consumption, the system provides administrators with the necessary tools to manage both internal and external resource constraints effectively. This framework ensures that resource-intensive tasks are executed within safe boundaries while providing detailed feedback when limits are reached.