Cost-Based Limiting
Relevant source files
The following files were used as context for generating this wiki page:
Cost-Based Limiting is a production feature in FastLimit that allows developers to assign specific weights or "costs" to different operations. Instead of treating every request as equal, this system enables more granular control by decrementing the available rate limit based on the resource intensity or priority of a specific execution.
This mechanism is particularly useful for protecting expensive operations, such as ML inference, large data exports, or complex database queries, ensuring that a few heavy requests do not consume the same quota as many lightweight requests.
Sources: README.md:27, README.md:213-219, examples/fastapi_app.py:161-171
Architecture and Core Logic
The cost-based limiting logic resides within the RateLimiter.check and RateLimiter.check_with_info methods. When a check is performed, a cost parameter is passed to the backend, which is then processed by Redis Lua scripts. To maintain high accuracy, FastLimit uses integer-only math with a 1000x multiplier.
Sources: README.md:29, fastlimit/limiter.py:186-187, fastlimit/limiter.py:202-203
Integer Precision Multiplier
To prevent precision issues common with floating-point math, all limits and costs are scaled by 1000 before being sent to Redis. For example, a standard request with a default cost of 1 is treated as 1000 units within the Lua scripts.
Sources: fastlimit/limiter.py:202-203, fastlimit/limiter.py:253-254
The following diagram illustrates the data flow for a cost-based rate limit check:
Implementation Methods
Cost-based limiting can be implemented either declaratively through decorators or manually via direct method calls.
Decorator-Based Usage
The @limiter.limit decorator supports a cost argument that can accept an integer or a callable (lambda) that inspects the incoming request to determine the weight dynamicially.
Sources: fastlimit/limiter.py:285-325, examples/fastapi_app.py:161-171
| Parameter | Type | Description |
|---|---|---|
rate | str | The rate limit pattern (e.g., "100/minute"). |
cost | Callable[[Request], int] | A function that returns the cost of the request based on context. |
Example: Dynamic Priority Costing
@app.post("/api/expensive")
@limiter.limit(
rate="20/minute",
cost=lambda req: 5 if req.headers.get("X-Priority") == "high" else 1
)
async def expensive_operation(request: Request):
# High priority requests consume 5 units of the 20/minute limit
return {"status": "ok"}
Sources: examples/fastapi_app.py:161-171, tests/test_middleware.py:112-120
Manual Check Usage
For more complex logic where decorators are insufficient, the check method can be used directly.
Sources: fastlimit/limiter.py:143-150, examples/algorithms_demo.py:166-174
Example: Direct Check
# Consumes 10 units from the quota in a single call
await limiter.check(key="user:123", rate="100/minute", cost=10)
Sources: examples/algorithms_demo.py:173, tests/test_fixed_window.py:147-150
Algorithm-Specific Behavior
Cost-based limiting is supported across all three core algorithms, though the internal implementation varies.
Sources: fastlimit/limiter.py:207-251, README.md:29
Fixed Window
In the Fixed Window algorithm, the cost is added to the counter for the current epoch-aligned window. If the INCRBY operation causes the counter to exceed the max_requests (also scaled by 1000), the request is denied.
Sources: fastlimit/scripts/fixed_window.lua:18-35, tests/test_fixed_window.py:153-156
Token Bucket
For the Token Bucket algorithm, the cost represents the number of tokens to be removed from the bucket. A request is allowed only if the bucket contains at least the number of tokens specified by the cost.
Sources: tests/test_token_bucket.py:84-93, fastlimit/limiter.py:228-232
| Algorithm | Cost Effect | Key Benefit |
|---|---|---|
| Fixed Window | Increments window counter by cost * 1000. | Simplest enforcement; low overhead. |
| Token Bucket | Removes cost tokens from current balance. | Allows bursts while maintaining smooth refill. |
| Sliding Window | Affects the weighted average of the current window. | Most accurate for strict SLA requirements. |
| Sources: README.md:104-124, fastlimit/limiter.py:207-251 |
Impact on Middleware and Headers
When using the RateLimitHeadersMiddleware, the X-RateLimit-Remaining header correctly reflects the cost of the operation. If a request has a cost of 5, the "Remaining" header will decrease by 5 units rather than 1.
Sources: tests/test_middleware.py:112-125, fastlimit/limiter.py:253-260
Conclusion
Cost-Based Limiting provides a flexible way to manage API resources by accounting for the varying "weight" of requests. By leveraging high-precision integer math in Redis Lua scripts, FastLimit ensures that even with complex costing logic, rate enforcement remains atomic, performant, and accurate across distributed environments.
Sources: README.md:213-219, benchmarks/performance.py:228-251