perf(worker): cache bot list and model pricing reads

Merged
TP
8d ago
+243-15

Cuts the hottest D1 readers to near zero.

Bot list (GET /api/bots/, worker/src/routes/bots.ts)

  • Per-user infinite in-memory cache (single isolate), write-through invalidation on POST/PUT/PATCH/DELETE, OAuth default-bot provisioning, and onboarding bootstrap
  • Per-bot SELECT model_id FROM bots_aimodel N+1 replaced with one batched WHERE id IN (...)

Model pricing (worker/src/usage.ts)

  • bots_aimodel is runtime-static (no worker route writes it), so rows load once per isolate and are shared by rawCostForToday + baselinePricingSnapshot (previously 2 full scans per quota check, on every chat turn)

Tests (worker/tests/api.test.ts)

  • Reset both caches in beforeEach — fresh DB per case would otherwise leak rows through module state

Verified

  • tsc --noEmit clean, vitest 51/51 pass (local seeded DB; no prod reads)

Staleness notes

  • Bot list: entries live until invalidated. Direct D1 console edits won't show until redeploy; any future bots_bot writer must call invalidateBotListCache.
  • Pricing: any future runtime writer must call `invalidateModelsCache()$.

Summary by cubic

Reduces the hottest D1 reads to near zero by caching the per-user bot list and the bots_aimodel pricing table in worker memory. The bot list's per-bot N+1 lookup becomes a single batched query, and pricing rows load once per isolate instead of twice per quota check on every chat turn.

Cache notes

  • Bot list entries live until invalidated; POST/PUT/PATCH/DELETE, OAuth provisioning, and onboarding bootstrap trigger invalidation, but direct D1 console edits won't appear until redeploy. An epoch guard drops rows a GET started reading before a mid-read bot write.
  • Pricing rows are safe to cache because no worker route writes bots_aimodel; any future runtime writer must call invalidateModelsCache(). An empty table is never cached, so costs can't pin at zero and silently stop enforcing the daily cap.
  • Tests reset both caches in beforeEach; a new cache.test.ts covers the epoch guard, eviction cap, and empty-pricing behavior.