perf(db): serve workspace usage drill-downs index-only on usage_log

Merged
WA
12d ago
+31773-6

Summary

  • Replaces usage_log_workspace_created_at_idx (workspace_id, created_at) with usage_log_workspace_created_at_cost_idx (workspace_id, created_at, billing_entity_type, billing_entity_id, user_id, source, cost, workflow_id), built concurrently before the old one is dropped (0446, replay-safe in the 0441/0442 pattern)
  • Every Workspaces drill-down read (totals, time series, source and member series, workflow ranking) filters on the entity, one workspace and a created_at window. With only (workspace_id, created_at) to work with, the planner either fetched a heap row per workspace row in the window, or scanned usage_log_billing_entity_created_at_cost_idx and fetched a heap row per organization row to check workspace_id. 0441 dropped the compact entity/period index that used to narrow the second through a BitmapAnd, so a large organization's workflow ranking went from ~4.6s to ~13.7s, reading thousands of rows per row returned. Each of these reads is now an index-only scan bounded by the workspace's rows in the window
  • workspace_id, created_at still lead, so the readers that name only the workspace keep their plans and order: the workspace_id FK's ON DELETE SET NULL, and the ledger list filtered to a workspace (which now also checks user_id in the index instead of on the heap)
  • Comments in usage-analytics.ts / usage-analytics-queries.ts that said a narrowed read heap-fetches per row are updated

Why this shape

  • Entity after created_at, not before. (workspace_id, billing_entity_type, billing_entity_id, created_at) would bound the same drill-downs, but the readers without an entity predicate (FK cascade, workspace ledger list) would lose the created_at order and bound. A workspace's rows billed to another entity (personal usage before it joined the org) are rare, and the index drops them without a heap fetch either way
  • Replace, don't add. Inserts maintain the same number of indexes; only this entry gets wider. The ledger insert is a low-rate single-row statement, so the wider entry is noise next to the heap tuple and the other index writes
  • Not widening usage_log_billing_entity_created_at_cost_idx instead. Adding workspace_id/workflow_id there would make the drill-down scan the whole organization's window and slow the org-level totals and series, which are by far the heaviest readers of that index
  • No INCLUDE. drizzle cannot declare one, and drizzle generate must stay in sync with the schema; trailing key columns serve the same scans (same as 0441)
  • Size. About 3x the index it replaces, measured on a seeded table and consistent with a hypothetical-index estimate; the cost of making every drill-down column available without the heap

Left alone

  • The per-user sum in an org (user_id + entity + window): rare, and already index-only on usage_log_billing_entity_created_at_cost_idx. The analogous change would widen usage_log_user_created_at_idx, the most-scanned index on the table, for little gain
  • A drill-down over a Stripe period (stamp-matched, opt-in preset): the period stamps are not in this index, so it keeps the plans it has today; not seen in production traffic
  • Model/BYOK drill-down dimensions still read description/metadata from the heap, now only for the workspace's rows

Type of Change

  • Other: performance (index)

Testing

  • New apps/sim/lib/billing/core/usage-analytics-plans.integration.ts: EXPLAINs the exact SQL of the totals, time series, source series, member series and workflow ranking narrowed to a workspace holding most and a small share of its organization's usage (with an unrelated organization and rows billed to another entity in the same workspace) and asserts an Index Only Scan of the new index bounded on workspace_id + created_at. All 10 cases fail on the old schema; dropping user_id and workflow_id from the key fails exactly the member series and workflow ranking cases. Passes with both push and INTEGRATION_DB_PROVISION=migrate
  • Production EXPLAIN (read-only) of the drill-down fingerprints: today they pick the narrow workspace index plus heap, a BitmapAnd with usage_log_workflow_id_idx, or the entity index with a heap filter on workspace_id (row estimates are skewed for large organizations). The competing index-only scan bounded on workspace_id + created_at costs a fraction of each of those
  • Local Postgres 17 seeded with a skewed org/workspace distribution: every drill-down read becomes an index-only scan with zero heap fetches, touching 7x fewer buffers for a workspace holding half its org and 38x fewer for a small one; the FK cascade and the workspace ledger list use the new index, and org-level reads keep their plans
  • Migration replayed twice and after a simulated INVALID build: rebuilds the index and ends valid
  • bun run check:migrations, drizzle-kit generate (no changes), bun run lint, bun run check:audits, type-check (apps/sim, packages/db), usage-analytics.test.ts

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)