perf(db): serve workspace usage drill-downs index-only on usage_log
Merged
WA
12d ago+31773+31773+01234567890123456789012345678901234567890123456789-6-6-0123456789
Summary
- Replaces
usage_log_workspace_created_at_idx(workspace_id, created_at)withusage_log_workspace_created_at_cost_idx(workspace_id, created_at, billing_entity_type, billing_entity_id, user_id, source, cost, workflow_id), built concurrently before the old one is dropped (0446, replay-safe in the 0441/0442 pattern) - Every Workspaces drill-down read (totals, time series, source and member series, workflow ranking) filters on the entity, one workspace and a
created_atwindow. With only(workspace_id, created_at)to work with, the planner either fetched a heap row per workspace row in the window, or scannedusage_log_billing_entity_created_at_cost_idxand fetched a heap row per organization row to checkworkspace_id. 0441 dropped the compact entity/period index that used to narrow the second through a BitmapAnd, so a large organization's workflow ranking went from ~4.6s to ~13.7s, reading thousands of rows per row returned. Each of these reads is now an index-only scan bounded by the workspace's rows in the window workspace_id, created_atstill lead, so the readers that name only the workspace keep their plans and order: theworkspace_idFK'sON DELETE SET NULL, and the ledger list filtered to a workspace (which now also checksuser_idin the index instead of on the heap)- Comments in
usage-analytics.ts/usage-analytics-queries.tsthat said a narrowed read heap-fetches per row are updated
Why this shape
- Entity after
created_at, not before.(workspace_id, billing_entity_type, billing_entity_id, created_at)would bound the same drill-downs, but the readers without an entity predicate (FK cascade, workspace ledger list) would lose thecreated_atorder and bound. A workspace's rows billed to another entity (personal usage before it joined the org) are rare, and the index drops them without a heap fetch either way - Replace, don't add. Inserts maintain the same number of indexes; only this entry gets wider. The ledger insert is a low-rate single-row statement, so the wider entry is noise next to the heap tuple and the other index writes
- Not widening
usage_log_billing_entity_created_at_cost_idxinstead. Addingworkspace_id/workflow_idthere would make the drill-down scan the whole organization's window and slow the org-level totals and series, which are by far the heaviest readers of that index - No
INCLUDE. drizzle cannot declare one, anddrizzle generatemust stay in sync with the schema; trailing key columns serve the same scans (same as 0441) - Size. About 3x the index it replaces, measured on a seeded table and consistent with a hypothetical-index estimate; the cost of making every drill-down column available without the heap
Left alone
- The per-user sum in an org (
user_id+ entity + window): rare, and already index-only onusage_log_billing_entity_created_at_cost_idx. The analogous change would widenusage_log_user_created_at_idx, the most-scanned index on the table, for little gain - A drill-down over a Stripe period (stamp-matched, opt-in preset): the period stamps are not in this index, so it keeps the plans it has today; not seen in production traffic
- Model/BYOK drill-down dimensions still read
description/metadatafrom the heap, now only for the workspace's rows
Type of Change
- Other: performance (index)
Testing
- New
apps/sim/lib/billing/core/usage-analytics-plans.integration.ts: EXPLAINs the exact SQL of the totals, time series, source series, member series and workflow ranking narrowed to a workspace holding most and a small share of its organization's usage (with an unrelated organization and rows billed to another entity in the same workspace) and asserts an Index Only Scan of the new index bounded onworkspace_id+created_at. All 10 cases fail on the old schema; droppinguser_idandworkflow_idfrom the key fails exactly the member series and workflow ranking cases. Passes with bothpushandINTEGRATION_DB_PROVISION=migrate - Production
EXPLAIN(read-only) of the drill-down fingerprints: today they pick the narrow workspace index plus heap, a BitmapAnd withusage_log_workflow_id_idx, or the entity index with a heap filter onworkspace_id(row estimates are skewed for large organizations). The competing index-only scan bounded onworkspace_id+created_atcosts a fraction of each of those - Local Postgres 17 seeded with a skewed org/workspace distribution: every drill-down read becomes an index-only scan with zero heap fetches, touching 7x fewer buffers for a workspace holding half its org and 38x fewer for a small one; the FK cascade and the workspace ledger list use the new index, and org-level reads keep their plans
- Migration replayed twice and after a simulated INVALID build: rebuilds the index and ends valid
bun run check:migrations,drizzle-kit generate(no changes),bun run lint,bun run check:audits,type-check(apps/sim, packages/db),usage-analytics.test.ts
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing (new tests pass the
test-auditauthoring gate) - No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)