perf(logs): order public log pages on an indexed column and retire a superseded document index

Merged
WA
21d ago
+31413-45

Summary

  • Public logs keyset: the v1/v2 log list ordered by date_trunc('milliseconds', started_at), which no index serves, so each page read and sorted every run of the workflow (and computed the handled-error flag for all of them). It now orders by the raw started_at with a microsecond cursor, using the existing (workspace, started_at, id) indexes, and filters workflowIds on workflow_execution_logs.workflow_id so multi-workflow scopes stay index-driven.
  • Legacy millisecond cursors keep paging under the old ordering until their walk ends; one classifier decides cursor format for the decoder and the keyset, so a client cursor never reaches the query as a 500 (invalid ones get the same 400 as before). The leading inclusive bound is opt-in for the precise key only — every other keyset list renders the same SQL.
  • Drop doc_connector_reconciliation_idx (DROP INDEX CONCURRENTLY): superseded by the v2 index in 0388; it was the only document index on source_seen_at, so it made every connector seen-stamp a non-HOT update.
  • Add kcsl_terminal_started_at_idx (CREATE INDEX CONCURRENTLY, partial on the prune's terminal statuses) so the sync-log retention prune range-scans instead of walking the whole connector index.

Type of Change

  • Performance improvement

Testing

  • Replica EXPLAIN (ANALYZE, BUFFERS, SERIALIZE): dense workflow 7.6 s → 1.0 ms; three-workflow scope 9.6 s cold → 2.1 ms; page-2 cursor 158 ms → 1.1 ms (backward index scan, ~100 buffers)
  • Old vs new pagination compared over 1,440 page boundaries (ties, sub-millisecond collisions, both orders, job-run union, workflow filters, resumed legacy cursors): identical
  • New public-queries.integration.ts: limit-1 paging across a shared millisecond, job-run union, workflow filter, legacy cursor resume (no-fraction / 4-digit / ms forms); unit tests for cursor classification and rejection
  • Confirmed the queries that occasionally used the dropped index already plan on the v2 index
  • check:migrations, drizzle-kit generate (no drift), biome on changed files

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)