fix(schema): batch DuckDB introspection and bound indexing

AN
9h ago
+759-117

Issue for this PR

Closes #1482

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Why: indexing a large warehouse was slow and could hang a session, and DuckDB introspection returned wrong metadata when other catalogs were attached.

Root cause: the cache crawler issued schema -> table -> per-table column requests with one SQLite transaction per table. DuckDB's schema/table/column queries had no catalog restriction or DISTINCT. The schema_index tool did not forward cancellation and nothing bounded indexing time.

Before -> after:

  • DuckDB: repeated schemas, duplicate tables and merged columns from same-named tables in attached catalogs -> metadata is scoped to the current catalog and unique. A new optional introspectSchema(signal) connector capability fetches everything in two queries regardless of table count.
  • Cache: per-table transactions -> one transaction replacing rows and the warehouse summary after collection; the old cache stays readable while collecting. Other connectors keep their discovery path with schema/table de-duplication.
  • Indexing: no deadline/cancel -> 30s default deadline (covers connection acquisition), session cancellation forwarded, partial counts and stopping reason reported, no late writes after cancellation.

Deferred:

  • Native DuckDB cannot abort a running query (Database::Interrupt() is a no-op); we stop waiting and block late writes, but the query may finish in the background.
  • Names-only indexing above a size threshold / lazy descriptions.
  • Bulk introspection for engines other than DuckDB; catalog restriction on other engines.

Review follow-ups (P2, not fixed here): DuckDB temp tables are excluded by the current-catalog filter; timeout_ms is not exposed on the tool; an interrupted refresh replaces the snapshot with completed tables without persisting a partial flag; a synchronous crawl can starve the deadline timer; the new DuckDB suites need explicit CI invocations (they require ALTIMATE_DUCKDB_E2E=1).

How did you verify your code works?

  • packages/opencode: schema-indexing, schema-index, schema-cache, feature-discovery-e2e: 76 pass, 0 fail; test/altimate/schema*: 140 pass, 1 skip, 0 fail
  • ALTIMATE_DUCKDB_E2E=1 schema-index-duckdb.test.ts (3,000 tables, 6,000 columns, attached-catalog conflicts): 1 pass
  • packages/drivers ALTIMATE_DUCKDB_E2E=1 duckdb-schema.test.ts: 5 pass, 0 fail (5 fail before the change)
  • bun run typecheck (opencode) and tsgo (drivers): clean
  • Marker check: no upstream-shared files modified; git diff --check clean
  • Independent Codex review: no P0/P1; five P2 items listed above

Screenshots / recordings

N/A

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Changes schema cache write semantics and indexing cancellation/timeouts; incorrect partial handling could leave stale or incomplete search metadata, though existing snapshots are preserved on failed refreshes.

Overview
Fixes slow/hanging warehouse schema indexing and incorrect DuckDB metadata when other catalogs are attached.

DuckDB driver scopes listSchemas / listTables / describeTable to current_database(), deduplicates schema names, and adds optional introspectSchema(signal) that loads schemas, tables, and columns in two queries. Internal query() honors AbortSignal (stops waiting; native query may still finish).

Schema cache indexing prefers bulk introspectSchema when available; otherwise falls back with schema/table de-duplication. Metadata is collected first, then the SQLite snapshot is replaced in one transaction so the prior cache stays readable during refresh. Partial runs report partial + note; a failed refresh over an existing complete cache keeps the old snapshot.

Indexing bounds: new index-control applies a 30s default deadline and merges session AbortSignal through schema.index, the schema_index tool (ctx.abort), and connector calls. Telemetry marks introspection unsuccessful when results are partial.

Reviewed by Cursor Bugbot for commit adb3450cd64049d985678d5d3f226d23e670853f. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

Fixes schema indexing so large warehouses no longer hang a session, DuckDB introspection no longer mixes metadata from attached catalogs, and interrupted refreshes keep the last good cache.

  • Adds a batched introspectSchema path for DuckDB that fetches schemas, tables, and columns in two queries regardless of table count and scopes results to the current catalog.
  • Wraps cache replacement in a single transaction so the previous snapshot stays readable while new metadata is collected; a partial run never replaces an existing snapshot, while a failed first index is saved with an epoch timestamp so freshness checks treat it as stale and re-index.
  • Adds a 30s default deadline for indexing (covering connection acquisition), forwards session cancellation, and stops all cache writes after cancellation or timeout.
  • Deduplicates schema and table names for connectors without the batched path.

Summary by CodeRabbit

  • New Features

    • Schema indexing can be cancelled or given a time limit, and incomplete results are clearly identified.
    • DuckDB indexing covers schemas, tables, and columns in the current database.

  • Improvements

    • Bulk metadata collection speeds up indexing, including for large databases.
    • If indexing is interrupted, available metadata can be retained, and existing cached schema data is preserved when a complete replacement is unavailable.