feat: richer column value profiles in `schema_inspect` (ranked values, uniqueness, value shapes, MongoDB)

Merged
AN
4d ago
+828-50

Issue for this PR

Closes # (feature; no linked issue)

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

schema_inspect already adds a value profile for SQLite, DuckDB and Postgres tables. That profile had gaps that hide what a column actually contains:

  • Values came from DISTINCT … LIMIT 21 with no ordering, so they were arbitrary, not the common ones.
  • Nothing indicated whether a column has one row per value.
  • Code-like formats were invisible.
  • Long text was clipped at 40 characters.
  • MongoDB collections got no profile at all, and nested fields did not appear.

This PR fills those gaps. It adds no new tool parameter and changes nothing for other describeTable consumers.

SQL tables (still one query per table):

  • Low-cardinality values are listed most frequent first, with counts. High-cardinality text columns show their 3 most frequent values: status: 4 distinct — open (812), closed (130), ….
  • A column is flagged unique in sample when its distinct count equals its non-null count.
  • Code-like text columns get value shapes, e.g. H01M4/9066 is shown as A9+A9/9+. Free text, uuid columns and single letters-only shapes are skipped.
  • Long text columns get one example of about 160 characters; other examples stay at 40.

MongoDB collections:

  • A value profile is built from one sample of up to 100 documents, read through a new optional Connector.sampleDocuments. It has the same format as SQL: distinct count, ranked values, null share, uniqueness and shape.
  • BSON types render as text: UUID, ObjectId, Decimal128, Long, Timestamp, regex, Code, Date and MinKey/MaxKey. Present values that cannot be rendered say values not profiled (<type>) instead of being counted as nulls.
  • A Nested fields (sampled N docs): section lists nested paths in MongoDB dot notation (items.sku). Array elements and partial presence are noted, e.g. string (array element, in 67% of docs).
  • describeTable output is unchanged, so the schema cache, PII detection and schema-sync see exactly what they saw before.
  • The sample read is bounded by maxTimeMS (at most 15 s or the connection timeout) and honours the tool call's abort. It still asks for the sql_execute permission and runs the precedence check.

Profiling fails safe as before: if the profile query fails or times out, the schema is still returned.

The output is larger. A text-heavy 40-column table went from 1,899 to 2,858 characters (1.5x); typical tables grow less.

Known limits:

  • For an all-unique column, the 50 fetched values are the lexicographically smallest, so its shapes and examples come from that slice. There is no portable random ordering across the three dialects.
  • For fields inside arrays, value counts are element occurrences.

How did you verify your code works?

Unit and integration tests:

  • packages/opencode: bun test --timeout 30000 on all 39 test files that reference the changed exports: 1172 pass, 0 fail. The new SQL cases run against real in-process SQLite and DuckDB: frequency order, the uniqueness flag, shapes (including no shape for free text), the long example, and quoting.
  • MongoDB unit tests use real bson values: UUID, ObjectId, Decimal128, Long, Timestamp, BSONRegExp, Code, Date, MinKey, binary and dotted keys. They assert the exact rendered lines.
  • packages/drivers: 268 pass. This includes a regression test that describeTable output for nested documents is unchanged, and the sampleDocuments timeout and abort options.
  • Real MongoDB 7.0 server: DRIVER_E2E_DOCKER=1 bun test test/altimate/drivers-mongodb-e2e.test.ts gave 91 pass, 0 fail, including a new nested-path and profile case. The suite skips without Docker.
  • Real Postgres: I ran the generated profile SQL on a local Postgres scratch database. It returned ranked values, null and distinct counts, and handled a quoted mixed-case column, jsonb and uuid correctly. The scratch database was then dropped.
  • Typecheck: bun run typecheck in packages/opencode and tsc --noEmit in packages/drivers are both clean.
  • Tool schemas: test/altimate/tools/tool-schema* passes, since no tool parameters changed.

Independent review: a separate reviewer reviewed 3b81de816e. Its six findings were all in the MongoDB path:

  • UUID, regex and Code rendering;
  • dotted keys;
  • the [] notation;
  • nested paths leaking into describeTable consumers;
  • abort and double sampling;
  • mock-only tests.

All six are fixed in deb2828de7.

Not run: an in-flight cancellation of a MongoDB sample read. The already-aborted case is tested.

Screenshots / recordings

Not a UI change. Example schema_inspect output from the MongoDB test:

Nested fields (sampled 6 docs):
address.city: string
items.sku: string (array element)

Value samples:
_id: 6 distinct, unique in sample — 6ad3bb06eb46b9102e51e3dc, …
filter: 1 distinct — /^ab+c/i (6)
ref: 6 distinct, unique in sample — 1eb83974-d3bc-4af2-a1cf-49b685295417, …
status: 2 distinct — open (4), closed (2)
items.sku: 2 distinct — S1 (6), S2 (6)

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Adds bounded MongoDB collection reads and substantially changes profiling SQL/output for schema inspection; behavior is read-only and fail-safe, but touches connector APIs and agent-facing inspect output.

Overview
Enriches schema_inspect value profiles for SQL (SQLite, DuckDB, Postgres) and adds equivalent profiling for MongoDB collections, without new tool parameters.

SQL profiling still uses one bounded query per table, but the SQL now returns frequency-ranked values with counts (not arbitrary DISTINCT samples). Rendered output adds unique in sample, per-value counts, value shapes for structured text (e.g. patent codes), and a longer first example for prose columns.

MongoDB gets a parallel path: optional Connector.sampleDocuments on the driver (raw finds capped by maxTimeMS, cancellable via AbortSignal). schema_inspect passes database/collection into profileValues, which samples up to 100 documents and builds the profile in TypeScript via profileDocuments—including a nested fields section (dot paths, array-element markers, doc presence %) and BSON-aware value text. describeTable stays top-level only; profiling failures still return the schema with a one-line note.

Reviewed by Cursor Bugbot for commit 4edad6ca39cffe6437d5321ec5dc9b44f4b001e2. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

Enriches the schema_inspect value profile for SQLite, DuckDB, Postgres, and MongoDB. Values are now listed most frequent first with counts instead of arbitrary DISTINCT samples, columns get a unique in sample flag, code-like text gets value shapes, and long text shows a longer example.

MongoDB profiling

  • Builds the profile from a sample of up to 100 documents via a new optional Connector.sampleDocuments, capped by maxTimeMS and the smaller of the connection timeout, and cancelled with the tool call's abort.
  • The abort signal is passed to the driver find, and a read that rejects after abort is suppressed instead of surfacing as unhandled.
  • Renders BSON types (UUID, ObjectId, Decimal128, Long, Timestamp, regex, Code, Date, MinKey/MaxKey) as text and lists nested fields in dot notation with presence shares and array-element markers.
  • describeTable output is unchanged; unrenderable values report values not profiled (<type>) instead of being counted as nulls.

Behavior and limits

  • No tool parameters changed; profiling still requires the sql_execute permission and fails safe, returning the schema if the profile query errors or times out.
  • Ranking and counts use the full value; only the output is clipped at 200 characters, so long values sharing a prefix stay distinct.
  • Output grows: a text-heavy 40-column table went from 1,899 to 2,858 characters.
  • For all-unique columns the fetched values are the lexicographically smallest 50, and counts for array elements are per occurrence.
  • A nested path seen both inside and outside an array stays an array element, so it is never flagged unique.

Summary by CodeRabbit

  • New Features
    • Schema inspection now profiles MongoDB collections, including nested fields, array elements, sampled value distributions, and document uniqueness.
    • SQL value profiles include frequency-ranked values and counts, uniqueness, null percentages, examples, and useful text-shape summaries.
    • MongoDB document sampling supports limits, timeouts, and cancellation.
  • Bug Fixes
    • Inferred schema details remain available when value sampling fails.