feat: dbt_cost_estimate on Snowflake and Databricks from the planner's EXPLAIN

Merged
HK
4h ago
+968-8

Issue for this PR

No GitHub issue. Related to #906 (pre-execution cost firewall). This is the CLI side of the Claude Code plugin's scan check on Snowflake and Databricks; the plugin PR, AltimateAI/altimate-claude-plugin#39, depends on this release.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

dbt_cost_estimate only read BigQuery's dry run. On any other warehouse it answered supported: false, so the plugin's scan check never held anything on Snowflake or Databricks. Neither has a dry run, so this PR reads the query planner's estimate from EXPLAIN, which compiles the query without running it:

WarehouseStatementFigure readAccuracy
BigQuerydry run (unchanged)bytes_processedunchanged: maxBytes bound, or lowerBound for scripts and zeros
SnowflakeEXPLAIN USING JSON <sql>GlobalStats.bytesAssigned (plus partitionsAssigned / partitionsTotal)Snowflake documents it as an upper-bound estimate: the bytes in the micro-partitions left after compile-time pruning. Runtime pruning (join filters) can scan less, and a cached result scans nothing.
DatabricksEXPLAIN COST <sql>the sum of Statistics(sizeInBytes=…) over the leaves of == Optimized Logical Plan == (the relations scanned)An estimate, not a bound either way. These are table and file sizes (Delta log, catalog) before column pruning and runtime file skipping, so a run usually reads less. A table without statistics has no estimate.
  • No estimate, never a block. If a Databricks leaf has no Statistics, or shows Spark's default for a table without statistics (spark.sql.defaultSizeInBytes, Long.MaxValue, printed as 8.0 EiB; anything at or above 2^62 is treated the same), the tool answers success: false. It names the relation when Spark prints a plain name (letters, digits, underscores, dots) and suggests ANALYZE TABLE <relation> COMPUTE STATISTICS NOSCAN, which records the size from file metadata without reading the data (a full ANALYZE scans the table). A failed EXPLAIN or an unreadable plan also gives success: false with the reason. None of these is read as a count.
  • Result fields: source: "explain", bound: "upper" on Snowflake and "estimate" on Databricks, bytes, display (the dry run's units, e.g. 2.0 GiB), and partitionsAssigned / partitionsTotal on Snowflake. There is no maxBytes, so a consumer compares bytes itself. BigQuery metadata is unchanged (no source field), so a consumer that sees no source treats the result as a dry run.
  • How EXPLAIN runs: the compiled model or SQL is wrapped in inline Jinja, {%- if execute -%}{%- set altimate_plan = run_query('EXPLAIN … <sql>') -%}{{ tojson(altimate_plan.columns[0].values() | list) }}{%- else -%}[]{%- endif -%}, and passed to unsafeCompileQuery. The execute guard matters because dbt compile --inline renders the query once at parse, where run_query returns None; that pass renders [] and runs nothing. It runs on dbt's own connection, the same way dbt-integration's getColumnsOfSource does. It goes through compile rather than immediatelyExecuteSQL because that path is dbt show --limit N, which wraps the query in a limit subquery that an EXPLAIN can't take. The SQL is written into a Jinja string literal, so quotes and backslashes are escaped and trailing semicolons and whitespace are dropped.
  • Never a script. dbt-snowflake runs every statement that the connector's split_statements splits off, so a second statement after the EXPLAIN would run for real. mayBeScript reads the SQL in one pass, the way that splitter does: single- and double-quoted text where a backslash escapes, backticks on Databricks, -- and /* */ comments. Anything it doesn't model exactly counts as a script and gets no estimate, with nothing sent: a Databricks -- comment continued past a trailing backslash, $$ text, // comments, a line starting with ! (a Snowflake client command, which the splitter sends on its own), Databricks raw strings, nested or unclosed comments, unclosed quotes, EXECUTE IMMEDIATE, or anything after a ; but whitespace, comments or more semicolons. Line endings are normalised first, as Jinja does (\r\n and a lone \r become \n), so the text checked is the text the driver gets.
  • No EXPLAIN for queries over files on Databricks. read_files(…), cloud_files(…), path queries over a known format (json.`…` , libsvm, …; a format word in the middle of a dotted name such as main.json.`t` is a table; a column qualified by an alias named like a format is refused, since that can't be told apart without parsing), and any backtick-quoted name that looks like a path (contains / or :), whatever the source before it (a format, a provider class), unless it follows Delta's own provider, get no estimate, because Databricks infers their schema by reading the files while it plans. Detection uses the same single scan as the script check (comments and quoted text blanked, backtick-quoted names kept) and accepts backtick-quoted names and comments between tokens (read_files/**/(, `json`.`…`). Delta paths and tables are estimated.
  • Other warehouses still answer supported: false. The message now names the three supported warehouses.

Supported input (EXPLAIN parsers)

  • parseSnowflakeExplain(content): the content column of EXPLAIN USING JSON, as a string or already parsed. Reads GlobalStats.bytesAssigned (a finite number ≥ 0). Partition counts are optional. Anything else (the TABULAR/TEXT formats, a missing or non-numeric bytesAssigned) returns undefined.
  • parseDatabricksExplainCost(plan): the plan column of EXPLAIN COST, i.e. Spark's TreeNode.generateTreeString with verboseStringWithSuffix. Each tree level is a 3-character prefix (: / for ancestors, :- / +- for the node). Inner children such as scalar subqueries sit at depth+2, and their relations count. Sizes use Utils.bytesToString: %.1f with B, KiB … EiB, Spark 2's KB … EB, or 1.23E+21 B past a long. Only the == Optimized Logical Plan == section is read, up to the next Spark section header (Parsed/Analyzed/Optimized Logical Plan, Physical Plan). CRLF is accepted.
  • mayBeScript(sql, type) is conservative by design: a false "script" gives no estimate, never a run.
  • A Databricks leaf's size is read only from the , Statistics(sizeInBytes=…[, …]) suffix at the end of the node line, so a column named like it is ignored.
  • Documented limits:
    • A node whose text spans lines (a string literal containing a newline) is joined to its node when the continuation line has no tree prefix. One that happens to start with tree-like spaces may still be read as an extra node, giving no estimate or a higher one, never a lower one.
    • If spark.sql.defaultSizeInBytes is set below 2^62, a table without statistics is read at that size.
    • An inner-child plan printed without statistics gives no estimate.

How did you verify your code works?

  • bun test test/altimate/dbt-cost-estimate.test.ts: 195 pass. New cases cover:
    • parsers on documented-shape outputs: the Snowflake JSON from the EXPLAIN docs, and Spark plans built from Statistics.simpleString, Utils.bytesToString and generateTreeString (joins, subqueries, default-size and missing stats, units, CRLF, malformed input)

    • the Jinja builder and its escaping

    • the tool on Snowflake and Databricks: compile, EXPLAIN, and the metadata above; no stats, a refused EXPLAIN, an unreadable plan, a failed compile, cancellation

    • from review:

      • the parse-pass guard
      • script refusal (select 1; delete from orders is never sent), including separators hidden by $$, //, raw strings, escaped double quotes, and nested or unclosed comments
      • statistics-like column names in Databricks plans
      • a lone \r that Jinja turns into a line break
      • ! commands at a line's start, nested and overlapping (/*/*/) comments, quoted text after a ;
      • relation names Spark prints unquoted with a comma, hyphen or space: the advice names only plain ones
      • multi-line plan nodes, and Databricks queries over files
      • stray == … == lines inside a plan

      Each fails on the code before its fix.

  • Real dbt, real Jinja: the generated template went through dbt compile --inline (dbt-core 1.11, dbt-duckdb, local DuckDB, where plain EXPLAIN runs).
    • Without the guard, dbt failed at parse: 'None' has no attribute 'table'. With it, the compile returns the EXPLAIN's first column as JSON.
    • A SQL with 'it''s; a \ test', a newline and a literal {{ not jinja }} round-trips exactly: ["it's; a \\ test"].
  • Existing BigQuery tests pass unchanged. The only edited test is the one that listed snowflake and databricks as unsupported; it now uses duckdb, redshift and postgres.
  • dbt-adapter-access and tool-cli suites: 150 pass. bun typecheck: green.
  • Live warehouse: none configured on this machine (no Snowflake or Databricks connection, no Spark runtime). Parsers are tested on the documented formats and were not run against a live warehouse.

Cost side effects

Yes, one warehouse statement per estimate, compile-only, with no data scanned:

  • Snowflake: EXPLAIN doesn't need a running warehouse and uses no compute credits. Per Snowflake's docs it uses cloud-services credits for the compilation, which usually fall within the daily 10% cloud-services adjustment.
  • Databricks: EXPLAIN COST runs on the SQL warehouse or cluster that dbt's profile connects to. If that is stopped, it starts it, and it is billed for the DBUs while it is up (auto-stop applies as usual).
  • Both: compiling the model renders its Jinja, as before. Macros such as run_query in the model can run billed queries, which the tool description already says. One estimate runs per call, and the plugin runs at most one per project at a time.

Screenshots / recordings

N/A (CLI tool output).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code