fix: remove the `text_label` bulk labeling tool and add a cost side-effects rule

Merged
AN
20h ago
+64-1249

Issue for this PR

Closes #1547

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Why. The analyst's text_label tool was registered by default and labeled every row of a query result through the session's model, one call per batch of about 50 rows. Its cost grows with the table, not the conversation: about 20,000 model calls for a million-row table, per question. In one internal evaluation day it billed about $680 on Opus, roughly 7x the agent sessions themselves. Its calls did not appear in traces or session cost, and each one-off batch prompt was written to the prompt cache and never read back. The accuracy it bought in that evaluation was within run-to-run noise. Full-table labeling through the session model has no safe default for real tables, so this removes it rather than gating it.

What changed (before → after).

  • text_label tool, its registry entry, its data-explorer permission and its tests: present and on by default → removed. No agent can call it.
  • data-question skill: told the agent to label every row with text_label, splitting larger populations into ranges → count the population first; label at most 500 rows by any path (the agent's own reading, subagents or warehouse AI functions) without the user's confirmation; for larger populations, label a documented random sample of up to 500 rows and report an estimate with its sample size.
  • tool-schema-anthropic.test.ts: asserts no agent registers text_label; the generic empty-sentinel schema check stays.
  • Analyst system prompt (analyst.txt): demanded full label coverage and suggested batching through data-explorer subagents → the same 500-row cap and sample-and-estimate default, with more only after the user confirms.
  • New rule, checked on every PR: REVIEW.md focus area 12 (read by the review bots), a "Cost side effects" section in packages/opencode/AGENTS.md (read by coding agents) and in CONTRIBUTING.md (repo-wide; a root AGENTS.md is forbidden by the upstream guard tests), a required ### Cost side effects section plus checkbox in the PR template, and a pr-standards.yml check that flags a missing or empty section on contributor PRs (team members are skipped by that workflow, as before). That check now also requires every checklist box, since the template has three. Any path whose spend grows with user data (per-row or per-item model calls, hidden model calls, unbounded queries, cloud resources) must be off by default or hard-capped, estimated or confirmed above the cap, on the cheapest adequate model with no cache breakpoint on one-off content, and counted in session cost and traces.

The provider stream-error surfacing that shipped in the same earlier PR (altimate/llm/stream-text.ts) is shared by other features and stays.

Deployment readiness. Self-contained. No config, data or migration.

User impact. Analyst sessions no longer offer bulk text labeling. Questions that need semantic labels over more than 500 rows get a sampled estimate, or a full run only after the user confirms, instead of an exact full-table count.

How did you verify your code works?

  • bun run typecheck: clean.
  • Unit: bun test test/agent test/altimate/sql-validation-e2e.test.ts test/altimate/tools/tool-schema-anthropic.test.ts test/altimate/stream-text.test.ts: 236 pass, 0 fail (rerun after the cap: test/agent and sql-validation-e2e, 217 pass). The tool-schema test validates every registered tool's schema for the analyst and builder agents on Anthropic and Bedrock, and asserts text_label is absent.
  • script/upstream/analyze.ts --markers --base main --strict: all custom code in upstream-shared files is marked.
  • Upstream guard tests (test/branding/upstream-guard.test.ts, upstream-merge-guard.test.ts): 222 pass.
  • git grep for text_label, TextLabel, text-label and ALTIMATE_TEXT_LABEL: only the new rules' historical example and the negative test remain.
  • No live model run: removal makes no model calls.
  • New pr-standards.yml cost check exercised locally: unfilled template, missing section and a prose-only mention are flagged; "None", a nested #### heading and a section at the end of the body pass. Workflow YAML parses and its embedded script passes node --check.
  • Reviews: no P0/P1 from a local read-only Codex review or the independent review. Their medium findings are fixed in this head: sample-size vs full-population counting, warehouse AI functions without confirmation, the analyst prompt's full-coverage fan-out, an uncapped sample, and the PR-template section the rule asks for.

Cost side effects

Removes one: full-table model labeling through the session model. Adds none.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code

[!NOTE]
Medium Risk
Removes a shipped analyst tool and changes labeling defaults, so sessions that relied on full-table text_label will behave differently; PR workflow enforcement is additive only.

Overview
Removes the analyst text_label bulk-classification tool (implementation, registry wiring, data-explorer permission, and tests) because session-model labeling scaled with table size, was on by default, and was not visible in session cost/traces.

Analyst behavior now caps semantic labeling at 500 rows without user confirmation (reading, subagents, or warehouse AI functions). Larger populations default to a documented random sample of up to 500 rows reported as an estimate; more labeling requires disclosing row count and job size first. The /data-question skill, analyst.txt, and data-explorer.txt prompts were updated to match—no more full-table text_label batching.

Adds ongoing cost side-effect governance: required Cost side effects section and checklist item in the PR template, CONTRIBUTING.md / REVIEW.md focus area 12 / packages/opencode/AGENTS.md, and pr-standards.yml validation (non-empty cost section; all three checklist boxes checked).

Reviewed by Cursor Bugbot for commit f2840b3a9e00967a0a0f943ebe868ef8e5197382. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Changes
    • Removed the dedicated text-labeling tool.
    • Data analysis guidance now limits labeling to 500 rows unless you confirm a larger job. For larger datasets, it directs the assistant to use a random sample and report an estimate rather than an exact population count.
  • Documentation
    • Added guidance to disclose and cap costs that scale with data, and to request confirmation before exceeding a cap.