Harness output limits held to model bounds; max_tokens is the whole budget

SH
1d ago
+262-31

DRAFT — ships alongside RunanywhereAI/InferenceInfra#5737 (server: max_tokens becomes a hard total for every caller, thinking included).

No behaviour change in wally. The prod ledger shows Wally's harnesses already ask 16k–64k, so they keep working when the server stops adding hidden thinking room. This PR pins that down:

  • Test: every harness config wally writes holds its output limit to the model's own bounds (≥ min(16,384, max_output), ≤ max_output), so a later change cannot quietly drop a harness to a small limit.
  • Docs: X-RA-Harness is attribution only; max_tokens is the whole budget. Notes on Hermes (writes no limit) and on the Anthropic bridge dropping thinking.budget_tokens (mapping it to reasoning_effort is a later decision).

Tests: full ctest suite via CMake/Ninja, as CI runs it. Independently reviewed: approve (4 low notes: comment wording, one bound the test does not pin).

🤖 Generated with Claude Code

Summary by cubic

No behavior change in Wally. As the API moves to treating max_tokens as a hard total (thinking included) with no extra reasoning allowance, Wally pins down its part: harness configs already declare the model's full output capacity, the X-RA-Harness header is attribution only, and the Anthropic bridge crosses max_tokens 1:1.

Output limits

  • The four harness config writers that declare an output limit now share one helper, output_limit, instead of each copying the "max_output, else window-capped fallback" logic.
  • A model with a reported max_output gets that cap even when no context window is listed; a window-only model gets min(window, fallback); one with neither gets no limit and the server applies its model default.
  • OpenCode alone writes no limit when no window is reported, because its limit field needs a context size; Shannon writes maxTokens only when the model has a cap.
  • Adds a test pinning every written limit to the model's own bounds: at least min(16,384, max_output), at most max_output, and no limit when the catalogue lists none.
  • Documents that a reasoning model may spend its entire max_tokens thinking, so the client owns making the budget large enough.

Attribution and bridging

  • Clarifies that X-RA-Harness is usage attribution only and never changes what a request is served; the reasoning allowance is a pending server change Wally does not rely on.
  • Adds a test showing the Anthropic bridge sends max_tokens as the whole budget and drops thinking.budget_tokens; Hermes receives no limit and relies on the API default.

Summary by CodeRabbit

  • Bug Fixes
    • Output limits now cover all generated content, including reasoning.
    • OpenCode, OpenClaw, Prime Agent, and dsh set output limits using each model’s available limits. OpenCode omits the limit when no context window is known; other harnesses use a known output cap where available.
    • Anthropic requests use max_tokens as the total generation limit. Reasoning-budget settings do not increase it, and an omitted max_tokens remains unset.