fix(commons): preserve default-format tool calls with malformed utf-8

SH
1d ago
+417-67

Description

Default-format <tool_call>{...}</tool_call> output with malformed UTF-8 can parse into a call whose arguments fail validation. Repair the bytes before JSON parsing and unquoted-key normalization, preserving decoded values, nested types, valid Unicode, and escaped backslashes.

Actual executor dispatch also needs repair before the protobuf boundary: the run loop and session serialize generated text into ToolParseRequest.text, and strict protobuf decoding rejects malformed bytes before the normalizer runs. The shared generation helper now repairs that text for both orchestration paths.

For example, a location containing bytes C3 28 now reaches the executor as �(, completes validation, and proceeds to final synthesis.

Fixes #1131.

Dependency

Stacked on #1154, which introduces the shared sanitizer and repaired-key collision checks. This branch includes #1154 through 006105ed74c1ebee2f58c8087e1b7d444ce93607. Maintainers should merge #1154 first; this branch can then be rebased onto main. The full PR diff includes that unmerged parent; review the #1131-only changes in the focused comparison.

Type of Change

  • Bug fix

Testing

  • Linux Debug protobuf-enabled targeted rebuild of all affected production translation units and the three test executables, reusing unchanged parent dependency artifacts.
  • Protobuf parser binary (previous follow-up): 36/36 cases passed.
  • Previous follow-up: three CTest suites passed — tool parser, tool-calling run loop, and tool-calling session proto. The original implementation also passed LLM proto service, LLM stream proto, and tool-value JSON.
  • New normalization cases cover malformed bytes, incomplete sequences, unquoted keys, nested arrays/objects, escaped backslashes, and valid Unicode.
  • Default-format parsing and schema validation preserve the exact repaired value.
  • Run-loop regression verifies validation stays enabled, the executor runs exactly once with repaired arguments, and final synthesis completes.
  • Session regression verifies the repaired call reaches the host and the supplied tool result completes the session.
  • Removal checks: deleting normalization repair fails both parser regressions; deleting shared generation repair fails the run-loop and session regressions. These checks use temporary copies.
  • Latest test-only follow-up: 35/35 non-protobuf parser cases passed. Both argument-key parse calls select the loop's format explicitly; removing duplicate-key rejection in temporary copies fails the collision regression for LFM2 and bare Pythonic independently.
  • Direct caller-owned malformed tool names exercise both validation APIs without parser sanitization; existing parsed-name coverage is preserved.
  • Removing validation-error replacement handling in temporary copies fails the new raw-name regression for each API while the parsed-name regression still passes.
  • Changed-line formatting, whitespace, PII-logging guard, and deprecated-surface gate passed.
  • Repository-wide formatting remains unclean: the same 81 files are flagged as in the parent checkout, including the generated defaults header. No newly flagged file.

A normalization-only experiment passed the parser suite while both orchestration regressions still failed. This confirms why both repair points are necessary.

Platform-Specific Testing

Native Linux Debug only. Apple, Android, Windows, and Web integration were not run locally.

Labels

  • Commons — shared native code in core/

Checklist

  • Self-review completed
  • Changed regions follow project formatting
  • Internal comments explain the two repair boundaries

Summary by cubic

Fixes default-format tool calls with malformed UTF-8 so their arguments survive parsing, schema validation, and executor dispatch.

Previously, <tool_call>{...}</tool_call> output with invalid UTF-8 bytes could fail validation or produce arguments with lost types and extra quotes. The normalizer now repairs bytes before JSON parsing and unquoted-key normalization, and the shared generation helper repairs text before it crosses the protobuf boundary in both run-loop and session orchestration paths. A shared rac::llm::sanitize_utf8 replaces each invalid byte with U+FFFD while preserving valid Unicode, nested types, and escaped backslashes.

  • Pythonic and LFM2 argument keys that become duplicates after repair now reject the whole call instead of silently dropping an argument.
  • Tool-result error payloads and validation error messages are also sanitized so malformed host- or model-provided text stays valid JSON.

Dependency

Stacked on #1154, which introduces the shared sanitizer and repaired-key collision checks. Merge #1154 first; this branch can then be rebased.

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of malformed text in AI-generated responses and tool arguments by replacing invalid characters, while preserving valid Unicode.
    • Improved formatting of tool results and errors in follow-up prompts, including when result text contains invalid characters.
    • Prevented ambiguous tool arguments from being accepted when text repair causes argument keys to collide.
    • Improved parsing of tool arguments to preserve array elements after malformed keys.