fix: UTF-16BE skill files keep their byte order during setup

Merged
HA
3mo ago
+52-2

Summary

  • UTF-16 big-endian skill files now keep their byte order when valid content resembles a little-endian carriage return.
  • BOM-less UTF-16 line-ending detection remains unchanged.

User Impact

  • Previously, a UTF-16BE text skill file with an explicit BOM could be routed through the little-endian heuristic when its payload contained bytes such as 0D 00. Setup could then rewrite valid content using the wrong byte order.
  • Explicit UTF-16 BOMs are now authoritative, so generated skill copies preserve their declared encoding while still normalizing CRLF to LF.

Root Cause

  • The little-endian BOM and line-ending heuristic were combined in the first branch. A BE BOM did not match the LE BOM check, but the LE heuristic still scanned the BE payload before the explicit BE BOM branch ran.

Changes

  • Evaluate explicit little-endian and big-endian BOMs before either BOM-less line-ending heuristic.
  • Keep the existing LE-then-BE heuristic order for content without a BOM.
  • Add a regression test containing UTF-16BE U+0D00 followed by CRLF.
  • Add BOM-less LE and BE characterization tests to protect the existing heuristic paths.

Verification

  • TDD Red: the new ambiguous UTF-16BE case failed while the other 8 normalizer tests passed.
  • TDD Green: SkillFileContentNormalizerTests 9 passed.
  • Repo-local Unity compile: 0 errors, 0 warnings.
  • SkillInstallLayoutTests: 18 passed.
  • ToolSkillSynchronizerTests: 52 passed.
  • git diff --check.

Compatibility

  • Only local skill file byte normalization changes. Wire formats and protocol declarations are unchanged.