fix: UTF-16BE skill files keep their byte order during setup
Merged
HA
3mo ago+52+52+01234567890123456789-2-2-0123456789
Summary
- UTF-16 big-endian skill files now keep their byte order when valid content resembles a little-endian carriage return.
- BOM-less UTF-16 line-ending detection remains unchanged.
User Impact
- Previously, a UTF-16BE text skill file with an explicit BOM could be routed through the little-endian heuristic when its payload contained bytes such as
0D 00. Setup could then rewrite valid content using the wrong byte order. - Explicit UTF-16 BOMs are now authoritative, so generated skill copies preserve their declared encoding while still normalizing CRLF to LF.
Root Cause
- The little-endian BOM and line-ending heuristic were combined in the first branch. A BE BOM did not match the LE BOM check, but the LE heuristic still scanned the BE payload before the explicit BE BOM branch ran.
Changes
- Evaluate explicit little-endian and big-endian BOMs before either BOM-less line-ending heuristic.
- Keep the existing LE-then-BE heuristic order for content without a BOM.
- Add a regression test containing UTF-16BE
U+0D00followed by CRLF. - Add BOM-less LE and BE characterization tests to protect the existing heuristic paths.
Verification
- TDD Red: the new ambiguous UTF-16BE case failed while the other 8 normalizer tests passed.
- TDD Green:
SkillFileContentNormalizerTests9 passed. - Repo-local Unity compile: 0 errors, 0 warnings.
SkillInstallLayoutTests: 18 passed.ToolSkillSynchronizerTests: 52 passed.git diff --check.
Compatibility
- Only local skill file byte normalization changes. Wire formats and protocol declarations are unchanged.