Decode server-sent events as UTF-8 with BOM handling

TH
2d ago
+134-3

Summary

A leading UTF-8 BOM currently hides the first data, event, or id field in an SSE response. It can also trigger a false event-size error. A non-UTF-8 response charset or client default encoding corrupts UTF-8 event data.

The SSE parsing rules require UTF-8 decoding and ignoring exactly one leading BOM. Decode response content bytes through the existing incremental TextDecoder before line buffering and size checks. Later U+FEFF characters remain intact, and Response.encoding is unchanged.

Added 42 parameterized sync/async regression cases covering fragmented BOMs and UTF-8 characters, event metadata, exact size limits, later/double BOMs, and charset/default-encoding overrides. Updated the SSE documentation.

Local validation on Python 3.15.0:

  • Against untouched upstream b77e3e8, nine sync regressions fail and five controls pass.
  • All 2,138 tests pass with the fix.
  • scripts/coverage reports 100% coverage; scripts/check passes formatting, typing, lint, lockfile and unasync checks.

Hosted CI and maintainer review remain separate gates.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

Implementation and tests were prepared with Codex assistance.