fix(python): base64-encode raw bytes passed to base64 file input fields

Merged
2Y
10h ago
+202-39

Why

Fields opted into x-speakeasy-base64-input-mode: file accept a pre-encoded base64 str, a PathLike, or a binary stream. Raw bytes, bytearray and memoryview values were passed through encode_base64_file_input unchanged. Pydantic's lax str coercion then decoded them as UTF-8:

  • non-UTF-8 payloads (most binary content) failed validation with string_unicode
  • ASCII payloads were accepted and sent on the wire unencoded
  • memoryview failed with string_type

Binary streams whose read() returns a memoryview raised TypeError.

What changed

  • encode_base64_file_input base64-encodes bytes-like values. Paths and binary streams are read into bytes and go through the same encoding step.
  • memoryview values must have a byte-sized item format (B, b, c, with or without a byte-order prefix). Other formats raise ValueError, which pydantic reports as a validation error, so the wire bytes never depend on the host's int size or byte order.
  • bytearray and contiguous memoryview values are encoded without an intermediate copy; only non-contiguous views are copied.
  • In a union that also has a bytes member (e.g. Union[Base64EncodedString, bytes, int], generated from a oneOf of a file-mode format: byte string, a format: binary string and another type), bytes input resolves to the bytes member regardless of member order, as on main. Values encoded from bytes-like input are returned as a str subclass that pydantic scores as a strict but non-exact match, so an exact bytes member wins.
  • Base64FileInput includes bytes, bytearray and memoryview, so type checkers accept them in the input TypedDicts.
  • Review Python SDK regenerated; only types/base64fileinput.py and its gen.lock checksum change.
  • New feature test request-bodies-base64-file-input-raw-bytes (pythonv2 only; skipped on the other targets, which do not implement the extension).

Encoding contract

The input type decides the encoding: str is sent as already-encoded base64, bytes-like values are treated as raw content. Content alone cannot tell the two apart, since short alphanumeric payloads are also valid base64.

[!WARNING]
⚠️ Migration: already-encoded base64 held as bytes (e.g. the result of base64.b64encode(...)) must now be passed as str (value.decode("ascii")). Passing it as bytes encodes it a second time, silently. That input was never part of the declared Union[str, Base64FileInput] type, and the changeset states the migration.

Known limitations: mixed unions

Rare shapes where a base64 file-input member shares a union with a member that also accepts bytes-like input. Pydantic's smart union has three match levels, so the helper cannot rank the base64 member above every other member and below bytes at the same time.

  • ⚠️ bytearray with a bytes member: for Union[Base64EncodedString, bytes], bytearray(b"abc") is sent through the base64 member as "YWJj" (main sent "abc"). The bytes member only accepts bytearray in lax mode, so the base64 member matches first. Exact bytes input is unaffected.
  • ⚠️ bytes with an Any member: for a union generated from oneOf: [{}, <file-mode string>], bytes input follows member order: Union[Any, Base64EncodedString] keeps the raw bytes ("abc", as on main), Union[Base64EncodedString, Any] sends "YWJj". str, PathLike and stream input resolve the same in both orders.

Why not a new extension value

file already means "pass content, the SDK encodes it", and file content is bytes elsewhere in the Python SDK (File.content: Union[bytes, IO[bytes], io.IOBase]). Raw bytes had no working behaviour under file, so keeping it and adding a separate value would leave existing specs on the broken path. A new value would also not avoid the bytes-as-pre-encoded break: SDK users would hit it as soon as a spec author switched values.

Testing

  • pytest tests/test_request_bodies_additional.py -k base64 in the generated primary pythonv2 SDK: 5 passed.
    • SDK round-trips cover non-UTF-8 bytes, ASCII bytearray (sent encoded), base64-looking b"YWJj" (sent as "WVdKag==", not passed through), and a strided memoryview.
    • Helper assertions cover contiguous, strided and ctypes ('<B') memoryview, a stream returning memoryview, a non-byte memoryview raising ValueError, and "YWJj" passing through unchanged.
    • Union assertions cover bytes, str and int input against Union[Base64EncodedString, bytes, int] in both member orders, PathLike and stream input against Union[Any, Base64EncodedString], and a non-byte memoryview falling through to List[int] in Union[List[int], Base64EncodedString].
  • Union resolution checked on pydantic 2.11.2 (the SDK's minimum) through 2.14.3.
  • mypy, pyright, pylint on the generated base64fileinput.py: clean.
  • ./scripts/build-review-sdk.sh pythonv2: integrity validation passed.
  • prettier --check on touched features.ts files; go build ./internal/features/.
  • Full make test-pythonv2 not run; the change is confined to the base64 input helper.

Public-safety check

  • This change contains no credentials, customer documents, private repository URLs, private filesystem paths, or unredacted private logs.
  • Title, body, comments, and commit messages name no customers or customer-derived identifiers, private paths or trackers, or workflow provenance, and are understandable without private context (.claude/skills/public-repo-communication/SKILL.md).
  • Generated fixtures and review SDK changes are public-safe.
  • I reviewed git diff --check.