fix(server): don't fail completions on invalid UTF-8 in model output

AY
12h ago
+3-2

The OpenAI-compatible server serializes model text with plain Json::dump(). nlohmann throws type_error.316 from dump() when a string holds invalid UTF-8, and model output can: a llama.cpp model sampling with a wide top_k emits byte tokens like \xE5\xE5\xBF (repro in #1317). When that happens:

  • Non-streaming: the throw escapes processNonStreaming and lands in the Json::exception handler in handleChatCompletions, so the client gets 400 invalid_request_error with Invalid request: [json.exception.type_error.316] invalid UTF-8 byte at index 4: 0xE5. The request was fine. The whole completion is thrown away and the client is told it sent something wrong.
  • Streaming: formatSSE throws inside the token callback, generation is reported as failed, and the stream ends with {"error":{"code":500,"message":"Local generation failed or was cancelled"}} partway through the answer.

I reproduced both with the existing fixture in test_openai_server.cpp by setting backend.output to "Hi \xE5\xE5\xBF there":

mainthis PR
non-streaming400, body is the type_error above200, "content":"Hi �� there"
streamingH, i, , then the 500 error chunkevery chunk, then finish_reason: "stop"

The fix passes error_handler_t::replace to the two dump() calls that carry model text (formatSSE and the non-streaming response), so invalid bytes come out as U+FFFD. That's the same call tool_calling.cpp and json_schema_to_gbnf.cpp already use for model-derived JSON. The other dump() calls in the server only serialize our own strings or the already-parsed request.

The server calls rac_llm_generate_stream directly, so it doesn't go through the llm_module stream funnels and wouldn't be covered by sanitizing there.

Testing: test_openai_server passes 84/84. I didn't add a case for this since it's a two-line change using an existing idiom, but I'm happy to add one to that file if you'd like it pinned. I checked it with the fixture probe above, not against a real model through runanywhere-server.

I'm a first-year student and used Claude Code to dig through the server code and set up the repro. I ran the before/after myself and read the diff line by line.

Summary by cubic

Stops the OpenAI-compatible server from failing completions when model output contains invalid UTF-8.

  • Non-streaming requests returned a 400 invalid_request_error; streaming requests stopped mid-answer with a 500 error chunk.
  • The two dump() calls that serialize model text now use error_handler_t::replace, so invalid bytes become U+FFFD, matching the idiom tool_calling.cpp and json_schema_to_gbnf.cpp already use.

Summary by CodeRabbit

  • Bug Fixes
    • Server-sent events and non-streaming completion responses now use compact JSON formatting.
    • Responses can handle invalid UTF-8 sequences by replacing them, helping prevent serialization failures.