Withdrawn

Closed
GR
3d ago
+6232-380

Serve several named models from one luce_server process and swap models that share a GPU while retaining their host prefix snapshots. A named request drains conflicting residents, runs eviction and reinstatement on each model's serial worker, and then admits work to the requested model. Failed swaps restore displaced models and apply a retry cooldown.

  • --luce-mix invokes the installed pair/topology launcher before GPU/model initialization, forwarding its options and this engine's executable path. The launcher resolves model blocks, then execs the normal engine/gateway path. LUCE_MIX_LAUNCHER overrides the launcher on PATH; --print-profile exports the existing engine profiles for reuse. CLI tests verify argument preservation and profile export; the complete handoff was exercised in hardware-detection/dry-run mode.
  • --routing-gateway /path/to/executable starts and supervises an installed local gateway on POSIX hosts. The gateway serves the requested --host/--port; the engine moves to loopback port 8491 (--routing-engine-port overrides it). The child receives --engine-url, --host, and --port. Shutdown reaps the gateway, and unexpected child exit stops the engine with failure. Routing policy remains external. See managed launch.
  • --model-routing name selects a model by the request's model field; omitted/auto requests use resident capacity. --unknown-model reject|primary controls alias fallback.
  • --swap-residency supports Qwen3.5 cold/warm eviction and DeepSeek V4 partial/cold eviction. Partial DeepSeek eviction retains the secondary expert tier and streamed cache; reinstatement preserves expert placement and router biases. Retained devices participate in conflict handling. --swap-keep-weights keeps selected Qwen weights warm between single-device models.
  • Non-streaming Qwen chat completions can return logprobs / top_logprobs. /v1/hidden_states reports selected prefill residual streams and can seed the prefix cache. Out-of-range unsigned layer indices are rejected before signed conversion.
  • Qwen loading supports tied embeddings and the eight capture layers used by the Qwen3.5-9B DFlash drafter.

The listener owns routing reservations and residency bookkeeping; each model's worker executes ControlOp transitions; each backend owns device teardown, reconstruction, and retained host snapshots. Startup loads and evicts inactive blocks before loading the resident set.

Swapping currently requires the serial worker (--max-concurrency 1). Qwen eviction requires a supported single-device configuration with host snapshots; DeepSeek eviction requires the supported in-process two-GPU expert placement. Streaming and unsupported backends continue to ignore logprobs. See the design and usage for configuration restrictions and earlier performance measurements.

Validation on the current branch, based on Luce-Org/lucebox main 5740ef0a3:

  • HIP build for gfx1151;gfx1201 passed.

  • ctest -L cpu: 753/753 passed, including gateway lifecycle coverage; the CLI validation suite passed. The unsigned-layer regression failed before the fix and passed afterwards.

  • R9700 Qwen3.8-27B + DFlash2: three cold eviction/reinstatement cycles preserved greedy snapshot continuations with no device-memory growth.

  • Real HTTP checks with Qwen3.5-0.8B passed in serial, batched, cold-swap, and warm-swap modes: hidden-state readouts, prefix seeding, logprobs, invalid-layer rejection, and identical greedy replies across swaps. Concurrent hidden-state readouts had cosine similarity above 0.998 against the single-request readout (the reference probe's criterion is >0.99).

  • R9700 + Strix Halo DeepSeek V4.1 + DSpark: ten partial cycles and a cold reload passed. The secondary tier survived partial eviction without re-upload, frozen placement survived environment changes, restored greedy output matched, and primary free memory was identical after the first and last partial eviction.

  • Managed gateway integration with two Qwen3.5-0.8B slots: engine-owned startup, internal URL forwarding, streamed and non-streamed consultation, eviction/reinstatement, switch back, graceful child reaping, and gateway-failure propagation passed. The installed gateway determines the public API endpoints.

Final review fix: capture LUCE_DS4_HYBRID_F16_VERIFY_KV at initialization and preserve it through partial and cold reinstatement. The GPU regression now changes the environment after loading and checks the captured mode during restores. Engine and GPU test compilation are checked; this extended GPU case has not run because an unrelated long-running benchmark occupies the machine. Earlier GPU results above predate this extension.